.. _task-configuration:

==================
Task configuration
==================

See :ref:`explanation-task-configuration` for an introduction to the concept
and :ref:`manage-task-configuration` for a practical guide on how to create
and maintain task configuration.

Task configuration entries
==========================

Tweaks and exceptions from normal task processing are recorded as
:bare-data:`debusine:task-configuration` entries in a
:collection:`debusine:task-configuration` collection, and then later used to
feed the relevant workflows and work requests.

Before a task becomes pending, relevant entries (if any) are looked up, merged
in a well defined order and finally applied.

Tasks can be any :ref:`type of Tasks <explanation-tasks>`, but for all
practical purposes, ``Worker`` and ``Workflow`` are the most likely targets for
configuration.

The result of applying a task configuration entry can be:

* key/value pairs added or replaced in the ``task_data`` field of
  :ref:`explanation-work-requests`
* Extra task-provided scheduler tags added to the task's computed set.
* Extra task-required scheduler tags added to the task's computed set.


When task configuration is applied
==================================

To be able to apply changes to the submitted ``task_data`` configuration,
we need to be able to know the *subject* and the *context*, which may depend on
information not available when the task is created. For example, the subject
may be derived from an artifact that is the output of a previous work request
in a workflow.

Task configuration can thus be applied only when a task becomes pending, and
subject and context are generated at that time using the task's
:py:meth:`debusine.db.tasks.DBTask.get_task_configuration_subject_context`
method.

This also ensures that at the time a task is pending, the final set of
scheduler tags has been computed, including those contributed by task
configuration entries, so that the scheduler has all the information needed to
look for a suitable worker.


How task configuration is applied
=================================

Looking up task configuration entries for a task
------------------------------------------------

When looking up task configuration entries for a task, these task attributes
are used for matching:

* Task *type*
* Task *name*
* Task *subject*, that is, a string describing what is being processed by the
  task
* Task *configuration context*, which is a string that can usefully be
  leveraged to apply some consistent configuration across all work requests
  sharing the same configuration context value
* Scheduler tags *provided* by the task
* Scheduler tags *required* by the task

Those values are used to lookup the various snippets of configuration, which
are retrieved and processed in a well defined order (see
:ref:`task-configuration-ordering`).

The collection can host partial or full configuration data. But it is
expected to be mainly useful to store overrides, i.e. variations compared
the defaults provided by the task or its containing workflow.

For example, for the ``debian-pipeline`` workflow, ``subject`` would typically be
the source package name while ``context`` would be the name of the target
suite.

.. _task-configuration-ordering:

Ordering matching entries
-------------------------

The final configured task data is generated by merging multiple snippets of
configuration, each stored in its :bare-data:`debusine:task-configuration`
entry. The ordering is well defined and intended to allow maintaining entries
that apply at different levels of granularity.

The entries to be applied to a task are sorted according to:

1. The value of the ``path`` and ``position`` fields, if present

2. Subject and context, in this order:

    1. entries with context and no subject
    2. entries with subject and no context
    3. entries with both subject and context

3. The database ID of the entry, as a tie-breaker to make ordering deterministic

If an entry uses templates in ``use_templates``, the referenced template
entries are placed *immediately after* the entry in the resulting ordered list.

About templates
---------------

Template entries follow the same structure as other entries, except that the
task matching parts are replaced by a template name.

They are meant to share some common configuration across multiple similar
packages, are only used indirectly: when a normal configuration entry, or
another template, refers to them as part of its ``use_templates`` field.


Building the set of changes to apply
------------------------------------

Once the entries are sorted, they are processed to build a set of default
values and override values, with the following operations::

    default_values = dict()
    override_values = dict()
    locked_values = set()
    provide_tags = set()
    require_tags = set()

    for config_item in all_items:
        # Drop all the entries referenced in `delete_values` (except
        # locked values)
        for key in config_item.delete_values:
            if key in locked_values:
                continue
            del default_values[key]
            del override_values[key]

        # Merge the default/override values in the response
        # (except locked values)
        for key, value in config_item.default_values:
            if key in locked_values:
                continue
            default_values[key] = value
        for key, value in config_item.override_values:
            if key in locked_values:
                continue
            override_values[key] = value

        # Update the set of locked values
        locked_values.update(config_item.lock_values)

        # Update the sets of provided tags
        provide_tags.update(config_item.provide_tags)
        require_tags.update(config_item.require_tags)

    return (default_values, override_values, provide_tags, require_tags)

Applying changes
----------------

Once we have a set of ``default_values``, ``override_values``, and tags to
provide/require, they get applied to the data available in ``task_data``::

    new_task_data = task_data.copy()
    default_values, override_values = get_merged_task_configuration()

    # Apply default values (add missing values, but also replace explicit
    # None values)
    for k, v in default_values:
        if new_task_data.get(k) is None:
            new_task_data[k] = v

    # Apply overrides
    new_task_data.update(override_values)

The result is stored in :py:attr:`WorkRequest.configured_task_data`, which will
be used from that point on as the task's data, while
:py:attr:`WorkRequest.task_data` remains untouched as documentation for the
initial task input.

Moreover, provided and required tags from task configuration entries are merged
with the set of computed scheduler tags, using the ``WORKSPACE`` provenance
(see :ref:`scheduler-tags-provenances`)::

    # Add provided/required tags
    tags_provided.add(
        ttags.ProvenanceProvided.WORKSPACE, provide_tags
    )
    tags_required.add(
        wtags.ProvenanceRequired.WORKSPACE, require_tags
    )


Reducing workflow complexity
============================

Having the ability to store overrides at the worker task level saves us from
adding too many configuration parameters on the workflows, so that the only
required parameters are those that are important to control the
orchestration step.

For example, we can have configuration for the sbuild worker
task next to the configuration for the debian-pipeline workflow::

  - task_type: Workflow
    task_name: debian-pipeline
    default_values:
      ...

  - task_type: Worker
    task_name sbuild
    context: stretch
    override_values:
      - backend: incus-lxc

This shows how the ``sbuild_backend`` parameter might no longer be a needed
input for the ``debian-pipeline`` workflow, though it is still available.


Task configuration examples
===========================

See :ref:`example-task-configuration` to see some realistic examples.
