DockerExecutorOptions

kedro_dagster.config.models.DockerExecutorOptions

Bases: MultiprocessExecutorOptions

Options for the Docker-based executor.

Parameters

Name	Type	Description	Default
retries	RetriesEnableOptions or RetriesDisableOptions	Retry configuration for the executor.	required
max_concurrent	int or None	Maximum number of concurrent processes.	required
image	str or None	Docker image to use.	required
network	str or None	Name of the network to connect the container at creation time.	required
registry	dict[str, str] or None	Information for using a non local/public docker registry.	required
env_vars	list[str]	Environment variables for the container.	required
container_kwargs	dict[str, Any] or None	Key-value pairs for containers.create.	required
networks	list[str]	Names of the networks to connect the container at creation time.	required

Examples

executors:
    docker_exec:
        docker_executor:
            image: "myrepo/app:latest"
            max_concurrent: 3
            env_vars: ["ENV=prod", "LOG_LEVEL=INFO"]
jobs:
    docker_job:
        pipeline:
            pipeline_name: batch_pipeline
        executor: docker_exec

See Also

kedro_dagster.config.models.MultiprocessExecutorOptions : Base class providing concurrency and retry settings. kedro_dagster.dagster.ExecutorCreator : Builds Dagster executor definitions from these options.

Source Code

View on GitHub

Show/Hide source

class DockerExecutorOptions(MultiprocessExecutorOptions):
    """Options for the Docker-based executor.

    Parameters
    ----------
    retries : RetriesEnableOptions or RetriesDisableOptions
        Retry configuration for the executor.
    max_concurrent : int or None
        Maximum number of concurrent processes.
    image : str or None
        Docker image to use.
    network : str or None
        Name of the network to connect the container at creation time.
    registry : dict[str, str] or None
        Information for using a non local/public docker registry.
    env_vars : list[str]
        Environment variables for the container.
    container_kwargs : dict[str, Any] or None
        Key-value pairs for ``containers.create``.
    networks : list[str]
        Names of the networks to connect the container at creation time.

    Examples
    --------
    ```yaml
    executors:
        docker_exec:
            docker_executor:
                image: "myrepo/app:latest"
                max_concurrent: 3
                env_vars: ["ENV=prod", "LOG_LEVEL=INFO"]
    jobs:
        docker_job:
            pipeline:
                pipeline_name: batch_pipeline
            executor: docker_exec
    ```

    See Also
    --------
    `kedro_dagster.config.models.MultiprocessExecutorOptions` :
        Base class providing concurrency and retry settings.
    `kedro_dagster.dagster.ExecutorCreator` :
        Builds Dagster executor definitions from these options.
    """

    image: str | None = Field(
        default=None, description="The docker image to be used if the repository does not specify one."
    )
    network: str | None = Field(
        default=None, description="Name of the network to which to connect the launched container at creation time."
    )
    registry: dict[str, str] | None = Field(
        default=None, description="Information for using a non local/public docker registry."
    )
    env_vars: list[str] = Field(
        default=[],
        description=(
            "The list of environment variables names to include in the docker container. "
            "Each can be of the form KEY=VALUE or just KEY (in which case the value will be pulled "
            "from the local environment)."
        ),
    )
    container_kwargs: dict[str, Any] | None = Field(
        default=None,
        description=(
            "Key-value pairs that can be passed into containers.create. See "
            "https://docker-py.readthedocs.io/en/stable/containers.html for the full list "
            "of available options."
        ),
    )
    networks: list[str] = Field(
        default=[], description="Names of the networks to which to connect the launched container at creation time."
    )