CeleryDockerExecutorOptions

kedro_dagster.config.models.CeleryDockerExecutorOptions

Bases: CeleryExecutorOptions, DockerExecutorOptions

Options for the Celery-based executor which launches tasks as Docker containers.

Uses fields from both CeleryExecutorOptions and DockerExecutorOptions to configure Celery workers running in Docker.

Examples

executors:
    celery_docker_exec:
        celery_docker_executor:
            image: "myrepo/celery-worker:latest"
            broker: "redis://redis:6379/0"
            backend: "rpc://"
            include: ["my_project.workers"]
            env_vars: ["WORKER_POOL=default"]
jobs:
    celery_docker_job:
        pipeline:
            pipeline_name: async_docker_pipeline
        executor: celery_docker_exec

See Also

kedro_dagster.config.models.CeleryExecutorOptions : Provides Celery broker and backend configuration. kedro_dagster.config.models.DockerExecutorOptions : Provides Docker container configuration. kedro_dagster.dagster.ExecutorCreator : Builds Dagster executor definitions from these options.

Source Code

View on GitHub

Show/Hide source

class CeleryDockerExecutorOptions(CeleryExecutorOptions, DockerExecutorOptions):
    """Options for the Celery-based executor which launches tasks as Docker containers.

    Uses fields from both ``CeleryExecutorOptions`` and ``DockerExecutorOptions``
    to configure Celery workers running in Docker.

    Examples
    --------
    ```yaml
    executors:
        celery_docker_exec:
            celery_docker_executor:
                image: "myrepo/celery-worker:latest"
                broker: "redis://redis:6379/0"
                backend: "rpc://"
                include: ["my_project.workers"]
                env_vars: ["WORKER_POOL=default"]
    jobs:
        celery_docker_job:
            pipeline:
                pipeline_name: async_docker_pipeline
            executor: celery_docker_exec
    ```

    See Also
    --------
    `kedro_dagster.config.models.CeleryExecutorOptions` :
        Provides Celery broker and backend configuration.
    `kedro_dagster.config.models.DockerExecutorOptions` :
        Provides Docker container configuration.
    `kedro_dagster.dagster.ExecutorCreator` :
        Builds Dagster executor definitions from these options.
    """

    pass