CeleryExecutorOptions

kedro_dagster.config.models.CeleryExecutorOptions

Bases: BaseModel

Options for the Celery-based executor.

Parameters

Name	Type	Description	Default
broker	str or None	Celery broker URL.	required
backend	str or None	Celery backend URL.	required
include	list[str]	List of modules every worker should import.	required
config_source	dict[str, Any] or None	Additional settings for the Celery app.	required
retries	int or None	Number of retries for the Celery tasks.	required

Examples

executors:
    celery_exec:
        celery_executor:
            broker: "pyamqp://guest@localhost//"
            backend: "rpc://"
            include: ["my_project.workers"]
            retries: 2
jobs:
    async_job:
        pipeline:
            pipeline_name: async_pipeline
        executor: celery_exec

See Also

kedro_dagster.config.models.CeleryDockerExecutorOptions : Combines Celery with Docker container settings. kedro_dagster.config.models.CeleryK8sJobExecutorOptions : Combines Celery with Kubernetes job settings. kedro_dagster.dagster.ExecutorCreator : Builds Dagster executor definitions from these options.

Source Code

View on GitHub

Show/Hide source

class CeleryExecutorOptions(BaseModel):
    """Options for the Celery-based executor.

    Parameters
    ----------
    broker : str or None
        Celery broker URL.
    backend : str or None
        Celery backend URL.
    include : list[str]
        List of modules every worker should import.
    config_source : dict[str, Any] or None
        Additional settings for the Celery app.
    retries : int or None
        Number of retries for the Celery tasks.

    Examples
    --------
    ```yaml
    executors:
        celery_exec:
            celery_executor:
                broker: "pyamqp://guest@localhost//"
                backend: "rpc://"
                include: ["my_project.workers"]
                retries: 2
    jobs:
        async_job:
            pipeline:
                pipeline_name: async_pipeline
            executor: celery_exec
    ```

    See Also
    --------
    `kedro_dagster.config.models.CeleryDockerExecutorOptions` :
        Combines Celery with Docker container settings.
    `kedro_dagster.config.models.CeleryK8sJobExecutorOptions` :
        Combines Celery with Kubernetes job settings.
    `kedro_dagster.dagster.ExecutorCreator` :
        Builds Dagster executor definitions from these options.
    """

    broker: str | None = Field(
        default=None,
        description=(
            "The URL of the Celery broker. Default: "
            "'pyamqp://guest@{os.getenv('DAGSTER_CELERY_BROKER_HOST',"
            "'localhost')}//'."
        ),
    )
    backend: str | None = Field(
        default="rpc://",
        description="The URL of the Celery results backend. Default: 'rpc://'.",
    )
    include: list[str] = Field(default=[], description="List of modules every worker should import.")
    config_source: dict[str, Any] | None = Field(default=None, description="Additional settings for the Celery app.")
    retries: int | None = Field(default=None, description="Number of retries for the Celery tasks.")