CeleryK8sJobExecutorOptions

kedro_dagster.config.models.CeleryK8sJobExecutorOptions

Bases: CeleryExecutorOptions, K8sJobExecutorOptions

Options for the Celery-based executor which launches tasks as Kubernetes jobs.

Parameters

Name	Type	Description	Default
job_wait_timeout	float	Wait time in seconds for a job to complete before marking as failed.	required

Examples

executors:
    celery_k8s_exec:
        celery_k8s_job_executor:
            broker: "pyamqp://guest@broker//"
            backend: "rpc://"
            job_namespace: "dagster"
            job_wait_timeout: 43200
            env_vars: ["ENV=prod"]
            include: ["my_project.workers"]
jobs:
    celery_k8s_job:
        pipeline:
            pipeline_name: hybrid_async_pipeline
        executor: celery_k8s_exec

See Also

kedro_dagster.config.models.CeleryExecutorOptions : Provides Celery broker and backend configuration. kedro_dagster.config.models.K8sJobExecutorOptions : Provides Kubernetes job configuration. kedro_dagster.dagster.ExecutorCreator : Builds Dagster executor definitions from these options.

Source Code

View on GitHub

Show/Hide source

class CeleryK8sJobExecutorOptions(CeleryExecutorOptions, K8sJobExecutorOptions):
    """Options for the Celery-based executor which launches tasks as Kubernetes jobs.

    Parameters
    ----------
    job_wait_timeout : float
        Wait time in seconds for a job to complete before marking as failed.

    Examples
    --------
    ```yaml
    executors:
        celery_k8s_exec:
            celery_k8s_job_executor:
                broker: "pyamqp://guest@broker//"
                backend: "rpc://"
                job_namespace: "dagster"
                job_wait_timeout: 43200
                env_vars: ["ENV=prod"]
                include: ["my_project.workers"]
    jobs:
        celery_k8s_job:
            pipeline:
                pipeline_name: hybrid_async_pipeline
            executor: celery_k8s_exec
    ```

    See Also
    --------
    `kedro_dagster.config.models.CeleryExecutorOptions` :
        Provides Celery broker and backend configuration.
    `kedro_dagster.config.models.K8sJobExecutorOptions` :
        Provides Kubernetes job configuration.
    `kedro_dagster.dagster.ExecutorCreator` :
        Builds Dagster executor definitions from these options.
    """

    job_wait_timeout: float = Field(
        default=86400.0,
        description=(
            "Wait this many seconds for a job to complete before marking the run as failed."
            f" Defaults to {86400.0} seconds."
        ),
    )