K8sJobConfig

kedro_dagster.config.models.K8sJobConfig

Bases: BaseModel

Configuration for Kubernetes jobs.

Parameters

Name	Type	Description	Default
container_config	dict[str, Any]	Configuration for the Kubernetes container.	required
pod_spec_config	dict[str, Any]	Configuration for the Pod specification.	required
pod_template_spec_metadata	dict[str, Any]	Metadata for the Pod template specification.	required
job_spec_config	dict[str, Any]	Configuration for the Job specification.	required
job_metadata	dict[str, Any]	Metadata for the Job.	required

Examples

executors:
    k8s_exec:
        k8s_job_executor:
            step_k8s_config:
                container_config:
                    image: "python:3.11-slim"
                    env:
                        - name: "KEDRO_ENV"
                          value: "prod"
                pod_spec_config:
                    nodeSelector:
                        nodepool: cpu
                pod_template_spec_metadata:
                    labels:
                        app: dagster-step
                job_spec_config:
                    backoffLimit: 3
                job_metadata:
                    labels:
                        team: platform
            per_step_k8s_config:
                op_name_overridden:
                    container_config:
                        resources:
                            limits:
                                cpu: "2"
                                memory: "2Gi"

See Also

kedro_dagster.config.models.K8sJobExecutorOptions : Uses this as its step and per-step configuration.

Source Code

View on GitHub

Show/Hide source

class K8sJobConfig(BaseModel):
    """Configuration for Kubernetes jobs.

    Parameters
    ----------
    container_config : dict[str, Any]
        Configuration for the Kubernetes container.
    pod_spec_config : dict[str, Any]
        Configuration for the Pod specification.
    pod_template_spec_metadata : dict[str, Any]
        Metadata for the Pod template specification.
    job_spec_config : dict[str, Any]
        Configuration for the Job specification.
    job_metadata : dict[str, Any]
        Metadata for the Job.

    Examples
    --------
    ```yaml
    executors:
        k8s_exec:
            k8s_job_executor:
                step_k8s_config:
                    container_config:
                        image: "python:3.11-slim"
                        env:
                            - name: "KEDRO_ENV"
                              value: "prod"
                    pod_spec_config:
                        nodeSelector:
                            nodepool: cpu
                    pod_template_spec_metadata:
                        labels:
                            app: dagster-step
                    job_spec_config:
                        backoffLimit: 3
                    job_metadata:
                        labels:
                            team: platform
                per_step_k8s_config:
                    op_name_overridden:
                        container_config:
                            resources:
                                limits:
                                    cpu: "2"
                                    memory: "2Gi"
    ```

    See Also
    --------
    `kedro_dagster.config.models.K8sJobExecutorOptions` :
        Uses this as its step and per-step configuration.
    """

    container_config: dict[str, Any] = Field(default={}, description="Configuration for the Kubernetes container.")
    pod_spec_config: dict[str, Any] = Field(
        default={}, description="Configuration for the Kubernetes Pod specification."
    )
    pod_template_spec_metadata: dict[str, Any] = Field(
        default={}, description="Metadata for the Kubernetes Pod template specification."
    )
    job_spec_config: dict[str, Any] = Field(
        default={}, description="Configuration for the Kubernetes Job specification."
    )
    job_metadata: dict[str, Any] = Field(default={}, description="Metadata for the Kubernetes Job.")