MultiprocessExecutorOptions

kedro_dagster.config.models.MultiprocessExecutorOptions

Bases: InProcessExecutorOptions

Options for the multiprocess executor.

Parameters

Name	Type	Description	Default
retries	RetriesEnableOptions or RetriesDisableOptions	Retry configuration for the executor.	required
max_concurrent	int	Maximum number of concurrent processes.	required

Examples

executors:
    local_multi:
        multiprocess:
            max_concurrent: 4
jobs:
    heavy_job:
        pipeline:
            pipeline_name: heavy_pipeline
        executor: local_multi

See Also

kedro_dagster.config.models.InProcessExecutorOptions : Base class providing retry configuration. kedro_dagster.dagster.ExecutorCreator : Builds Dagster executor definitions from these options.

Source Code

View on GitHub

Show/Hide source

class MultiprocessExecutorOptions(InProcessExecutorOptions):
    """Options for the multiprocess executor.

    Parameters
    ----------
    retries : RetriesEnableOptions or RetriesDisableOptions
        Retry configuration for the executor.
    max_concurrent : int
        Maximum number of concurrent processes.

    Examples
    --------
    ```yaml
    executors:
        local_multi:
            multiprocess:
                max_concurrent: 4
    jobs:
        heavy_job:
            pipeline:
                pipeline_name: heavy_pipeline
            executor: local_multi
    ```

    See Also
    --------
    `kedro_dagster.config.models.InProcessExecutorOptions` :
        Base class providing retry configuration.
    `kedro_dagster.dagster.ExecutorCreator` :
        Builds Dagster executor definitions from these options.
    """

    max_concurrent: int = Field(
        default=1,
        description=(
            "The number of processes that may run concurrently. "
            "By default, this is set to be the return value of `multiprocessing.cpu_count()`."
        ),
    )