PipelineOptions

kedro_dagster.config.models.PipelineOptions

Bases: BaseModel

Options for filtering and configuring Kedro pipelines within a Dagster job.

Parameters

Name	Type	Description	Default
pipeline_name	str	Name of the Kedro pipeline to run. Defaults to __default__.	required
from_nodes	list[str] or None	List of node names to start execution from.	required
to_nodes	list[str] or None	List of node names to end execution at.	required
node_names	list[str] or None	List of specific node names to include in the pipeline.	required
from_inputs	list[str] or None	List of dataset names to use as entry points.	required
to_outputs	list[str] or None	List of dataset names to use as exit points.	required
node_namespaces	list[str] or None	Namespace(s) to filter nodes by.	required
tags	list[str] or None	List of tags to filter nodes by.	required

Examples

jobs:
  sales_etl:
    pipeline:
      pipeline_name: etl
      node_namespaces: ["sales", "shared"]
      tags: ["daily", "priority"]
      from_nodes: ["extract_raw_sales"]
      to_nodes: ["publish_clean_sales"]
      from_inputs: ["raw_sales"]
      to_outputs: ["clean_sales"]

See Also

kedro_dagster.config.models.JobOptions : Wraps this model alongside executor and schedule settings. kedro_dagster.utils.get_filter_params_dict : Extracts filter parameters from this configuration.

Source Code

View on GitHub

Show/Hide source

class PipelineOptions(BaseModel):
    """Options for filtering and configuring Kedro pipelines within a Dagster job.

    Parameters
    ----------
    pipeline_name : str
        Name of the Kedro pipeline to run. Defaults to ``__default__``.
    from_nodes : list[str] or None
        List of node names to start execution from.
    to_nodes : list[str] or None
        List of node names to end execution at.
    node_names : list[str] or None
        List of specific node names to include in the pipeline.
    from_inputs : list[str] or None
        List of dataset names to use as entry points.
    to_outputs : list[str] or None
        List of dataset names to use as exit points.
    node_namespaces : list[str] or None
        Namespace(s) to filter nodes by.
    tags : list[str] or None
        List of tags to filter nodes by.

    Examples
    --------
    ```yaml
    jobs:
      sales_etl:
        pipeline:
          pipeline_name: etl
          node_namespaces: ["sales", "shared"]
          tags: ["daily", "priority"]
          from_nodes: ["extract_raw_sales"]
          to_nodes: ["publish_clean_sales"]
          from_inputs: ["raw_sales"]
          to_outputs: ["clean_sales"]
    ```

    See Also
    --------
    `kedro_dagster.config.models.JobOptions` :
        Wraps this model alongside executor and schedule settings.
    `kedro_dagster.utils.get_filter_params_dict` :
        Extracts filter parameters from this configuration.
    """

    model_config = ConfigDict(extra="forbid")

    pipeline_name: str = "__default__"
    from_nodes: list[str] | None = None
    to_nodes: list[str] | None = None
    node_names: list[str] | None = None
    from_inputs: list[str] | None = None
    to_outputs: list[str] | None = None
    node_namespaces: list[str] | None = None
    tags: list[str] | None = None