Configuration Reference¶

Kedro-Dagster expects a standard Kedro project structure. The main configuration file is dagster.yml, located in conf/<ENV_NAME>/.

dagster.yml¶

Defines jobs, executors, schedules, and loggers for your project.

schedules:
  my_job_schedule:
    cron_schedule: "0 0 * * *"

executors:
  my_executor:
    multiprocess:
      max_concurrent: 2

loggers:
  my_logger:
    log_level: INFO
    handlers:
      - class: logging.StreamHandler
        stream: ext://sys.stdout
        formatter: simple
    formatters:
      simple:
        format: "%(asctime)s - %(levelname)s - %(message)s"

jobs:
  my_job:
    pipeline:
      pipeline_name: __default__
      node_namespace: my_namespace
    executor: my_executor
    schedule: my_job_schedule
    loggers: [my_logger]

Jobs¶

Each job maps a Kedro pipeline to a Dagster job, with optional filtering. A job can reference a pre-defined executor, schedule, and list of loggers by name.

Accepted pipeline parameters: PipelineOptions.

Executors¶

Define how jobs are executed: in-process, multiprocess, Docker, Celery, Kubernetes, etc. Each entry corresponds to a Dagster executor.

Configuration models per executor type are documented in the API reference.

Multiprocess example (MultiprocessExecutorOptions):

executors:
  my_multiprocess_executor:
    multiprocess:
      max_concurrent: 4

Docker example (DockerExecutorOptions):

executors:
  my_docker_executor:
    docker_executor:
      image: my-custom-image:latest
      registry: "my_registry.com"
      network: "my_network"
      networks: ["my_network_1", "my_network_2"]
      container_kwargs:
        volumes:
          - "/host/path:/container/path"
        environment:
          - "ENV_VAR=value"

wc -l /home/gigi/Workspace/stateful-y/kedro-dagster/docs/pages/reference/configuration.md! note The docker_executor requires the dagster-docker package.

Schedules¶

Cron-based schedules for jobs. See the Dagster scheduling documentation and ScheduleOptions.

Loggers¶

Custom loggers for Dagster runs. See the logging guide for configuration details and LoggerCreator.

definitions.py¶

Auto-generated by the plugin. Serves as the main entry point for Dagster to discover all translated Kedro objects. Contains the Dagster Definitions object registering all jobs, assets, resources, schedules, and sensors.

In most cases, you should not manually edit definitions.py; instead, update your Kedro project or dagster.yml.