How to Deploy to Production¶

This guide shows you how to move from kedro dagster dev to a production Dagster deployment. Use this when your pipelines are working locally and you need to run them on a schedule, with monitoring, in a remote environment.

Prerequisites¶

A working Kedro-Dagster project (Getting Started)
Familiarity with your target infrastructure (Docker, Kubernetes, or a cloud VM)

1. Create a production environment¶

Kedro's environment system lets you maintain separate configurations per deployment target. Create a production dagster.yml alongside your existing base or local one:

kedro dagster init -e production

Edit conf/production/dagster.yml to configure production-appropriate settings:

executors:
  prod_executor:
    multiprocess:
      max_concurrent: 8

schedules:
  nightly:
    cron_schedule: "0 2 * * *"

jobs:
  training_pipeline:
    pipeline: data_science
    executor: prod_executor
    schedule: nightly

Run with the production environment:

kedro dagster dev -e production

2. Choose an executor¶

The executor determines where and how your pipeline steps run. Pick based on your infrastructure:

Executor	Use when
`in_process`	Single-machine debugging, simplest setup
`multiprocess`	Single machine, parallel steps
`docker_executor`	Steps need isolated environments or specific system dependencies
`k8s_job`	Running on Kubernetes with per-step pods
`dask`	Large-scale distributed computation on a Dask cluster
`celery`	Celery-based task queues are already part of your infrastructure

See the executor configuration how-to for YAML examples of each executor.

3. Build a deployable code location¶

Dagster deploys projects as code locations. The generated definitions.py in your Kedro source directory is the entry point.

Package your project as a Python package:

uv build

If deploying with Docker, create a Dockerfile:

FROM python:3.12-slim

WORKDIR /app
COPY . .
RUN pip install .

# Dagster loads the definitions module
ENV DAGSTER_HOME=/opt/dagster/dagster_home
ENV KEDRO_ENV=production

4. Configure the Dagster instance¶

See the Dagster deployment documentation for complete instance configuration options.

5. Verify before deploying¶

Before deploying, confirm that the production configuration translates correctly:

# List all definitions with the production environment
kedro dagster list-defs -e production

# Start the UI locally with production config to inspect
kedro dagster dev -e production

Check that jobs, schedules, and executor assignments appear as expected in the Dagster UI.