YAML Workflow

A lightweight workflow engine for CI/CD pipelines, data processing, and DevOps automation. Define reproducible, version-controlled workflows in YAML — run them locally, in CI, or on any machine with Python installed.

Why yaml-workflow?

Most workflow tools require servers, databases, and complex infrastructure. yaml-workflow takes a GitOps approach — workflows are plain YAML files, version-controlled alongside your code:

	yaml-workflow	Airflow / Prefect / Dagster
Setup	`pip install yaml-workflow`	Server, database, scheduler, workers
Configuration	Plain YAML files	Python DAGs + infrastructure config
Dependencies	2 (PyYAML, Jinja2)	50+ packages, Docker, PostgreSQL
Use case	Local automation, scripts, CI/CD, data pipelines	Enterprise orchestration at scale
Learning curve	Minutes	Hours to days
State	File-based, resumable	Database-backed

Choose yaml-workflow when you need:

Simple task automation without infrastructure overhead
Reproducible pipelines defined in version-controlled YAML
Batch processing with parallel execution
State persistence and workflow resume after failures
A lightweight alternative to shell scripts with better error handling
GitOps-friendly pipelines that live in your repo alongside the code
A single tool that runs the same pipeline locally and in CI

Features

YAML-driven workflow definition with Jinja2 templating
Multiple task types: shell, Python, file, template, HTTP, batch
Workflow composition via imports — reuse steps across workflows
Plugin system via entry points — pip install yaml-workflow-myplugin
Watch mode — --watch to re-run on file changes
Dry-run mode to preview without executing
Workflow visualization (ASCII branching DAG and Mermaid)
Parallel execution with configurable worker pools
State persistence and resume capability
Retry mechanisms with configurable strategies
Namespaced variables (args, env, steps, batch)
Flow control with custom step sequences and conditions
Extensible task system via @register_task decorator
Parallel step execution via depends_on — run independent steps concurrently
Secrets validation — fail fast if required environment variables are missing
Structured output (--format json) for CI integration and scripting
MCP server — expose workflows as AI agent tools (pip install yaml-workflow[mcp])
Web dashboard — monitor runs and trigger workflows (pip install yaml-workflow[serve])
GitHub Action — run workflows in CI with uses: orieg/yaml-workflow@v0.9.3

Use Cases

CI/CD pipelines — multi-step build, test, deploy workflows in YAML
Data processing — batch ETL pipelines with retry and resume on failure
DevOps automation — infrastructure tasks with secrets management and notifications
AI/LLM pipelines — orchestrate API calls with auth, retry, and batch processing
Local automation — replace shell scripts with reproducible, parameterized workflows

Quick Start

bash

# Install (isolated CLI — recommended)
pipx install yaml-workflow            # Core CLI
pipx install 'yaml-workflow[all]'     # + web dashboard + MCP server

# Or with pip
pip install yaml-workflow

# Initialize example workflows
yaml-workflow init

# Run a workflow with parameters
yaml-workflow run workflows/hello_world.yaml name=Alice

Example workflow (hello_world.yaml):

yaml

name: Hello World
description: A simple greeting workflow

params:
  name:
    type: string
    default: World

steps:
  - name: create_greeting
    task: template
    inputs:
      template: "Hello, {{ args.name }}!"
      output_file: greeting.txt

  - name: show_greeting
    task: shell
    inputs:
      command: cat greeting.txt

Visualize workflows

bash

yaml-workflow visualize workflows/data_pipeline.yaml

Code

  Workflow: Data Pipeline

  ┌─────────────────┐
  │  detect_format  │
  │   python_code   │
  └─────────────────┘
           │
           ▼
  ┌────────────────┐  ┌────────────────┐  ┌────────────────┐  ┌────────────────┐
  │  process_json  │  │  process_csv   │  │  process_xml   │  │ handle_unknown │
  │     shell      │  │     shell      │  │     shell      │  │     shell      │
  └────────────────┘  └────────────────┘  └────────────────┘  └────────────────┘
           │
           ▼
  ┌─────────────────┐
  │ generate_report │
  │   python_code   │
  └─────────────────┘

Adjacent conditional steps are automatically grouped as branches. Use --format mermaid to export for docs or GitHub rendering.

Dry-run mode

Preview what a workflow would do without executing anything:

bash

yaml-workflow run workflows/hello_world.yaml name=Alice --dry-run

Code

[DRY-RUN] Workflow: Hello World
[DRY-RUN] Steps: 2 to execute

  [DRY-RUN] Step 'create_greeting' — task: template — WOULD EXECUTE
    template: Hello, Alice!
    output_file: greeting.txt
  [DRY-RUN] Step 'show_greeting' — task: shell — WOULD EXECUTE
    command: cat greeting.txt

[DRY-RUN] Complete. 2 step(s) would execute, 0 would be skipped.
[DRY-RUN] No files were written. No tasks were executed.

Workflow composition

Reuse steps across workflows with imports:

yaml

# main.yaml
imports:
  - ./shared/logging_steps.yaml
  - ./shared/common_params.yaml

steps:
  - name: my_step
    task: shell
    inputs:
      command: echo "runs after imported steps"

Imported steps are prepended. Imported params provide defaults that the main workflow can override. Supports transitive imports with circular detection.

Parallel Steps

Run independent steps concurrently with depends_on:

yaml

steps:
  - name: fetch_api
    task: http.request
    inputs: {url: "https://api.example.com/data"}

  - name: fetch_db
    task: python_code
    inputs:
      code: "result = query_database()"

  - name: merge
    task: python_code
    depends_on: [fetch_api, fetch_db]
    inputs:
      code: |
        api_data = steps["fetch_api"]["result"]
        db_data = steps["fetch_db"]["result"]
        result = {"merged": True}

Watch mode

Automatically re-run on file changes during development:

bash

yaml-workflow run workflows/hello_world.yaml name=Alice --watch

Monitors the workflow file and all imported files. Press Ctrl+C to stop.

GitHub Actions

Run workflows in CI with the yaml-workflow action:

yaml

- name: Run pipeline
  uses: orieg/yaml-workflow@v0.9.3
  id: pipeline
  with:
    workflow: workflows/deploy.yaml
    params: |
      env=production
      version=1.2.0
    format: json

- name: Use results
  run: echo '${{ steps.pipeline.outputs.result }}'

Docker & Kubernetes

Run anywhere without installing Python:

bash

# Run a workflow in Docker
docker run --rm -v $(pwd)/workflows:/app/workflows \
  ghcr.io/orieg/yaml-workflow run /app/workflows/pipeline.yaml

# Start the web dashboard
docker run -p 8080:8080 -v $(pwd)/workflows:/app/workflows \
  ghcr.io/orieg/yaml-workflow

Deploy on Kubernetes with the Helm chart:

bash

helm install my-workflows ./helm/yaml-workflow \
  --set-file workflows.files.pipeline\\.yaml=workflows/pipeline.yaml

Compatible with ArgoCD (GitOps) and Argo Workflows. See the Kubernetes guide.

More commands

bash

# List available workflows
yaml-workflow list

# Validate a workflow (with JSON output for CI)
yaml-workflow validate workflows/hello_world.yaml --format json

# Resume a failed workflow
yaml-workflow run workflows/hello_world.yaml --resume

# Structured output for scripting
yaml-workflow run workflows/pipeline.yaml --format json --output results.json

Documentation

Full documentation is available at orieg.github.io/yaml-workflow.

Getting Started - Installation and first workflow
Task Types - Shell, Python, file, template, and batch tasks
Workflow Structure - YAML configuration reference
Templating - Jinja2 variable substitution
State Management - Persistence and resume
Task Development - Creating custom tasks
API Reference - Full API documentation

Contributing

Contributions are welcome! See the Contributing Guide for development setup and guidelines.

License

This project is licensed under the MIT License - see the LICENSE file for details.