Orchestration Overview

orchestration/ is an Astronomer CLI project (Apache Airflow) generated via astro dev init. It runs the scheduled pipelines that move and persist MX fuels reporting data, and owns the analytics database schema.

Layout

Path	Contents
`dags/`	DAG definitions: `mx_fuels_reporting.py`, `mx_fuels_backfill.py`, `db_migrations.py`, `db_rollback.py`, `test_connection.py`
`include/db/`	The active `db` package — SQLModel tables, Alembic migrations, persistence helpers (see Packages → db)
`include/mx_fuels_reporting/`	Shared business logic used by the reporting DAGs (assembly, XML/JSON schema validation, models)
`include/wheels/`	Pre-built wheels of root-workspace packages (`utils`) vendored into the orchestration image
`Dockerfile`, `requirements.txt`, `packages.txt`	Astro Runtime image build inputs
`Makefile`	`make bin` (rebuild wheels + relock), `make build`/`start`/`restart` for local Astro dev

Orchestration has its own UV workspace, separate from the root: [tool.uv.workspace] members = ["include/mx_fuels_reporting", "include/db"], with utils vendored as a wheel rather than a workspace member — see Architecture for why.

DAGs

DAG	Purpose	Trigger
`mx_fuels_reporting`	Assembles fuel volume/commercialization reports (XML/JSON), triggered by Service Bus queue events	Asset-driven (queue event)
`mx_fuels_backfill`	Backfills historical report data	Manual/scheduled
`db_migrations`	Runs Alembic upgrade/downgrade against the analytics database	Manual trigger only
`db_rollback`	Downgrades the schema (optionally destructively, to `base`)	Manual trigger only
`test_connection`	Smoke-tests the local/Azure Postgres connection	Manual

Both db_migrations and db_rollback select their Postgres connection based on ENVIRONMENT: Azure-Analytics-App-Postgres in production, Local_Analytics_Postgres otherwise.

Local development

cd orchestration
astro dev start

This spins up five containers (Postgres metadata DB, Scheduler, DAG Processor, API Server, Triggerer) and opens the Airflow UI at http://localhost:8080/.

Next

DAGs & Deploy — writing new DAGs and how deploys work
Database & Migrations — the Alembic migration/rollback workflow
Architecture → CI/CD — the orchestration-deploy.yaml pipeline