Skip to content
Visory LLC

DAGs & Deploy

Adding a new DAG

Section titled “Adding a new DAG”

DAGs live under orchestration/dags/, one file per DAG, using the Airflow DAG context manager and TaskFlow @task decorator (see db_migrations.py for a minimal example, mx_fuels_reporting.py for a more complete one with async I/O, asset watchers, and Pydantic models).

Conventions observed in existing DAGs:

  • Select the Postgres connection ID based on ENVIRONMENT (Azure-Analytics-App-Postgres in production, Local_Analytics_Postgres otherwise) rather than hardcoding one.
  • Manual-trigger-only DAGs (migrations, rollback) set schedule=None and document dag_run.conf options in the module docstring (doc_md=__doc__ surfaces this in the Airflow UI).
  • Never pass secrets through XCom — db_rollback.py’s upgrade_after_rollback task re-fetches its Postgres URI directly rather than accepting it from an upstream task, to avoid persisting it in the Airflow metadata DB.
  • Import shared business logic from include/ packages (e.g. from mx_fuels_reporting.assembly import ...) rather than duplicating logic in the DAG file.

Shared code in include/

Section titled “Shared code in include/”
  • include/mx_fuels_reporting/ — the reporting domain logic: report assembly (assembly.py), column name normalization, JSON/XML schema validation and serialization, Pydantic models (ReporteVolumenes, JsonReporteComercializacion), and trigger classes (AsbQueueEventTrigger for Service Bus-driven DAG runs).
  • include/db/ — the database package (models, Alembic migrations, persistence helpers) — see Database & Migrations and Packages → db.

Both are workspace members of orchestration/pyproject.toml’s own UV workspace, imported directly (from db import ..., from mx_fuels_reporting... import ...) rather than via the root workspace.

Local testing before opening a PR

Section titled “Local testing before opening a PR”
Terminal window
cd orchestration
astro dev start

Trigger DAGs manually from the Airflow UI (http://localhost:8080/) with any needed dag_run.conf, and check task logs for errors before pushing. orchestration/tests/dags/ holds automated DAG tests — run these too if you touch DAG structure.

Deploying

Section titled “Deploying”

Deploys are automatic on push to orchestration-production via .github/workflows/orchestration-deploy.yaml: make bin rebuilds wheels, then a Docker image is built and pushed to Astro along with the DAGs. See Architecture → CI/CD for the full pipeline and Conventions → Branches for the orchestration-development → orchestration-production branch pairing.