Environment & Secrets

This repo has three secret-management surfaces. They use different vaults and different credentials:

1. Python orchestration/pipelines — Devbox → .envrc → export_settings.py → Dynaconf hooks → Azure Key Vault → OS keyring cache
2. Next.js dashboard (apps/mx_fuels_reporting_app/apps/web/) — loads secrets directly from its own vault via @azure/identity
3. CNE regulator credentials — dashboard writes credential files to a dedicated vault; Airflow reads them through its routed secrets backend

Prerequisite

Assumes direnv and devbox are already installed. See Fresh Computer Setup first if direnv allow errors with “command not found”.

Python pipelines: Devbox → direnv → Dynaconf

devbox shell (invoked automatically by direnv via the root .envrc) bootstraps the Nix-based environment in devbox.json, running the provisioning scripts in scripts/workspace-setup/ in order — venv creation, Homebrew, uv install, Jupyter kernel registration, git config, symlinks, Astronomer CLI.

The root .envrc loads devbox, then evaluates export_settings.py to export resolved secrets as shell variables:

.envrc

use_devbox() {
    watch_file devbox.json devbox.lock
    eval "$(devbox shellenv --init-hook --install --no-refresh-alias)"
}
use devbox
export ASTRO_TELEMETRY_DISABLED=1
export DISABLE_DYNACONF_CACHE=1

eval "$(python3 "$DEVBOX_PROJECT_ROOT/scripts/environ/export_settings.py" \
  --cache .cache/env_cache azure)"

export_settings.py maps short hook names to modules in packages/utils/src/utils/config/:

Hook	Module	Provides
azure	azure_hooks.py	AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_USERNAME
astronomer	astronomer_hooks.py	WORKSPACE_ID (Astronomer workspace)
postgres	postgres_hooks.py	AZURE_PG_TOKEN (fresh token from az account get-access-token)
web_app	web_app_hooks.py	WEB_KEY_VAULT_URL, CNE_CREDENTIALS_KEY_VAULT_URL

Azure-backed hooks fetch values via the az CLI, then cache them — first in .cache/env_cache (TTL-based), then in the OS keyring (default 1h TTL). The web_app hook derives non-secret vault URLs from vaults.py and does not call Azure. Secrets ultimately resolve through Dynaconf:

packages/utils/src/utils/settings.py

def load_global() -> Dynaconf:
    return Dynaconf(
        envvar_prefix="PIPELINES",
        settings_files=[
            CONFIG_DIR / "settings.py",
            CONFIG_DIR / "paths.py",
            CONFIG_DIR / "vaults.py",
        ],
        post_hooks=[azure_settings],
    )

Access from code:

from config import settings
settings.AZURE_CLIENT_ID

The vault names for this configuration live in packages/utils/src/utils/config/vaults.py. Two vaults back the Python settings: visory-devbox-scus-kv and visory-dev-scus-ds-kv (data science / orchestration secrets, e.g. Airflow storage account/container).

Dashboard secrets

apps/mx_fuels_reporting_app/apps/web/ is a separate trust boundary: it reads application secrets from visory-dev-scus-web-kv and writes regulator credentials to visory-dev-scus-cne-kv. It uses the production UAMI or the local developer’s Azure CLI identity. Route handlers/server components import named loaders from src/lib/keyVault.ts (loadWorkOSSecrets, loadPGSecrets, loadAzureStorageSecrets, loadDBSecrets, loadVDRSecrets) rather than using SecretClient directly, keeping the secret surface auditable and out of client bundles.

The dashboard app .envrc uses the web_app Dynaconf hook to export WEB_KEY_VAULT_URL and CNE_CREDENTIALS_KEY_VAULT_URL. Local compose.js.yaml passes both values to the web container. keyVault.ts requires both values and has no vault URL fallback.

The dashboard reads existing application secrets from visory-dev-scus-web-kv. It can only write CNE credentials to visory-dev-scus-cne-kv. The Airflow routed secrets backend reads CNE credentials. The dashboard never reads CNE secret values.

In production, the dashboard uses its attached UAMI. Locally, it uses the mounted developer Azure CLI session. Missing vault URLs fail explicitly.

Tip

The dashboard’s Postgres connection also supports a developer identity mode distinct from the service-principal mode — see RBAC & Identity.

If secrets aren’t resolving

1. Confirm you’re authenticated — any hook that shells out to az fails (or returns empty) otherwise:

   az login

2. Reload your shell environment:

   direnv reload

3. If a secret still looks stale (e.g. an expired AZURE_PG_TOKEN), clear the file cache and reload again:

   rm -f .cache/env_cache
   direnv reload

DISABLE_DYNACONF_CACHE=1 is set in .envrc to skip Dynaconf’s own cache layer — the file cache (24h TTL) plus OS keyring cache (1h TTL) are sufficient. To clear a single stuck key during debugging: keyring.delete_password(service, key).

Security rules

• Never cross-load application secrets — pipelines read visory-dev-scus-ds-kv / visory-devbox-scus-kv; the dashboard reads visory-dev-scus-web-kv. The dedicated CNE vault permits dashboard writes and Airflow reads only.
• Never commit secrets; .env, .envrc.local, secrets/ are gitignored.
• Never log sealed session cookies, WorkOS payloads, or raw secret values.
• Rotate service principal secrets quarterly; prefer managed identities where Azure supports them.

See docs/ai-experts/03-environment-security-engineer.md for the full architectural reference this page is based on.