Data Engineering & Architecture Specialist

Full source: docs/ai-experts/07-data-engineering-architect.md

Scope

ETL pipeline design under projects/*/src/<project>/etl/, database schema management (SQLModel for Python, Prisma for web/site/), and high-performance data processing (Pandas/Polars/PySpark/Dask).

Key patterns

• Polars for datasets >1GB (5-10x faster than Pandas); PySpark for >10GB; lazy evaluation (pl.scan_csv + .sink_parquet) for memory efficiency.
• SQLModel combines SQLAlchemy + Pydantic for typed DB models; Pydantic-only models for non-table data transfer objects.
• DVC pipelines (dvc.yaml stages) for reproducible, dependency-tracked data transformations.
• Parquet + zstd compression for columnar storage; batch inserts (executemany/copy_from) for bulk loads.

Note

This persona’s code examples reference packages/db/src/db/conn.py, auth.py, and PostgresDBHelper.py — these are now entirely commented-out scaffolding, not in active use. The real, active database package for schema/migrations is orchestration/include/db/ — see Packages → db for the up-to-date picture.

When to consult

• Designing ETL pipelines in projects/*/etl/
• Managing database schemas
• Optimizing data processing performance
• Implementing data quality validation with Pydantic

See also: Packages → db, Orchestration → Database & Migrations.