Skip to content
Visory LLC

MX COM Reporting

What it does

Section titled “What it does”

Processes historical CRE (“Comisión Reguladora de Energía”) fuel commercialization reports (“Reporte Estadístico de Comercialización”) into a clean, consolidated dataset, plus a parallel core/ module that models the current reporting format against its SAT/CRE XML schema.

Pipeline (DVC-orchestrated, dvc.yaml)

Section titled “Pipeline (DVC-orchestrated, dvc.yaml)”

A historical-cleanup pipeline (src/mx_com_reporting/historical/):

  1. unzip_historical — unzips the raw historical archive, hashes it for change detection.
  2. build_metadata — walks unzipped files, builds a copy plan / metadata catalog (metadata.csv, summary.json, disregarded.txt).
  3. completeness — audits metadata for completeness gaps (completeness.csv).
  4. select_sheets — extracts the relevant sheet(s) from each workbook into per-file CSVs.
  5. cleanup — removes intermediate multi-sheet CSV artifacts.
  6. build_dataset — assembles selected sheets into a fuels dataset.
  7. dataset_dtypes — audits column dtypes across the dataset.
  8. split_by_kind / consolidate — splits records by report kind and consolidates into partitioned Parquet via polars (polars_consolidate.py).

Other modules

Section titled “Other modules”
  • core/xls_parsing_v2.py — parses the current (non-historical) Reporte de Volúmenes format against a proper XML schema (xmlschema, xsdata), producing typed ReporteVolumenes models with Producto/SubProducto/Sitio/SectorEconomico mappings. Uses utils.azure.keyvault for secrets and utils.system.cache for caching.
  • models/ — generated dataclasses from the CRE XSD (xsd_reporte_estadistico_comercializacion.py) plus lookup mappings.
  • historical_v2/full_pipeline.py — a newer, consolidated rewrite of the historical pipeline stages above.
  • notebooks/test.py — exploratory script exercising the separate orchestration/include/mx_fuels_reporting column-normalization/loading/cleaning code (not this project’s own modules) against sample import workbooks.

Data

Section titled “Data”

data/{raw,preprocessed,out,assets}; historical outputs land under data/out/historical_filing/ (metadata, completeness report, per-kind CSVs, consolidated Parquet).

Stack

Section titled “Stack”

polars, xmlschema, xsdata/xsdata-pydantic, openpyxl/python-calamine/xlrd (multi-format Excel reads), deltalake, dask[complete,dataframe], pyspark, prophet, shared utils package.

Relationship to other MX fuels work

Section titled “Relationship to other MX fuels work”

Distinct from both the MX Fuels Reporting App (apps/mx_fuels_reporting_app, the production reporting web app) and Axfaltec Variance Validation (projects/2026-05-11_axfaltec_variance_validation, a client-specific variance audit) — this project focuses on cleaning/consolidating historical CRE commercialization filings as a standalone dataset-building exercise. The exact data lineage between this project and the orchestration mx_fuels_reporting include package needs author confirmation.