The problem
Event data does not arrive neatly ready for analysis. Records can be malformed, duplicated, or inconsistent with a contract. A useful pipeline needs to make those cases visible while moving valid events toward usable tables.
THE MEDALLION PIPELINE
What I built
The Bronze streaming job consumes JSON events from Kafka, checks contracts and quality, and separates valid from invalid records. Valid events go to Iceberg tables; invalid records have a quarantine table and a Kafka dead-letter path.
The writer checks repeated event identifiers against existing data. Silver transformations flatten nested cart, product, transaction, and event fields, then use bounded ingestion windows and upserts. dbt models build staging and analytics tables over the prepared data.
Make the workflow explicit
The repository contains an Airflow DAG that sequences Bronze, Silver, and dbt jobs through Kubernetes operators, alongside deployment manifests and a CI definition. These make orchestration and intended deployment reproducible in code.
Engineering tradeoffs
Quarantine preserves the possibility of investigation and repair. Identifier checks help manage repeated events. Writes to several storage systems still need careful coordination: these components alone do not establish an exactly-once guarantee.
This is an engineering prototype. The repository demonstrates pipeline and deployment structure; it is not evidence of a production service, measured throughput, or operating uptime.