🚀 Just finished my #DEZoomcamp project! I built an end-to-end pipeline to process population frequencies from the European Variation Archive (EVA).

The Stack:
🛠️ Orchestration: #Bruin (Asset-based & lightweight)
📥 Ingestion: On-the-fly Python filtering
⚡ DWH: #ManticoreSearch for sub-second variant lookups
📊 UI: #Gradio dashboard
🐳 Env: #Docker & Codespaces & Cloud Run

Efficiency > Big Budgets. 🧬

🔗 https://github.com/tnotstar/data-engineering-zoomcamp-2026-project-attempt-1

#DataEngineering #Python #OpenSource #LearningInPublic #DataTalksClub

Module 7 of Data Engineering Zoomcamp done!

- Kafka producers and consumers
- PyFlink tumbling and session windows
- Real-time taxi data analysis
- Redpanda as Kafka replacement

My solution: https://github.com/tnotstar/data-engineering-zoomcamp-2026-07-streaming

Free course by @DataTalksClub: https://github.com/DataTalksClub/data-engineering-zoomcamp/

⚡ Module 6 of Data Engineering Zoomcamp done!

- Batch processing with Spark 🔥
- PySpark & DataFrames
- Parquet file optimization
- Spark UI on port 4040

My solution: https://github.com/tnotstar/data-engineering-zoomcamp-2026-06-batch

Free course by @DataTalksClub: https://github.com/DataTalksClub/data-engineering-zoomcamp

#dezoomcamp

dlt Workshop of Data Engineering Zoomcamp done!

- REST API pipelines with @dltHub
- AI-assisted pipeline building
- DuckDB as local data warehouse
- dlt Dashboard & marimo notebooks

My solution: https://github.com/tnotstar/data-engineering-zoomcamp-2026-workshop-1

Free course by @DataTalksClub: https://github.com/DataTalksClub/data-engineering-zoomcamp/

Module 5 of Data Engineering Zoomcamp done! 🫡

- Data Platforms with Bruin
- End-to-end ELT pipelines
- Data quality & lineage
- Deployment to BigQuery

Free course by @DataTalksClub: https://github.com/DataTalksClub/data-engineering-zoomcamp/

#dezoomcamp

Just wrapped up Module 4 of the #dezoomcamp with @DataTalksClub!

Dived deep into Analytics Engineering with dbt:
* Built production-ready models for NYC taxi data
* Managed lineage for 43M+ FHV records
* Wrote data tests to catch schema drifts

The "T" in ELT is powerful! 🚀

#DataEngineering #dbt #learninginpublic