just finished a painful legacy data migration at work and honestly the amount of things that can silently go wrong during ETL is underrated as a topic

spent three days debugging a pipeline that was "working" but dropping ~4% of records with no error logs. fun times

if you're going into something like this, worth reading up before you start:
https://cobit-solutions.com/en/services/etl-migration-consulting/

saved us from a few architectural mistakes we would've made otherwise

#data #ETL #dataengineering #softwareengineering