💡 #DWDsTips use pv to monitor the loading of data into a database table
command-to-generate-data |
pv --line-mode --size ESTIMATED_NO_OF_LINES --progress --rate |
psql DB_NAME -c "\copy TABLE_NAME (...COLUMNS... FROM STDIN with (format 'csv', header false)"
My data is generated from a command which reads and transforms files - last time it took a few seconds shy of 90 minutes.
Having a progress estimate is really helpful
