Everything is a lot right now. Want to tune it all out for ~10 minutes and help us better understand your energy data needs? We're kicking off our first (hopefully annual!) PUDL community survey. Please boost! Both current and possible future PUDL users encouraged.

https://forms.gle/cT1ZY4yunhza3YLK9

#EnergyTransition #PUDL #OpenData #EnergyData #OpenSource

For the energy data nerds: we've got a new data release out. PUDL v2024.11.0 includes quarterly updates to EIA 860M, EIA 923 year-to-date, EIA 930, EPA CEMS, and final 2023 data for the EIA 861. Comment in this GitHub discussion if you find anything weird. (or just to say Hi 👋)

https://github.com/orgs/catalyst-cooperative/discussions/3967

#EnergyTransition #OpenData #PUDL

We want to apply to the Google Season of Docs for #PUDL but have never worked with an outside technical writer before. Does anybody have someone to recommend? It's a #Python project focused on producing open data describing the US energy system.

Cc: @turingway @choldgraf @yabellini @leahawasser

#PyData #WriteTheDocs #EnergyTransition #OpenData #OpenSource #EnergyMastodon

Now that we're putting all our denormalized output tables and analyses into the #PUDL DB, we've got a lot more #metadata to manage, and are trying to figure out how to best combine existing tools to do it.

GitHub Discussion: https://github.com/orgs/catalyst-cooperative/discussions/2546

Currently we store column, table, and dataset level information in big JSON-ish #python data structures, which are converted into objects using @pydantic models based (loosely) on the #FrictionlessData tabular data package abstractions.

#datadon

The @dagster folks interviewed us and did a write-up of our migration of #PUDL from a messy DIY #Python ETL to using their orchestration framework, which has thus far been a very positive experience. Unlike most of their users we are producing #OpenData outputs. Very curious to see if other non-profit / open-data users will adopt the platform:

https://dagster.io/blog/catalyst-cooperative-case-study

#DataEngineering #datadon #EnergyMastodon #OpenSource #EnergyTransition

As #PUDL moves toward distributing only data (and much more of it) rather than expecting everyone to run the software (with its 500+ dependencies...) we're going to deprecate our output management layer.

We see two possible deprecation paths. Should we go slow? Or rip the band-aid off now?

#OpenData #EnergyTransition #OpenSource #EnergyMastodon #datadon #pydata #EnergyTwitter

Discussion on GitHub here: https://github.com/orgs/catalyst-cooperative/discussions/2503

I did not realize you can post up to 100GB of data to #Kaggle and they provide access to computational resources and #Jupyter notebooks.

We're thinking about automatically posting all our #PUDL data there, and maybe running community competitions to help solve entity matching, anomaly detection, and imputation problems. Is there any downside to doing this?

#OpenData #MachineLearning #DataScience #EnergyTransition #EnergyTwitter #EnergyMastodon

https://www.kaggle.com/datasets/zaneselvans/catalyst-cooperative-pudl

A few #PUDL announcements!

https://github.com/orgs/catalyst-cooperative/discussions/2475

Our migration to @dagster is progressing rapidly. If you use PUDL and run the ETL yourself, and need help getting Dagster set up, feel free to sign up for office hours:

https://calendly.com/catalyst-cooperative/pudl-office-hours

Or ask for help in our GitHub discussions:

https://github.com/orgs/catalyst-cooperative/discussions

#OpenSource #OpenData #EnergyTransition #EnergyMastodon #datadon #DataEngineering #EnergyTwitter