SQLite for Everyone!
My talk from the 2024 Research Software Engineering conference is up on YouTube.
It's a 25-minute intro to the benefits relational databases and #SQL. The target audience is "past me", from when I was working with data as a research scientist and learning #python.
It includes a tour of useful tools including #sqlitebrowser, @qgis, @datasette and #etlhelper.
ETL Helper is a Python ETL (Extract, Transform, Load) library to simplify data transfer into and out of databases.
It makes it easier to run #SQL queries via #Python.
Documentation: https://britishgeologicalsurvey.github.io/etlhelper/
v1.0.2 is out now. It has a minor bug fix and also updates the test suite for Python 3.12 and 3.13.
Release notes: https://github.com/BritishGeologicalSurvey/etlhelper/releases/tag/v1.0.2
ETL Helper is primarily developed by staff (past and present) at the British Geological Survey. These include myself, @ximenes, @decvalts, @ultrazool
We use it for managing geological data. As it solves a generic problem, we decided to make it #opensource.
There is a news item on the BGS website about the v1 release. It describes how ETL Helper is now used all around the world.
https://www.bgs.ac.uk/news/update-released-for-bgs-open-source-database-software-etl-helper/
#etlhelper 3/3
There is a full list of changes and upgrade tips on the release notes.
These include:
+ Replacing cx_Oracle with python-oracledb driver
+ Data transfer uses dictionaries by default
+ Type hints
+ `log_to_console()` command
+ Integration tests run in GitHub actions
https://github.com/BritishGeologicalSurvey/etlhelper/releases/tag/v1.0.0
ETL Helper is available on PyPI: https://pypi.org/project/etlhelper/
The source code is on GitHub: https://github.com/BritishGeologicalSurvey/etlhelper
#etlhelper 2/3
ETL Helper is a Python ETL (Extract, Transform, Load) library to simplify data transfer into and out of databases.
It makes it easier to run #SQL queries via #Python.
This month we released Version 1. 🥳
One of the biggest changes is a proper documentation site. Check it out:
https://britishgeologicalsurvey.github.io/etlhelper/
#etlhelper #oracle #postgresql #sqlite #database #gis #opensource 1/3
@ultrazool On the GitHub page? I'm back at work next week, so will add then.
Coincidentally, I chatted with HW computer scientist about Airflow for their #dataengineering course. I confessed that I haven't really used it beyond the 2019 proof-of-concept we did.
#GitLab scheduled pipelines plus #etlhelper Python scripts cover many needs, although there are some jobs that I'd like to try with queues.
Dr John A Stevenson
