ETL Helper is a Python ETL (Extract, Transform, Load) library to simplify data transfer into and out of databases.
It makes it easier to run #SQL queries via #Python.
This month we released Version 1. 🥳
One of the biggest changes is a proper documentation site. Check it out:
https://britishgeologicalsurvey.github.io/etlhelper/
#etlhelper #oracle #postgresql #sqlite #database #gis #opensource 1/3
There is a full list of changes and upgrade tips on the release notes.
These include:
+ Replacing cx_Oracle with python-oracledb driver
+ Data transfer uses dictionaries by default
+ Type hints
+ `log_to_console()` command
+ Integration tests run in GitHub actions
https://github.com/BritishGeologicalSurvey/etlhelper/releases/tag/v1.0.0
ETL Helper is available on PyPI: https://pypi.org/project/etlhelper/
The source code is on GitHub: https://github.com/BritishGeologicalSurvey/etlhelper
#etlhelper 2/3
ETL Helper is primarily developed by staff (past and present) at the British Geological Survey. These include myself, @ximenes, @decvalts, @ultrazool
We use it for managing geological data. As it solves a generic problem, we decided to make it #opensource.
There is a news item on the BGS website about the v1 release. It describes how ETL Helper is now used all around the world.
https://www.bgs.ac.uk/news/update-released-for-bgs-open-source-database-software-etl-helper/
#etlhelper 3/3
@brunogirin
ETLHelper is based on Python's DBAPI2 specification that defines how a database driver should work: https://www.python.org/dev/peps/pep-0249/
It is possible to add support for any other DBAPI2 compliant database. Unfortunately, it looks like PyMongo has a different API.
This API has been defined to encourage similarity between the Python modules that are used to access databases. By doing this, we hope to achieve a consistency leading to more easily understood modules, code that is generally more portable across datab...
@jlayt Good luck! Send any questions here or via GitHub.
From your bio, it seems like your work has a lot in common with the data management that we do in BGS Informatics group e.g. https://www.bgs.ac.uk/geological-data/data-search/
@volcan01010 Thanks, will look at it shortly.
We do indeed make use of BGS data, especially the borehole service which we pull into our GIS when assessing sites. We should look again to see what else you have we could use!
John Layt
Dr John A Stevenson
Bruno Girin