Neelesh Salian

Some of the foundational pieces like the Hive Metastore in my previous role were powerful and clunky. But, at the time it made sense.

We built a whole ecosystem to sustain us in the long term. Lessons learned and grown.

Link to the Berlin Buzzwords talk:
https://2021.berlinbuzzwords.de/session/building-metadata-ecosystem-using-hive-metastore

#data #datadon #metastore #metadata

Nov 5, 2022 at 9:34pmWeb
Show threadAnanth PackkilduraiNov 5, 2022
@neelesh_salian one advantage of Hive metastore is that; regardless of the processing framework, there is one metastore to integration. I feel this is where the Lakehouse systems & cloud data warehouses broke the data processing interface. Hive metastore failed on its own success :-(
Show threadDavid JayatillakeNov 5, 2022
@ananth @neelesh_salian I wonder if DB will try to do this with unity catalog, it's quite a strange catalog right now already.
Show threadAnanth PackkilduraiNov 5, 2022
@jayatid @neelesh_salian I heard the opposite. The unity catalog is not open enough for integration with other query engines like Trino because of the competing market share.
Show threadNeelesh SalianNov 5, 2022
@ananth @jayatid the demo I saw with the unity catalog brought in the Metastore as a sub-catalog (I guess?).
I just can't think of how to abstract the 3 levels of table naming to users. <catalog>.<sub-catalog>.<db>.<table>.
Show threadDavid JayatillakeNov 5, 2022
@ananth @neelesh_salian I suppose I meant one metastore to integration inside Databricks.
Show threadNeelesh SalianNov 5, 2022
@jayatid @ananth hello there vendor lock-in
Show threadDavid JayatillakeNov 5, 2022

@neelesh_salian @ananth yup 🤷‍♂️

At least you can access your files unlike with Snow