#ApacheHudi 1.0 is now generally available!

The release introduces new features aimed at transforming data lakehouses into what the project community considers a fully-fledged "Data Lakehouse Management System" (DLMS).

Details on #InfoQ 👉 https://bit.ly/3E5AXZi

#AI #DataLake #opensource #DataAnalytics

New Blog on Table Format Interoperability:

https://medium.com/data-engineering-with-dremio/when-to-use-apache-xtable-or-delta-lake-uniform-for-data-lakehouse-interoperability-d89b120e2163

#dataengineering #apacheiceberg #deltalake #apachehudi #datalakehouse

New Blog on Table Format Interoperability:

https://medium.com/data-engineering-with-dremio/when-to-use-apache-xtable-or-delta-lake-uniform-for-data-lakehouse-interoperability-d89b120e2163

#dataengineering #apacheiceberg #deltalake #apachehudi #datalakehouse

More good stuff from Grab - this time writing about how they are building a realtime datalake with tools including #apacheFlink, #apacheHudi, #apacheSpark and #TrinoDB

https://engineering.grab.com/enabling-near-realtime-data-analytics

#dataEngineering #dataArchitectures #openSource

❓❓❓HOW LAKEHOUSE TABLE FORMAT WORKS❓❓❓

1. Engine reads table format metadata
2. Builds list of files with relevant data based on metadata
3. Scans those files and executes query

#DataEngineering #DataAnalytics #BigData #DataLakehouse #ApacheIceberg #ApacheHudi #DeltaLake

🔧🔨 How to Implement Write-Audit-Publish (WAP) - an exploration of how WAP can be done on @apacheicebergdevs, #apacheHudi, @deltalakeoss, @lakeFS , or #projectNessie

https://lakefs.io/blog/how-to-implement-write-audit-publish/?utm_campaign=Social%20media%20activity&utm_source=Mastodon&utm_medium=social&utm_content=blog_rm-wap2

🙋🏻 What is WAP?
👉🏻 Check out yesterday's blog: https://lakefs.io/blog/data-engineering-patterns-write-audit-publish/?utm_campaign=Social%20media%20activity&utm_source=Mastodon&utm_medium=social&utm_content=blog_rm-wap1

#dataEngineering #data #writeAuditPublish #datadon

Get a detailed overview of #DeltaLake, #ApacheHudi, and #ApacheIceberg as we discuss their data storage, processing capabilities, and deployment options https://dzone.com/articles/delta-hudi-and-iceberg-the-data-lakehouse-trifecta

#analytics #spark

How Infomedia built a serverless data pipeline with change data capture using #AWS Glue and #ApacheHudi

https://aws.amazon.com/blogs/big-data/how-infomedia-built-a-serverless-data-pipeline-with-change-data-capture-using-aws-glue-and-apache-hudi/

This blog from Onehouse about #ApacheHudi is interesting.

My eye was caught by the chart showing which organisations and companies contribute to the #opensource projects. We all know that DB dominates DL. I wonder if the balance on the other two will stay over time or if Onehouse and Tabular (circled) will start to grow.

https://www.onehouse.ai/blog/apache-hudi-vs-delta-lake-vs-apache-iceberg-lakehouse-feature-comparison

My Medium adventure enters a new phase: the first post for a Medium-held publication, Plumbers of Data Science, just got published :)

It's also more technical than my previous writings. The point is to introduce Apache Hudi in a softer way than the official documentation does at the moment. So, if you're interested in starting with Hudi, look no further :)

#apachehudi #apachespark #dataengineering

https://medium.com/plumbersofdatascience/apache-hudi-copy-on-write-explained-563f1d23d34f