💡 Databricks Advanced Security with RBAC, RLS & ABAC

Our newest blog-post summarizes authorization patterns on the Databricks platform. How do role-based and attribute-based access control mix with row- and column-level security? All you need to know in a concise little write-up:

🔗 https://www.nextlytics.com/blog/master-databricks-security-with-rbac-rls-abac

#databricks #dataengineering #datascience #sapdatabricks #businessintelligence #blog #azuredatabricks #datawarehouse #datagovernance #unitycatalog

The OpenSearch Data Prepper maintainers are happy to announce the release of Data Prepper 2.15. This version adds the ability to ingest data from Apache Iceberg, making it easier to keep OpenSearch in sync without custom pipelines. It also extends Prometheus support with a remote-write source and the ability to send data to open source Prometheus.

Big kudos to the community for keeping this project moving forward. Read the full blog here: https://bit.ly/4sotUhD

#OpenSearch #DataEngineering

The tricky part of machine learning isn’t training the model, but putting it into production.
The serverless pattern using Docker, FastAPI, AWS Lambda, API Gateway and ECR offers a clean, scalable architecture with no servers to manage.

A modern approach to deploying models as real, reproducible and easy-to-maintain APIs.
#MachineLearning #MLOps #AWS #Serverless #Docker #FastAPI #DataEngineering #FediTech

ETL In-Flight vs At Rest

In-Flight (Streaming):

Transform while data moves
Real-time results
Higher cost, lower latency
Kafka, Flink, Spark Streaming

At Rest (Batch):

Store first, transform later
Scheduled processing
Lower cost, higher latency
SQL, dbt, Spark Batch

Real-time or cost-effective? Your call!

#ETL #DataEngineering #Streaming #Batch

Data Inlining in DuckLake: Unlocking Streaming for Data Lakes – DuckLake
https://ducklake.select/2026/04/02/data-inlining-in-ducklake/

#DuckLake #DataEngineering

AI coding tools generate plausible but wrong SQL constantly. The fix isn't waiting for a smarter model.

AI skills are markdown files that encode domain knowledge into coding tools. No framework, just structured text in a repo.

The loop: code → review → handoff → skills update. Every session makes the next one smarter. One aggregation bug became a permanent rule enforced automatically.

Dori Wilson broke it all down at Data Debug SF. Full writeup:
https://blog.reccehq.com/ai-skills-for-analytics-eng

#DataEngineering #AI

Project Manager for Data Science -- Arnaout Lab @UCSF
Arnaout Lab @UCSF

See the full job description on jobRxiv: https://jobrxiv.org/job/arnaout-lab-ucsf-27778-project-manager-for-data-science-arnaout-lab-ucsf/

#computerengineering #computerscience #computervision #dataengineering #healthdatascience #medicalimaging #Projectmanagement #ScienceJobs #hiring #research
https://jobrxiv.org/job/arnaout-lab-ucsf-27778-project-manager-for-data-science-arnaout-lab-ucsf/?fsp_sid=10841