Igor De SouzaApache Spark vs. Apache Kafka: A Comprehensive Technical Comparison
https://www.automq.com/blog/apache-spark-vs-kafka-comparison-event-streaming-processing
InfoQDatabricks is contributing the tech behind Delta Live Tables (DLT) to the #ApacheSpark project!
It will now be known as Spark Declarative Pipelines, making it easier to develop & maintain streaming pipelines for all Spark users.
đź”— Learn more: https://bit.ly/3IkaM3a
Andreas ScherbaumToday is the DBA Appreciation Day!
Bring your DBAs a cake and a coffee, please. And don't drop any tables in production, pretty please. It's weekend ...
#PostgreSQL #SQLServer #Oracle #DB2 #MySQL #MariaDB #Snowflake #SQLite #Neo4j #Teradata #SAPHana #Aerospike #ApacheSpark #Clickhouse #Informix #WarehousePG #Greenplum #Adabas
Overture Rede Private LimitedTraining Requirement: Freelance Trainer – Big Data & Spark
Location: Pune, Mumbai | Duration: Project-Based / Part-Time
Experience: 10+ years
đź“© Email: amritk1@overturerede.com
đź“ž Call/WhatsApp: +91 9289118667
You can also explore and apply to current openings here:
đź”— https://zurl.co/3fAbr
#BigData #ApacheSpark #FreelanceTrainer #DataEngineering #HiringNow #RemoteJobs #SparkSQL #DataFrames #KafkaStreaming #TechTraining #HadoopEcosystem #MLlib #RealTimeAnalytics
Ronan
Why Apache Spark is often considered as slow?
The question about why Apache Spark is "slow" is one of the most often questions I'm hearing from junior engineers and peoples I'm mentoring. While that is partially true, it should be clarified. TLDR – OSS Spark is a multi-purpose engine that is designed to handle different kinds of workloads. Under the hood of Spark is using a data-centric code generation but also it has some vectorization as well as option to fallbak to a pure Volcano-mode. Because of that Spark can be considred as a hybrid engine, that can benefit from all the approaches. But because of it's multi-purpose nature it will be almost always slower compared to pure vectorized engines like Trino on OLAP workloads on top of columnar data, except rare cases of big amount of nulls or deep branching in the query. In this blogpost I'm trying to explain the statement above.
Dirk Van den PoelLast week, the 2025 Edition of our “Current Data Science for Business Students Meet Alumni” Event took place at the Facultyof Economics and Business Administration (Ghent University). #ORMS #DataScience #DataAnalytics #Python #ApacheSpark #SQL
Current DS4B Students Meet Alumni (2025 Edition)
Last week, the 2025 Edition of the “Current Data Science for Business Students Meet Alumni” event was held at the Faculty of Economics and Business Administration of Ghent University, Belgium. Four DS4B graduates (Elke, Karac, Lieselot, and Charles) shared their work experience in Data Analytics and
Markus Breuer🚀 From 24h to 20min – A Small Change, Huge Impact!
A Spark query ran almost a full day on a large dataset. Stats showed 300GB traffic between worker nodes! 🔍 The Explain Plan revealed the culprit: a costly JOIN causing shuffles.
The fix? No JOIN needed! A simple filter replaced it—resulting in a 20-minute runtime instead of 24h.
đź’ˇ Lesson: Always check the Explain Plan!
Pedro FariaNew post about how to write data from a Apache Spark DataFrame into a Elasticsearch/Opensearch database #datascience #databricks #elasticsearch #opensearch #bigdata #apachespark #spark #tech #programming #python:
https://pedro-faria.netlify.app/posts/2025/2025-03-16-spark-elasticsearch/en/
heise online EnglishEasier to use: DuckDB gets local web user interface
As of version 1.2.1, the DuckDB in-process database can be conveniently operated via a local UI, which is installed as an extension, as an alternative to CLI.
heise DeveloperEinfacher bedienen: DuckDB erhält lokale Web-Benutzeroberfläche
Die In-Process-Datenbank DuckDB lässt sich ab Version 1.2.1 alternativ zur CLI komfortabel über ein lokales UI bedienen, das als Extension installiert wird.