Rini M6d ago
Built an end-to-end data pipeline using GCP, Airflow, PySpark, and BigQuery to analyze thermal anomaly data (India 🇮🇳 vs USA 🇺🇸).
Uncovered patterns in fire frequency, intensity, and seasonality through interactive dashboards.
#DataEngineering #GCP #Airflow #BigQuery #PySpark
Robert WatkinsFeb 10

How to run #PySpark on Windows.

1. Install it using `pip`
2. Make sure you're using Java 17 or Java 21. It won't work with Java 25.
3. Do _not_ fuck around with Hadoop Winutils, no matter how many articles or pieces of LLM "advice" you get about it.

https://twasink.net/2026/02/10/how-to-run-pyspark-on-windows/

How to run PySpark on Windows

TL;DR version: install either Java 17 or Java 21. Install PySpark (e.g. via pip). Run it. Longer Version Story Time As part of work for a client, we’ve been looking at using Microsoft Fabric …

Software is too expensive to build cheaply....
Marcel-Jan KrijgsmanJan 17
Now that I got a #datalakehouse in a #Europeancloud I want more: some actual data to query, #pyspark and notebooks and #datavisualisation. In this blogpost I get Project Jupyter Hub running, I get open data from the #EuropeanParliament and try to visualise that data in #PowerBI. Getting the connection working turned out to be an expensive affair.
https://marcel-jan.eu/datablog/2026/01/16/data-engineering-in-the-european-cloud-part-3-adding-data-and-visualisations/
Data engineering in the European cloud – Part 3: Adding data and visualisations – Expedition Data

Python Job SupportJan 11

ADF Interview Questions | Cloud Data Engineer #databricks #pyspark #adf #datafactory #microsoft

Q2. What are the main components of Azure Data Factory? Learn about the main components of Azure Data Factory! This video ... source

https://quadexcel.com/wp/adf-interview-questions-cloud-data-engineer-databricks-pyspark-adf-datafactory-microsoft-5/

ADF Interview Questions | Cloud Data Engineer #databricks #pyspark #adf #datafactory #microsoft - QuadExcel.com

Q2. What are the main components of Azure Data Factory? Learn about the main components of Azure Data Factory! This video ... source

QuadExcel.com
Python Job SupportDec 29

Spark SQL for Data Engineering 1 : I am going to start spark sql sessions as series. #sparksql

Spark SQL Part 1 : I am going to start spark sql sessions as series. #sparksql #deltalake #pyspark ' Databricks Notebooks code for ... source

https://quadexcel.com/wp/spark-sql-for-data-engineering-1-i-am-going-to-start-spark-sql-sessions-as-series-sparksql/

Python Job SupportDec 10

ADF Interview Questions | Cloud Data Engineer #databricks #pyspark #adf #datafactory #microsoft

Q13. How can you optimize the performance of an Azure Data Factory pipeline? Boost the performance of your Azure Data ... source

https://quadexcel.com/wp/adf-interview-questions-cloud-data-engineer-databricks-pyspark-adf-datafactory-microsoft-4/

ADF Interview Questions | Cloud Data Engineer #databricks #pyspark #adf #datafactory #microsoft - QuadExcel.com

Q13. How can you optimize the performance of an Azure Data Factory pipeline? Boost the performance of your Azure Data ... source

QuadExcel.com
Python Job SupportNov 17

ADF Interview Questions | Cloud Data Engineer #databricks #pyspark #adf #datafactory #microsoft

Q23. How did you handle a situation where a pipeline needed to run based on specific business events? Running pipelines ... source

https://quadexcel.com/wp/adf-interview-questions-cloud-data-engineer-databricks-pyspark-adf-datafactory-microsoft-3/

ADF Interview Questions | Cloud Data Engineer #databricks #pyspark #adf #datafactory #microsoft - QuadExcel.com

Q23. How did you handle a situation where a pipeline needed to run based on specific business events? Running pipelines ... source

QuadExcel.com
La Experimental NewslatterNov 14

Ya esta disponible La Experimental #14

🌐 Tendencias #Internet
💻 Gestión de #Git hooks
🧑🏻‍💻 Diseño #TUI con #GoLang
🐍 #Python sin GIL
💾 Guía de #PySpark SQL
🤖 Agente #IA local
🐧 Guía de seguridad #Linux
🌩️ Servicio de monitorización SelfHosted
💼 Informe laboral Tech en España de #manfred

Link: https://open.substack.com/pub/laexperimental/p/le-14?r=54y3on&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true

#tech #technology #selfhosting #gnulinux #cli #trabajo #opensource #development

La Experimental #14

Analisis web, gestor de Git hooks, diseño TUI en Go, Python sin GIL, guía PySpark SQL, agente de IA local, seguridad en Linux, monitorización de servidores self-hosted, informe de empleo tech España.

La Experimental
Python Job SupportOct 26

GCP Interview Questions | Cloud Data Engineer #gcpdataengineer #databricks #pyspark #gcp

Q12. How to handle schema evolution and versioning in a data lake architecture on GCP? Learn how to manage schema ... source

https://quadexcel.com/wp/gcp-interview-questions-cloud-data-engineer-gcpdataengineer-databricks-pyspark-gcp/

GCP Interview Questions | Cloud Data Engineer #gcpdataengineer #databricks #pyspark #gcp - QuadExcel.com

Q12. How to handle schema evolution and versioning in a data lake architecture on GCP? Learn how to manage schema ... source

QuadExcel.com
Python Job SupportOct 15

ADF Interview Questions | Cloud Data Engineer #databricks #pyspark #adf #datafactory #microsoft

Q10. How do you implement parameterization in an Azure Data Factory pipeline? Learn how to implement parameterization in ... source

https://quadexcel.com/wp/adf-interview-questions-cloud-data-engineer-databricks-pyspark-adf-datafactory-microsoft-2/

ADF Interview Questions | Cloud Data Engineer #databricks #pyspark #adf #datafactory #microsoft - QuadExcel.com

Q10. How do you implement parameterization in an Azure Data Factory pipeline? Learn how to implement parameterization in ... source

QuadExcel.com