Diving deep into Spark batch processing!⚡️

Learned how to:
✅ Optimize data pipelines with filtering, repartitioning & grouping
✅ Design efficient ETL pipelines with Spark
✅ Understanding when and how to use partitioning strategies
✅ Use Google Cloud Storage (GCS) as a data source for Spark applications and configuring Spark to read Parquet or other formats from GCS
✅ Visualize execution plans for efficient coding
✅ Review the Spark UI for performance monitoring

💡 Key takeaway: One thing that amazes me about distributed computing is how we've transformed from struggling with massive datasets to generating insights in near real-time. As an analyst who has dealt with long wait times in processing data, spark saves so much time in getting results faster and make data-driven decisions more quickly.

Review my work here: https://github.com/ammartin8/data_engineering_zoom_camp/blob/main/modules/module_6/project_06/README.md

#mastodon #fediverse #data #spark #dataengineering #ai #technology #opensource #datatools #datapipelines #fedihire #wednesday #sql #observability #etl #python

Need to shrink or extract data quickly?

Explore Compress & Decompress Tools that help you reduce file size, encode data efficiently, and unpack compressed content with ease. Fast, simple utilities designed for developers and anyone working with data online.

🔗 https://99tools.net/category/compress-decompress-tools

#Compression #DeveloperTools #WebDevelopment #CodingTools #DataTools #Programming #DevUtilities #OnlineTools #TechTools #99Tools

Just completed a project building an end-to-end data pipeline for NYC taxi data using dlt 🚕📊! What a ride! 😅 The REST API extraction was particularly fun (in a challenging way) but dlt's modular design made it manageable. Here’s what I learned:

✅ Full life cycle: From REST API extraction to DuckDB loading, all in one framework
✅ Reproducibility: Tracked every transformation with dlt's lineage features
✅ Modular design: Defined reusable components for extracting and normalizing data
✅ Handles complexity: Seamlessly handled pagination from the API
Big takeaway: dlt isn't just tooling, it's a framework for thinking about data pipelines that emphasizes transparency and reproducibility which is essential for any modern data stack

#dlt #dataengineering #datapiplines #etl #fediverse #mastodon #opensource #oss #ai #data #linux #technology #duckdb #datatools

🛠️ Đã ra mắt CODED FLOWS – công cụ kéo‑thả Python trực quan, biến code AI thành các block tái sử dụng. Người dùng có thể kéo‑thả, chia sẻ package (ML, query DB, chuẩn dữ liệu) và xuất ra script Python. Giúp giảm việc viết lại mã lặp lại trong các dự án nội bộ. #CodedFlows #AI #Python #NoCode #Automation #CôngCụ #LậpTrình #DataTools

https://www.reddit.com/r/SideProject/comments/1qr6qna/i_built_a_visual_python_tool_that_turns_ai/

Building a Simple Tech Stack: Avoid Costly Mistakes and Boost Efficiency! #data #datatools

How to should you choose be best data tools to make an efficient stack? Keep it simple! source

https://quadexcel.com/wp/building-a-simple-tech-stack-avoid-costly-mistakes-and-boost-efficiency-data-datatools/

📰 Open Source 2026: Why OpenTofu, Biome, and Ollama Change Everything

Explore the seismic shifts in open source for 2026. From Rust-powered toolchains to local LLMs, discover why OpenTofu, Biome, and Ollama are the new standards.

#opensource #datatools #github #news

🌍 Also in: 🇪🇸 🇫🇷 🇩🇪 🇧🇷 🇮🇹

🔗 https://dataformathub.com/blog/open-source-2026-why-opentofu-biome-and-ollama-change-everything-lwu?utm_source=mastodon&utm_medium=social&utm_campaign=blog

Một công cụ phân tích dữ liệu thời gian miễn phí mới đã ra mắt! **Prophetize** cho phép dự báo nhanh bằng thuật toán Holt-Winters trực tiếp trên trình duyệt, không cần tải lên đám mây. Hỗ trợ phát hiện mùa vụ tự động, xử lý ngày tháng phức tạp và xuất kết quả sang Excel. Dễ dàng, không cần đăng nhập. #Technology #TimeSeries #DataTools #PhầnMềmMiễnPhi #PhânTíchDữLiệu

https://www.reddit.com/r/SideProject/comments/1pu0v1d/i_made_a_browserbased_alternative_to_excel_for/