10 Essential Tools Every Data Engineer Needs to BUILD Effective Cloud Pipelines

AWSDataEngineering #AWSGlue #Redshift AWS Data Engineering – Ultimate Guide to Master Cloud Data Pipelines Ready to ... source

https://quadexcel.com/wp/10-essential-tools-every-data-engineer-needs-to-build-effective-cloud-pipelines/

10 Essential Tools Every Data Engineer Needs to BUILD Effective Cloud Pipelines - QuadExcel.com

AWSDataEngineering #AWSGlue #Redshift AWS Data Engineering – Ultimate Guide to Master Cloud Data Pipelines Ready to ... source

QuadExcel.com
PostgreSQL Connection Timeout Error: Troubleshooting JDBC Connection Issues in AWS Glue
PostgreSQL Connection Timeout troubleshooting for AWS Glue & JDBC connections. Learn how to fix common data integration issues caused by network config, firewalls, and driver compatibility. Get practical solutions now! #PostgreSQLConnection #JDBCConnectionTimeout #AWSGlue #DataIntegration #DatabaseConnectivity #Troubleshooting
https://tech-champion.com/database/db2luw/postgresql-connection-timeout-error-troubleshooting-jdbc-connection-issues-in-aws-glue/
How to Become an AWS Data Engineer in 2024 : A Quick Guide - Career Swami

Learn the essential AWS services every data engineer should master, from S3 and Redshift to Glue and Lambda to become an AWS Data Engineer.

Career Swami
AWS GlueのJobをAWS CDKで管理する - Qiita

はじめに私たちのチームでは、企業が利用している人事システムなどのデータを取り込み、集計して、人的資本にまつわる様々な指標を表示したり、業界平均などと比較することのできるWebアプリケーションを開発…

Qiita

that feeling when you realizing you might be one of about six people in the world attempting to use an enterprise technology and it isn't clear that any of the six have ever gotten it to work.

(In this case an #AWSGlue to #Opensearch java library)

Amazon Q Data Integration in AWS Glue Simplifies Data Transformation on AWS
#amazonq #aws #awsglue

https://www.infoq.com/news/2024/02/amazon-q-aws-glue-etl/

Amazon Q Data Integration in AWS Glue Simplifies Data Transformation on AWS

Recently, AWS announced the preview of a new feature for AWS Glue, enabling customers to use natural language for authoring and troubleshooting data integration jobs. With Amazon Q data integration in AWS Glue, developers can provide a description of their data integration workload, and the service will generate an ETL script.

InfoQ

Why developer experience doesn't matter, as expressed by #AWSGlue and #Oracle

- Dev-ex is expensive, only developers enjoy it
- Punching the developer in the face (metaphorically) is immensely satisfying to product managers and is free
- If you don't metaphorically punch developers in the face they will get uppity and use your product.
- They don't deserve to use your product

You thought you were done with Python3.9 and Python3.10? Nope, because you decided to skill up & try to use #AWSGlue

You think maybe at least the 2 flavors of glue (spark and ray) could at least both get to 3.10, but nope.

#AWSGlue doesn't use regular python packaging, it asks the user to create a zip of dependencies and the users own code. (I count 3 different ways & they're documented in English not code.)

So I started writing a #python packaging tool for that

#AWSLambda s kind of do the same thing, but I already named the package. Sigh.

https://pypi.org/project/raypack/

raypack

Unofficial AWS Glue Ray.io packaging tool. Pure python or cross platform.

PyPI
@schizanon I understand the feeling but I am curious about what you hate specifically. Since I am into bioinformatics and data engineering, there are other services I find disturbingly buggy and badly designed and documented (like #awsglue and #awsOmics ) but always curious to know what other users experience. I am relatively new to #AWS but sometimes it surprises me how much time I invest helping #AWS debug software I am paying for. I understand small bugs but we have found major ones!