How to Become an AWS Data Engineer in 2024 : A Quick Guide
#dataengineeringonAWS #AWSservicesfordataengineers#bigdataonAWS #AWSGlue #AmazonRedshift #AWSLambda #AWSdataprocessing #careerswami #career #jobs #AWS
that feeling when you realizing you might be one of about six people in the world attempting to use an enterprise technology and it isn't clear that any of the six have ever gotten it to work.
(In this case an #AWSGlue to #Opensearch java library)
Recently, AWS announced the preview of a new feature for AWS Glue, enabling customers to use natural language for authoring and troubleshooting data integration jobs. With Amazon Q data integration in AWS Glue, developers can provide a description of their data integration workload, and the service will generate an ETL script.
Why developer experience doesn't matter, as expressed by #AWSGlue and #Oracle
- Dev-ex is expensive, only developers enjoy it
- Punching the developer in the face (metaphorically) is immensely satisfying to product managers and is free
- If you don't metaphorically punch developers in the face they will get uppity and use your product.
- They don't deserve to use your product
You thought you were done with Python3.9 and Python3.10? Nope, because you decided to skill up & try to use #AWSGlue
You think maybe at least the 2 flavors of glue (spark and ray) could at least both get to 3.10, but nope.
#AWSGlue doesn't use regular python packaging, it asks the user to create a zip of dependencies and the users own code. (I count 3 different ways & they're documented in English not code.)
So I started writing a #python packaging tool for that
#AWSLambda s kind of do the same thing, but I already named the package. Sigh.