brittany bennettit should not be be this difficult to learn data engineering. there are so many resources for learning SQL. and for leaning Python. and courses for becoming an analyst or analytics engineer. but i find there’s a dearth of resources for mastering the best practices for data engineering — common patterns, pagination, working with APIs. i can’t be the only one
i’ve taken half a dozen “intro to python” courses. i know how to use virtual envs and the debugger and how to write tests with pytest. i’ve worked with APIs and i’ve written a few ELT scripts myself. but i find myself still at the beginner level!! it’s maddening
SirLeeJackwagon@thebbennett I've been a data engineer for several years and have shifted into full stack swe and now doing some data science. It has been maddening throughout. I think @jsonbecker makes some good points. I think another piece that adds significant complexity here has to do with differing architectures and their associated costs. Docker+AWS ECR+AWS Lambda+AWS RDS(SQL) has dollar and learning costs, but that's but only one way to build a pipeline.