brittany bennettit should not be be this difficult to learn data engineering. there are so many resources for learning SQL. and for leaning Python. and courses for becoming an analyst or analytics engineer. but i find there’s a dearth of resources for mastering the best practices for data engineering — common patterns, pagination, working with APIs. i can’t be the only one
i’ve taken half a dozen “intro to python” courses. i know how to use virtual envs and the debugger and how to write tests with pytest. i’ve worked with APIs and i’ve written a few ELT scripts myself. but i find myself still at the beginner level!! it’s maddening
Jason Becker@thebbennett I think there’s a boundary between “scripting” and “developing” that’s really hard to bridge because there’s bad dialog between the two “camps” and little good work that understands one bridging to the other. (This is also at the heart of R v Python language wars)
@jsonbecker can you say more? i’m really interested in what you have to say here
@thebbennett I think there’s a world of scripts, where software largely has an input and an output and runs sequentially. We may organize in files or even classes and have tests, but there’s an entry and exit point with clear steps and goals. Then there’s a separate world where software exists unattended taking concurrent inputs and almost unknowable state. You can get surprisingly far on both sides without understanding the other. And each has valuable lessons for the other.
@thebbennett that may be the best I can do in 500 characters or less, but I’d love to discuss this at anytime. I think a lot of folks on the data side start in one world and adopt and adapt many tools from the other, but it’s incredibly hard to find tutorials appropriate to highly skilled people at one piece that help to bridge. The beginner materials don’t help.