Building data pipelines with Java and open source
I'm not feeling the warm fuzzies about trying to do GCP Dataflow (aka Apache Beam) with Typescript. Doesn't feel fully baked.
Also doesn't feel like a lot of doc/examples nor many people using it. I don't think our org should be on the bleeding edge of something like this for a core piece of our infrastructure.
Dunno. Will keep working on it for a bit.
Do you have a penchant for fine whiskey! This presentation embarks on a quest talk to analyze whiskeys produced by the world’s top 86 distilleries to identify the perfect single-malt Scotch. For fun, several Apache projects will be used. Groovy simplifies your data science code. Commons Math and Commons CSV let you write code for reading your data and your processing logic. Beam, Flink, Ignite, Spark and Wayang let you scale your machine learning applications.
#CaseStudy - Discover how #Yelp reworked its data streaming architecture with #ApacheBeam & #ApacheFlink!
The company replaced a fragmented set of data pipelines for streaming transactional data into its analytical systems, like Amazon Redshift and in-house data lake, using Apache data streaming projects to create a unified and flexible solution.
Dive into the details: https://bit.ly/3WgkTL7
#InfoQ #SoftwareArchitecture #EventDrivenArchitecture #DataPipelines #Streaming
Good morning ☕☀️
I am #recruiting a Data Engineer into my team.
We're building pipelines in #trivago that analyse, verify and normalise the content coming from partners, to deliver them clean to the rest of the company.
I am looking for a pragmatic engineer engaged for quality and stability, that takes the challenge to process big volumes of data. We use ATM #Python and #ApacheBeam in #GCP, but you don't need them to apply if you are using other data tooling.
We offer a competitive salary, a constant challenge, a very enthusiastic team and an authentic atmosphere of multicultural colleagues.
We're based in #Düsseldorf, and we work in english in an hybrid scheme of 2 days #homeOffice / 3 days in person. Also unlimited vacation days, 20 days per year fully remote, kitchen, coffee, daily fruits...
Tony Salerno
Marcus Fihlon (McPringle)
Denis 🌏
Mike Porter
Paul King
『 บูเบ๋อ 』
InfoQ
Xavi