The recording of my @berlinbuzzwords presentation "All the DataOps, all the paradigms" is now online. I have observed that most teams are not aware of differences between data processing paradigms and their practical consequences, so I tried to contribute some order and structure. As usual, I tried to squeeze in too much and rambled, but I hope that it is comprehensible.

In terms of scope, this is the presentation that best describes what I have spent the last 10 years on - trying to help companies to adopt the practices and methods that evolved during the big data era. Methods that took us beyond analytics, enabled production quality data-powered products, unlocked machine learning for practical use, and ended the AI winter. Methods that most of the incumbents and enterprises never managed to master.

The Silicon Valley companies are ahead of us here, and I see this as a component of European sovereignty. The amount of money that is squandered is astronomical. On a regular basis, I encounter individual use cases within close reach where the stakes are 10s of MEUR. We need to capture those by stop accepting what the large American data vendors are spoonfeeding us - their incentive is to increase client spending, not revenue - and apply software engineering and our own thinking and innovation to leave data warehousing (now aka lakehousing) and other inadequate methods from the 90s behind us.

https://youtu.be/uev_27z3-1s

The slides from today's presentation "All the DataOps, all the paradigms" at Berlin Buzzwords are now online: https://www.slideshare.net/slideshow/all-the-dataops-all-the-paradigms/280671667

Video is expected in the next few days.

I always assumed that parameterised query implementations pack the query and values in some binary structure with field offset and length information, to properly separate the control and data planes. How naive.

It turns out that the MySQL python connector developers instead figured "Surely we can get this quoting right. What could possibly go wrong?"

Now I have a record of death that refuses to be inserted, resulting in "error in your SQL syntax," pointing to the data inserted, not to the query, since the connector mashes them both up in one big query.

So, one of the most costly design mistakes in the history of computing was remedied ... by repeating the same mistake.

My grandma passed away this week, at the age of 109, as the fifth oldest person alive in Sweden.

She moved from Narvik to Sweden in 1939. Her brother was killed in battle in WWII. She was probably the last person in Sweden who could say so. I am glad that she was unable to see the nazi salute at a presidential event, likely the first of many to come. Her generation saved us with weapons in hand. Now it is our generation that must be decisive, brave, and prepared to make sacrifices, or the next generation will have to do it holding weapons.

Everyone can contribute, but it is primarily up to those of us that have significant influence in economics, technology, or society to make active choices and use our talent. Rutger Bregman puts it better than I can. https://lnkd.in/dCHv9z8g

First post of the year on a somewhat sad note. I have decided to shelve public speaking and want to explain why.

For the same reasons, we have concluded that we cannot build a sustainable business on our value proposition. We have succeeded a few times to create the kind of collaborations we aim for and then managed to reach data engineering productivity KPIs normally reserved for the technical giants. We have achieved > 10x speed and cost efficiency compared to other data efforts we encounter. It has been really valuable in a few contexts with the necessary prerequisites in place, but they are too far apart, and we are now looking to put our capabilities to better use than we have done in the last few years.

At this point, we are open to all kinds of opportunities and want to see if our network has suggestions.

https://www.linkedin.com/pulse/three-conversations-data-lars-albertsson-ptluf/

Idag skriver vi en debattartikel i Ny Teknik där vi kommenterar AI-kommissionens rapport.

Våra kommentarer gäller att framdriften och innovationen inom AI inte kan vara huvudsakligen samhällets ansvar. Det mesta av AI-innovationen sker i våra företag. Och vilken tid vi lever i just nu. Å ena sidan pratar vi om att innovera inom artificiell intelligens medan vi å andra sidan ser företag och myndigheter misslyckas kapitalt med att leverera stora projekt som drivs med metoder från 1990-talet.

AI förutsätter data och det är lätt att se att de företag som leder AI-innovation också leder innovationen inom både datahantering och förmåga att leverera mjukvara. Skillnaderna mellan företag är enorma - detta kallas Data Divide eller numera AI Divide - och de växer.

Du kan läsa mer om Data Divide här: https://www.linkedin.com/pulse/data-divide-success-factors-vs-friction-lars-albertsson-vdrof/ där vi listar faktorer som vi har sett skilja framgångsrika företag från mängden.

Hur kommer man då ikapp? Det är en lång resa, men dessa är de åtgärder vi har sett göra störst skillnad:

1. Arbeta i korsfunktionella team, längs värdekedjan, så att de kan lösa uppgifter inom teamet och därmed iterera och innovera snabbare. Boken Team Topologies är en bra guide: https://teamtopologies.com/

2. Lämna 90-talets paradigmer med data warehousing bakom er och angrip utmaningar inom data / AI som den mjukvaruutveckling det faktiskt är. Boken Accelerate (https://itrevolution.com/product/accelerate/) beskriver vilka faktorer inom mjukvarutveckling som har visat sig leda till framgång och ger bra vägledning. Samma principer och faktorer är tillämpliga inom data och AI. Att skapa ett nytt dataflöde ska inte ta 6-8 veckor (vilket är vanligt), utan förväntan borde vara att kunna leverera funktionalitet t.o.m. fortare än för annan mjukvara.

3. Etablera vanor att ta bort det som har lite värde. De företag som lyckas rör sig inte bara fortare, de gör också färre saker. Tekniken value stream mapping är användbar för att identifiera arbete som saknar värde. https://www.marcusoft.net/2018/03/a-simple-diagram-on-flow-efficiency.html