Used an app to measure the sound levels of my electric razor: It is 80+ dB. I had always felt that it was very loud but hadn't done anything about it. Now, I have started using it with regular earplugs and it is much better!

The blender in my kitchen is likely 90+ dB - planning to use earplugs for it.

I realized only recently that the decibel scale is a logarithmic scale. So, compared to 60 dB, 80 dB is 100 times more intense and 4 times louder.

Musical instruments that my kids play are at 90 - 95 dB - dangerous sound levels. Ordered some "music earplugs" that won't muffle the sound...

Many people think of "Distributed Tracing" as being valuable only when MANY apps/services in a system participate in it.

But even if you enable it for just a SINGLE service in your system, it can save you hours...

Here is WHY (based on a real issue):

We have one service that uses an open-source SDK to achieve a part of its functionality.

One day, we saw an issue where the calls made to this SDK are taking several seconds to complete.

Our service itself is instrumented using #OpenTelemetry Tracing APIs. But how do we diagnose this problem happening in the library it uses?

(Thread 1/3)

Earlier this week, I read the "The Tail At Scale" paper by Jeffrey Dean and Luiz André Barroso.

I really liked the intuitive techniques described in it.

I wrote the below blog post to try draw an analogy with a physical world example, and to summarize my main takeaways from it on:

- What is tail latency
- Why should we care about it
- Why reducing component level variability is not sufficient
- Two classes of patterns to become tail-tolerant

https://blog.techlanika.com/reducing-tail-latency-three-patterns-to-improve-responsiveness-of-large-scale-systems-47b5664baf61

The recent problems with Southwest Airlines is a good example of a Metastable failure at scale in the physical world:

TRIGGERs: Capacity reducing triggers (reduced staff capacity due to sickness, snow storms at Denver, Chicago, and the rest of the country).

AMPLIFICATION: Capacity degradation amplification caused by a combination of factors such as:
—-point to point business model meant the crew is not in the right places,
—-scheduling software breaking down resulting in manual matching of flights to crews - (can’t even imagine how tedious this would have been…kudos to the manual schedulers)
—-crew not able to communicate with the airlines (!) due to phone systems being down, likely due to a metastable failure of the phone system caused by overload due to customers trying to reach the airline for rescheduling..

So, even if the matching of a flight to a crew was done, the crew might not have been aware of that assignment! So, even as “system capacity” (airport, flights, crew) started becoming available, they couldn’t be used effectively…

MITIGATION: As with many metastable failure mitigations, load shedding was the mitigation- they temporarily reduced the number of flights to 1/3rd of the usual number…

Looks like the airline was running the system in an extremely vulnerable state (optimizing for high turnaround time to improve efficiency and packing the schedule without any headroom to handle overloads caused by capacity degradation).

Hope they do a thorough incident analysis using the metastable failure framework and make improvements…

References:

https://www.cnn.com/2022/12/27/business/southwest-airlines-service-meltdown/index.html

https://www.cnn.com/2022/12/29/business/southwest-airlines-service-meltdown/index.html

Just published this blog post on the beauty of the 4 positive feedback loops that writing brings to the table:

https://blog.techlanika.com/4-reasons-why-you-should-write-online-487e1b5831dd

Feedback/comments welcome.

Happy Holidays!

To read an industry/research paper, to understand/internalize it reasonably well, and more importantly to *retain* better the learnings from it (to be able to apply them later), two good forcing functions I have found useful are:

- Presenting/talking about it in a forum; committing to doing this creates an accountability/deadline to get it done.

(and/or)

- Writing/blogging about it in my own words.

[Edit: And of course, the above should be done with the goal that attendees or readers get good value out of it (since they are "paying" for it with their time)].

Along those lines, I had the opportunity to share my learnings from the TAOBench paper (https://www.vldb.org/pvldb/vol15/p1965-cheng.pdf) last month in the "Distributed Systems Reading Group" (thanks for the opportunity!) that's organized by Aleksey Charapko (http://charap.co/).

In case you find it useful, here's a link to my presentation: https://www.youtube.com/watch?v=PClXmtEetgA.

The same channel (DistSys Reading Group channel) also has recordings to a bunch of other presentations as well.