Specific line of thought to illustrate my general point:
Consider an LLM that helps manage email correspondence. It writes emails! It summarizes emails! Less reading! Less typing! More messages faster! Productivity boost!! Except:
- You have to babysit the LLM, guide it and check it to make sure it’s accurately preserving human intent (which is, after all, the whole point of communication…right??). That’s new work, and likely cancels out the slim time savings of reduced reading and typing.
2/
- But it's an LLM, so it’s still often wildly, convincingly incorrect. Miscommunication increases. Miscommunication has costs. Miscommunication generates new work. Which now gets done faster! And generates yet more work!
- IT staff has to administer the LLM, support the LLM, evaluate vendors, yada yada.
- People have to maintain the LLM itself, and the infra that supports it. Those costs are •large•.
3/
And if by some magic all of this actually spins up and gets working, then (1) the barrier to communication decreases (why not just send another email if it’s automated?), (2) individual communication load increases (because you can answer emails at a faster rate), and (3) the net efficiency of communication decreases (because of everything in the previous two posts).
Sound and fury, signifying nothing.
4/
I severely doubt many real orgs measure actual desired large-scale outcomes well enough to spot that net efficiency decrease. All this is going to look like increased productivity. Will •be• increased productivity in the ways that most folks actually measure it.
But here, with the bird’s-eye view of a hypothetical, it’s clear: the total amount of work happening to achieve the same ends has •increased•.
5/
I said “reduce total workload.” What are some thing that accomplish •that•?
“Do we really have the problem we think we have?”
“There’s a simpler way.”
“Work from home!”
“Hmm, I’m going to think about my reader, and edit for clarity and emotional impact before sending this email.”
“We’re willing to pay for experience / expertise.”
“Things are going well. Head home for the day!”
“Maybe we don’t need to do this thing anymore. We can just choose not to have this problem.”
/6
@inthehands Fair points, but you seem to be comparing an LLM-based system against one with perfect efficiency, instead of the existing human-based system (which I’m certain has its own set of failings).
While it’s useful to know how an LLM system deviates from the ideal, I’d be far more interested in how it compares against the existing system. Personally, I don’t need a system to be perfect - I just need it to be better
@inthehands Hmm. Having been on the receiving end of thousands of emails throughout my career, I’m not convinced of this. I’m not ready to believe that the “failure rate” of LLM-generated emails is higher than that of human-generated emails.
This actually sounds like it’d be a really cool study to run