GlyphJul 11, 2024

I don't know why I've been thinking so much about "Agile". Maybe looking at LinkedIn too much (for unrelated reasons) has broken some demon seal within me, but despite the fact that I'm not interacting with it now, I find myself annoyed about it more than I have been in a while. Today I guess I've got a milquetoast defense of Agile & Scrum: a lot of what they get blamed for is just management trying to go too fast, without explaining.

(ICYMI, see previous for context <https://mastodon.social/@glyph/112735695495968878>)

Show threadGlyphJul 11, 2024
Imagine, if you will, a frictionless spherical development team in simple harmonic motion: they receive requirements from the business, they implement the requirements, they have the standups, they plan the poker, they complete the sprint, they do the retro, they improve the process for the next one. One can *imagine* how this works very nicely. Business gets regular results, developers get to communicate realistic estimates, nobody gets overworked, everybody's happy.
Show threadGlyphJul 11, 2024
The goal of the "agile" process in such a team is to provide a bidirectional communication channel. The business wants to say "we would like feature A", and to have engineering say "A will be delivered after time interval T" and to have *known error bars* on T. The engineers want the ability to negotiate with credibility. Engineering estimates are hard to do accurately but they are always better than uninformed business guesses about delivery timelines, so engineers want a say in that.
Show threadGlyphJul 11, 2024
In this model of the world, the business knows resourcing and the engineers know engineering, and everyone has enough information to stay in their lane and make good decisions. Why do planning poker? Why compute "velocity"? It's not to have a metric for _this_ sprint, the only use for such a metric is, in the _next_ sprint, for engineering to be able to say "no" in a structured, useful way.
Show threadGlyphJul 11, 2024
There are two forms of this "no". The business can commit to one of two things: they can either say A is immutable, in which case T has to flex, or T is immutable, in which case A has to flex. Agile — and Scrum in particular — is a set of deliberate practice exercises for engineers to get as accurate as possible with their estimates, so that these tradeoffs can be made into commitments, and the business can allocate resources without doing stuff like project cancellations and layoffs.
Show threadGlyphJul 11, 2024

There are a few unfortunate features of reality that interfere with this perfect vision, however:

The business does not *want* to flex on A or T. They want A to be "everything" and T to be "zero". Whenever asked to allocate more time (budget) or to cut features (revenue) the impulse is to say "no". Their job is to maximize A and minimize T, which means a group of people constantly saying "how about more T and less A" is a frustration, *even if* they're super good at their job.

Show threadGlyphJul 11, 2024

We engineers also don't want to give estimates. There is a hard upper bound on how good estimates can be; if we knew what we were doing, we would not call it "software". Software development is chaotic, in the "chaos theory" sense: highly sensitive to tiny, barely-observable differences in initial conditions. Consider being asked at the start of a project, "how many days will you be stuck debugging a mysterious issue that you do not understand at all?"

You know that number is not going to be 0.

Show threadGlyphJul 11, 2024

So the dynamic set up by Agile is to divide the business into two classes of person, each with a discrete responsibility. Management (a class of person who definitionally does not want to hear estimates) asking for estimates from engineering (a class of person who definitionally does not want to give estimates).

You can see how this might be a challenging environment in which to build mutual trust.

Show threadGlyph

The second aspect of the Scrum dynamic is that *none of these estimation-improving tools are free*.

Engineers go faster when we concentrate, uninterrupted, for long stretches of time. We are happier when we are faster. There's lots of science on how bad interruptions are. And for the neurodivergent like myself, it's even worse.

Also, managers want to manage risk, but they also want *speed*. There's no point in mitigating risk with a mitigation that costs more than the risk's downside.

Jul 11, 2024 at 5:14pmWeb
Show threadGlyphJul 11, 2024

Third, while the things that scrum injects into the development process necessarily have costs:

1. the meetings take time
2. they are interruptions
3. they create stress in the form of constant meta-cognition and work-about-the-work

They do not *necessarily* have any benefits *at all*. To understand why it is important to understand the concept of "deliberate practice" https://www.apa.org/education-career/k12/practice-acquisition

Show threadGlyphJul 11, 2024
Very briefly, the concept of "deliberate practice" is the idea that in order to be useful, time spent doing an activity must involve active feedback and the intention to incorporate it into your practice. If you spend 100 hours taking guitar lessons where a teacher is giving you instruction and carefully giving feedback on performance, you will probably improve a lot. If you aimlessly pluck the strings for the same amount of time, you will probably not.
Show threadGlyphJul 11, 2024
Many Scrum processes quickly become worthless ceremony; they are the aimless banging on the guitar strings without regard to the music. In order for estimates to get better, you can't just ask engineers for their estimates, you have to meticulously record each estimate, record the amount of time the task actually took to do, and plot the delta historically, then give feedback to the original estimator so they can get a sense of their own biases. And this adds *even more work*!
Show threadGlyphJul 11, 2024

So this is asking engineers to do a LOT of work to improve a skill which will only help management, without management building the trust that gives engineers confidence that it will be worth their while.

This begins with explicit reward structures.

Show threadGlyphJul 11, 2024

At the most basic level, engineers are typically rewarded for "impact"—which is to say "delivering features"—not for accurate estimates. In the Scrum model of the world, where risk-reduction is the most important thing, one of the most valuable things an engineer can do is say "I can confidently say that is impossible in the given time budget, and we shouldn't do it".

If your organization uses Scrum, do you know anyone who has been promoted into a senior or leadership role for having done that?

Show threadGlyphJul 11, 2024

But at a higher level, sometimes even the idea of "risk reduction" starts to fall apart as even a *logical* construct, before you get to questions of priority and rewards.

Consider a green field project, whose ultimate goal is to achieve goal "G", which needs A B C D E and F prerequisites. It has a budget of $M money.

The goal of estimating this project accurately is to avoid running out of money. You don't have revenue, you don't have an existing product.

Show threadGlyphJul 11, 2024
Let's even ignore the fact that this team hasn't worked together yet and Scrum will even tell you that you can't really get an accurate team-level velocity without history. Pretend we have dead-on accurate velocity measurements. We do the estimates. We're agile, so we won't estimate G, we're still just doing A. We spend M/3 on engineer time to (highly accurately!) estimate that itself A will take M/3. We are now 1/3 of our way through our budget and we know we are destined to fail.
Show threadGlyphJul 11, 2024
A more "agile" process would be to start with zero ceremony, to say "we are going to go as fast as we can, and maybe we are going to run out of money, but let's see what we can do". Let the engineers focus for long uninterrupted stretches of time. Only do meetings when necessary. Maybe this will result in a clever hack to ship G more minimally. This is much "riskier" in the sense that the variance on outcomes is higher, but it is less risky in that it is *much more likely to succeed*.
Show threadGlyphJul 11, 2024

Even in established organizations, a lot work is like this. Management says: "We absolutely need 'A'. It is a regulatory requirement, we cannot skip any of it." and then a ton of effort goes into estimating every sub-part of 'A'.

Why? There is no decision that an accurate estimate of 'A' could inform! It just has to get done, it's existential for the product, so engineers should be left alone to do it. Everybody knows it. Everybody knows the ceremony is useless. But it proceeds due to inertia.

Show threadGlyphJul 11, 2024
This erodes engineering's trust in management. Clearly all the estimates are pointless busy-work, they don't listen to feedback in the retros when engineers say there are too many meetings. So engineers produce garbage estimates to tick the busy-work box as quickly as they can, and everyone is miserable. To say nothing of the impacts that the inevitable layoffs have.
Show threadGlyphJul 11, 2024
If you are a manager and you want to lead an organization that uses something like Scrum effectively, here are the steps to take:
Show threadGlyphJul 11, 2024
1. Be honest with yourself. Why are you doing this process? Do you value absolute speed, or do you want accurate estimates? Does delivering functionality in "sprints" even make sense for your target market? Are you soothing your own feelings of uncertainty and risk or are you providing solid plans for the business?
Show threadGlyphJul 11, 2024
2. Treat your engineers as colleagues, not resources. If you have a business question that needs answering *tell them what the question is*, don't collect a bunch of metrics and tell them mysteriously that it will be "taken into consideration" during the "leadership offsite" or whatever. Tell them clearly, something like: "we don't know if we can get enough revenue from this feature to justify a team to maintain it. Here's the revenue we project, here's how many engineers that pays for."
Show threadGlyphJul 11, 2024
3. Close the loop. If you're doing a ton of ceremony to improve estimates and maintain consistent progress and feature delivery in sprint-sized increments, then *measure that shit, and tell your people about it*. Allow them to engage in deliberate practice. Reward accuracy in estimations. If you're paying for the costs, pay enough to actually get the benefits. And if you're not, then *stop*. Recognize the cost, and drop the useless meetings.
Show threadGlyphJul 11, 2024
3(a). I've been talking a lot about estimation, but a lot of this applies equally to the "sprint" concept, in that they're supposed reduce risk by delivering features in smaller increments. But often agile teams are *not* doing that, just shipping "refactorings", because the sprint feedback loop is actually too short for significant value-delivery. Some features just take a few months to bake and if you don't allow for that you make it impossible to develop them without subverting your process.
Show threadGlyphJul 11, 2024
The only way you can tell if you're delivering that value is by having a way of measuring concrete value delivered, and ruthlessly classifying "stories" (value directly delivered to users) vs. "tasks" (things engineering needs to do *in order* to deliver that value) and actually caring about the difference. It is extremely difficult to maintain the institutional will to both do this *and* not start rewarding one over the other so that engineers won't start stealthily reclassifying them.
Show threadGlyphJul 11, 2024
Whew. Okay. That was a lot of feelings about Agile which I hope were useful to somebody. That probably should have been a blog post, and maybe I'll wrap it up as one, but I guess doing it live meant that it didn't die in the drafts as I fussed over wording and finding citations forever.
Show threadChristopher NeugebauerJul 11, 2024
@glyph It's very good, thank you. I'd love it as a blog post, and I think a "how to read this post critically" instruction at the top might be handy. I think mine is close, but not quite right?
Show threadGlyphJul 11, 2024
@chrisjrn OK time to start fussing over wording forever! ;-)
Show threadGlyphJul 11, 2024
@chrisjrn I think this is the beta version of the post, but perhaps not the alpha. I do really like your addition and I will try to incorporate it … somehow
Show threadAlmadJul 11, 2024

@glyph Haha, I do have a scrum post in my draft folder since 2020, so I can emphatize.

Good summary.

Show threadDan SloanJul 12, 2024
@glyph I enjoyed reading through this thread. It matches much of my experiences on both sides of the divide, and you articulated a lot of thoughts I’ve had far more clearly than I’m able, as well as filling in some gaps for me.
Show threadChristopher NeugebauerJul 11, 2024
@glyph my team recently stopped doing sprints because of this exact reason. I work in healthcare, and our project's delivery timeframe is subject to regulatory compliance (surprise). In cases like that, a methodology that assumes that all constraints are internal doesn't work. We seem to be a lot happier not kidding ourselves about delivering "deliverables" (not actually) every two weeks.
Show threadAndy ToddJul 13, 2024
@chrisjrn @glyph that's certainly something that resonates for me - coming up with 'deliverables' every 2 weeks always feels a little disingenuous to me, if something is going to take a couple of months then it should be doable. The question, I guess, is how do you measure progress and ensure you are on target to hit your estimate - or change it when reality intrudes?
Show threadGlyphJul 13, 2024
@andy47 @chrisjrn sadly I do not have a ready-made drop-in solution for this, only the observation that scrum (or more generally "agile") is *not* a drop-in solution. in order to do what you're saying, you need *some* structure that builds in frequent check-ins and re-estimation, but that also allows for a blame-free "oops, that approach didn't work, let's abandon it quickly) rather than trying to keep all the work that is already on the sprint board graven into stone
Show threadChristopher NeugebauerJul 13, 2024
@glyph @andy47 yeah, that's it. Check-ins on a regular basis (every two weeks, for instance) are necessary, possibly even with pre-defined fortnightly progress markers ("how close are we to delivery"?). I've seen 6-week deliverable cycles with fortnightly checkins work well before, too.
Show threadAN/CRM-114Jul 12, 2024

@glyph It’s also terrible for infrastructure because like a bridge or tunnel most of the work is below the surface and of zero value until 95% in and the shark tank managers hate that.

This is how you end up with five thousand hello world repos instead of a codebase

Show threadJason Petersen (he)Jul 11, 2024
@glyph is this universal advice? I’m not sure I would care about these things. I don’t think everyone does. Or should. Sometimes silos are fine.
Show threadGlyphJul 11, 2024
@jason I think this is super nuanced, I also think silos are actually pretty good and it's easy to require too much communication to get anything odne, but decision-making like this needs to be integrated vertically *into* your silo. You don't need to know exactly how the decision is made, but if you're at the pointy end of the data-gathering process to support a decision and you don't know what that decision is, you will often be collecting the data in subtly wrong ways that make it gibberish
Show threadJason Petersen (he)Jul 11, 2024
@glyph ah yeah that makes sense. Context-sharing, I can see that angle here.
Show threadAndrew BartlettJul 11, 2024

@glyph A value I can see in the estimates and then measuring against estimates is to locate the developer stuck in the weeds.

But only if knowing that actually can change things, eg bring in a second look at the problem.

Otherwise the interminable "so how long from here" is just stress.

Show threadGlyphJul 11, 2024
@abartlet it is intensely difficult to set up a system where a programmer getting stuck will be quickly identified but *also* not feel constantly surveilled.
Show threadAndrew BartlettJul 11, 2024

@glyph I think I did well with that as tech lead rather than management, just asking them to chat about their work and perhaps if I could help, but my fellow engineers might disagre.

I think this shows that this is the role of the standup not the estimation process.

Show threadLyallJul 12, 2024

@glyph @abartlet I've had a great experience on some teams recently keeping developers unstuck with a constellation of: psychological safety; collaborate early; collaboration is more valuable than working on another card; learning/experiment culture; leaders demonstrating learning; chat about what annoys you today.

It's mostly about engineers supporting engineers, and management trusting that collaboration is more effective than individuals working in parallel.

Show threadjamesJul 12, 2024
@LyallMorrison dude I’d love to hear more about this.
Show threadLyallJul 12, 2024

@james Hmm... what could I say? We're doing a performance review round right now so I'm reflecting a lot on processes and impact.

Would you like to hear more about the team practices I mentioned or about how that fits into company structure?

Show threadjamesJul 12, 2024
@LyallMorrison are you still at the place we met? I would love to be able to move our culture (which is not in any way bad) more towards these ideals but I feel that in a consultancy when the rubber hits the road it’s more about keeping people billable than keeping them safe.
Show threadLyallJul 12, 2024

@james I've moved on since then (working for Envato; opinions are my own etc.) but I have a great fondness for some of the practices and ideals we had.

At Envato we're working on internal software for a digital storefront so the pressures are different - no external contracts to fulfill. Teams still have to justify their existence but it's not as cutthroat.

For example, it's comparatively easy to assign work to developers to maximise their career growth rather than to ship faster.

Show threadjamesJul 12, 2024
@LyallMorrison yeah I think that’s the rub. I’ve been working with our leadership to try and add some product to our portfolio which as you know is a slow road to success but we have just launched a premium support offering for Ash which at the moment is staffed by principals but I’m hoping to use it as a skill ladder for team members who are just treading water on their current project.
Show threadLyallJul 12, 2024

@james So, one concrete thing I would fight against is the idea that one developer = one card (in Trello, Jira, whatever your planning tool is).

That caused so much trouble for my teams in the past by creating an artificial isolation and preventing collaboration (because all the other developers were 'committed' to marching their own cards across the board, for no real reason)

Show threadLyallJul 12, 2024
@james We should have a coffee some time - I work remotely from Tawa these days
Show threadjamesJul 12, 2024
@LyallMorrison I’m in Wellington for a few hours on Thursday - dropping one kid off at the airport stupid early and picking another up around 11am. I was going to find a cafe and fuck around on my laptop. Otherwise I’m doing the opposite on Sunday night. Also if you come to Wairarapa for any reason you should both stop by - we’d love to see you. We’re in Featherston.
Show threadPotato Enthusiast 🇵🇸Jul 12, 2024

@james @LyallMorrison

Hi! Want me to pick u u from airport? I live next door

Show threadjamesJul 12, 2024
@Br3nda @LyallMorrison nah I’m driving in from Featherston with my son and swapping him for his older sister later that morning.
Show threadLyallJul 12, 2024
@Br3nda @james Brenda, what cafe do you recommend these days for remote working? I used to like Polo but I haven't been there for a long time
Show threadLyallJul 12, 2024
@glyph @abartlet it's also not about estimates or predictability at all - just doing the work faster and less painfully which is difficult to sell to management since it doesn't _measurably_ benefit project timelines.
Show threadGlyphJul 13, 2024
@LyallMorrison @abartlet yah it's that trust from management that is key, and I've never seen that fall out the other end of a scrum training; in my experience it's a cultural value that management either has or doesn't
Show threadMatt BoydJul 11, 2024
@glyph My former boss (a mutual acquaintance actually) told us to read the agile manifesto at https://agilemanifesto.org/ and the 2014 "Agile is dead, long live agility" essay at https://pragdave.me/thoughts/active/2014-03-04-time-to-kill-agile.html and that was the beginning and end of our agile training.
Manifesto for Agile Software Development

We are uncovering better ways of developing software by doing it and helping others do it. These are our values and principles.

Show threadJason Petersen (he)Jul 11, 2024

@3psboyd @glyph this probably puts you in the 99% percentile of teams “using agile” in terms of any training whatsoever, I’d guess.

Usually it’s something like: show up, be told “this is when the meetings are and what we do in them, that’s the training, play along”

Show threadjohnaldisJul 11, 2024
@glyph What it tells you (at best) is that “A” will take long enough that we absolutely have to cut one of “B” and “C” if we want to have “D”. Now the business which has committed to D needs to decide whether B or C is more important, and go about managing client expectations, while engineering cranks out “A” as fast as possible.
Show threadGlyphJul 11, 2024
@johnaldis But, what if — just spitballing here — management & sales *instead* all had a collective freak-out and just yelled at the engineers until they got browbeaten into acquiescing and lied about being able to deliver B and C on a timeline that would still make the customer happy? Bam! Now you're steve jobs!
Show threadjohnaldisJul 13, 2024
@glyph You’re only Steve Jobs if you can *also* make the engineers who lied that they could deliver B and C actually deliver those things. (Which mainly involves finding the right engineers in the first place, but may also require you to bribe them with pineapple pizza…)
Show threadjohnaldisJul 13, 2024
@glyph I have no special information here but my feeling is 1980-era Steve Jobs would not have accepted cargo-cult agile—but his strategy of “internal disruption” by making the Mac compete with the Lisa and Apple III, while deeply unpleasant for many people, was “real agile” at some level. By accounts I’ve read, Apple III got mired in “design meetings” with little to no engineer input, whereas Mac development was engineers doing what they thought best, looped into requirements.
Show threadGlyphJul 13, 2024
@johnaldis while I think your observations here are *probably* pretty accurrate, the evocation of steve jobs as an avatar was a sarcastic jab at doing analysis like this — it doesn't actually matter what Steve Jobs thought, he was an asshole, he was building different products with different technology and very few of the lessons of his life really generalize. if you meet the macintosh on the road, kill it
Show threadgoedelchenJul 14, 2024

@glyph
> Management says: "We absolutely need 'A'. It is a regulatory requirement,
> we cannot skip any of it." and then a ton of effort goes into estimating
> every sub-part of 'A'.

> Why? There is no decision that an accurate estimate of 'A' could inform!

Of course there are decisions. If you have a regulatory requirement, you also have a deadline by when the requirement has to be met. If we are not confident that A can be implemented in time we have to decide for a tactical solution...

Show threadgoedelchenJul 14, 2024

@glyph ... which often contains some manual process, which we want to avoid with solution A.

There are other situations, where the regulator asks by when you can deliver (e. g. in cases where the regulator also have a word on HOW you implement the regulatory requirements). They might or might not agree with your answer, but you definitely have to be able to say something, which looks realistic. For that you need an estimation.

Show threadGlyphJul 14, 2024
@goedelchen you are not totally wrong here, and in real life there is no discrete atomic “A” which gets implemented all at once, but also, if a certain task *must* get done, and scope cannot be meaningfully cut, a planning process that assumes all constraints are internal and all customers are open to constant renegotiation will not be successful in providing the kind of estimates or the kind of velocity that this sort of interaction requires
Show threadgoedelchenJul 15, 2024

@glyph "you are not totally wrong here" I hope so 🙂

" a planning process [...] will not be successful in providing the kind of estimates..."
A planning process should provide a plan - an estimation process provides estimations. These are prerequisite for any meaningful planning. Estimates are also independent of whether constraints are internal or external or whether negotiations are possible.

Planning is the process of replacing chance with error 😉

Show threadgoedelchenJul 14, 2024

@glyph
...If we are not confident that A can be implemented in time we have to decide for a tactical solution...

and/or inform the regulator about possible delays as early as possible. You want to build trust with the regulator otherwise you'll have a quite hard time. Coming up late with delays is destroying trust.

Show threadgoedelchenJul 15, 2024

@glyph I almost overlooked "accurate estimate". IMHO, that's an oxymoron. Either it's accurate, then it's a prediction. Or it's an estimate, which is not accurate.

If I take one of the explanations from Merriam-Webster, an accurate estimate would be an "accurate rough or approximate calculation"