Waldo Jaquith

In order to keep high-dollar government software projects from going off the rails, projects are often required to undergo an “independent verification & validation” (IV&V) process. The theory is that if government is incapable of evaluating the quality of software being built by a vendor, they can pay another company to review it, and they can say if it's any good. In the abstract, this is not a terrible idea. But in reality it is, in fact, a terrible idea. But what if it wasn’t?

Walk with me.

Dec 13, 2024 at 5:20pmIvory for Mac
Show threadWaldo JaquithDec 13, 2024

Fundamentally, I think that full-time government employees should be ensuring that procurement software is up to snuff. That's a north star for me. But I also recognize that's not always plausible, and it would be good to have a backup option.

IV&V, as it stands now, is not a backup option. I don't want to say that IV&V is always useless, but I have only ever seen it be useless, and I have never heard of an instance where it's good (but I'd sure like to hear about such instances).

Show threadWaldo JaquithDec 13, 2024
In my experience, IV&V vendors are mostly concerned with whether the requirements have been met, and the requirements are things like "there shall be a login form," "there shall be a password recovery function," "passwords shall be no less than 8 characters," etc. Those requirements are harmful garbage, so enforcing those more rigorously is only going to result in getting rigorously defined harmful garbage.
Show threadWaldo JaquithDec 13, 2024
But what if IV&V vendors—and this sounds wacky but hear me out—actually read the code written by the vendor? What if they enforced the Quality Assurance Surveillance Plan (https://guides.18f.gov/derisking-government-tech/resources/quality-indicators/) at the end of each sprint? What if they read through every test carefully, to ensure that they're well-written, and testing the right things? What if they reviewed the user research to ensure that it's being done well? What if they provided an Agile coach to ensure the team is functioning well?
Sample Quality Assurance Surveillance Plan (QASP) | 18F De-risking Guide

Sample Quality Assurance Surveillance Plan (QASP) for a performance-based services contract to build custom software for a government agency.

Show threadWaldo JaquithDec 13, 2024
I have no idea if federal funders would go for IV&V that isn't crap. I imagine it depends on existing regulatory and policy language. But I'd love to see both demand for good IV&V, and vendors getting into the space of providing good IV&V. If that could be done without any change in regs—just slipping good work into the place of bad work—it could be a great lever for change.
Show threadJeff Miller (orange hatband)Dec 13, 2024
@waldoj I like the move of interpreting official processes in terms that are pertinent and useful to their stated intention.
Show threadMax FentonDec 13, 2024
@waldoj This is making me think of how many really top-notch “fedihire” posts I saw this year of people who would probably thrive at that exact work.
Show threadPresent GoodDec 13, 2024
@waldoj As a bookkeeper who has had to interact with VA and other states' taxing websites, I have seen far to many bad 90's built sites 'enhanced' by building parentheses around them for security: we still have to interact with the tiny textboxes of yesteryear after we've plowed through what is ostensibly today's 'security'.