Mastodawn

100% code coverage is near-meaningless - but is there a good measure to use?

https://feddit.uk/post/443660

100% code coverage is near-meaningless - but is there a good measure to use? - Feddit UK

Is there some formal way(s) of quantifying potential flaws, or risk, and ensuring there’s sufficient spread of tests to cover them? Perhaps using some kind of complexity measure? Or a risk assessment of some kind? Experience tells me I need to be extra careful around certain things - user input, code generation, anything with a publicly exposed surface, third-party libraries/services, financial data, personal information (especially of minors), batch data manipulation/migration, and so on. But is there any accepted means of formally measuring a system and ensuring that some level of test quality exists?

MagicShel Jul 9, 2023

Pit testing is useful. It basically tests how effective your tests are and tells you missed conditions that aren’t being tested. pitest.org

PIT Mutation Testing

robotdna Jul 9, 2023

Does something like this exist for Python?

v_krishna Jul 9, 2023

mutatest.readthedocs.io/en/latest/

Mutatest: Python mutation testing — Mutatest 3.1.0 documentation

robotdna Jul 11, 2023

Oh sweet! This introduced a whole new world to me. Also seeing mutmut, is one better than the other?

vampatori Jul 9, 2023

This is really interesting, I’ve never heard of such an approach before; clearly I need to spend more time reading up on testing methodologies. Thank you!

mattburkedev Jul 10, 2023

The most extreme examples of the problem are tests with no assertions. Fortunately these are uncommon in most code bases.

Every enterprise I’ve consulted for that had code coverage requirements was full of elaborate mock-heavy tests with a single Assert.NotNull at the end. Basically just testing that you wrote the right mocks!

MagicShel Jul 10, 2023

That’s exactly the sort of shit tests mutation testing is designed to address. Believe me it sucks when sonar requires 90% pit test pass rate. Sometimes the tests can get extremely elaborate. Which should be a red flag for design (not necessarily bad code).

Anyway I love what pit testing does. I hate being required to do it, but it’s a good thing.

Deely Jul 14, 2023

Yeah. All the same. Create lazy metric - get lazy and useless results.

xthexder Jul 10, 2023

I’d never heard of mutation testing before either, and it seems really interesting. It reminds me of fuzzing, except for the code instead of the input. Maybe a little impractical for some codebases with long build times though. Still, I’ll have to give it a try for a future project. It looks like there’s several tools for mutation testing C/C++.

The most useful tests I write are generally regression tests. Every time I find a bug, I’ll replicate it in a test case, then fix the bug. I think this is just basic Test-Driven-Development practice, but it’s very useful to verify that your tests actually fail when they should. Mutation/Pit testing seems like it addresses that nicely.

Zoe Codez Jul 9, 2023

There are tools to detail the code coverage if your tests. I’ve worked with Istanbul in the past, and it’s helped to point out parts of the code that could use more attention

istanbul.js.org

Istanbul, a JavaScript test coverage tool.

Istanbul instruments your ES5 and ES2015+ JavaScript code with line counters, so that you can track how well your unit-tests exercise your…

istanbuljs-website

vampatori Jul 9, 2023

I use coverage tools like nyc/c8, but I can easily get 100% coverage on buggy, exploitable, and unstable code. You can have two projects, both with 100% coverage, and one be a shit show and the other be rock solid - so I was wondering if there’s a way to measure quality of tests, or to identify code that really needs extra attention (despite being 100%). Mutation testing has been suggested and that’s really interesting, I’m going to give it a go tomorrow and see what it throws up!

raspberry_confetti Jul 9, 2023

80%. Much beyond that and you get into a decreasing return on the investment of making the tests.

canpolat Jul 9, 2023

I think this is a good rule-of-thumb in general. But I think the best way to decide on the correct coverage is to go through uncovered code and make a conscious decision about it. In some classes it may be OK to have 30%, in others one wants to go all the way up to 100%.

raspberry_confetti Jul 9, 2023

Bingo, exactly this. I said 80 because that’s typically what I see our projects get to after writing actually useful tests. But if your coverage is 80% and it’s all just tests verifying that a constant is still set to whatever value, then yeah, thats a useless metric.

Psilves1 Jul 10, 2023

God I fucking wish my projects were like this

ilega_dh Jul 10, 2023

The 80-20 rule is for everything. Don’t waste 80% of effort to get that last 20% of coverage.

Sibbo Jul 9, 2023

Maybe fraction between money spent on writing code versus money spent on testing code?

lordxakio Jul 9, 2023

I am not sure what the common/agreed upon rules are. Seems like it depends on the team lead or manager to decide. Some orgs have better engineers, experience, systems and others don’t.

I used to follow the 100% coverage because I was told to do so in my start. I found myself chasing semi-colons rather than null references. Luckily, I had a team mate with which we argued a lot about what we did, do, and will do and he helped me. (In a friendly manner, not like Dinesh and Gilfoyd from Silicon Valley).

Now, I start my tests by going over how the user will use it, e.g. the happy path. Then happy path away. It seems to cover most cases. It helps if you know the business too. (Think messaging system that is intentionally and strictly simple, or one that has a lot of Unicode and language support… fucking emojis hurt me cause I forget they exist even though I use them all the time, I always forget).

Alas, no matter what, I always miss some test case or a very imaginative user will find a way to show me how wrong I am.

In the end, I think the best, no matter how big or small the project you’re building is, to do many small PRs (with tests) to your team. This way, things are tested in increments and helps prevent PR burnouts. This I need to get better at myself.

MR_GABARISE Jul 9, 2023

On top or, better, in addition to mutation testing, some amount of property-based testing is always great where it counts.

bluGill Jul 9, 2023

I.prefer to count and report total tests run as part of each build. We get impressive large numbers, but there is no way to put any specific goal on the exact number, we can always go higher.

cgtjsiwy Jul 9, 2023

Different applications require different tests, so no measure is going to please everyone. If you’re making embedded devices for an airplane, the buyer might ask you to provide a formal proof that the program works. In contrast, web apps tend to simply use end users as testers, since it’s cheaper.

RandomBit Jul 10, 2023

“When a measure becomes a target, it ceases to be a good measure”. — Goodhart’s law

Goodhart's law - Wikipedia

tatterdemalion Jul 10, 2023

This might not be exactly what you’re looking for, but there is verifiably correct software. You can use proof assistants or work in limited computational models (i.e. always-terminating, non-Turing-complete).

One example: statebox.org/what-is/

Compositional Diagrammatic Programming Language

austin Jul 10, 2023

Write tests. Not too many. Mostly integration.

Write tests. Not too many. Mostly integration.

[Guillermo Rauch](https://x.com/rauchg) [tweeted](https://x.com/rauchg/status/807626710350839808) this a while back. Let's take a dive into what it means.

Psilves1 Jul 10, 2023

Great read

snowe Jul 10, 2023

Mutation testing. Someone else mentioned it as PIT testing, but its actual name is mutation testing. It accomplished exactly what you’re looking for here.

UFO Jul 11, 2023

I’d like to see state space coverage instead of line coverage. That, at least, catches silly “100%” cases.

I don’t know of a tool that provides this metric. I don’t even think such a thing could be made for most languages. still, useful to think about when reviewing code.