vampatoriJul 9, 2023

100% code coverage is near-meaningless - but is there a good measure to use?

https://feddit.uk/post/443660

100% code coverage is near-meaningless - but is there a good measure to use? - Feddit UK

Is there some formal way(s) of quantifying potential flaws, or risk, and ensuring there’s sufficient spread of tests to cover them? Perhaps using some kind of complexity measure? Or a risk assessment of some kind? Experience tells me I need to be extra careful around certain things - user input, code generation, anything with a publicly exposed surface, third-party libraries/services, financial data, personal information (especially of minors), batch data manipulation/migration, and so on. But is there any accepted means of formally measuring a system and ensuring that some level of test quality exists?

Show threadMagicShel
Pit testing is useful. It basically tests how effective your tests are and tells you missed conditions that aren’t being tested. pitest.org
PIT Mutation Testing

Jul 9, 2023 at 7:48pmWeb
Show threadrobotdnaJul 9, 2023
Does something like this exist for Python?
Show threadv_krishnaJul 9, 2023
mutatest.readthedocs.io/en/latest/
Mutatest: Python mutation testing — Mutatest 3.1.0 documentation

Show threadrobotdnaJul 11, 2023
Oh sweet! This introduced a whole new world to me. Also seeing mutmut, is one better than the other?
Show threadvampatoriJul 9, 2023
This is really interesting, I’ve never heard of such an approach before; clearly I need to spend more time reading up on testing methodologies. Thank you!
Show threadmattburkedevJul 10, 2023

The most extreme examples of the problem are tests with no assertions. Fortunately these are uncommon in most code bases.

Every enterprise I’ve consulted for that had code coverage requirements was full of elaborate mock-heavy tests with a single Assert.NotNull at the end. Basically just testing that you wrote the right mocks!

Show threadMagicShelJul 10, 2023

That’s exactly the sort of shit tests mutation testing is designed to address. Believe me it sucks when sonar requires 90% pit test pass rate. Sometimes the tests can get extremely elaborate. Which should be a red flag for design (not necessarily bad code).

Anyway I love what pit testing does. I hate being required to do it, but it’s a good thing.

Show threadDeelyJul 14, 2023
Yeah. All the same. Create lazy metric - get lazy and useless results.
Show threadxthexderJul 10, 2023

I’d never heard of mutation testing before either, and it seems really interesting. It reminds me of fuzzing, except for the code instead of the input. Maybe a little impractical for some codebases with long build times though. Still, I’ll have to give it a try for a future project. It looks like there’s several tools for mutation testing C/C++.

The most useful tests I write are generally regression tests. Every time I find a bug, I’ll replicate it in a test case, then fix the bug. I think this is just basic Test-Driven-Development practice, but it’s very useful to verify that your tests actually fail when they should. Mutation/Pit testing seems like it addresses that nicely.