Xarn3d ago

There are some advantages to how GTest does value parametrization, but the amount of boilerplate to get even the most simple parametrization going means that we have ton of basically duplicated tests.

With Catch2 these 12 tests I am looking at would've been 1, because the barrier to go from 1 concrete test to a value parametrized test is tiny.

Show threadBilly O'Neal3d ago
@horenmar This kind of parameterization has always been one of those things that I think makes sense but every time I actually end up looking at it I convince myself to just write the for-loop instead of using the test harness' features to accomplish that. :/
Show threadXarn3d ago

@malwareminigun The advantage of for loop is that it is dead simple. But it is opaque to the test framework, so if you write test like that and want to investigate failure for specific input, you have to edit the sources.

With value param., you can filter down to the failing input just from the runner. This means that, e.g. if you know that test foo, input #23 fails on ARM and that's ok, you can specify this in your CI script, without having to hardcode the detection into the test itself.

Show threadBilly O'Neal
@horenmar I guess I consider known failing tests to be cancer and the thought of making it easier to need extra information to get a working test run like that is kind of an anti-feature IMO. However, [...]
Mar 19 at 8:14amWeb
Show threadXarn3d ago

@malwareminigun Sure, but the practical reality is that for large crossplatform projects you end up needing this.

For example where the tests are not failing, right now we disable 5 tests in our valgrind CI job. 1 because it causes internal Valgrind error, 4 because even the 6 hour timeout we have is not enough (Valgrind...).

Of those, 2 are in huge table-driven test sets, so being able to disable just 1 test from the table saves bunch of complexity in workarounds.