Mastodawn

Do you know of examples of a software library's test suite catching a bug in its upstream dependencies? I've seen a few of these over the years, and I'd like to put together a small list. Things like:

- A programming language implementation's test suite uncovering a bug in other implementations
- A library's test suite uncovering a bug in the language implementation itself
- A framework addon's / extension's test suite uncovering a bug in the framework

Show thread

Jon Banafato

I'm viewing this from a perspective of "many hands make light work" or "with enough eyes, all bugs are shallow". I'm also interested in the outcomes of these examples, e.g. did they affect the upstream test suite, were the bugs impactful enough to add other tests, something else?

Show thread

Stephen Rosen Jun 19

@jonafato There's been some very interesting activity over in JSON Schema over the past couple of years, getting a harness setup to run the official test suite against many implementations.

@JulianWasTaken might be able to point you at a good reference point, as I think he did a lot of that work.

Show thread

Julian Berman Jun 19

@sirosen @jonafato yeah we've had a ton of these over the years both inside Python and in other languages I think. The one that most recently comes to mind is https://github.com/python/cpython/issues/127065#issuecomment-2817064962 I think but I'm sure if I thought for 5 more seconds I'd remember more of them.

`methodcaller` is not thread-safe (or re-entrant) · Issue #127065 · python/cpython

Bug report EDIT: edited to clarify that the issue is in the C implementation of operator.methodcaller. Originally reported by @ngoldbaum in crate-py/rpds#101 Reproducer from operator import methodc...

GitHub

Show thread

Jon Banafato Jun 19

@JulianWasTaken @sirosen Thanks! I figured JSON Schema might have some examples here, especially with that common test suite approach. This topic pops into my head periodically, and I find myself wondering if / how one could proactively approach it.