Nicolas Fränkel 🇺🇦🇬🇪
Hacker NewsWhy Property Testing Finds Bugs Unit Testing Does Not (2021)
https://buttondown.com/hillelwayne/archive/why-property-testing-finds-bugs-unit-testing-does/
#HackerNews #PropertyTesting #UnitTesting #SoftwareBugs #BugHunting #ProgrammingInsights
Jan 
In the second screenshot I've highlighted some interesting parts:
Key field indices are 2 and 3, so when diffing the records, where key fields are highlighted, they'll be compared as `Modify`, because:
- key fields are equal between left and right record
- other fields are unequal between left and right record
The other two records on the right have no corresponding left record - so those are `Add`ed records
Wow, #FuzzTesting/ #PropertyTesting is actually harder than doing an automatic proof.😓
I didn't expect that!😮
#SoftwareEngineering #Unexpected #Testing #FormalVerification
Ramin HonaryProperty-based testing in Haskell with QuickCheck
falsify A few days ago, Edsko de Vries of Well-Typed published an in-depth article on property-based software testing, with a focus on the concept of “shrinking.”
In brief, property-based testing is sort-of like fuzz testing but for algorithms and protocols. Like fuzz testing, random test cases are procedurally generated, but unlike fuzz testing, the test cases are carefully designed to verify whether a software implementation of an algorithm satisfies a specific property of that algorithm, such as:
- “this function always fails if the index is larger than the array”
- “this function always returns a result in
n*log(n)number of iterations for input dataset of sizen“ “the sequence of log messages is guaranteed to obey this rules of this particular finite-state automata: (connect | fail) -> (send X | fail) -> (receive Y | receive Z | fail) -> success .”
Shrinking is the process of simplifying a failed test case. If you have found some input that makes your function return a value when it should have thrown an exception, or produce a result that does not satisfy some predicate, then that input is a “counterexample” to your assertion about the properties of that function. And you may want to be able to “shrink” that counterexample input to see if you can cause the function to behave incorrectly again but with a simpler input. The “QuickCheck“ library provides a variety of useful tools to let you define property tests with shrinking.
Defining unit tests with such incredible rigor takes quite a lot of time and effort, so you would probably do not want to use property-based testing for your ordinary, every-day software engineering. If you are, for example, being scrutinized by the US Department of Government of Efficiency, you would likely be fired if you were to take so much time to write such high-quality software with such a strong guarantee of correctness.
But if you are, for example, designing a communication protocol that will be used in critical infrastructure for the next 10 or 20 years and you want to make sure the reference implementation of your protocol is without contradictions, or if you are implementing an algorithm where the mathematical properties of the algorithm fall within some proven parameters (e.g. computational complexity), property-based testing can give you a much higher degree of confidence in the correctness of your algorithm or protocol specification.
#tech #software #haskell #WellTyped #QuickCheck #UnitTesting #PropertyTesting
David Wynn#PropertyTesting Pattern: Some things never change
Examples:
- Adding the same permissions to a user
- Removing a user from a group
- Changing a user preference
#PropertyTesting Pattern: There and back again
Write properties that pair an operation with its opposite, so that you get the results you started with.
- Serializing and deserializing
- Toggling preferences / settings
- Writing and deleting an object from storage
- Adding and removing a fixed value from a Number
#PropertyTesting Pattern: Different paths, same destination
If order truly doesn't matter, you should confirm that. Notable examples:
- Adding items in a cart (esp w/ & w/o sales)
- Changing user permissions
- Adding data sources
To test, generate a random set of operations to add and then check that the end state is what you expect.
If you're looking at putting #propertytesting into practice and drawing a blank once you open VSCode, this is the best article for direct advice.
I'll dive deeper into it over the week, because the examples can absolutely be expanded to services instead of just functions.
jnpnSeems like I found what I was looking for
