Testing at the Boundaries (2019)

이 글은 Kent Beck의 TDD 원저서가 실제 산업 현장에서 오해받는 부분들을 짚으며, TDD가 모든 상황에 적합하지 않음을 설명한다. 특히 시스템 경계에서 테스트를 작성하는 중요성을 강조하며, 정적 타입 언어와 컴파일러 개발 사례를 통해 전통적 단위 테스트가 기술 부채가 될 수 있음을 보여준다. 또한 게임, 동시성, 분산 시스템 등에서는 단위 테스트 외에 프로퍼티 테스트, 프로덕션 모니터링 등 다양한 테스트 전략이 필요하다고 지적한다. TDD는 강력한 도구이나 모든 개발 상황에 맞는 만능 해법은 아니며, 상황에 맞는 테스트 방법론 선택이 중요하다.

https://www.tedinski.com/2019/03/19/testing-at-the-boundaries.html

#tdd #testing #softwareengineering #propertytesting #systemdesign

Anyone using the brick #Haskell library and what to shore hints / lecture me about how to test functions that "live in" `EventM` ?

In particular, I'd like to be able to take an initial state (generated in whole or part randomly) feed some events (mainly keystrokes) in an then validate the final state. But, maybe I'm thinking about it all wrong.

The rendering functions are "easy enough" to test, but I feel like most of the real "logic" is in the event handling.

#SoftwareTesting #PropertyTesting

Why #PropertyTesting Finds Bugs #UnitTesting Does Not

https://buttondown.com/hillelwayne/archive/why-property-testing-finds-bugs-unit-testing-does/

#testing

Why Property Testing Finds Bugs Unit Testing Does Not (2021)

https://buttondown.com/hillelwayne/archive/why-property-testing-finds-bugs-unit-testing-does/

#HackerNews #PropertyTesting #UnitTesting #SoftwareBugs #BugHunting #ProgrammingInsights

#Fuzzing along in #CSVDiff  

In the second screenshot I've highlighted some interesting parts:

Key field indices are 2 and 3, so when diffing the records, where key fields are highlighted, they'll be compared as `Modify`, because:
- key fields are equal between left and right record
- other fields are unequal between left and right record

The other two records on the right have no corresponding left record - so those are `Add`ed records

#Rust #FuzzTesting #RustLang #PropertyTesting

Wow, #FuzzTesting/ #PropertyTesting is actually harder than doing an automatic proof.😓

I didn't expect that!😮

#SoftwareEngineering #Unexpected #Testing #FormalVerification

A few days ago, Edsko de Vries of Well-Typed published an in-depth article on property-based software testing, with a focus on the concept of “shrinking.”

In brief, property-based testing is sort-of like fuzz testing but for algorithms and protocols. Like fuzz testing, random test cases are procedurally generated, but unlike fuzz testing, the test cases are carefully designed to verify whether a software implementation of an algorithm satisfies a specific property of that algorithm, such as:

• “this function always fails if the index is larger than the array”
• “this function always returns a result in n*log(n) number of iterations for input dataset of size n“

• “the sequence of log messages is guaranteed to obey this rules of this particular finite-state automata: (connect | fail) -> (send X | fail) -> (receive Y | receive Z | fail) -> success .”

Shrinking is the process of simplifying a failed test case. If you have found some input that makes your function return a value when it should have thrown an exception, or produce a result that does not satisfy some predicate, then that input is a “counterexample” to your assertion about the properties of that function. And you may want to be able to “shrink” that counterexample input to see if you can cause the function to behave incorrectly again but with a simpler input. The “QuickCheck“ library provides a variety of useful tools to let you define property tests with shrinking.

Defining unit tests with such incredible rigor takes quite a lot of time and effort, so you would probably do not want to use property-based testing for your ordinary, every-day software engineering. If you are, for example, being scrutinized by the US Department of Government of Efficiency, you would likely be fired if you were to take so much time to write such high-quality software with such a strong guarantee of correctness.

But if you are, for example, designing a communication protocol that will be used in critical infrastructure for the next 10 or 20 years and you want to make sure the reference implementation of your protocol is without contradictions, or if you are implementing an algorithm where the mathematical properties of the algorithm fall within some proven parameters (e.g. computational complexity), property-based testing can give you a much higher degree of confidence in the correctness of your algorithm or protocol specification.

#tech #software #haskell #WellTyped #QuickCheck #UnitTesting #PropertyTesting

#PropertyTesting Pattern: Some things never change

Examples:

- Adding the same permissions to a user
- Removing a user from a group
- Changing a user preference

#PropertyTesting Pattern: There and back again

Write properties that pair an operation with its opposite, so that you get the results you started with.

- Serializing and deserializing
- Toggling preferences / settings
- Writing and deleting an object from storage
- Adding and removing a fixed value from a Number

#PropertyTesting Pattern: Different paths, same destination

If order truly doesn't matter, you should confirm that. Notable examples:

- Adding items in a cart (esp w/ & w/o sales)
- Changing user permissions
- Adding data sources

To test, generate a random set of operations to add and then check that the end state is what you expect.