JamesMar 17
Stef Walter

Statistical significance tells you a result is real. Practical significance tells you if it is worth caring about. As UX researchers, we need both so we do not overreact to tiny effects or ignore big, noisy signals.

This is why context matters. I have this exact case at the moment. A very small sample, but 90% of my users go towards one specific direction. Those users are representing their own business lines.
https://www.nngroup.com/articles/practical-significance/

Statistical Significance Isn’t the Same as Practical Significance

Statistical significance helps establish whether a result is reliable, while practical significance helps determine whether it is worth acting on.

Nielsen Norman Group
Mar 17 at 2:30pmWeb
Show threadStef WalterMar 17

Meaning, even if we only have a small sample, their direction makes indeed sense in the global business value.

Statistical Significance Isn’t the Same as Practical Significance (8min) https://www.nngroup.com/articles/practical-significance/

Statistical Significance Isn’t the Same as Practical Significance

Statistical significance helps establish whether a result is reliable, while practical significance helps determine whether it is worth acting on.

Nielsen Norman Group