Exploratory Data Alex
Peter EllisI wrote an #rstats blog post to remind myself that just because a log transform improves an ordinary least squares regression in conventional terms (eg makes it more linear and residuals less heteroskedastic), doesn't mean it is more fit for purpose. http://freerangestats.info/blog/2023/07/30/log-transforms
Ben Bolker@peter_ellis would doing log transformed analysis plus bias correction (i.e., compute mean of predicted lognormal distribution, exp(mu + sigma^2/2)) give you the best of both worlds?
@bbolker maybe I will do a sequel with that and iterative weighted least squares.
Cameron Patrick@peter_ellis @bbolker (Best of both worlds not guaranteed! AFAIK the standard bias correction is only correct if the underlying population is roughly lognormal; I think this is discussed in Permutt’s paper “Do covariates change the estimand”)
Nikolas Kuschnig@bbolker @peter_ellis Yes, if the variable is roughly log-normal, as mentioned below. Interestingly, a small bias remains in this case, but the variance of the estimate is improved (σ at 35 vs 36).
@peter_ellis Nice! This is something where it depends on the task at hand - in your example of population totals the geometric mean is bad, if you wanted something closer to a typical individual in a skewed population the geometric mean can be better than the arithmetic.
tippy_topThanks @peter_ellis That's a great read. Fascinating how "better model" leads to a worse estimate.
Julia Silge@peter_ellis Super clear and interesting, as always 🙌