Gaël VaroquauxDec 20, 2022

✨Preprint in causal inference: finite sample error and variable selection when reweighting randomized experiments (RCTs) to generalize them to other populations

Exciting because informs on bias-variance tradeoffs, eg including or not a variable
https://arxiv.org/abs/2208.07614

🧵👇 Thread below

Reweighting the RCT for generalization: finite sample error and variable selection

Randomized Controlled Trials (RCTs) may suffer from limited scope. In particular, samples may be unrepresentative: some RCTs over- or under- sample individuals with certain characteristics compared to the target population, for which one wants conclusions on treatment effectiveness. Re-weighting trial individuals to match the target population can improve the treatment effect estimation. In this work, we establish the exact expressions of the bias and variance of such reweighting procedures -- also called Inverse Propensity of Sampling Weighting (IPSW) -- in presence of categorical covariates for any sample size. Such results allow us to compare the theoretical performance of different versions of IPSW estimates. Besides, our results show how the performance (bias, variance, and quadratic risk) of IPSW estimates depends on the two sample sizes (RCT and target population). A by-product of our work is the proof of consistency of IPSW estimates. Results also reveal that IPSW performances are improved when the trial probability to be treated is estimated (rather than using its oracle counterpart). In addition, we study choice of variables: how including covariates that are not necessary for identifiability of the causal effect may impact the asymptotic variance. Including covariates that are shifted between the two samples but not treatment effect modifiers increases the variance while non-shifted but treatment effect modifiers do not. We illustrate all the takeaways in a didactic example, and on a semi-synthetic simulation inspired from critical care medicine.

arXiv.org
Show threadGaël VaroquauxDec 20, 2022

We consider an experiment (trial) applied to a population different from the target population where the intervention will be applied.

Effect heterogeneous for covariates (features) that differ in the trial vs target population, leads to differing average treatment effect.

The effect can be extrapolated (generalized) by reweighting the trial (IPSW methods): giving an importance to each sample proportional to the density ratio.
2/n

Show threadGaël Varoquaux

When reweighting a trial, a practical question is: which variables should one include in this density ratio to best compute the shift?

Including all variables minimizes the chances that there is a unaccounted-for shift. But it also means density ratios more difficult to estimate (higher dimensional statistics).

We give explicit estimation-error formula, for discrete covariates, thus answering this question
👇 3/n

Dec 20, 2022 at 5:21pmWeb
Show threadGaël VaroquauxDec 20, 2022

🥁 Reweighting a trial for #ExternalValidity, the target average treatment is best estimated with covariates = treatment effect modifiers (shifted or not)

Covariates that are shifted between the 2 samples but not treatment effect modifiers increase the variance while non-shifted but treatment effect modifiers do not

4/n

Show threadGaël VaroquauxDec 20, 2022

Many other interesting results on generalizing #RCT:

⋄ On the trial, the observed probability of being treated should be used, rather than the theoretical one in the study design

⋄ We give ISPW variance function of size of trial and target population samples

⋄ Surprising result: the estimated generalized effect (reweighting the trial) can by smaller variance than that estimated on the trial. This happens with heteroscedasticity when the target population explore more low-variance strata
5/n

Show threadGaël VaroquauxDec 20, 2022

A fundamental statistical-theory result in #CausalInference : the IPSW estimator is shown to be consistent without oracle hypothesis (causal results are often established using knowledge of distributions, not concluding on estimators)

https://arxiv.org/abs/2208.07614

The work was led by the amazing Benedicte Colnet, in collaboration with Julie Josse and Erwan Scornet at Inria Saclay, Inria Sophia and CMAP – École Polytechnique

6/6

Reweighting the RCT for generalization: finite sample error and variable selection

Randomized Controlled Trials (RCTs) may suffer from limited scope. In particular, samples may be unrepresentative: some RCTs over- or under- sample individuals with certain characteristics compared to the target population, for which one wants conclusions on treatment effectiveness. Re-weighting trial individuals to match the target population can improve the treatment effect estimation. In this work, we establish the exact expressions of the bias and variance of such reweighting procedures -- also called Inverse Propensity of Sampling Weighting (IPSW) -- in presence of categorical covariates for any sample size. Such results allow us to compare the theoretical performance of different versions of IPSW estimates. Besides, our results show how the performance (bias, variance, and quadratic risk) of IPSW estimates depends on the two sample sizes (RCT and target population). A by-product of our work is the proof of consistency of IPSW estimates. Results also reveal that IPSW performances are improved when the trial probability to be treated is estimated (rather than using its oracle counterpart). In addition, we study choice of variables: how including covariates that are not necessary for identifiability of the causal effect may impact the asymptotic variance. Including covariates that are shifted between the two samples but not treatment effect modifiers increases the variance while non-shifted but treatment effect modifiers do not. We illustrate all the takeaways in a didactic example, and on a semi-synthetic simulation inspired from critical care medicine.

arXiv.org
Show threadSoda - InriaDec 22, 2022
@GaelVaroquaux Congratulations to Bénédicte Colnet, our team member, for leading this great project 👏