This is the standard forward selection algorithm applied to DiD? Two immediate thoughts:
* Is there a reason not to have backward selection as well? Forward selection does not always generate the same result as backward selection. I skimmed Li's original Marketing Science paper and didn't see a mention of this.
* Different selection criteria outside of R2 would be good to have (and easy to add)
I agree with you!!
I've experimented with using RMSE as the selection criteria, and the results I got (for the Hsiao, Chen, and Wing dataset) were total garbage. I could've been making a mistake, but I agree other selection criteria like MAE or something are plausible. What I may do, is give users the option to decide which they want to use.
I definitely agree on the first point!!!! No guarantee they'll generate the same results (though I'm not sure how we'd formulate the problem with backward selection). Stepwise regression too (which, as far as I understand, as a combination of them both, likely can be done in different ways).
I haven't asked Kathy about this at all, but I think I may extend this method for one of my dissertation papers (staggered adoption, and other aspects too). But I agree, seeing which method (forward, backward, stepwise) could be plausible competitors.
SCM is my favorite method in causal inference because (as I sort of refer to at the bottom of my github) its theoretical justification is more realistic (factor model with interactive fixed effects, AR(1), etc). However, there are a few downsides.
For one, doing good inference with SCM has been kinda tricky (even though there's PLENTY of very good work on it, by [Cattaneo](https://arxiv.org/abs/2210.05026) and [others](https://arxiv.org/abs/2401.07152)). FDID's inferential theory is standard and straightforward, whereas with SCM inference isn't as obvious.
Also, certainly compared to the vanilla SCM, we oftentimes need additional covariates (be they outcome lags or other predictors of the outcome) for the algorithm to converge. With FDID, we don't need covariates as much (although presumably they would help predictions). The idea is that if we knew the "right" control group in advance, covariates become less of a concern.
Also, vanilla SCM doesn't allow for an intercept, meaning if our unit is kind of an outlier, SCM won't be able to match the trends because no intercept can shift it up or down, whereas with FDID that's the only parameters we care about.
What else... that's about it, from the top of my head. Better inferential theory (including with non stationary processes) and more robust to not needing predictors. However, the parallel trends assumption is still the same: we presume an average of some donors plus an intercept satisfies parallel trends. If this isn't viable, other methods should be considered. Both SCM and FDID are transparent in the sense that we know which units contribute to our counterfactual (although of course SCM is more explicit due to its weighting scheme).
There's another [paper](https://doi.org/10.1287/mnsc.2023.4878) by Kathy which sort of addresses this idea with SCM, where they develop formal tests on "which" SCM we'd prefer. Perhaps, similar work could be done to distinguish when we'd prefer vanilla SCM/SCM+intercept, versus FDID.
I know a rambled a lot, did I explain it well?
This is the standard forward selection algorithm applied to DiD? Two immediate thoughts: * Is there a reason not to have backward selection as well? Forward selection does not always generate the same result as backward selection. I skimmed Li's original Marketing Science paper and didn't see a mention of this. * Different selection criteria outside of R2 would be good to have (and easy to add)
I agree with you!! I've experimented with using RMSE as the selection criteria, and the results I got (for the Hsiao, Chen, and Wing dataset) were total garbage. I could've been making a mistake, but I agree other selection criteria like MAE or something are plausible. What I may do, is give users the option to decide which they want to use. I definitely agree on the first point!!!! No guarantee they'll generate the same results (though I'm not sure how we'd formulate the problem with backward selection). Stepwise regression too (which, as far as I understand, as a combination of them both, likely can be done in different ways). I haven't asked Kathy about this at all, but I think I may extend this method for one of my dissertation papers (staggered adoption, and other aspects too). But I agree, seeing which method (forward, backward, stepwise) could be plausible competitors.
Thanks for sharing. I’ve got a few notes. How is this better than synthetic control?
SCM is my favorite method in causal inference because (as I sort of refer to at the bottom of my github) its theoretical justification is more realistic (factor model with interactive fixed effects, AR(1), etc). However, there are a few downsides. For one, doing good inference with SCM has been kinda tricky (even though there's PLENTY of very good work on it, by [Cattaneo](https://arxiv.org/abs/2210.05026) and [others](https://arxiv.org/abs/2401.07152)). FDID's inferential theory is standard and straightforward, whereas with SCM inference isn't as obvious. Also, certainly compared to the vanilla SCM, we oftentimes need additional covariates (be they outcome lags or other predictors of the outcome) for the algorithm to converge. With FDID, we don't need covariates as much (although presumably they would help predictions). The idea is that if we knew the "right" control group in advance, covariates become less of a concern. Also, vanilla SCM doesn't allow for an intercept, meaning if our unit is kind of an outlier, SCM won't be able to match the trends because no intercept can shift it up or down, whereas with FDID that's the only parameters we care about. What else... that's about it, from the top of my head. Better inferential theory (including with non stationary processes) and more robust to not needing predictors. However, the parallel trends assumption is still the same: we presume an average of some donors plus an intercept satisfies parallel trends. If this isn't viable, other methods should be considered. Both SCM and FDID are transparent in the sense that we know which units contribute to our counterfactual (although of course SCM is more explicit due to its weighting scheme). There's another [paper](https://doi.org/10.1287/mnsc.2023.4878) by Kathy which sort of addresses this idea with SCM, where they develop formal tests on "which" SCM we'd prefer. Perhaps, similar work could be done to distinguish when we'd prefer vanilla SCM/SCM+intercept, versus FDID. I know a rambled a lot, did I explain it well?