CI Stability - how do we get it green?
Brainstorm approaches to getting CI green.
Daily builds always fail, and since master can't build Green in a single shot, neither can a PR. Having to re-run CI repeatedly for even a simple change to confirm it doesn't introduce a regression is burdensome, and risks people starting to ignore our CI because its false positive rate is too high.
We have improved the chance of success a bit lately by aggressively marking tests as flaky, but this amounts to ignoring tests so is a problematic strategy.
