Decide which incremental upgrade paths we want to manually test as part of our release process
Before 1696d48cbdfab990b09ad7ac45a2078dc904e7fe we would test all incremental upgrade paths. To take an extreme example, when we'll release Tails 4.20 there will be more than 2 dozens such upgrade paths. I felt this would take too much of our resources to me, so in 1696d48cbdfab990b09ad7ac45a2078dc904e7fe I changed the doc to only test the "current stable installed from scratch → version being prepared" upgrade path (which uses the v2 upgrader, v2 UDFs, and a v2 IUK). But perhaps that's not good enough and we need to find a middle ground.
Another question is whether we want to test "Tails << 4.2 → 4.2 or 4.2.1 → version being prepared", which is a bit different: the first step uses v1 UDFs and IUKs, while the 2nd step switches to v2 UDFs and IUKs.
After the 4.2.1 release, depending on how things go (now that the IUKs and UDFs have been fixed, presumably, and now that all the UDFs are on the test channel), we can discuss what a suitable trade-off could look like.
Food for thoughts: #17425
Yep, that's definitely something to take into account.
FWIW, here's an update on this front: we now have regression tests in place for #17425, one in the Upgrader codebase itself, the other in the Tails integration test suite (f46d5c504fac878a43de91e74ad9e000c68cb137 and f122cc078ef4421f1f4d94ecf716620993e5a2ea). Had they been added as part of #15281, we would not have hit #17425 at the worst possible time.
- Status changed from Confirmed to Resolved
- We thinks it's now enough to do the minimal (as documented) test as long as the Upgrader and IUK generation do not change
- For less simple situations, we try to figure out in advance what can go wrong, with developers + RM + other relevant stakeholders, we test risky cases before merging, and then the RM adds whatever extra manual tests they feel are needed during the release process.