@broli basically yes! Your intuition here is 100% correct. Muscles adapt to three types of stressors: 1. Mechanical tension 2. Metabolic load 3. Fatigue. The three can be distinct sometimes but usually they are related through the intersection of pathways through which muscles get fuel and get rid of by-products and the brain commands the muscles to move through the Central Nervous System (CNS).
If you were to do any of the workouts without any break whatsoever you would be, in most cases, still not hitting all three stressors. This is because the body adapts automatically to the experienced load so it can keep on performing. So suppose, for argument's sake, that you were to do "Power 20" with zero breaks and you just go from one exercise to the other the subsequent lack of recovery time for the muscles would deliver an increased (and increasing metabolic load). And you would get fatigued, but the mechanical tension your muscles would experience would be less than if you did take breaks because your body automatically adjusts the intensity of the exercise even if your perceived effort goes up.
Now suppose, again for argument's sake, that you took a 20 sec rest between sets (as prescribed int he workout) and a full 2 minutes between exercises. What happens now is that you give your body time to reduce the metabolic load but mechanical tension goes up (because you're fresh) and fatigue would still be experienced (because you're going so hard and with such controlled precision).
So, basically, rest time between exercises is the variable that you introduce that best fits your fitness level. If you're fit already and want to make it harder for yourself you allow just 30 seconds for example. This doesn't allow for full recovery from the experienced metabolic load so now you're adding this stressor on top of the mechanical tension you apply. Fatigue is going to kick in, it introduces its own stress and, at the same time, forces you to adjust mechanical tension. Every time you do the workout even if your level of fitness remains the same there will be variables that make a difference: the time of the day, your hormonal profile, when you last ate, which day of the week it is, your emotional state, etc.
Most times these variables make a little difference, sometimes, they make a lot. Most of them we cannot control.
What we do control is rest time. If you want to create a test for yourself that allows you to see how you are progressing the rest time variable is by far the best. If, on the same workout, you go from a two minute break between exercises to just 30 seconds and still manage to maintain form (i.e. mechanical tension) you're then making real, measurable progress in your fitness. But you have to try hard to make sure all the other variables are as consistent as possible.
Now, when we put workouts together and we do not specify time we allow a self-selected interval that is anywhere between 30 seconds and two minutes between exercises. What we've found out through experience is that it doesn't matter what people choose. Some will go for the full two minutes even though they could have gone for less. But their intensity will be higher so mechanical tension gets jacked up and, eventually, fatigue will appear. Others go for less when they shouldn't and then metabolic load becomes a thing pretty quickly and fatigue sets in. Regardless of how good or bad their choice of rest time was ultimately they all benefit from it (which is why we don't get too prescriptive).
I hope this explanation has taken some of your anxiety of hitting the rest time 'spot on' away. You're training consistently and you are, I am sure, seeing gains. Monitoring how you feel and how much rest time you need in different workouts is the easiest way there is to increase the stressors you apply to your body. It is also an 'easy' way to do more with less. For instance, with sufficient reps and no rest time even a 4kg dumbbell can be made to deliver serious strength gains because it allows fatigue and metabolic load to kick in.
I really hope this has helped, feel free to tag me any time you want more clarification or if you feel there is something I said here you could do more with by way of examples or explanation.