Rik,
Your explanation was great and clarifies a lot of things in my mind.
But after thinking during long hours(

) i still find it compatible with my previous reasonning...
I really regrets that english is not my native language as i fear that what i will try to explain will seem confuse.
Let's forget the FOV as you demonstrated that it's misleading.
What I understand, now, is that every pixel on the sensor is the result of the focusing rays comming from everywhere on the lens.
I also understand that, if i'm near the lens the shape of the important rays is a circle and if i'm near the sensor it's a rectangle.
In the following modification of your diagram, i did the following modification :
- reduced the size of the sensor to the red line
- drawed in green the cone of rays containing the famous
sort of truncated cone made of the rays hitting the sensor.
On the sensor plane, the green cone is a circle. And this circle is circumscribing the sensor rectangle. It's why i represented its profile larger than the sensor (no scale respected).
Now, in blue, i drawed 2 positions for a circle baffle.
I believe that, to avoid vignetting, the openning of the baffle must be adapted to the green cone, and this, whatever it's position on the lens-sensor axis.
Am I correct?
If this is the case, then i see absolutely no difference, in term of passing rays , between the left and the right baffle.