Thanks for everyone's input.
I've done some more thinking, reading and experimenting with my gear, and I've made some progress but I'm not yet completely happy.
So, these problems first emerged whilst using the studio strobe (but also earlier with the Sigma 150mm). I also noticed that I can see sharp detail in live view, but can't capture that in an image. I can take perfectly fine images with straight macro lenses, the issues are restricted to using objectives but several different combinations of prime lens / objective were all generating the problem. Subsequently, I noticed that if I use ISO 400 and underexpose, even ISO 800, I get a considerably sharper image, better looking than ISO 100. Those were all big clues. But I also did some reading about the studio strobe that I've been using, the Elinchrom D-Lite RX 4.
Overall, this has exposed a couple of false assumptions.
Studio strobe flash duration is faster than speedlite flash duration = false assumption. The Elinchrom D-Lite RX 4 is rated to 1/800 but that is at max power. I couldn't find data for the lower power settings, but it could be as slow as 1/100 according to something I read. I saw some frames that seemed to indicate vibration (but not others). With 20MP at 20x, on a not-super-stable wooden table and wooden floorboards, all prone to vibration, 1/100 is probably not always fast enough to quell signs of vibration. (Any) flash is sufficiently fast to suppress vibration when photographing stationary objects = second false assumption. I found data indicating that the Canon 580EX flash gives 1/4000 at full power but still 1/500 at 1/4 power, and I try to keep my flash power below 1/4 since I've read that variability can increase close to max power, hence it's not good whilst stacking. You won't see vibrations with flash in regular macro photography like 1:3 or 1:2 but at 20x all bets are off I guess.
But, the bigger issue is that of internal reflections. I don't think there's anything wrong with my optics, rather it's a terrible problem with - call it what you like - flare, glare, veiling, etc. I did several things to counter this, such as shoot on black cloth rather than on the Perspex stand, and put an improvised lens hood on the front of the objective. But most importantly, I flocked over the the gold threads at the back of the objective, and this produced a significant improvement in the image. Clearly there was some sort of internal issue there.
I then went back and retested the Mitutoyo 20x on the Sigma 150mm and it was significantly improved. So that lens does perhaps have some potential as a tube lens.
However, my images still aren't quite right. Not a sharpness issue so much as one of deteriorated contrast / microcontrast. They just have a veiled, dull, ghosted sort of appearance to them. When compared to the previous images, they have much worse clarity. And I still can't explain why I didn't cop these issues before, considering that nothing much has changed. Perhaps I have more than one 52-25mm ring lying around and one is better than the other somehow, I just don't know.
So overally, as far I can tell, the swap to the Elinchrom brought several interacting factors to the fore, the new strobe was putting a) more light into the system and b) putting it in slower, and this both overwhelmed/exposed the (rather low) tolerance of the system for internal flare and also (sometimes) allowed vibration to become visible.
It's a shame to lose the benefits of the studio flash. I suppose I could still use it at higher power settings to get faster duration and use some kind of
ND to bring the intensity down to what I need.
I'm still planning to switch to using a bellows or extension tube system using a Raynox, since I'm skeptical that I can completely fix the images using my current gear. Something is still badly wrong. But hopefully this info might be useful for anyone else who strikes similar problems. Aside from that, I guess I could try photographing somewhere with a concrete slab floor and solid benchtop, or swap in some different tube lenses if I can get access to them. But sadly, I have to go away for work for 4 weeks so everything must be shelved for a while.