Comparing macro lenses using MTF - invitation

Miljenko · Post by **Miljenko** » Wed Aug 28, 2019 6:13 am

After more than 50 macro lenses tested for resolution and CA using standardized method I've found some real gems, some of them being already known for great results. Of course, I have selected the best of the best for my own macro work, although there are few weak points (magnifications) where feeling says there must be some better optics out there.

I have bought many of those lenses on Ebay auctions, some of them locally and some were borrowed from friends and distributors. There are still few lens "milestones" worth testing and comparing but so far I didn't have a chance to get samples like Minolta Dimage 5400 lens, Mitutoyo 5x objective or Zhongyi (Mitakon) Super Macro 1x-5x lens.

I wonder would it be unrealistic to ask you guys if you can part from your precious lens worth testing for a week or two, send it to me for testing and get it back with charts? Unfortunately, postage from USA or Asia to Croatia and back is way to high so only Europeans qualify for this challenge! I believe small package postage across Europe is in order of €10 or so.

mawyatt · Post by **mawyatt** » Thu Aug 29, 2019 8:09 am

I would send the DI 5400 & Mitutoyo 5X if you were in US, but postage to Europe is ~$24!! I know because I just shipped a couple of my controllers and both got rejected by customs, so I had to redo the documents and ship again. So 2 packages cost me $96

How would you think the Laowa 25mm would compare to the DI 5400 or other better lenses around 2~4X?

Best,

Mike

Miljenko · Post by **Miljenko** » Thu Aug 29, 2019 9:51 am

mawyatt wrote:I would send the DI 5400 & Mitutoyo 5X if you were in US, but postage to Europe is ~$24!! I know because...

I know too Mike that US to Europe postages become to high in the last couple of years. This is why I'm very rarely buying anything from USA except when friends who live there are visiting.

mawyatt wrote:How would you think the Laowa 25mm would compare to the DI 5400 or other better lenses around 2~4X?

My main argument when insisting on MTF way of testing lenses is repeatibility and easy comparison even when the lenses are miles or years apart. My series of macro lens tests was divided by magnification factor and 3x episode was published in September last year: www.photomacrography.net/forum/viewtopic.php?t=38109. So let's compare!
It is easy to find that Laowa 25mm equals lens #4 (Lomo 3.7 objective @3x) when comparing weighted average resolution but Laowa has more balanced resolution across the frame. However, when Lomo was mounted via tube lens, it's edge resolution improved so it can be considered as better (and much cheaper) solution in 3x to 4x range.
Untill someone borrows me Dimage 5400 lens for testing I wouldn't be able to compare it to Laowa so I can only speculate about their differences....

mawyatt · Post by **mawyatt** » Thu Aug 29, 2019 10:34 am

Thanks for the link, what tube lens did you use with the Lomo 3.7?

I agree with your testing setup, having repeatable results with numbers is the proper "scientific" way to evaluate, anything else is "speculation".

Best,

Miljenko · Post by **Miljenko** » Sat Aug 31, 2019 5:23 am

Lomo 3.7 measured 2505 on average when extended for 3x but went over 2600 when relayed with Raynox DCR-250. Central resolution doesn't change much but edges improve substantially. It's 125mm focal length gives almost exact 3x magnification.

mawyatt · Post by **mawyatt** » Sat Aug 31, 2019 8:15 am

Thanks!!

Best,

Miljenko · Post by **Miljenko** » Thu Sep 05, 2019 11:56 pm

First package with lenses and objectives for MTF testing has arrived from our fellow photomacrographer Filip (jurkovicovic): three scanner lenses and three microscope objectives. I believe second group will arouse broader attention. There is Amscope "knurled ring" Plan 4x/0.10 and two infinite Zeiss/Jena GF Planachromats: 6.3x/0.12 and 12.5x/0.25.
This is going to be one intense weekend in my lab!

jurkovicovic · Post by **jurkovicovic** » Fri Sep 06, 2019 1:21 am

Nice photo

Miljenko · Post by **Miljenko** » Wed Sep 11, 2019 5:18 am

This testing episode came to a stall unexpectedly. One of my „calibrated“ tools turned out to be way off the expected values. I become aware of it thanks to few reference objectives tested earlier and now added to this new batch. Since 2016 I was using Capture One as part of MTF testing workflow. Comparison to other raw converters revealed this piece of software is second to none when it comes to extracting the last pixel of resolution while fighting nicely demosaicing with just the right amount of presharpening. Version 10.0 was the first to read (compressed) Fuji raw files so I started to convert raw test patterns into files accepted by Imatest program. All was done by using default values in order to maintain cross-platform compatibility. When C1 v.11.0 was introduced in November 2017. I've upgraded so my „Comparing macro lenses using MTF” series published here at the end of 2018. was done using that later version. In April this year I upgraded to v.12.0 but didn’t do any testing since recently. Comparing old figures from my Lomo 3.7x, Nikon 5x NA 0.15 LU Plan and Nikon 10x NA 0.30 Plan EPI to new values brought me to shocking 10-15% differences. Since most of HW and SW components are well controlled and frequently checked, Capture One came under suspicion at once. Here are the results of Imatest resolution test performed with the same lens, on the same raw file but processed with C1 versions 10.0, 11.0 and 12.0 respectively:

You can easily see equal vales for 10.0 and 11.0 but substantially lower values for v.12.0. Obviously, Phase One team decided to weaken presharpening algorithms but without any notice to their users.
Now I have rolled back to v.11.0 because of MTF testing. Of course, at work with Nikon D850 I’m staying with the latest version due to some new features introduced.
The testing continues….

rjlittlefield · Post by **rjlittlefield** » Wed Sep 11, 2019 8:54 am

Miljenko, thank you for this latest report about the significance of sharpening applied during raw conversion.

I wonder if you have done any similar tests regarding the effect of using different sensors.

My understanding of slanted edge analysis is that what it's really determining is system MTF, not just lens MTF. If that's correct, then it seems like sensor resolution, presence/absence of AA filter, and maybe even smaller details like microlens characteristics could have some significant effect on the results.

Is this something that you've looked at?

--Rik

Miljenko · Post by **Miljenko** » Thu Sep 12, 2019 1:02 am

rjlittlefield wrote: I wonder if you have done any similar tests regarding the effect of using different sensors.

Never directly but since 2004. I have "imatested" literally hundreds of lenses using different cameras I owned: Nikon D200, D300, D7000, D7100 and Fuji X-T1 & X-T2. On some occasions I used the same lens with different cameras and the results were so different that I archived them strictly separated. For some time my resolution reference was (great copy of) Sigma 70mm f2.8 Macro so I have tested in on several cameras. And yes, resolution values were different.

rjlittlefield wrote: My understanding of slanted edge analysis is that what it's really determining is system MTF, not just lens MTF. If that's correct, then it seems like sensor resolution, presence/absence of AA filter, and maybe even smaller details like microlens characteristics could have some significant effect on the results.

You are absolutely right. All those components play it's role in final resolution score. The most prominent being AA filter so when manufacturers in general decided to omit this questionable add-on, resolution figures measured jumped by a huge margin.
As a side note: although I do my best to stick to "standardized" testing procedure and gear I am very well aware there is hardly such thing. But nevertheless if someone does MTF testing using the same hardware and software as I do, results will be similar and within lens variation values.

jurkovicovic · Post by **jurkovicovic** » Thu Sep 26, 2019 11:44 am

Today I got my objectives back!
More informations here:
http://www.photomacrography.net/forum/v ... hp?t=40303
About cooperation with Miljenko I can write only one thing: cooperation is PERFECT!
The whole action tooks about one month, including shipping time + time for making of adapters.
So if you like Miljenko´s idea, but you have doubts or you are not 100% sure, how it works. Do not hesitate and ask him or rather send him objectives for measurement

Again I would like to repeat, measurement of objectives is important and database of measured objectives can help to a lot of people!
Miljenko is real gentleman, and I would like to thank him for cooperation and measurement. Thank you!

Miljenko · Post by **Miljenko** » Fri Sep 27, 2019 12:25 am

Thank you Filip for the nice words. Measuring lenses the MTF way is both fun and useful activity IMHO. Since I have tested dozens of lenses and didn't publish all the results here (those are contained within one gigantic spreadsheet), I'm considering cutting it into smaller chunks and publish them with comments. You have helped to enlarge that database and increase community knowledge base so thank you for that.
BTW, we have exchanged p.m's about relation between measurements and subjective picture quality. My opinion is that although MTF is absolutely precise method of resolution and CA lens evaluation, it does have it's limitations. For instance, MTF50 is accepted as kind of standard criteria stating LW/PH resolution where contrast ratio is exactly 50%. It is well known that human eye can distinguish black to white transitions at brightness differences as low as 10%. So 50% is very conservative figure often underrating otherwise excellent lenses. Before MTF50 standard was accepted, 30% contrast was considered as realistic figure and I lean to embrace that opinion. But in order to make my measurements as standard as possible, I'm publishing MTF50 figures.
Thanks to pretty huge Imatest use experience, when final charts appear at the end of measuring process, I can recognize lenses that might perform better in real shooting than MTF figures suggest. Here is one of more recent examples: during 3x shootout I have tested Minolta Rokkor 45mm f2.8 microfiche lens: https://www.photomacrography.net/forum/ ... ht=#238491
It charted as #7 only with just average resolution figures. But MTF curve looked promising, extending very far with somewhat lower MTF values. So these days I have done quick and dirty 5-fold lens comparison at 3x with hornet head stacks. When Rokkor's shots were sharpened adequately prior stacking, end results were among best three pics. So, at the end of the day I believe that combined lens evaluation with both MTF measurement way and visual assessment way is the way to go.

RDolz · Post by **RDolz** » Fri Sep 27, 2019 3:54 am

Miljenko wrote:
rjlittlefield wrote: I wonder if you have done any similar tests regarding the effect of using different sensors.
Never directly but since 2004. I have "imatested" literally hundreds of lenses using different cameras I owned: Nikon D200, D300, D7000, D7100 and Fuji X-T1 & X-T2. On some occasions I used the same lens with different cameras and the results were so different that I archived them strictly separated. For some time my resolution reference was (great copy of) Sigma 70mm f2.8 Macro so I have tested in on several cameras. And yes, resolution values were different.
rjlittlefield wrote: My understanding of slanted edge analysis is that what it's really determining is system MTF, not just lens MTF. If that's correct, then it seems like sensor resolution, presence/absence of AA filter, and maybe even smaller details like microlens characteristics could have some significant effect on the results.
You are absolutely right. All those components play it's role in final resolution score. The most prominent being AA filter so when manufacturers in general decided to omit this questionable add-on, resolution figures measured jumped by a huge margin.
As a side note: although I do my best to stick to "standardized" testing procedure and gear I am very well aware there is hardly such thing. But nevertheless if someone does MTF testing using the same hardware and software as I do, results will be similar and within lens variation values.

Miljenko: Thank you very much for providing such an interesting macro lens comparison.

But, this point seems especially relevant to me. From what I have read, when measuring MTF in video systems, even the cables that were used to obtain the signal influenced the result. Thus, it seems more appropriate to measure the MTF of the complete system (lens + sensor + illumination).

Since what is especially relevant to the forum is the quality that is finally obtained, I think it would be very interesting to compare the same lens (with the same magnification conditions, lighting, shutter speed, etc.) mounted on different sensors (different sensors APSC or different Full Frame). Do you think it's right?

I understand that it is very complicated to have several bodies with their adapters, but, for example: Do you have any comparative measurement of the same lens mounted on the different APSC sensors of the Fuji series (16, 24 or 26 Mpx) ?.

(I think you are now using a Fuji X-T2 to obtain MTF).

The best

chris_ma · Post by **chris_ma** » Fri Sep 27, 2019 4:52 am

I enjoy these comparisons a lot and I can see how getting everything controlled and comparable is a huge undertaking, so thanks a lot for all the hard work!

One thing that I've noticed on my tests is that with very small detail the debayer artefacts overlap the real image, and I get much more detail with a camera with pixel shift, or even high resolution pixel shift.
Maybe the influence is not that big on tests targets where we just use the slanted edge to calculate MTF, but it would certainly be interesting to see how much it changes.

Is there a way to replicate your setup or is there any special gear/test target needed?
I could probably make some tests with an S1R and some high res lenses to see what your software readings are with those.

chris