Super Resolution at 2x Magnification?

A forum to ask questions, post setups, and generally discuss anything having to do with photomacrography and photomicroscopy.

Moderators: rjlittlefield, ChrisR, Chris S., Pau

ray_parkhurst
Posts: 3417
Joined: Sat Nov 20, 2010 10:40 am
Location: Santa Clara, CA, USA
Contact:

Super Resolution at 2x Magnification?

Post by ray_parkhurst »

I've been assuming that the Super Resolution (SR) technique would not be useful at magnifications where diffraction would obscure sub-pixel detail. However, as Rik has pointed out many times, there is still information to be had even at apertures well beyond the Diffraction Limited Aperture (DLA) of the sensor. Maybe there is enough sub-pixel information in the "gray area" just beyond the DLA to make the SR technique useful.

To test this, I shot 3 stacks at 2x using the 95mm Printing-Nikkor (95PN). Nominal aperture was f3.3, effective f9.9, which is where the 95PN is at its sharpest for single images. For each stack the subject was moved approx 20um in both X and Y with no regard to exact amount of movement. I used PhotoAcute3 to SR-process the 3 shots, producing a double-resolution final image that when downsized 2x is comparable in size to the original source images.

Comparing one of the source images to the SR composite shows that the SR technique introduces significant sharpening, either intentionally, as part of the SR process, or simply due to the 2x downsize. To make a good comparison, I took one of the source images and sharpened it to match the SR. Shown below are 200% crops at the center of the resulting SR and sharpened source images:

200% crop of SR composite:
Image

200% crop of single source image, sharpened to match SR composite:
Image

The SR composite, to my eye, shows significant added, believable, and more natural looking detail. This seems very promising to me, and a bit unexpected. I thought for sure that effective f9.9 would obscure most or all of the sub-pixel detail, but there indeed seems to be more info which can be extracted.

I plan to try f2.8, and perhaps a 4, 5, or 6 shot composite to see if I can get an improved result. Ultimately this shows that I can still get more from the 95PN. I may also try the same test at 3.5x with the 105mm Inspec.x L, though it is operating at f18 and thus seems less likely to benefit from the SR technique, but it would be good to see the limit.

rjlittlefield
Site Admin
Posts: 23563
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Post by rjlittlefield »

What's the pixel pitch of the sensor?

--Rik

ray_parkhurst
Posts: 3417
Joined: Sat Nov 20, 2010 10:40 am
Location: Santa Clara, CA, USA
Contact:

Post by ray_parkhurst »

rjlittlefield wrote:What's the pixel pitch of the sensor?

--Rik
It's my T2i, 4.3um.

rjlittlefield
Site Admin
Posts: 23563
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Post by rjlittlefield »

The relevant theory is explained at http://www.photomacrography.net/forum/v ... 831#124831 .

Thinking out loud...

Assuming lambda = 0.55 micron, then the cutoff frequency nu_0 at f/9.9 is 1/(0.55*9.9) = 0.1837 cycles/micron, or 5.445 microns per cycle. That would require 2.72 microns per pixel just to meet the minimum Nyquist requirement of 2 pixels per cycle.

Working the equations the other way around, 4.3 microns per pixel gives Nyquist sampling at 1/(2*4.3) = 0.116 cycles/micron, for which f/9.9 gives MTF 25.1% .

So yes, with 4.3 microns per pixel, your sensor should definitely be failing to resolve some of the finest detail.

By the same token, your SR images, downsampled to the same resolution as the original sensor, will also have the same restriction!

From my standpoint, what you've really done here is to compare the images captured by two methods that both have the same resolution (4.3 micron pixels), neither of which is capable of resolving all the detail that diffraction theory says is present in the optical image.

I pulled your two images into Photoshop, layered them, and did a flash-to-compare.

I definitely agree that the SR image is better than the sharpened single capture.

For me, the most obvious difference is that flash-to-compare makes the single capture look like it is slightly smeared, both horizontally and vertically. I can easily imagine the smearing is somehow introduced in demosaicing, but I expect there are other possibilities also.

Anyway, I think I'm seeing cleaner detail in the SR image, but I'm not convinced that I'm seeing more detail.

I hope this helps, somehow or other.

--Rik

ray_parkhurst
Posts: 3417
Joined: Sat Nov 20, 2010 10:40 am
Location: Santa Clara, CA, USA
Contact:

Post by ray_parkhurst »

rjlittlefield wrote:The relevant theory is explained at http://www.photomacrography.net/forum/v ... 831#124831 .

Thinking out loud...

Assuming lambda = 0.55 micron, then the cutoff frequency nu_0 at f/9.9 is 1/(0.55*9.9) = 0.1837 cycles/micron, or 5.445 microns per cycle. That would require 2.72 microns per pixel just to meet the minimum Nyquist requirement of 2 pixels per cycle.

Working the equations the other way around, 4.3 microns per pixel gives Nyquist sampling at 1/(2*4.3) = 0.116 cycles/micron, for which f/9.9 gives MTF 25.1% .

So yes, with 4.3 microns per pixel, your sensor should definitely be failing to resolve some of the finest detail.

By the same token, your SR images, downsampled to the same resolution as the original sensor, will also have the same restriction!

From my standpoint, what you've really done here is to compare the images captured by two methods that both have the same resolution (4.3 micron pixels), neither of which is capable of resolving all the detail that diffraction theory says is present in the optical image.

I pulled your two images into Photoshop, layered them, and did a flash-to-compare.

I definitely agree that the SR image is better than the sharpened single capture.

For me, the most obvious difference is that flash-to-compare makes the single capture look like it is slightly smeared, both horizontally and vertically. I can easily imagine the smearing is somehow introduced in demosaicing, but I expect there are other possibilities also.

Anyway, I think I'm seeing cleaner detail in the SR image, but I'm not convinced that I'm seeing more detail.

I hope this helps, somehow or other.

--Rik
Yes, good description...the single image looks "smeared" compared with the SR. I also see some details that are "hinted at" in the single image which show good contrast in the SR. This could easily be due to demosaicing interpolation and phasing issues, which are improved by the SR method.

It's a bit generous to do these calcs on a Bayer sensor with 50% coverage for G and 25% for R and B. Although the stated pitch of the sensor is 4.3um, the reality is the effective pitch is larger than that.

What I am looking for in the final output is better IQ at 100% for the original image size. The SR method seems to "fill in" some of the missing detail and provide some IQ improvement, albeit for at least a 3x increase in required source images. It might be worth the effort!

Lou Jost
Posts: 5948
Joined: Fri Sep 04, 2015 7:03 am
Location: Ecuador
Contact:

Post by Lou Jost »

As in Pentax/Sony pixel-shifting, maybe you are getting better sharpness because of noise reduction (you've tripled the light) and reduced Bayer filter effects (since each point in the image may have real measured values for two or three colors).

ray_parkhurst
Posts: 3417
Joined: Sat Nov 20, 2010 10:40 am
Location: Santa Clara, CA, USA
Contact:

Post by ray_parkhurst »

Lou Jost wrote:As in Pentax/Sony pixel-shifting, maybe you are getting better sharpness because of noise reduction (you've tripled the light) and reduced Bayer filter effects (since each point in the image may have real measured values for two or three colors).
Yes, these are likely the reasons for the improvement due to SR when viewing the image at same size. In my earlier tests of SR I saw improvements with only 2 samples, a little more improvement with 3, then diminishing returns with 4 or more when viewing same size. It's actually amazing to me that such a small number of additional samples makes these improvements, though in reality the change is somewhat modest.

Lou Jost
Posts: 5948
Joined: Fri Sep 04, 2015 7:03 am
Location: Ecuador
Contact:

Post by Lou Jost »

I'm amazed that Zerene is so consistent between stacks!

ray_parkhurst
Posts: 3417
Joined: Sat Nov 20, 2010 10:40 am
Location: Santa Clara, CA, USA
Contact:

Post by ray_parkhurst »

Lou Jost wrote:I'm amazed that Zerene is so consistent between stacks!
That's another thing that I was able to show with this exercise. I didn't know if the stack-SR workflow would be effective. I worried that I'd have to do SR first, then stack the SR outputs. If I had a clue about doing batch processing it wouldn't be such a concern, but running the SR and stacks manually makes it more daunting to think of the SR-stack workflow.

Lou Jost
Posts: 5948
Joined: Fri Sep 04, 2015 7:03 am
Location: Ecuador
Contact:

Post by Lou Jost »

By the way, I'm sorry I still haven't done the pixel-shifting tests I promised. I've been too busy lately with field work and other stuff.

ray_parkhurst
Posts: 3417
Joined: Sat Nov 20, 2010 10:40 am
Location: Santa Clara, CA, USA
Contact:

Post by ray_parkhurst »

Lou Jost wrote:By the way, I'm sorry I still haven't done the pixel-shifting tests I promised. I've been too busy lately with field work and other stuff.
No worries Lou, though I did ponder asking you about it when you mentioned pixel shift. I'm still using the T2i, and obviously searching for ways to get the most out of it and my lenses. I still want to see the single, shifted, and objective views to know what the Oly pixel shift would buy me, so appreciate if it stays on your to-do list.

ray_parkhurst
Posts: 3417
Joined: Sat Nov 20, 2010 10:40 am
Location: Santa Clara, CA, USA
Contact:

Post by ray_parkhurst »

For completeness regarding the SR output, here is a crop of the original double-resolution image. This shows the detail lost by down-converting/upconverting by 2x:

Edited to add: to my eye there is not much more detail in the full-size SR vs the 1/2x-2x processed one, it just looks sharper and with better contrast in the details.

Edit #2: Forgot to mention that I shot RAW, then kept them in TIFF until the final output in jpg. However, I did check and even outputting in TIFF vs the required jpg quality "3" to maintain file size, the differences in IQ are miniscule.

Image

Post Reply Previous topicNext topic