View previous topic :: View next topic 
Author 
Message 
rjlittlefield Site Admin
Joined: 01 Aug 2006 Posts: 20652 Location: Richland, Washington State, USA

Posted: Sun Mar 01, 2020 8:12 pm Post subject: Sample density needed to resolve Rayleigh features 


This post is to again address the question, "What pixel density is needed to capture an optical image?"
More precisely, this time I'm going to look at what sample density is needed to resolve "Rayleigh features", by which I mean Airy disks that are spaced at the Rayleigh criterion.
Motivation for writing this up is that in recent memory I've seen a couple of instances where people argued that two pixels per Rayleigh separation distance is enough.
I gather this was based on combining the ideas that (a) Rayleigh separation is the smallest thing we care about, and (b) the Nyquist sampling theorem says that two samples per cycle is enough.
Unfortunately, that line of reasoning is not correct. Here I'll attempt to illustrate and explain why that is, and in the end talk about some aspects of Nyquist that you may not have realized.
To begin, let's define a test problem.
Here is a line cut through a simulated optical image that consists of four Airy disks organized in two pairs. In each pair, the separation between the two Airy peaks is set to the Rayleigh criterion. But there's no particular relationship between one pair and the other pair, and in fact I'm going to adjust the positions of the pairs so as to create vivid examples.
Graph title: "This is the optical image to be captured, two pairs of Airy disks at Rayleigh separation."
What I'm now going to do is "capture" that image by sampling to discrete pixels. I'll do that under three conditions, and in each condition I'll tweak the positions of the two pairs so as to show the best and worst cases.
First, I'll sample at two pixels per Rayleigh separation. This works great IF the pixels happen to line up properly with the peaks and valleys, but it totally fails to resolve the separation if the pixels do not happen to line up well.
In my tests, the captured pixel values showed no valley at all in something over 1/4 of random positions. Losing features is not a rare occurrence.
Graph titles: "At two samples per Rayleigh separation, many pairs are not resolved at all. Here are a couple more cases where no valley is seen."
It turns out that there are a couple of things wrong with casually applying the Nyquist sampling theorem this way.
One of them, very important, is that even though we might only be interested in Rayleighsized features, the optical image actually contains higher resolution information that can mess up the sampling. The actual cutoff frequency for a diffractionlimited image corresponds to the Abbe separation, about a factor of 1.22 smaller than the Rayleigh separation.
So, to properly apply Nyquist, we need to use a smaller spacing of our samples, about 2.44 samples per Rayleigh separation.
If we do that, then this is what happens:
Graph title: "At Nyquist = two samples per cycle at cutoff, all pairs are resolved but contrast varies widely."
Well, that's certainly a lot better  we don't actually miss the separation even in the worst case. On the other hand, we do find a pretty serious variation in contrast, from about 16.1% in the best case down to about 5.7% in the worst case. (This is calculating contrast=(peakvalley)/peak in both cases.)
No surprise, we can do better by increasing the sampling density.
Here is the same exercise, done with 3 samples per cycle at cutoff, rather than Nyquist's 2 samples per cycle:
Graph title: "At 1.5*Nyquist, three samples per cycle at cutoff, contrast is much more uniform."
With this increase in sampling density, we're now looking at a contrast variation only from 19.1% to 15.6%.
From here, a simple intuition is correct: higher sample density gives a more accurate capture, lower sample density gives a less accurate capture, and all of that is on a continuum with diminishing returns. Many people would agree that 3 samples per cycle is a lot better than 2, while 4 samples per cycle is better than 3 but maybe not worth the cost, and 10 samples per cycle would be crazy.
While I was writing this up, I realized that there's a very important aspect of the Nyquist sampling theorem that I'm hoping is subtle and frequently overlooked. I'm hoping that because it took me a long time to get a good grip on it.
Here is an illustration of it:
Graph title: "Given these values sampled at Nyquist density, which show two very different patterns, we can reconstruct the original signal, which has two copies of the same pattern, slightly shifted."
Now, in words...
With some additional fine print, what the sampling theorem guarantees is that a bandwidth limited signal can be perfectly reconstructed from a set of discrete samples taken at 2 samples per period of the bandwidth limit.
The theorem does not say that the samples themselves will "look like" the original signal in any particular way.
To recover the original signal from Nyquist samples, you have to go through a Fourierbased reconstruction process that implicitly depends on prior knowledge of the bandwidth limit. In the above example, the reconstruction process would be figuring out that the original signal must have been overshooting and undershooting between the sample points, because that's the only way it could exactly meet the sample points given the bandwidth restriction.
If you do any other sort of reconstruction, for example the casual fillinginthegaps interpolation that we humans do without even thinking, then the signal you think you see may be significantly different from the one that was actually measured.
I hope this helps somebody else. I'm sure I've learned a lot in trying to figure out how to explain it.
Rik 

Back to top 


Lou Jost
Joined: 04 Sep 2015 Posts: 4269 Location: Ecuador

Posted: Mon Mar 02, 2020 4:47 am Post subject: 


Excellent examples.
It's important to note also that this is true for monochromatic light. In color, the "pixels" for each color are quite far apart. If the Bayer filter were perfect (it's not) and the subject had blue or red features, the effective pixel density would be half the density of pixels on the sensor. So the number of pixels you desire based on Rik's example needs to be increased for Bayer sensors, up to a factor of 2 depending on the subject. A reasonable average would be something slightly less than 2. I think I've seen this problem treated mathematically somewhere, will look for it. _________________ Lou Jost
www.ecomingafoundation.wordpress.com
www.loujost.com 

Back to top 


mawyatt
Joined: 22 Aug 2013 Posts: 2452 Location: Clearwater

Posted: Mon Mar 02, 2020 7:15 am Post subject: 


Rik,
Nice explanation, well done!!
There's a very good article on Wiki about the NyquistShannon Sampling Theorm, that includes an optical section. Also an excellent visulation of how bandwidth limits in the frequency domain affects reconstructed signals in the time domain.
https://en.wikipedia.org/wiki/Nyquist–Shannon_sampling_theorem
In communications (where these theorems originated) recent work has been pushing the limits beyond Nyquist. Most have thought of Nyquist as a limit, but as far back as 1975 Mazo at Bell Labs had shown it is not an absolute limit barrier, but could be "broken" in fact under certain signaling conditions. The recent interest in all this is simply because there is so much value ($) in "pushing" more signals through bandwidth limited channels (all channels are effectively bandwidth limited in some way) in ever aspect of communications (5G & others).
Google "Faster than Nyquist" will return a number of links.
J. B. Anderson, F. Rusek, and V. wall, “FasterThan Nyquist Signaling,” Proceedings of the IEEE, vol. 101, no. 8, pp. 1817–1830, Aug. 2013.
J. E. Mazo, “FasterThanNyquist Signaling,” Bell System Technical Journal, vol. 54, no. 8, 1975.
Best, _________________ Research is like a treasure hunt, you don't know where to look or what you'll find!
~Mike 

Back to top 


wpl
Joined: 28 Jun 2012 Posts: 17 Location: New Mexico, USA

Posted: Mon Mar 02, 2020 2:12 pm Post subject: 


A camera sensor does not really sample the image at a finite number of points. It histograms the image into a finite number of bins. This is not the same thing and I wonder if the sampling theorem even strictly applies to this case. 

Back to top 


rjlittlefield Site Admin
Joined: 01 Aug 2006 Posts: 20652 Location: Richland, Washington State, USA

Posted: Mon Mar 02, 2020 3:48 pm Post subject: 


wpl wrote:  A camera sensor does not really sample the image at a finite number of points. It histograms the image into a finite number of bins. 
True, and what I didn't tell you is that all of my graphs except the last one were constructed with binning, just like a camera sensor with 100% coverage for each pixel.
The effect of binning does not change the overall message, though it can either increase or decrease local contrast depending on how pixels happen to line up with features in the image.
Here s a copy of one graph from above, plus the same alignment but not binned.
Quote:  This is not the same thing and I wonder if the sampling theorem even strictly applies to this case. 
It depends on what you mean by "strictly". You can think of pixel values as being point samples of a function that is formed by convolving the optical image with a second sensorrelated function that represents the acceptance profiles of the pixels. The Fourier transform of a convolution is equal to the product of the Fourier transforms of the component functions, so if the optical image is bandwidth limited then the convolution is bandwidth limited also. Nyquist then applies to sampling the bandwidth limited convolution. The bottom line is that if Nyquist would have applied to the optical image, then it also applies to the binned function that is actually sampled.
I have seen this line of analysis written up in a lot more detail, in one of the several long articles that I've read about slantededge MTF measurement. Unfortunately I can't remember which one, and I cannot quickly locate the reference.
Rik 

Back to top 


wpl
Joined: 28 Jun 2012 Posts: 17 Location: New Mexico, USA

Posted: Mon Mar 02, 2020 4:36 pm Post subject: 


Rik,
Thanks for helping me understand how binning should be handled. The effect may be small but you did say that the difference between 1 and 1.22 is significant, so maybe small effects make a difference and should be considered.
Sincerely,
Walter 

Back to top 


palea
Joined: 17 Dec 2018 Posts: 34

Posted: Tue Mar 03, 2020 6:31 am Post subject: 


Lou Jost wrote:  So the number of pixels you desire based on Rik's example needs to be increased for Bayer sensors, up to a factor of 2 depending on the subject. 
2sqrt(2) if the feature of interest's oriented along the diagonal of a grid of square pixels. And, as the sensor's two dimensional, the total number of pixels one would be looking for in an image scales like (2sqrt(2))^2. Which isn't the point you were making but it's a likely area of confusion, particularly as the entrylevel superresolution camera manufacturers are calling pixel shift is probably the easiest way to get the pixel count.
rjlittlefield wrote:  I realized that there's a very important aspect of the Nyquist sampling theorem that I'm hoping is subtle and frequently overlooked. I'm hoping that because it took me a long time to get a good grip on it. 
I think this particular aspect of perfect reconstruction's easy to miss if you're selfstudying. Every class I've had where Nyquist comes up has had some problem set/homework exercise of looking at what happens when pushing the Nyquist "limit", though (I just did this for like the fourth time in an assignment yesterday, actually). You probably have the code already set up to do the same in whatever's producing the binning graphs.
A related hang up, which seems to be the motivation for this thread, is people get fixated on the factor of two associated with perfect reconstruction and start treating the Nyquist frequency like a wall rather than a mathematically idealized statement about the bandwidth of a lowpass filter. Sensors and ADCs are noisy and DSP has finite numerical precision, meaning we never get perfect pixels anyway. Mostly what we actually care about is avoiding empty magnification, whether it's happening from moving up in objective magnification without adequate increase in NA or from shrinking pixels.
It's perhaps a little early, but I feel like I should mention RichardsonLucy deconvolution is a fairly easy method of mitigating the diffraction limits being studied here. I'd suggest checking it out once the Nyquist bit starts feeling done. 

Back to top 


rjlittlefield Site Admin
Joined: 01 Aug 2006 Posts: 20652 Location: Richland, Washington State, USA

Posted: Tue Mar 03, 2020 11:25 am Post subject: 


palea wrote:  ...easy to miss if you're selfstudying... 
Good point, and certainly applicable. I come from a math / computer science background with some early coursework in electronics, but most of my knowledge of Nyquist comes from books and experiments, plus discussions like this one. As far as I can remember, I never had any class that addressed Nyquist in a homework assignment.
A couple of other references...
There's a very old thread, started almost 13 years ago, where I come at the issue of sampling density from a different viewpoint. Despite its age, I still refer to that thread frequently, largely for one of its summary statements: Quote:  So, there's "sharp" and there's "detailed"  pick one or the other 'cuz you can't have both. What a bummer! 
In reviewing that old thread just now, I discovered a trailing post that I had overlooked. I've just now responded to that, so the thread will be popped near the top of the forum again.
Switching subjects a bit...
Quote:  It's perhaps a little early, but I feel like I should mention RichardsonLucy deconvolution is a fairly easy method of mitigating the diffraction limits being studied here. I'd suggest checking it out once the Nyquist bit starts feeling done. 
In that line, I suggest also reviewing earlier discussion in the thread and links starting at https://www.photomacrography.net/forum/viewtopic.php?t=33724 .
Rik 

Back to top 


rjlittlefield Site Admin
Joined: 01 Aug 2006 Posts: 20652 Location: Richland, Washington State, USA

Posted: Tue Mar 03, 2020 12:48 pm Post subject: 


mawyatt wrote:  In communications (where these theorems originated) recent work has been pushing the limits beyond Nyquist. Most have thought of Nyquist as a limit, but as far back as 1975 Mazo at Bell Labs had shown it is not an absolute limit barrier, but could be "broken" in fact under certain signaling conditions. The recent interest in all this is simply because there is so much value ($) in "pushing" more signals through bandwidth limited channels (all channels are effectively bandwidth limited in some way) in ever aspect of communications (5G & others).
Google "Faster than Nyquist" will return a number of links.
J. B. Anderson, F. Rusek, and V. wall, “FasterThan Nyquist Signaling,” Proceedings of the IEEE, vol. 101, no. 8, pp. 1817–1830, Aug. 2013.
J. E. Mazo, “FasterThanNyquist Signaling,” Bell System Technical Journal, vol. 54, no. 8, 1975. 
Good refs. I also found what appears to be a nice summary at https://dsp.stackexchange.com/questions/52730/whatisfasterthannyquistsignaling . It begins with
So, a bit different situation. FTN signalling applies when you have some choice about what you're pushing down the channel.
For optical images limited by diffraction, I'm thinking that the sampling theorem we've been discussing still applies.
Rik 

Back to top 


mawyatt
Joined: 22 Aug 2013 Posts: 2452 Location: Clearwater

Posted: Tue Mar 03, 2020 7:02 pm Post subject: 


rjlittlefield wrote:  mawyatt wrote:  In communications (where these theorems originated) recent work has been pushing the limits beyond Nyquist. Most have thought of Nyquist as a limit, but as far back as 1975 Mazo at Bell Labs had shown it is not an absolute limit barrier, but could be "broken" in fact under certain signaling conditions. The recent interest in all this is simply because there is so much value ($) in "pushing" more signals through bandwidth limited channels (all channels are effectively bandwidth limited in some way) in ever aspect of communications (5G & others).
Google "Faster than Nyquist" will return a number of links.
J. B. Anderson, F. Rusek, and V. wall, “FasterThan Nyquist Signaling,” Proceedings of the IEEE, vol. 101, no. 8, pp. 1817–1830, Aug. 2013.
J. E. Mazo, “FasterThanNyquist Signaling,” Bell System Technical Journal, vol. 54, no. 8, 1975. 
Good refs. I also found what appears to be a nice summary at https://dsp.stackexchange.com/questions/52730/whatisfasterthannyquistsignaling . It begins with
So, a bit different situation. FTN signalling applies when you have some choice about what you're pushing down the channel.
For optical images limited by diffraction, I'm thinking that the sampling theorem we've been discussing still applies.
Rik 
There might be some analogy there, I tend to think of this as a communication system where the transmitter is the subject, the receiver is the sensor and the channel is the lens, aperture and sensor filter. So trying to push as much information as possible from the transmitter thru the channel to the receiver.
Also tend to think of some image post processing as reducing some of the effects of the "channel" which is often referred to as "equalizing the channel" in communications.
Anyway, interesting stuff.
Best, _________________ Research is like a treasure hunt, you don't know where to look or what you'll find!
~Mike 

Back to top 


Ultima_Gaina
Joined: 28 Jan 2017 Posts: 108

Posted: Tue Mar 10, 2020 11:00 am Post subject: 


rjlittlefield wrote:  mawyatt wrote:  In communications (where these theorems originated) recent work has been pushing the limits beyond Nyquist. Most have thought of Nyquist as a limit, but as far back as 1975 Mazo at Bell Labs had shown it is not an absolute limit barrier, but could be "broken" in fact under certain signaling conditions. The recent interest in all this is simply because there is so much value ($) in "pushing" more signals through bandwidth limited channels (all channels are effectively bandwidth limited in some way) in ever aspect of communications (5G & others).
Google "Faster than Nyquist" will return a number of links.
J. B. Anderson, F. Rusek, and V. wall, “FasterThan Nyquist Signaling,” Proceedings of the IEEE, vol. 101, no. 8, pp. 1817–1830, Aug. 2013.
J. E. Mazo, “FasterThanNyquist Signaling,” Bell System Technical Journal, vol. 54, no. 8, 1975. 
Good refs. I also found what appears to be a nice summary at https://dsp.stackexchange.com/questions/52730/whatisfasterthannyquistsignaling . It begins with
So, a bit different situation. FTN signalling applies when you have some choice about what you're pushing down the channel.
For optical images limited by diffraction, I'm thinking that the sampling theorem we've been discussing still applies.
Rik 
The analogy between optical and telecom is obvious. After all, in both cases, we are dealing with electromagnetic waves as a means of information transport.
In telecom, there are ways to deal with some intersymbol interference, tolerate some errors and go beyond Nyquist, indeed.
In the case presented here, when the resolution/sampling is insufficient, details can be missed. That's equivalent to the damage done by ISI in the information theory.
But when our optical "intersymbol interference" is not dramatic, errors can be also tolerated and, for example, basic sharpening can enhance the transmitted signal, with results that are obviously not ideal, but good enough to transmit the desired detail/information. 

Back to top 


mawyatt
Joined: 22 Aug 2013 Posts: 2452 Location: Clearwater

Posted: Tue Mar 10, 2020 12:21 pm Post subject: 


Ultima_Gaina wrote:  rjlittlefield wrote:  mawyatt wrote:  In communications (where these theorems originated) recent work has been pushing the limits beyond Nyquist. Most have thought of Nyquist as a limit, but as far back as 1975 Mazo at Bell Labs had shown it is not an absolute limit barrier, but could be "broken" in fact under certain signaling conditions. The recent interest in all this is simply because there is so much value ($) in "pushing" more signals through bandwidth limited channels (all channels are effectively bandwidth limited in some way) in ever aspect of communications (5G & others).
Google "Faster than Nyquist" will return a number of links.
J. B. Anderson, F. Rusek, and V. wall, “FasterThan Nyquist Signaling,” Proceedings of the IEEE, vol. 101, no. 8, pp. 1817–1830, Aug. 2013.
J. E. Mazo, “FasterThanNyquist Signaling,” Bell System Technical Journal, vol. 54, no. 8, 1975. 
Good refs. I also found what appears to be a nice summary at https://dsp.stackexchange.com/questions/52730/whatisfasterthannyquistsignaling . It begins with
So, a bit different situation. FTN signalling applies when you have some choice about what you're pushing down the channel.
For optical images limited by diffraction, I'm thinking that the sampling theorem we've been discussing still applies.
Rik 
The analogy between optical and telecom is obvious. After all, in both cases, we are dealing with electromagnetic waves as a means of information transport.
In telecom, there are ways to deal with some intersymbol interference, tolerate some errors and go beyond Nyquist, indeed.
In the case presented here, when the resolution/sampling is insufficient, details can be missed. That's equivalent to the damage done by ISI in the information theory.
But when our optical "intersymbol interference" is not dramatic, errors can be also tolerated and, for example, basic sharpening can enhance the transmitted signal, with results that are obviously not ideal, but good enough to transmit the desired detail/information. 
The analogy as you say they are both EM waves has fascinated me, especially considering that all this is described by Maxwell's Equations.
Kirkhoff Voltage and Current laws, Ohms Law, Gauss & Ampere's Law are all special cases of Maxwell's Equations! The vision Maxwell had into EM physics is absolutely stunning, yet few know of his contributions.
Best, _________________ Research is like a treasure hunt, you don't know where to look or what you'll find!
~Mike 

Back to top 


palea
Joined: 17 Dec 2018 Posts: 34

Posted: Sat Mar 14, 2020 9:23 am Post subject: 


Any particular aspect of it? About half the thread is pretty solid, I think, but I chose not to bump it when I looked a couple years ago because attempting to explain components of graduate signal processing courses to interested folks over the internet is seldom particularly effective, especially from a low post count. I'm not seeing that would be different now.
I also feel maybe I shouldn't be casting a first stone. Three years later, limitations of RichardsonLucy implementations for photographic processing haven't changed. While most of them could be removed by straightforward software adjustments, my coding time is allocated elsewhere for the foreseeable future. 

Back to top 


rjlittlefield Site Admin
Joined: 01 Aug 2006 Posts: 20652 Location: Richland, Washington State, USA

Posted: Sat Mar 14, 2020 1:43 pm Post subject: 


palea wrote:  Any particular aspect of it? 
Not really. I just like people to be familiar with older discussions before revisiting the territory. That's not to say there's anything wrong with revisiting old territory. Sometimes revisiting becomes revising, or at least better understanding and better communicating, and those are both good things.
Quote:  ... attempting to explain components of graduate signal processing courses to interested folks over the internet is seldom particularly effective... 
Yes, that can be tricky. But sometimes it's worth the attempt. If you see something that I've messed up or seem to have a wrong understanding about, please let me know so we can get it fixed.
Rik 

Back to top 


palea
Joined: 17 Dec 2018 Posts: 34

Posted: Sat Mar 21, 2020 6:53 pm Post subject: 


rjlittlefield wrote:  If you see something that I've messed up or seem to have a wrong understanding about, please let me know so we can get it fixed. 
Hi Rik, I think I'd like to suggest a different framing. Zerene gives you a thorough knowledge of alignment and focus stacking. Similar depth and nuance exists in most other image formation and processing tasks. So, given reasonable diligence (as is the case here) I don't feel it's so much about right and wrong as it is completeness of understanding.
To make a broad generalization, technically minded photography forum threads are usually about selective exploration and therefore often resemble the content one might find in, say, part of an undergraduate lecture on a particular topic. That's cool but it does imply limits to what can be expected. I guess the main conceptual observation I'd make herebesides the Nyquist is not a wall remarks aboveis forum discussion usually neglects what's happening with information in signals, either specifically in a Shannon entropy sense or more generally.
For example, diffraction is routinely interpreted as a loss even though underlying issue is that the information is present but got redistributed in a way which both makes it cumbersome to access and impedes use of other information. This is also the case for energy aliased by sampling and haloing of depth discontinuities in focus stacks. If there's a pattern I see to changes in signal processing over the past 25 years or so it's use of increasing computing power to mitigate some of these limitations. 

Back to top 


