Stack-and-stitch: ordinary optics vs telecentric

A forum to ask questions, post setups, and generally discuss anything having to do with photomacrography and photomicroscopy.

Moderators: rjlittlefield, ChrisR, Chris S., Pau

rjlittlefield
Site Admin
Posts: 23562
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Stack-and-stitch: ordinary optics vs telecentric

Post by rjlittlefield »

[Edited to add: This thread is only the second of three closely related threads regarding telecentric optics. The other two are HERE and HERE.]

Some recent observations and questions by Steve Valley (in this thread) prompted me to write up this explanation of stack-and-stitch with ordinary optics versus telecentric systems.

The key parts were:
svalley wrote:Larger insects present another problem. They are too large to fit into the field of view of the scope. This means that they have to be shot in overlapping sections and these sections have to be stitched together to create an image of the whole specimen. ... I discovered that with our Leica imaging system, there is too much depth of field at low magnifications, despite always shooting with both internal apertures wide open. At magnifications less than 16x, the size of the specimen changes dramatically within the image frame of each layer in the stack, resulting in terrible fringing around every sharp edge in the composite.
rjlittlefield wrote:Regarding your problems of stitching stacks, you might be interested in http://www.photomacrography.net/forum/v ... php?t=1032 .
Each tile is a stack, 11.5 mm wide. The deepest stack is over 25 mm deep, and the total subject in-focus depth is 37 mm. To shoot the stacks, the camera was shifted laterally. But when I flash between stacks, there's no visible parallax. It's pretty cool.
svalley wrote:Rik, I had already read that thread and just reread it. ... I did not understand exactly what you are doing.... to shoot the individual tiles are you just moving the camera and rear lens with the front lens remaining stationary, relative to the subject? This is a very interesting concept, but would be hard to adapt to my scope setup at work. I have not had any noticable parallax problems with my stitched images.
I was confused for a long time about apparent inconsistencies in these various writings, but I think that eventually I figured out how to make everything fit together.

Let's consider a simple hypothetical situation. Our subject is a red circle above a green square, in different planes.

Image

Suppose we try to stack-and-stitch this subject using ordinary optics, where the center of perspective (the entrance pupil) is someplace around the center of the lens. Then we get this situation:

Image

Notice that with an ordinary lens, the red circle looks bigger than the green square, regardless of where the lens is focused. If the lens is moved to focus lower, then the ratio of sizes becomes a bit larger, but not much.

In this case, there are essentially two approaches that the software can take. Either 1) it can do nothing except select pixels from the appropriate images, or 2) it can adjust the scale of each image to compensate for the shift in focus, and then select pixels.

In general, the decision of how to select pixels varies from easy to difficult depending on the DOF per frame. At high magnification and large apertures, with very shallow DOF per frame, decisions are easy and the software generally gets them right. At lower magnifications and smaller apertures, with greater DOF per frame, decisions are difficult and the software may get them wrong.

The quality of pixel selection interacts strongly with scaling and stitching.

If the software does not scale (Cases A and B), then bad pixel selection quickly causes bad result quality because the source images just don't line up. Low magnification makes this problem worse because it encourages large focus steps that have correspondingly large changes in scale.

This problem can be almost completely avoided by having the software adjust images to a common scale (align image features) before selecting pixels. The solution is not quite perfect because the relative sizes of image features at different depths actually changes slightly as the focus shifts. But it's pretty good.

So, if you want to do just a single stack, then in most cases scaling the images to compensate for focus shift is the right way to go. (The one serious exception is if you're not prepared to accept the closer=bigger perspective that's naturally produced by this type of lens system.)

But if you try stitching multiple stacks, the picture changes.

In this case, the fundamental problem is that the lens looks outward in a cone. Subject features that appear to line up with the lens in one position, do not line up when the lens is moved to a different position. This is called parallax, and it's a Very Bad Thing for stitching. But what to do, what to do?

The "solution" to parallax in this case is simple: don't adjust image scale to compensate for focus shift. Taking that approach -- and assuming perfect pixel selection -- subject features will line up the same way in adjacent tiles, even though that's not the way the lens originally saw them.

Unfortunately, if there are errors in pixel selection, this becomes a Catch-22 situation. You cannot turn off scaling because subject edges go bad, but you cannot turn it on either because then you get parallax errors between tiles.

Normally I run with scaling turned on. That produces the best image quality in a single stack, but introduces parallax between tiles.

I'm thinking that Steve runs with scaling turned off. That avoids parallax, but causes worse problems around edges at lower magnifications.

OK, with that analysis as background, let's look at telecentric optics.


Image

You can think about telecentric optics as meaning "no change in scale with distance", or "only uses light rays parallel to the axis", or "entrance pupil at infinity". They're all roughly equivalent.

However you think of it, the key feature of telecentric optics for our purposes here is that the alignment of subject features does not change as you move the lens, either to step focus or to shift laterally for tiling.

Because the alignment of subject features does not change -- courtesy the telecentric optics -- there is no need for scale adjustment in software, and there is no parallax introduced between adjacent tiles.

Using telecentric optics makes it much easier to get a clean result from stack-and-stitch.

There is a downside, of course: telecentric optics are uncommon, potentially expensive, and necessarily have a small field of view (no greater than the front lens diameter minus the aperture size).

But dang, they sure are neat to work with for some subject sizes!

Does this explanation help? What remains confusing?

--Rik

Edit: Jan 2, 2007, to add link to earlier discussions.
Edit: Feb 21, 2015, adding links to other closely related threads
Last edited by rjlittlefield on Sat Feb 21, 2015 11:50 am, edited 2 times in total.

svalley
Posts: 343
Joined: Sun Dec 03, 2006 7:07 pm
Location: Albany, Oregon

Post by svalley »

Light bulb goes off in my head!! I got it.

I may have a solution to the problem of finding large diameter glass. I have optical setups from an old style map plotting apparatus. Some of these "lenses" are >4 inches in diameter. I wonder, given the original purpose of this apparatus if these were telecentric optics?

Anyway thanks, Rik. You describe my technique perfectly. High mag=shallow depth+ no scaling between layers in the stack. No obvious parallax when tiles are stiched.

You know everyone is going to think we are nuts, carrying on like this about weird optics, but you can't build a time machine without weird optics.
"You can't build a time machine without weird optics"
Steve Valley - Albany, Oregon

rjlittlefield
Site Admin
Posts: 23562
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Post by rjlittlefield »

svalley wrote:Light bulb goes off in my head!! I got it.
I love it when that happens. :D
I may have a solution to the problem of finding large diameter glass. I have optical setups from an old style map plotting apparatus. Some of these "lenses" are >4 inches in diameter. I wonder, given the original purpose of this apparatus if these were telecentric optics?
Hard to say, but even if they weren't, perhaps they can be used for the purpose. There's another configuration of lenses that's telecentric and is suitable for low mag work. I'll try to find time to sketch that up tonight.
You know everyone is going to think we are nuts, carrying on like this about weird optics, but you can't build a time machine without weird optics.
Trust me -- with this group, it's going to take a lot more than weird optics to make 'em think we're nuts. :wink:

--Rik

PS. The line about "time machine...weird optics" sounds familiar, but I can't place it. Is that quoted from something, and if so, what?

svalley
Posts: 343
Joined: Sun Dec 03, 2006 7:07 pm
Location: Albany, Oregon

Post by svalley »

I thought I made it up, but it could be someone else's quote buried in a tangled neuron cluster. hmmmm.....deciding to try it out as a signature
"You can't build a time machine without weird optics"
Steve Valley - Albany, Oregon

rjlittlefield
Site Admin
Posts: 23562
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Post by rjlittlefield »

I was thinking maybe "Back to the Future" or some such. But when I checked "time machine" "weird optics" with Google, no hits came back. So I guess it's either original, or might as well be. :D

--Rik

Bruce Williams
Posts: 1120
Joined: Mon Oct 30, 2006 1:41 pm
Location: Northamptonshire, England
Contact:

Post by Bruce Williams »

Hi Rik,

To help me catch up with the earlier postings, could you point me to the original thread where Steve wrote "Larger insects present another problem...".

Thanks,
Bruce

rjlittlefield
Site Admin
Posts: 23562
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Post by rjlittlefield »

could you point me to the original thread...
Sorry, thought I had linked that but I guess not.

It's well down in his introduction topic, which is here in the Community forum.

I also edited the first post in this thread to include the link. Thanks for the tickle!

--Rik

rjlittlefield
Site Admin
Posts: 23562
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Post by rjlittlefield »

svalley wrote:You describe my technique perfectly. High mag=shallow depth+ no scaling between layers in the stack.
Steve,

In Helicon Focus and CombineZ5, it's a user setting whether images in the stack get scaled and realigned.

Helicon actually has independent controls for whether you want adjustment in X, Y, magnification, and rotation, and if so, what limits to search over.

What options/controls are provided by the software you use?

--Rik

svalley
Posts: 343
Joined: Sun Dec 03, 2006 7:07 pm
Location: Albany, Oregon

Post by svalley »

Rik, Funny, I had not even considered that these options might be available. I will have to look tomorrow. Rotation does not seem like it would apply in my case, nor X & Y. Changing magnification for scaling seems like it would cause distortion, when I try to imagine the affect in my mind (feeble as it is). ImagePro is actually an image analysis package of which the extended depth routine is just a small part. One nice feature is it automatically plays the stack like a movie when it completes the last exposure, so I can see immediatly if the specimen has moved during the aquisition of the stack.

Sometimes this can have unexpected results. Once when i was shooting a scale insect, a dozen or so babies emerged out from under while I was aquiring the stack. When the stack played back there were all these little guys zipping around. It looked pretty cool.

I was thinking about our earlier discussions in this thread and it occurs to me that the reason I do not have a parallax problem (with my tiles) may be because size of the region I am shooting is much smaller than the diameter of my objective.
"You can't build a time machine without weird optics"
Steve Valley - Albany, Oregon

rjlittlefield
Site Admin
Posts: 23562
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Post by rjlittlefield »

svalley wrote:Rik, Funny, I had not even considered that these options might be available. I will have to look tomorrow. Rotation does not seem like it would apply in my case, nor X & Y.
I agree.

Rotation is mostly useful in setups that allow a little wiggle in the movement, such as the demo that I did for Carl by sliding my camera across the table by hand. X and Y shifts can occur for the same reason.

Shifts also occur if the axis of movement is not parallel to the optical axis. This happens, for example, if one shoots through the normal eyepieces of most stereo scopes. The camera port of a stereo scope may or may not have this problem; the best ones probably don't. Shifts occasionally happen with my apparatus if I'm not careful to get my bellows perfectly square to the milling table.
Changing magnification for scaling seems like it would cause distortion, when I try to imagine the affect in my mind (feeble as it is).
I hope that having difficulty with this does not indicate a feeble mind, because it took me literally weeks of dedicated effort to get all the concepts straight. With luck, we can get you through the process faster.

Start by going back to the second diagram at the top of this posting. Look at the little section labeled "Single Stack".

Because the lens is looking outward in a cone (as all normal lenses do), the apparent size of a feature depends on its distance away from the lens. The red circle and the green square have the same width, as measured by a ruler. But because the red circle is closer to the lens than the green square is, it appears to be bigger. This is just perspective: a very special type of "distortion" that we usually don't think about unless it's carried to extremes. You've probably seen hilarious pictures of people shot with a very wideangle lens from close up -- their noses look huge! Of course their noses are not really huge, they're just relatively much closer to the lens than the rest of the face is. Again, this is just basic perspective: closer=bigger.

Now consider what happens as the lens moves downward to shift focus from the red circle to the green square. Both images get bigger, by whatever factor they are now closer to the center of projection. The red circle and the green square do not get bigger by quite the same factor, but it's pretty close.

So, one can adjust the overall scale of the second image to make corresponding in-focus and out-of-focus features line up very closely. The sharp red circle from the first image can be aligned with the fuzzy red circle from the second image, and at the same time the fuzzy green circle from the first image will be aligned with the sharp green circle from the second image. Not exact, but pretty close. (In math, a/b does not equal (a+delta)/(b+delta), but it's pretty close if delta is small compared to a and b. For stacking, scaling can align features very well if the focus step is small compared to the distance from lens to subject.)

It's the fact that fuzzy and sharp images of the same feature are aligned with each other, that makes this arrangement tolerant of errors in pixel selection.

If you do the scaling perfectly (based on a priori knowledge of the geometry), and you select exactly the in-focus pixels, then no distortions are introduced beyond the basic perspective projection. The final composite is "distorted" only in the perspective sense that foreground features will appear bigger than background features of the same size. This is a bad thing if you're interested in precision measurements, but it doesn't affect visual perception very much. In fact, it corresponds exactly to the geometry that you would see if you were shrunk small and placed your eye at the scope's center of projection, someplace near the middle of its objective.

Unfortunately, one seldom knows the geometry a priori, so what most software actually does is to ignore geometry and determine scaling solely by analysing the images to line up visible features. This analysis is not an exact process, and errors accumulate as the software works through the stack. Interesting nonlinear distortions can easily be introduced into the composite.

However, those introduced distortions are so small that almost nobody ever notices them. The only reason I know about them is because during testing, I ran the same stack through Panorama Tools, Helicon Focus, CombineZ5, and AutoMontage, made a layered Photoshop file, and clicked the layers on/off to quickly toggle between them. I did this intending to look at pixel selection details, but the result was very surprising -- a strange "morphing" effect, like the subject had been made of modeling clay and had changed shape slightly from one stack to the next. But there was only one stack! Each of the software products had made slightly different alignment decisions as it worked through the stack, and the accumulation of those decisions had resulted in slight distortions of the geometry. With the images sitting side by side, I could not see the differences -- layering and clicking was the only way.
I was thinking about our earlier discussions in this thread and it occurs to me that the reason I do not have a parallax problem (with my tiles) may be because size of the region I am shooting is much smaller than the diameter of my objective.
Well, sort of, but not really.

With image alignment turned off (no scaling), you will never have parallax between tiles. That's because with image alignment turned off, the geometry of your reconstructions will always be orthographic, as shown in the first diagram. If those reconstructions are clean enough to keep you happy on a tile-by-tile basis, then you'll be happy with the composite because there's no parallax to mess up the stitching.

However (as you observed), you may not be happy with the reconstructions of individual tiles, particularly around edges. That's because you're trying to take images that actually have perspective geometry, and shoehorn them into orthographic geometry. Focused & slightly defocused versions of the same feature do not line up with each other, so pixel selection errors cause visible defects.

This problem is reduced with smaller DOF and smaller angles of view (field width versus distance from lens). Smaller subjects usually have both of these, so you're more likely to be happy with the reconstruction of each tile.

But that effect is due to DOF and angle of view, not (directly) to field width versus aperture diameter.

Perhaps you were thinking that your scope is acting telecentric because of subject size vs aperture diameter. Nope, it's not. But if it's looking at a small subject from a large distance, then it has a narrow angle of view, which is certainly on the path toward telecentricity. (A telecentric system has angle of view = 0 degrees. Yep, exactly zero, no matter how wide the field or how far between the lens and subject. Study this article if you want to follow up on that aspect.)

--Rik

Charles Krebs
Posts: 5865
Joined: Tue Aug 01, 2006 8:02 pm
Location: Issaquah, WA USA
Contact:

Post by Charles Krebs »

Great discussion!

While this moves away from telecentric optics, it might suggest another approach for dealing with subjects considerably larger than the largest field of view of a scope. Perhaps some one has already tried this.

I toyed around with doing this about a year ago. Since I am basically lazy and all this stacking and stitching is a lot of work, I thought it would be fun to take a decent flatbed scanner and remove the glass. Then mount a "stage" or "subject holder" that can be raised and lowered through the focal plane of the scanner via a scavenged microscope focus rack. (I know there are pricey flat beds that allow you to move the focal plane, but they are too much $$ for me).

Then create your z-stack, drop it into the software, and see what comes out!

svalley
Posts: 343
Joined: Sun Dec 03, 2006 7:07 pm
Location: Albany, Oregon

Post by svalley »

Interesting idea Charles. I have been pretty impressed with the scans of dragonflies that those guys in Texas (Digital Dragonflies) and Dennis Paulson in Washington have been getting.

Rik, I looked into the workings of ImagePro today and sure enough, it has all the scaling and postioning options that you can use on a stack. You can choose an anchor layer and all the other layers are optimized to that one. It also has automated stitching, that I tried a long time ago, but I can do it much faster and with less work in Photoshop. I tested the scaling routine today and compared it to the stacking routine that I have been using. they were exactly the same, I mean every single pixel! That was a surprise. The only thing I can conclude is that either: 1) my normal routine adjusts scaling or 2)scaling is not an important factor under the conditions I am using.
"You can't build a time machine without weird optics"
Steve Valley - Albany, Oregon

rjlittlefield
Site Admin
Posts: 23562
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Post by rjlittlefield »

svalley wrote:Interesting idea Charles. I have been pretty impressed with the scans of dragonflies that those guys in Texas (Digital Dragonflies) and Dennis Paulson in Washington have been getting.
Well, let's see... The sensor on a Canon Digital Rebel XTi has 3888 pixels in 22.4 mm. That's, um, 4408 pixels per inch. Good scanners now work at 4800 dpi = 4800 pixels per inch. So at least in theory, the scanner should deliver a little more resolution than the XTi with a 1:1 macro lens. And obviously the scanner has a lot wider working field. That's all on the positive side for the scanner.

On the negative side, the scanner is a lot slower per frame, and the mechanics are not necessarily as stable as you might hope. At work we once tried to scan the front and back sides of the same subject and line up those two images. The subject was riddled with fiduciary holes so we could tell how well we were doing. We were expecting to have a bit of trouble with skew in the scanner, but we had planned how to handle that. What we had not expected to find -- but did -- were small local deviations of the scanned grid. Apparently the scanner head was not moving perfectly evenly down both rails at the same time, but rather was wobbling back and forth just a bit. The deviations were not much, a few pixels at maximum, and they varied only slowly so that, again, the deviations were not visible in side-by-side comparison. But they were enough to have wreaked havoc with stacking. We tested several other scanners and discovered that they had similar problems, so we figured out how to work around it in software, using a technique similar to what I described above for aligning features, but with local warping instead of scaling.

I'll be interested to hear how stacking with a scanner works out, if anybody tries it.

Steve, the Digital Dragonflies site at http://www.dragonflies.org/hardware.htm claims that they're using only a 600 dpi scanner, but I wonder if that number is obsolete. Do you know if that's what they're currently using?
Rik, I looked into the workings of ImagePro today and sure enough, it has all the scaling and postioning options that you can use on a stack. You can choose an anchor layer and all the other layers are optimized to that one. It also has automated stitching, that I tried a long time ago, but I can do it much faster and with less work in Photoshop. I tested the scaling routine today and compared it to the stacking routine that I have been using. they were exactly the same, I mean every single pixel! That was a surprise. The only thing I can conclude is that either: 1) my normal routine adjusts scaling or 2)scaling is not an important factor under the conditions I am using.
I knew things were making too much sense! :lol:

Turn scaling off, and see what happens. If nothing changes, then you'll know that either scaling is not important or the control doesn't work. If something does change, then you might get some good insight by layering the results in Photoshop and clicking between them.

By the way, do you happen to have one or two of those images where
At magnifications less than 16x, the size of the specimen changes dramatically within the image frame of each layer in the stack, resulting in terrible fringing around every sharp edge in the composite.
I'd be interested to see how bad "terrible" is, and the general appearance might give up some clues about what's going on.

--Rik

rjlittlefield
Site Admin
Posts: 23562
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Post by rjlittlefield »

I thought it might be helpful to demonstrate the Perils of Parallax.

I shot the front end of a fly specimen using optics that could be switched between ordinary and telecentric simply by stopping down either one of two apertures. (The setup is shown at the end of this post.)

I took a total of only four shots: camera position shifted left or right, optics set to ordinary or telecentric.

All pictures were stopped down to nominal f/22. There's no stacking involved in this demo.

Here are the four pictures, arranged in side-by-side presentation.
Image

Looks OK, yes? Certainly no differences that will club you over the head.

But look what happens if we think about stitching these images together. (You should see this as an animated gif, flashing between left frame only, and right frame overlaid on it.)
Image

It should be apparent in this animation that the pair of pictures shot with ordinary optics have significant parallax -- the front end of the fly appears to rotate a couple of degrees between the two frames. In contrast, the pair of pictures shot with telecentric optics have essentially no parallax -- the front of the fly just appears in different places in each picture. (The rotation is even more obvious if you look at the blurred texture of the balsa in the background. Big shift with respect to the fly using ordinary optics, none with telecentric.)

Could the ordinary pair be stitched to produce an invisible seam?

Maybe, but the stitch wouldn't reflect reality. To avoid a visible seam, portions of the fly would have to be either warped to fit or masked to hide. (I've done plenty of both, working with large-world panoramas.)

To get clean natural stitches with such a deep subject, we really need to be working with tile images that don't have parallax. To get those, I can think of three approaches: 1) Rotate the camera+lens around its normal entrance pupil, someplace inside the lens. 2) Move the entrance pupil out to infinity, using telecentric optics, then just laterally shift the subject or camera+lens. 3) Compute parallax-free tiles from inherently parallax-ridden stacks, by turning off image scaling and relying on very precise pixel selection.

Option 1 is mechanically awkward. Option 2 requires unusual optics. Option 3 is convenient, but may not work well for reasons discussed in earlier posts in this thread. Unfortunately there's no free lunch, or if there is, I haven't found it yet.

The main point of this post is just to demonstrate that parallax is a real issue with stitching, even with small subjects viewed from fairly long distances. I'm not arguing for any particular approach here, just illustrating one issue.

--Rik

Image

svalley
Posts: 343
Joined: Sun Dec 03, 2006 7:07 pm
Location: Albany, Oregon

Post by svalley »

Rik, Nice demo of parallax. One of these days I am going to have to set my optical bench up and try some experiments. I have a nice 5-foot long, 200 lb. optical bench, but I will have to fabricate my own clamping and lens holding devices.

In your previous post:
Turn scaling off, and see what happens. If nothing changes, then you'll know that either scaling is not important or the control doesn't work. If something does change, then you might get some good insight by layering the results in Photoshop and clicking between them.
Scaling is a seperate routine in Imagepro and when I run it prior to my stacking composite routine there is no noticable improvement. By the way another way to see differences between 2 layers in PS is to change the mode of the top layer to Difference, then only the pixels that are different will show up, everything else will be black.
I'd be interested to see how bad "terrible" is, and the general appearance might give up some clues about what's going on.
Well, here are the "Bad and the Ugly". These are quick and dirty shots I dug out of my archive. I originally shot them for a presentation on imaging that my boss was giving at an Ent. Society of America meeting. The shot on the left is pretty crappy too, but the point was to illustrate the problem if you try to composite a stack with mag. too low and DOF too thick for each layer in the stack.

Image

Image
"You can't build a time machine without weird optics"
Steve Valley - Albany, Oregon

Post Reply Previous topicNext topic