Focus stacking algorithms: Zerene vs Photoshop

A forum to ask questions, post setups, and generally discuss anything having to do with photomacrography and photomicroscopy.

Moderators: rjlittlefield, ChrisR, Chris S., Pau

mrred
Posts: 4
Joined: Sun Jan 31, 2021 9:40 am

Focus stacking algorithms: Zerene vs Photoshop

Post by mrred »

Hi, i'm a student interested in stacking procedure.

Even if newbie in the field, after a little search seems already clear that the big actors in this field are Zerene/Helicon.

I'm here for the expertise that various member clearly shows, as also for Rik (his knowlegde is quite vaste of course).

Thanks to documentation of Zerene and also Helicon, i've at least a little idea of which kind of algo's and logic they use (sure, i know details are not public due to commercial reasons)
Instead, I can't find anything about Photoshop logic/procedure, or at least i'cant find anything with some "good level of trustness".


In some other forums, someone claims that Photoshop procedure is something that "resembles" the formally named "Method A" of Helicon.

a) Is there someone can confirm this info ? Or is a totally different logic? (maybe also someonce can infer/induct conclusions from his experience?)

b) Is there someone can address me about the photoshop method (in general i mean) and the differences between Zerene and Helicon ?

c) Are there some docs/papers which i'm missed (i mean also for Zerene /Helicon procedure which i can read to go more depth) ?

Thanks a lot, and sorry if there are already answers to these subject in some place/thread, but i can't find specifically.
Last edited by mrred on Tue Feb 02, 2021 7:10 am, edited 3 times in total.

elf
Posts: 1416
Joined: Sun Nov 18, 2007 12:10 pm

Re: Focus stacking algorithms: Zerene vs Photoshop

Post by elf »

No need to know the algorithms. Just stack the same image with Zerene and Photoshop. I'll guarantee the Zerene image will have fewer artifacts :D

rjlittlefield
Site Admin
Posts: 23564
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Re: Focus stacking algorithms: Zerene vs Photoshop

Post by rjlittlefield »

mrred, welcome aboard!

I can provide partial answers to your questions.

A good introduction to the "pyramid" methods that are used by Zerene Stacker PMax and (I presume) by Helicon Focus Method C is provided by https://www.researchgate.net/publicatio ... Processing , "Pyramid Methods in Image Processing", Adelson et.al., 1983. In that paper, look in particular at formula 7 and figure 9 (in which it looks like panels a and c got accidentally switched). This is not exactly what PMax does, and I doubt that it is exactly what HFMC does, but it should be close enough to get you started.

The most important concept of the pyramid methods is that you decompose each image into a bunch of different scales of detail (levels of the pyramid), you do some form of focus stacking on each layer independently, then you construct the output image as the recomposition of all the focus-stacked levels of the pyramid. The method described by Adelson et.al. exploits the fact that the Laplacian pyramid representation already directly encodes local contrast of details at each scale level, to create a particularly simple method for the focus stacking bit.

Zerene Stacker DMap and Helicon Focus Method B are both "depth map" methods. I do not have a handy reference for depth map methods, but I expect articles are pretty simple to find. The key idea in depth mapping is to assume that the stack frames are in depth order, figure out which frame has the best focus in each small neighborhood, then construct a single-valued depth map function that fits the best-focus estimates, and finally construct the output image by selecting pixel values from just a single frame, or a weighted average of two adjacent frames, as specified by the depth map. The initial estimates of best focus can be done using one of the kernel-based methods described in https://en.wikipedia.org/wiki/Edge_detection .

Where the physical structure of the subject matches the depth map concept, these methods work great. The difficulty is that many subjects do not match the depth map concept, for example having parts that overlap. This introduces an unavoidable "halo" problem, as illustrated at https://www.photomacrography.net/forum/ ... 42#p135042 and discussed in the surrounding thread. Another common problem is that many scenes contain areas where there is little or no focused detail, so some method is required to keep the depth map in those areas from being determined by random noise rather than meaningful image content. In Zerene Stacker, that's the function of the contrast threshold slider, to take advantage of human understanding to tell the software where depth from image content did the right thing. Other packages use other methods, typically examining the distribution of values in small neighborhoods of the depth map, and marking as not-to-be-trusted any places where too many different depth values are found. All of these processes result in a depth map that has holes, and then the holes are filled in by some interpolation method. Often the depth map is smoothed as a final step.

A third class are the "contrast weighted" methods. These form each pixel value in the output image as a weighted sum of the corresponding pixel values from some or all of the source images, with the weights determined by some method similar to what's used to find the single best frame in the depth map methods. The contrast weighted methods do not require any particular order of source frames, so these methods can work very well with transparent subjects. HF's Method A is described by them as being a contrast weighted method. Adobe has never described Photoshop's method, as far as I know, but surely we have to consider it as being a contrast-weighted method because one of its main outputs is a set of masks, one for each source image, and then the output image seems to be composed by Photoshop's ordinary masking process. There's some illustration of how Photoshop works, at slide 52 and preceding, in http://zerenesystems.com/presentations/ ... rTCDPC.pdf .

I hope this helps. Sorry I can't provide better pointers into the literature.

--Rik

mrred
Posts: 4
Joined: Sun Jan 31, 2021 9:40 am

Re: Focus stacking algorithms: Zerene vs Photoshop

Post by mrred »

Rik, thanks a lot for your effort, I must say that I did not expect an answer so fast and complete.
Your passion and availability is noticeable.

Your reply help me to put some fixed point to start;
I hope in next days to sort (better) all pieces of concepts you've pointed out, with the help of literature of course.

For sure new doubts will arise... but for now i've already some direction and things to read.

Saul
Posts: 1781
Joined: Mon Jan 31, 2011 11:59 am
Location: Naperville, IL USA
Contact:

Re: Focus stacking algorithms: Zerene vs Photoshop

Post by Saul »

elf wrote:
Sun Jan 31, 2021 3:08 pm
No need to know the algorithms. Just stack the same image with Zerene and Photoshop. I'll guarantee the Zerene image will have fewer artifacts :D
Maybe Rik will explain this - I have better luck with PS stacking transparent crystals.
Saul
μ-stuff

rjlittlefield
Site Admin
Posts: 23564
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Re: Focus stacking algorithms: Zerene vs Photoshop

Post by rjlittlefield »

Saul wrote:
Fri Feb 05, 2021 12:23 pm
elf wrote:
Sun Jan 31, 2021 3:08 pm
No need to know the algorithms. Just stack the same image with Zerene and Photoshop. I'll guarantee the Zerene image will have fewer artifacts :D
Maybe Rik will explain this - I have better luck with PS stacking transparent crystals.
Simple -- I disagree with elf. What I would guarantee is that PS can do a better job in some cases.

The "sweet spot" for PS is for transparent subjects where you want to retain the original colors, contrasts, and noise levels, and it's OK to maybe leave some parts blurred even though they were sharp in some source image.

Helicon Focus Method A is also a good match to those subjects, probably better than Photoshop.

--Rik

mrred
Posts: 4
Joined: Sun Jan 31, 2021 9:40 am

Re: Focus stacking algorithms: Zerene vs Photoshop

Post by mrred »

With the help of the Adelson's paper, now the pyramid method begin to appear a lot more clear.

Instead, i'm not so sure to catch well the foundamental difference between "Depth Method" (i.e. DMap/Helicon B) and the class of "contrast weighted" methods (PS/Helicon A).

Maybe Rik can address better the main difference.


DMap method :
- assume the stack frames are ordered;
- for each frame in the stack :
-> use a Focus Measure Operator ( example : Sum Modificed Laplacian ) for every pixel in image (or for every small window of pixel) ;

At the end of cycle, for each image we have :
(a) a FM value for each pixel (or for every small window of image), that can be also the "weighted result" also from "2 adjacent stack frames" AND
(b) the information about "the position in the stack order" for that specific source of "best focused pixel" found by Focue Measure Operator

At this point the algo can use (a) information but also (b) information to recostruct a "well adjusted" single value DepthMap;
finally the result image is created from this new Single-Value Depth Map.




"Contrast weighted" methods :
-no assumptions about stack frames order;
- for each image in stack :
-> use a Focus Measure Operator ( SML ) for every pixel in image (or for every small window of pixel) ;
(here the FM is computed only with data of the current image; no "eventual help" of the data of consecutive stack pic)

At the end, the algo chooses every single pixel (or group) to assign to the result image - with a rule like :
for each (i,j) pixel :
resultImage(i,j) := arg MAX ( Focus Measure (i,j) in all stack frames ) // simplified base idea



So :
both methods use Focus Measure Operator to "weight" the pixel in each image of stack, but:
- Dmap also use the info about "stack order", and use specifically this info to "weight pixel" , restrict (in case of necessity) the "field" between 2 consecutive frames (this should increment the "accuracy" i think);
- Contrast weighted class have no a lot of info to use, so the final pixel is a "simple weighted media" from all stack frames, without "a legitimate restricted field" of "admissible values"


Rik, could you give a little suggest to spot better the differences ?

rjlittlefield
Site Admin
Posts: 23564
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Re: Focus stacking algorithms: Zerene vs Photoshop

Post by rjlittlefield »

mrred wrote:
Tue Feb 09, 2021 5:41 am
Rik, could you give a little suggest to spot better the differences ?
I don't have much to add, beyond what you've written. I am glad to see "simplified base idea" in the description of contrast weighting. There are a lot of possibilities at that stage, notably using weighting functions that are not just 1 for the highest contrast image and 0 for all others. For example, where similar details overlap at different depths in a transparent subject, you'd probably prefer equal weights for each of the contributing depths.

--Rik

mrred
Posts: 4
Joined: Sun Jan 31, 2021 9:40 am

Re: Focus stacking algorithms: Zerene vs Photoshop

Post by mrred »

rjlittlefield wrote:
Mon Feb 15, 2021 9:36 pm
mrred wrote:
Tue Feb 09, 2021 5:41 am
Rik, could you give a little suggest to spot better the differences ?
I don't have much to add, beyond what you've written. I am glad to see "simplified base idea" in the description of contrast weighting. There are a lot of possibilities at that stage, notably using weighting functions that are not just 1 for the highest contrast image and 0 for all others. For example, where similar details overlap at different depths in a transparent subject, you'd probably prefer equal weights for each of the contributing depths.

--Rik
Yes, i'm conscious in my statement ("simplified base idea") there is a whole world of fusion rules. :)

Rik, for now i've read only a bit about the alignment phase (image registration in literature if i understand well...).
Seems that there are parametric and non parametric methods able to "do align".

Maybe you could give some clue about Zerene implementation of alignment ? (as always, i mean the general approach ).
And, for your experience, you can formulate hypothesis about Photoshop or Helicon alignment procedure ?


I hope in next days to read some literature, i've some doubts about
the parameters calculated, measure/error in alignment and the compatibility of this part of stacking process with the "photogrammetry theory".

I mean: i've found some papers where focus stacking is used in close range photogrammetry of small artifact, giving good metrical results (..seems at least).

I'm trying to do a university work where [close range photogrammetry + focus stacking] is used to reconstruct a 3d model of small object;
and the main point of view is a metrical point of view, not only a qualitative texture/model render
(indagate the possibility to retrieve very high precision measure).

Any suggest is welcome :)

rjlittlefield
Site Admin
Posts: 23564
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Re: Focus stacking algorithms: Zerene vs Photoshop

Post by rjlittlefield »

mrred wrote:
Thu Feb 18, 2021 11:56 am
Maybe you could give some clue about Zerene implementation of alignment ? (as always, i mean the general approach ).
The method used by Zerene Stacker is simple in concept. The alignment to be optimized is a restricted affine transformation that incorporates only X-shift, Y-shift, Scale, and Rotation. The goodness-of-fit metric is just the RMS difference between luminance values in image A and transformed image B, summed across all pixels in image A. The optimization problem is formulated as "minimize the difference by changing the shift/scale/rotation values", so, minimize a single-valued function of four variables. The numerical method used to do the optimization is classic Levenberg–Marquardt, using one of the public domain Java implementations. To make the computation run faster, and to avoid premature falling into local minima, the optimization is performed in a sort of pyramid framework that starts with low-resolution versions of the images and works its way to higher resolutions, at each stage using the result of the previous stage as a starting point.
And, for your experience, you can formulate hypothesis about Photoshop or Helicon alignment procedure ?
I suspect that Photoshop is using a feature-points method. The main reason for that thought is that in Photoshop, focus stacking and panorama stitching present the same user interface and progress indicators for alignment. For panorama stitching, matching feature points is the natural way to go, and once you have that in place, it works OK for focus stacking also. If you do focus stacking using the hugin/enfuse pipeline, the alignment function will be done by feature points.

I have no strong ideas about how Helicon does alignment.
i've some doubts about the parameters calculated, measure/error in alignment and the compatibility of this part of stacking process with the "photogrammetry theory".
For Zerene Stacker, see viewtopic.php?t=21508 , viewtopic.php?t=11519, and viewtopic.php?t=39677 .
I'm trying to do a university work where [close range photogrammetry + focus stacking] is used to reconstruct a 3d model of small object;
and the main point of view is a metrical point of view, not only a qualitative texture/model render
(indagate the possibility to retrieve very high precision measure).
To achieve highest accuracy in this process, you will want to make sure that the lens entrance pupil does not move when focus is changed. Two simple ways to do this are (1) step focus by changing the lens-to-camera distance, keeping the lens fixed in one place, or (2) use telecentric optics. Telecentric optics have the downside that the front lens element has to be wider than the object to be viewed, so there are pretty strict limits on the magnification ranges where they can be used. For very small subjects, typically viewed through a microscope objective, there is also the trick of simply turning off Scale correction in the software. This imposes orthographic projection, just the same that you would get from telecentric optics, with perhaps some reduction in image quality (sharpness) away from center if the optics are too far from actually telecentric.

In one paper that I have read sometime in the last couple of years, the authors incorporated prior knowledge of the optics into their alignment process, so that the alignment process could know without image comparisons what scale should go with each focus distance. Their idea was that this would allow them to construct a stacked image that has perfect pinhole geometry, even though the lens entrance pupil might have to move while they were shooting the stack. I do not have a handy link to the paper. Probably it will turn up in your literature search.

--Rik

Post Reply Previous topicNext topic