Automated FocusStacker

Have questions about the equipment used for macro- or micro- photography? Post those questions in this forum.

Moderators: rjlittlefield, ChrisR, Chris S., Pau

masquerade
Posts: 9
Joined: Tue Nov 08, 2016 12:51 am

Automated FocusStacker

Post by masquerade »

Hi Everyone,

I have been making automated macro rails for a while. I wanted to share my experiences and findings through my blog.

I split the serie into 4 parts, which each part has a different approach. My latest build is the closest one in my mind.

The most important aspects I tried to achieve are a super-compact and easy to use design. I guess FocusStacker v4 has it and ready to be in the field.

http://www.ardakutlu.com/focus-stacking ... es-part-i/

http://www.ardakutlu.com/focus-stacking ... s-part-ii/

http://www.ardakutlu.com/focus-stacking ... -part-iii/

http://www.ardakutlu.com/focus-stacking ... s-part-iv/
If it is not in frame, it does not exist
- Murnau

Saul
Posts: 1781
Joined: Mon Jan 31, 2011 11:59 am
Location: Naperville, IL USA
Contact:

Post by Saul »

Welcome aboard, Arda !

Your Del-Tron stage works great, I'm very happy with it , very compact & versatile !

Looking forward to see V4 ;)

Cheers,
Saul
Saul
μ-stuff

masquerade
Posts: 9
Joined: Tue Nov 08, 2016 12:51 am

Post by masquerade »

Hi Saul,

Thank you very much. v4 is out and working :) I am just making small fixes and tweaks. Once I gather all the materials, I will publish them at my website.

Image
If it is not in frame, it does not exist
- Murnau

Smokedaddy
Posts: 1954
Joined: Sat Oct 07, 2006 10:16 am
Location: Bigfork, Montana
Contact:

Post by Smokedaddy »

Arda in case you didn't notice, you have a private message. <g>

Chris S.
Site Admin
Posts: 4042
Joined: Sun Apr 05, 2009 9:55 pm
Location: Ohio, USA

Post by Chris S. »

Arda, welcome to the forum! :D And thanks for sharing with us your macro journey.

I would respectfully disagree with you, however, that having stacking software handle raw files is a useful feature.

Shooting raw has value if--and only if--the photographer takes the time to tweak conversion parameters in robust raw conversion software. Stacking software that uses default values to perform this conversion provides only the illusion of a benefit. This throws away and advantages of shooting raw. A photographer, in such a workflow, would be better shooting jpeg, and carefully tweaking them during in-camera acquisition.

For focus stacking, I sometimes shoot jpeg and other times shoot raw. In my workflow, jpeg is for those cases when careful control of camera settings can place all useful visual data in a jpeg file. For situations where important visual data exceeds the range of jpeg, I shoot raw--and process resultant raw files with care, in order to compress useful visual information the capacity of a tiff or jpeg output. Vitally, this judgment cannot currently be done without thoughtful human input.

So when stacking software is contrived to accept raw files and convert them based on default values, this is adds little or no value, no matter how good it looks on a features checkoff list.

Cheers,

--Chris

masquerade
Posts: 9
Joined: Tue Nov 08, 2016 12:51 am

Post by masquerade »

Hi Chris,

I see your point. Maybe I couldnt explain it well, sometimes I think I couldnt explain myself clear since English is not my native language, I apologize for that in advance.

Let me try to rephrase it.
What I meant by "the ability to handle the raw files" is that Helicon's feature of reading .dng files. Compared to tif conversion from original raw files (.raw, .nef etc) conversion to .dng is very fast. Because it keeps most of the data same, but only converts the shell which enables it to be identified by the softwares. And the size is smaller then tif files so writing speeds to disk is faster.

There is no data loss when you convert a .raw file to a 16bit .tiff file. Actually most raw formats are 14 bit, so you put it into a bigger container and there is still room for more data. The key point is your range is preserved. There is no clipping assuming you didnt excess the limits of the original raw data.

The advantage of using dng workflow of Helicon is merely a speed and storage issue and since dng files are smaller sizes with the same data range, you can go for higher resolutions with less RAM.

About the .jpg pipeline, I respect that approach if speed, bandwith, amount of RAM etc is an issue for you.
For me, the main priority is quality and sharpness. You will surely lose some data on the way even if you shoot with a perfect lighting which will require no color correction. Because jpg is not a lossless compression, meaning even with the highest quality you will be losing something. For most of the time it may not be noticable or important, but I am not willing to take that risk.

On the other hand, I totally agree with you that one should pay the utmost attention to the lighting conditions, framing and color balance while shooting to minimize the post process (or even to eliminate it completely) Believe me, working in the post production field for ten years, I am well aware it is much better to prevent the mistake than try to patch it up. But we have to face it, everyone makes mistakes. If you somehow require post process, may it be color correction, re-framing, spot cleaning or even reconstructions of large pieces every bit of data is a plus.
If it is not in frame, it does not exist
- Murnau

ChrisR
Site Admin
Posts: 8668
Joined: Sat Mar 14, 2009 3:58 am
Location: Near London, UK

Post by ChrisR »

but I am not willing to take that risk.
A number of us have looked, and not see an improvement with the raw>tiff route. There's nobody I can remember saying it's a better route to go via raw than from a maximum quality Jpeg - UNLESS - you want to edit the raw file.

(I have a nagging suspicion that Canon's in-camera Raw>Jpeg converter isn't quite as good as Nikon's, but it's from anecdotal observations on my part.)
Chris R

masquerade
Posts: 9
Joined: Tue Nov 08, 2016 12:51 am

Post by masquerade »

I am not sure if I am missing something or dont understand what you mean. Raw images holds the whole range (either it is 14bit or 16bit) But jpg images can only hold 8bit information + it adds lossy compression.

In my pipeline I open .nef (raw) images either in photoshop or in lightroom and send them to zerene using the zerene-lightroom connectivity plugin (which uses tiff conversion). This way the 16bit depth information has been retained. Then I save the stacked image again as 16bit tiff file. This way I have a 16 bit stacked file which contains the whole range as has been shot by the camera. If the images was converted to a jpg file at any step, the range would be cropped to 8 bit and you cannot get it back.

Are we talking about if the raw files has an advantage over jpg files or if the raw pipeline in Helicon Focus is a good thing or unnecessary thing?

Can you elaborate what you mean by editing the raw file? If removing spots, applying primary and secondary color corrections etc. is editing, I did not even have a single photo which is not edited.
If it is not in frame, it does not exist
- Murnau

ChrisR
Site Admin
Posts: 8668
Joined: Sat Mar 14, 2009 3:58 am
Location: Near London, UK

Post by ChrisR »

These posts of Rik's, of focus stacking workflow, and Raw file stacking, explain:
http://www.photomacrography.net/forum/v ... hp?t=27012

http://www.photomacrography.net/forum/v ... hp?t=25268

When I want to use Raw I want to use my choice of raw converter with whatever quality and bells and whistles it has. I have tried using raw source, and ex-camera tiff source, and ex camera fine Large Jpeg, with suitably adjusted lighting on subject, and found no significant difference in a stack output. YMMV.

Of course the output needs editing, but you don't edit every source frame. Spot removal etc can be done to the output just fine, I've found.

I won't swallow the common justification claim : "It's wrong to tolerate any loss" without a demo that the loss is detectable.
Workflow preference, and speed benefits, I accept. It's nice to have all the options.
Chris R

masquerade
Posts: 9
Joined: Tue Nov 08, 2016 12:51 am

Post by masquerade »

Of course the output needs editing, but you don't edit every source frame. Spot removal etc can be done to the output just fine, I've found.
That is exactly what I mean. Since I dont do that for every shot and since I dont want to do that on a 8bit image, I have no other option then either using the dng or tiff pipeline. There is no turning back once an image turned into jpg.

On the second post link, Rik says actually the same thing that I mentioned:
The downside of JPEG is mostly not from compression, but from the fact that sharpening, noise reduction, and color balance are baked into the images and cannot be adjusted much in postprocessing without introducing other problems. The fact that they have 8-bit color also limits flexibility to do curves adjustment in post.
What I try to say is going back to 8bit is not option for me. That is of course a choice, there is no rule about that. As a habit, as long as I dont have a pressing deadline or hardware issue, I never waste even the fraction of my color depth. Not at work, not at home.

I tried to point out in my blog that dng wrapping option in Helicon provides a much faster way of getting a 16bit unclamped stack output compared to tif pipeline in Zerene Stacker. I am not defending that everyone should follow that or another workflow. If you are not going to do any post processing, if you have a limited bandwith and/or hardware, if you dont mind the jpg compression and noise or if you dont have time for data transfers, of couse jpg pipeline stays as an option (which I will never practice :) )
If it is not in frame, it does not exist
- Murnau

ChrisR
Site Admin
Posts: 8668
Joined: Sat Mar 14, 2009 3:58 am
Location: Near London, UK

Post by ChrisR »

If you are not going to do any post processing, if you have a limited bandwith and/or hardware, if you dont mind the jpg compression and noise
Or, - if you try it and find those are theoretical rather than actual concerns.
16.8 million colors is usually enough. :)
Chris R

masquerade
Posts: 9
Joined: Tue Nov 08, 2016 12:51 am

Post by masquerade »

Apparently not enough for everyone. Otherwise everything would be jpg, there wont be anything like exr, dpx, tif.
I sometimes work with 32 bit rendered images for feature films. But you are right, no monitor can show that color depth even it was perceivable by the human eye (which is not). It is about having that room to work with, not seeing the whole range.

I used to think exactly like you do. In one night my whole idea changed, so I wont insist anymore :) Thanks for reading and listening.
If it is not in frame, it does not exist
- Murnau

rjlittlefield
Site Admin
Posts: 23564
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Post by rjlittlefield »

masquerade, I mostly agree with your position about pixel depth.

However, regarding the dng workflow in Helicon Focus, I think you have some misconceptions.

It appears, from your description at http://www.ardakutlu.com/zerene-stacker ... omparison/ , that you think Helicon is doing the stacking with raw sensor data. It's not.

When you feed Helicon Focus a .dng file, the first thing it does is to convert the image to 16-bit RGB TIFF, in a background process, using default conversion parameters. Then it stacks the 16-bit TIFFs, producing a 16-bit RGB output. When you save the output as DNG, it packs those RGB pixels back into a DNG wrapper, leading a casual observer to think "Ah, DNG in, DNG out, how efficient!"

Let me illustrate this with an experiment that I just now ran. What I did was to take three Canon raw files, type .CR2, and had Lightroom convert them to .dng . Then I fed the .dng files into Helicon Focus, stacked, and saved the output as .dng . While Helicon Focus was running, I also snapped a copy of its cache so that I could look behind the GUI.

Here's what I see when I look at the folder contents using Windows Explorer:

Image

Image

Quickly summarizing, in the input/output folder the original .CR2 files are about 18,017 KB, the .dng files produced by Lightroom are about 15,600 KB, and the .dng output produced by Helicon Focus is 53,125 KB, over 3 times larger than the .dng inputs. In the Helicon Focus cache folder, the TIFF files are even larger: 88,234 KB. That last number is completely consistent with uncompressed 16-bit RGB TIFF: 4752x3168 pixels at 6 bytes per pixel would be 88,209 KB, plus a bit for headers.

So... When your article talks about the speed advantages of raw workflow, it seems pretty badly misdirected. The speed advantages of Helicon Focus come from other causes.

As for why Zerene Stacker does not provide a raw workflow, the reasons for that are outlined in one of the FAQs:
A longer explanation is that no stacking software really works directly with raw files.

The structure of data in a typical raw image file, one value per photosite with color implied by a mosaic Bayer filter pattern, is fundamentally incompatible with the image alignment process that is required for stacking.

Some stacking software deals with this aspect by accepting raw files at the level of the user interface, then converting them to some RGB format, typically TIFF, in a background process that is easy to overlook and may be difficult to optimize.

Zerene Stacker goes the other route, exposing the conversion process and encouraging users to deal with it as what we think it really is: a key part of the overall workflow that needs to have some attention paid to it in order to get best results.
Adding a bit more information to that last point in the FAQ... As the fellow who writes Zerene Stacker, I'm acutely aware that stacking methods can benefit from appropriate amounts of sharpening and noise reduction applied before stacking. And as a frequent stacker at many different magnifications, I'm also frequently reminded that "appropriate amounts" vary quite a bit from one stack to another depending on setup. As a result, I'm not at all inclined to provide a workflow in which raw conversion is done invisibly in a background process, using parameters that are at best "out of sight, out of mind". It's a policy decision, to not spend my development time encouraging users to take a shortcut that I think is a bad idea.

To the extent that Zerene Stacker competes with Helicon Focus at all, I think it does so on the basis of output quality versus human time invested. If the nature of someone's stacking is such that computer time is the deciding factor, then they should definitely opt for Helicon Focus, and I'll be happy to make that recommendation.

That said, it's also true that there are ways to make Zerene Stacker run quite a bit faster than the trial configuration that I'm guessing you used for your tests. But that's a topic for another day, and probably a different thread. If you're curious, you might look at http://www.photomacrography.net/forum/v ... 699#141699 and the discussion continuing onto the following page.

--Rik

ChrisR
Site Admin
Posts: 8668
Joined: Sat Mar 14, 2009 3:58 am
Location: Near London, UK

Post by ChrisR »

I used to think exactly like you do. In one night my whole idea changed,
Point missed, I think. Perhaps not fully expressed.

I'm not arguing against the generality of benefit of having more data to work with (though it's frequently asserted where untested and not real), but the stacking case is somewhat atypical.

As I said, if I want Raw, I'll choose the converter and conversion, not as Rik says leave it to someone else's decisions.

When stacking, then most of the time, you're working with a standard setup, and standard lighting. If you turn off all the "adjustments" that the camera thinks are a good idea, control the lighting, then what you get is well within the Jpeg "space".

This is taking into account that process time was an issue when I started stacking, but it's not a bother now and memory has become almost free.
Chris R

masquerade
Posts: 9
Joined: Tue Nov 08, 2016 12:51 am

Post by masquerade »

I am actually not defending that Helicon uses the raw pixel data. None of them use it. Raw data holds the color information with a gamma curve applied. When it is converted to tiff it becomes linear and if there is no compression, it is normal that the tiff file becomes much larger. And actually 16bit Tiff file can hold more data then the Raw file.

About the conversion, you probably mean the Bayer interpolation and White Balance, Saturation, Contrast etc. value conversions right? These are converted automatically when you shoot jpg. But if you shoot raw and convert it to tiff (as long as you dont push the white and black levels beyond edges) since you will have 4096 intensity levels instead of 256 per pixel, you can get the exact same image as you would shoot jpg. And also a room that you can play with.

What I tested was actually more then the abilities of the softwares the time spend during the data transfers mostly. As you can see and download the test results, both of my test results have the same data range. So there is nothing different between them in terms of color depth.

You can try it yourself. When you send the files using Zerene Stackers plugin with a tif export, it takes 3-4 times longer then the .dng pipeline suggested in Helicon's website.

And Helicon does the stacking faster too when you compare these two methods. It may be related with file format (that was my initial guess) but It may be not, I dont know. The only thing I know is if you want to have a 16bit result, dng pipeline is much more faster.
If it is not in frame, it does not exist
- Murnau

Post Reply Previous topicNext topic