On the performance of Zerene Stacker

A forum to ask questions, post setups, and generally discuss anything having to do with photomacrography and photomicroscopy.

Moderators: rjlittlefield, ChrisR, Chris S., Pau

rjlittlefield
Site Admin
Posts: 23543
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Post by rjlittlefield »

I've just now posted out a new beta of Zerene Stacker that addresses some of the performance issues discussed above. This is Build T201403071805_beta (2014 March 7, 6:05 pm).

Quoting from the release documentation:
  • New option to overlap I/O and computation if there is sufficient memory (>100 megabytes per megapixel); see Options > Preferences > Multiprocessing > “Overlap I/O with computation if possible” (Pro-only).
  • New option to throttle the display of in-progress results while stacking; see Options > Preferences > Look & Feel > “Set output display interval when stacking” (Pro-only).
  • Reduced cpu overhead for memory management during stacking.
  • Improved parallelization of alignment computation.
  • Shortcut some computations when no alignment options are selected.
  • Avoid certain computations if screen image caching is not selected.
  • Much faster computation of depth map after threshold selection.
Here is a series of numbers from tests that I ran just now on my new 6-core machine. This is for PMax'ing 200 frames of 12 megapixels, all alignment options turned off, all other parameters at default except as noted.

1. Current full release T201402072140, 3:40 (min:sec)
2. Turned off caching, 3:30
3. New beta T201403071805_beta, 2:38
4. Turned on I/O overlap, 1:57
5. Throttled display to once per 15 seconds, 1:32 (min:sec)

As always, your mileage will vary. Many of the mods were motivated by this application and my new 6-core hyperthreaded machine, so they won't have as much effect in other situations. I'll be interested to hear what other people see with their different tests and configurations.

--Rik

johan
Posts: 1005
Joined: Tue Sep 06, 2011 7:39 am
Contact:

Post by johan »

Superb!
My extreme-macro.co.uk site, a learning site. Your comments and input there would be gratefully appreciated.

rovebeetle
Posts: 308
Joined: Thu May 22, 2008 4:21 am
Location: Vienna, Austria
Contact:

Post by rovebeetle »

Wow! I have only yesterday got my new Xeon workstation and ZS was the main reason for this upgrade. If these numbers hold true I will upgrade my home version of ZS to a pro license which i did not need so far because at home I only do the stacking and had no need for the pro functions. In my institute I mostly do the capturing and since the status of my lab PC is approximately medieval I do the stacking at home. But recently, time became an important factor. I have not tested stacking yet on the new machine but I expect that the new CPU (E3-1245-v3 vs C2Q 9450) and more RAM (32 GB vs 8GB) will already cut the time in half.

Do I have to upgrade to Pro to use the beta version to full extent, and will it replace my current version or will it run separately?

Anyway, thanks for the effort you put into improving ZS!

cheers
Harry

rjlittlefield
Site Admin
Posts: 23543
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Post by rjlittlefield »

rovebeetle wrote:Do I have to upgrade to Pro to use the beta version to full extent
All the new Pro-only features have their own 30-day trial periods, so there's no need to upgrade to check them out. With a non-Pro license they'll warn you on first access, asking for permission to start the trial.
will it replace my current version or will it run separately?
On Mac, every new download runs separately by default. To replace the old requires explicitly trashing the old and moving the new into its place.

On Windows, when you install the beta, the default is to replace your current version but you have the option of installing it to a new place. Just specify some folder other than the default C:\Program Files\ZereneStacker, and also some different Start Menu folder. Then you can go through the Start Menu to pick the version you want to execute.

To uninstall, go through the Start Menu and select the uninstall icon in the corresponding Start Menu folder.

If you have multiple versions installed at once, they will automatically share the license key and the configuration file. Old versions will simply ignore new options saved into the configuration file.
Anyway, thanks for the effort you put into improving ZS!
You're very welcome. I like writing code and seeing it get used. :)

--Rik

rjlittlefield
Site Admin
Posts: 23543
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Post by rjlittlefield »

This thread seems like a good place to add one more note about Zerene Stacker performance.

Quick summary is this:

1. Avoid compressed TIFF.

2. Avoid using the Properties > Details panel of Windows Explorer to change or remove any properties on TIFF files.

The reason for (1) is that ZS can read uncompressed files several times faster than compressed.

On my system, the total time to read a 36 megapixel 16-bit TIFF file and transform it for retouching or stacking is 2.1 seconds for uncompressed, but 7.5 seconds for LZW. CPU time for just the read is even more striking: 0.28 seconds for uncompressed, versus 5.9 for LZW.

The reason for (2) is that changing or removing properties using Windows Explorer will cause a TIFF image to be reformatted using LZW compression. Ironically, this will usually cause the file to become larger, not smaller, at least with 16-bit TIFF. But the important thing here is not the size, it's the fact that LZW is much slower to read, due to the cpu having to run the decompression algorithm.

Many thanks to Macro Photog for raising a red flag in the first place, and then spending several hours and iterations of email working with me to figure out why his files were agonizingly slow to select for retouching.

--Rik

mawyatt
Posts: 2497
Joined: Thu Aug 22, 2013 6:54 pm
Location: Clearwater, Florida

Post by mawyatt »

Rik,

Thanks,

Good information for Zerene users and probably other stacking software as well.

Cheers,

Mike

dance621
Posts: 1
Joined: Tue Aug 02, 2016 5:22 pm

Server farms and zerene

Post by dance621 »

I have a project at the moment where i need a lot of folders stacked (up to 2700 folders of 21 images per day). Each stack is taking about 2.3 mins. We are shooting with the canon 50mp 5ds so the files are quite heavy. i7 16gb etc. I would need a room full of beastly computers I have calculated to process these folders and I am wondering if remoter servers would be an option to process the large number of files. Has anyone else considered this option? Any suggestions where to start? Thanks

rjlittlefield
Site Admin
Posts: 23543
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Re: Server farms and zerene

Post by rjlittlefield »

dance621 wrote:I have a project at the moment where i need a lot of folders stacked (up to 2700 folders of 21 images per day).
...
I am wondering if remoter servers would be an option to process the large number of files. Has anyone else considered this option? Any suggestions where to start?
Yes, multiple servers have been done before, typically coordinated using the batch API described at http://zerenesystems.com/cms/stacker/docs/batchapi .

But that's just one out of many make-it-go-faster methods that might be applicable, all depending on the overall goals of your work.

For special ultra-large volumes like this, your process should include contacting support@zerenesystems.com . (That's me, but in a different role.)

--Rik

rjlittlefield
Site Admin
Posts: 23543
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Post by rjlittlefield »

Adding some recent issues to this thread...

I have seen a couple of cases where slow performance was caused by overheating, when the computer's processor slowed down its own clock to avoid frying itself. See the discussion at http://www.photomacrography.net/forum/v ... 930#220930 . As discussed there, the symptoms were especially mystifying because Windows Resource Manager gave the impression of just low utilization, as if processors were waiting for I/O or something.

Another cause of slow processing can be setting an unnecessarily high level of interpolation, at Options > Preferences > Alignment > "Advanced interpolators". In the case that comes to mind, the user had selected Lanczos4 (8x8) instead of the default Bicubic spline (4x4). The result of that was significantly slower image processing, for essentially no gain in image quality because the source images were not sharp enough to justify the higher level of interpolation.

--Rik

Post Reply Previous topicNext topic