Speeding up Zerene Retouching

[Macro Photog]

First, if this post is in the wrong forum please move to the correct one. I thought of this as a technique but may have classified it incorrectly.

As my stacks are often large I am always looking for ways to reduce the time it takes to get to a final image. One of the things that has always bothered me is the time between preview shots in "movie mode" in Zerene Stacker Retouching. When I press the up/down key from preview to preview is usually takes several seconds for the cursor to react and load the image. It is distracting and breaks my concentration during lengthy retouch sessions.

In my opinion, I/O speed seemed to be the issue. This weekend I had some time and tried a few things to speed this up. One was loading a RAM disk (RD) and installing Java and Zerene on it. I set the projects to load to the RD so the preview images would reside on the RD. All originals reside on the SSD because I did not have enough room to put them on the RD.

The results are amazing. I now have near instantaneous response time image to image in movie mode even when I press the up/down keys repetitively. In general, Zerene is "zippier" as well, although my alignment/stacking times are not substantively improved. I believe this is due to a. the files are large and b. they reside on the SSD that does not have the throughput of the RD.

I strongly believe that in addition to having the thumbnails on the RD AND installing Java/Zerene on the RD resulted in decreased response times.

The RD program I use will load an image of everything on the RD to another drive when the machine is shutdown so you do not lose work if you saved it in Zerene. I do not see why this would not work with other stacking software although I have not tried it.

Note: There are few if any bottle necks in my system as the operating system and data drives are all SSD and I have 32 G of system RAM.


CAVEATS

IF YOUR MACHINE "GOES DOWN" FOR VIRTUALLY ANY REASON ALL OF YOUR WORK IS LOST. A UPS backup is highly recommended for environments that are subject to power outages/spikes. In the case of outages it will give you time to shut the machine down in an orderly manner so the project and retouch work is saved to a hard drive and not lost

So far (a couple of days) this setup has been very stable but has not been tested during a heavy duty retouch session. I will post later if there are any issues or substantive reductions in response time. If this has been covered before my apologies, a cursory search did not pull up any similar information. I also loaded my caching and undo folder to the RD and expect better results.

[rjlittlefield]

For people who haven't done so recently, I recommend to review the Zerene Stacker tutorial "Tips for Retouching".

It specifically addresses the issue of rapid navigation, which is often the biggest problem.

In particular,

Use the fast methods for selecting source frames.

Every time you finish selecting a source file, it takes a few seconds of computation to prepare the image for use in retouching. So, the trick for fast selection is to not “finish” until you see that you're on the correct file. There are a couple of ways to do that:

1. Use press-and-drag in the list of Input Files, rather than press-and-release on each file. While the mouse button is held down, most of the computation is skipped so you can quickly zero in on the correct file before releasing the mouse button.
2. Hover over either image window, then press and hold the Shift key while using the mousewheel, up/down arrow keys, or press-and-drag with the mouse. As long as Shift is held down, most of the computation is skipped so you can quickly zero in on the correct file before releasing the Shift key. If you have a deep stack, probably press-and-drag with the mouse will be faster because it allows skipping across input files, while using the mousewheel or up/down arrow keys will require looking at every image.

Both of these methods rely on image caching, so be sure not to remove the checkmark on “Cache aligned screen images” at Options > Preferences > Image Caching.

If possible, use the advanced brush types, such as “Pixels”.

Compared to the default “Details” brush, the advanced brushes do a lot less computation to prepare each source image when it is finally selected. Tradeoffs are that (1) the default Details brush is better at doing seamless integration, and (2) the advanced brushes are “Pro-only” features that require Prosumer or Professional licenses for long term use. You'll have to experiment to figure out which is better overall for your situation.

In evaluating the effect of fast storage such as RAM disk and SSD, versus conventional rotating disk, it's important to keep in mind a couple of confounding factors:

1. Most operating systems will cache recently used files in otherwise unused memory, regardless of where the files are permanently stored. HERE, bullet #6, I document a case where simply copying the image files from one folder to another and then running ZS on the copy basically wiped out what otherwise appeared to be large advantage to the SSD. In those timing tests, the initial copy, either from hard disk to SSD or hard disk to hard disk, essentially served as a "pre-load" that moved a block of work from inside the timed interval to outside it. The result was that a quick test falsely indicated improvements that would not have been sustained over a longer session. I have no idea whether this is relevant to Macro Photog's situation, but it's something to keep in mind.

2. When using the fast navigation techniques with press-and-drag or holding down Shift, most of the time is not going into I/O. The "screen preview" cached files are relatively small highly compressed JPEG files, typically less than 1 megabyte per 10 megapixels. The visible time for accessing these images is almost entirely due to cpu manipulations in getting them ready for display.

3. Placing the Zerene Stacker executable on fast medium will affect mainly startup and initialization time. Once ZS is running and doing routine tasks, it will stay memory resident anyway.

--Rik

[Macro Photog]

First I want to thank Rik for his thoroughness, patience, and input to my problem. However, a bit of (negative) serendipity changed my thinking on this issue. To recap I was having rather extreme time delays (approx +1 sec.) switching between images in movie mode. After sending Rik some files and system information he confirmed the extended lag time. During this time another machine I have crashed (the above mentioned serendipity) and I had to redirect my time to recovering it. As part of the repair I took a hard drive from my graphics (Zerene) system and used it in my crashed PC. Fast forward and the crashed system was recovered and I could now turn my attention back to the retouching issue on my graphics system. To my surprise Zerene and movie mode in particular was much faster. The drive I removed was not the system boot drive but did have a copy of Windows on it, however, on a day to day basis it was not in the graphics editing workflow. I used it from time time in the graphics system as an extra storage drive. There is no reason the system should have accessed that drive when booting or performing any other system activity. Other than the drive removal nothing changed on this PC. I am in a quandary as to the actual cause of the issue, however, in a previous life I performed hardware/software maintenance on a large network. Rarely, very rarely, we would find hardware negatively impacting performance in unexpected and not readily discernible ways. This seems to be the case here.

All of that being said, I still use the RAM drive with a few modifications.
1. I run Zerene from a hard drive because, as Rik pointed out, Zerene loads in memory and does not refer back to the hard drive once loaded. Rik please correct me if I this is a mis-statement.
2. I do not have my image files loaded in the RAM disk since most of my stacks are too large for the RD, although I would if I could.
3. From time to time I do store the image thumbnails on the RD and believe they load slightly faster during retouch from there, however, it is more work integrating all the project files later.
4. I sometimes store retouch files on the RD as I feel they load faster from there.
5. I always store my retouch undo files on the RD as it feels snappier to me when I have to undo and edit.
6. Please note in all of the above I say operations FEEL faster, snappier etc. While there is a factual difference in bus speeds between RAM and and SSD I have not measured the differences and admit the ones I cite as noticeable are small and may fall into the category of "my car is faster after I wash it".

One more thing. Due to my incorrectly interpreting some readings in Windows resource manager I believed Zerene was loading a full version of Java. Rik, as part of our trouble shooting effort let me know Zerene loads a run time version. For those experimenting, there is no need to load Java separately.

In any event I am very happy with the speed of Zerene now.

[rjlittlefield]

1. I run Zerene from a hard drive because, as Rik pointed out, Zerene loads in memory and does not refer back to the hard drive once loaded. Rik please correct me if I this is a mis-statement.

What you say is correct. It would be a rare event, if ever, that code important to performance would be released from memory and reloaded from disk.

... Zerene loads a run time version. For those experimenting, there is no need to load Java separately.

Correct. To further clarify... The distribution of Zerene Stacker includes an embedded copy of some recent JRE (Java Runtime Environment), provided by Oracle and bundled with ZS. With a standard installation, ZS will use only that embedded copy and will ignore any other JRE that might be installed. It is possible to reconfigure the installation to use a separately installed JRE, but I generally recommend against doing that except in extreme cases such as to work around a newly introduced inconsistency between Oracle's JRE and the user's computer.

In any event I am very happy with the speed of Zerene now.

I am delighted, of course.

--Rik