Zerene Speed

A forum to ask questions, post setups, and generally discuss anything having to do with photomacrography and photomicroscopy.

Moderators: rjlittlefield, ChrisR, Chris S., Pau

ChrisR
Site Admin
Posts: 8662
Joined: Sat Mar 14, 2009 3:58 am
Location: Near London, UK

Zerene Speed

Post by ChrisR »

Sorry I know this has been done before but there may be quick answers.
I'm new to Windows 10.

For 21MP jpeg images I'm getting 1.85 secnds per pre-aligned frame for Pmax. Default alignments
Original files are here
https://drive.google.com/file/d/0B7WuJo ... sp=sharing , which were taken by a forum member, I'm sorry I forget who.

This is an i7 7700 at 4.25GHz

Image

Is there anything else I should be checking?
Chris R

rjlittlefield
Site Admin
Posts: 23543
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Post by rjlittlefield »

By itself the graph looks pretty good to me. As I read it, you're getting about 85% average CPU utilization while processing the stack. Max would be 100%, so you couldn't possibly make the thing run more than about 18% faster without removing some work.

That said, the time per frame is not wonderful. On my computer, i7-4930K, 6 cores at nominal 3.4 GHz with auto-overclocking to 4.125 GHz, doing those same images, with all preferences reset to default, 4000 MB memory allocated, Prosumer Edition license key, the first PMax took 47.2 seconds (2.33 seconds per frame) at about 75% utilization, and repeating the PMax (now already aligned) took 33.0 seconds (1.83 seconds per frame) at about 65% utilization.

So, on my system it seems that the "same" amount of work is getting done in fewer CPU cycles. The most likely explanation is that "same" doesn't really mean identical. The individual preference that I would look at first is Options > Preferences > Alignment > "Advanced interpolators", and be sure that's set to the default "Bicubic spline (4x4, default)". Just bumping the interpolator up to Lanczos3 raises my PMax repetition time to 43.4 seconds. BTW, I see that there's a bug in the preferences UI: based on execution times it looks like setting Lanczos and then just removing the checkmark does not cause using the default interpolator as it should.

If I were on your system, I would first text-edit the configuration file %APPDATA%\ZereneStacker\zerenstk.cfg so as to delete all settings except for ComputerConfiguration.PhysicalMemory and LaunchConfiguration.HeapSize.64bitJava, and re-run the test. Probably a good idea to save a copy of the old zerenstk.cfg, in case you need to go hunting for specific causes.

--Rik

ChrisR
Site Admin
Posts: 8662
Joined: Sat Mar 14, 2009 3:58 am
Location: Near London, UK

Post by ChrisR »

Thanks I checked averything was as you suggested - it was.
The numbers are the same for memory.
I re-ran a few times and checked the Console log between Done With frames #10 to #15
Each time I ran it for the first 5 tries, it got slower!
Frame rate went from 1.66 seconds each to 1.9.
Time to Load a frame went from about 0.85 to 1.2 seconds per frame, which appears to be the difference. Then it dropped back again on subsequent runs.

I tried moving the source files from the Data drive to the m.2 C drive where ZS lives, which slowed things very slightly.

Stack repeat rate was about 33 seconds, as yours. Your procesor is slightly up the benchmark listing I think, but not enough to account for the difference in utilisation. (Graph still bouncing between about 80 and 99%.) Something to do with the cores/processors difference?

I'm somewhat disappointed, it's "only" about 2x the speed of my 7 year old laptop.


The ZS files files weren't quite where I expected, I had to search ( which has bugs, thanks Microsoft). I've never fathomed out how MS thinks one is supposed to use their logical/physical folders.
I've added no users/accounts, but can't find things. I have four folders called "documents", without creating any. Later, later..
Chris R

rjlittlefield
Site Admin
Posts: 23543
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Post by rjlittlefield »

ChrisR wrote:Stack repeat rate was about 33 seconds, as yours. Your procesor is slightly up the benchmark listing I think, but not enough to account for the difference in utilisation. (Graph still bouncing between about 80 and 99%.) Something to do with the cores/processors difference?
I would expect higher utilization with fewer processors, but I'm hearing the opposite, higher with 8 cores than with 6.

Are you sure all that processor time is going to ZS, and not also to something like anti-malware scanning files?

--Rik

ChrisR
Site Admin
Posts: 8662
Joined: Sat Mar 14, 2009 3:58 am
Location: Near London, UK

Post by ChrisR »

I don't think so, unless MS was interrogating each image seeking marketing opportunities.

Photoshop wasn't open .
This is task manager, expanded.
I do have a strange orange blob:
Image

27%? There's 32GB RAM.
Chris R

rjlittlefield
Site Admin
Posts: 23543
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Post by rjlittlefield »

ChrisR wrote:27%? There's 32GB RAM.
Perhaps some combination of those 45 background processes?

--Rik

ChrisR
Site Admin
Posts: 8662
Joined: Sat Mar 14, 2009 3:58 am
Location: Near London, UK

Post by ChrisR »

There are about 89 Windows processes too, with no other programs running, but none are using more than 0.1% CPU or about 40MB memory.
I checked all 8 "processors" are showing the same utilization.

4GB Ram is as much as Java can use, yes?
Chris R

rjlittlefield
Site Admin
Posts: 23543
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Post by rjlittlefield »

ChrisR wrote:4GB Ram is as much as Java can use, yes?
No -- 64-bit Java can use unlimited; 32-bit Java can only use 2 GB on Mac and Linux, typically only about 1.6 GB on Windows.

With large physical memory, the current default for Zerene Stacker is nominal 4000 MB heap space, so a little more than that total. The 4270.9 that you're showing looks normal.

--Rik

ChrisR
Site Admin
Posts: 8662
Joined: Sat Mar 14, 2009 3:58 am
Location: Near London, UK

Post by ChrisR »

I increased the Java heap size, and Zerene memory allocated number together in steps 4GB, 8GB, 16GB, 24GB.
The time per frame has dropped from about 1.85 seconds to about 1.54 seconds.
It's not very consistent. It's not drifting longer. like before.
I haven't trried decreasing them again, separately, yet.
I can't find a short term memory usage utility. The native ones show a very slow increase in Used memory and Commit Charge over a few runs. (in %)
Chris R

Macro Photog
Posts: 92
Joined: Sat Dec 13, 2014 11:45 am

Post by Macro Photog »

This is a very interesting thread as workflow speed is one of my top interests/priorities. I'm replying because I noticed ChrisR's Task Manager capture showed the CPU running at almost 100% when mine is about 20%.

I'm not sure which is correct but I believe there are other processes, potentially running at a higher priority, that are taking up CPU time on ChrisR's system. One diagnostic resource I use is "Resource Manager". To run RM, in Task Manager, click on the "Performance" tab. At the bottom of the tab is a link to start Resource Manager. Resource Manager is a detailed view of CPU, Disk, and Network activities. It should show real time programs/processes that have a higher priority than Zerene. Very active, high priority programs could cause PMax to slow down.

If you do not know what a program does by its name there is a very nice feature that will look it up on the internet to provide additional information. To do this highlight a program and right click. A dialog box will appear and the last choice is "Look Online". Select this and it will open a browser window searching on that program name.

Another helpful tab is "Services" in Task Manager. This shows a relatively granular breakdown of all background processes running in your system. This a very powerful area because you can activate and disable these services in this section to stop unnecessary processes from sapping system resources. You may also find apps and programs that are using system resources even when the primary application is closed. To find out more about individual services you can go to the bottom of the Services tab and click on "Open Services", and click on/highlight a service and a brief (sometimes cryptic) definition of the service will appear in the far left hand side of the dialog box. Clicking the right mouse button will allow you to perform a variety of functions on the service such a stopping and starting it. DISCLAIMER: CHANGING THESE SETTINGS CAN NEGATIVELY IMPACT YOUR SYSTEM PERFORMANCE AND EVEN DISABLE THE SYSTEM. Be clear of what the impact is before you take an action.

My utilization is about 20% when running PMax. My machine is dedicated to photo processing and no other applications run during any photo process except Win 10 background processes. RM shows that Zerene is using +90% of the 20% CPU utilization. RM also shows activity of about 20% CPU utilization on all other cores. I'm not sure what this means as the graphs for all cores look identical. I have 22 background processes and all show 0% utilization except for NT Kernel Manager which shows less than 1% as it manages the system.

ChrisR, if you are running at 100%, either there is something "throttling" my system or you have a "rogue" process (legitimate or otherwise) that is utilizing a lot of CPU time. If so it may very well show up in RM. You can perform some simple but extremely helpful diagnostics by clicking on the checkbox next to various programs in the CPU section of RM. This filters RM activities for the application you have highlighted (clicked). You can reset priorities in the Task Manager under the "Details" tab and right click. It may be tempting to make Zerene run in "Real Time" but this will negatively impact performance. The best I was able to do was give it an "Above Normal" rating.

Alternately, if others think my CPU utilization is too low please let me know along with any suggestions of why. What is very puzzling to me is ChrisR indicates PMax is processing a 21MP aligned jeg image in under 2 seconds. I have 36MP non-aligned tif images my processing time is approximately 30 seconds. Is there that much difference between jpeg/tif - aligned/non-aligned images?

My system
Win 10 Pro
i7 4770K 3.50G
32 gig memory Z usually uses 15G of this
Normal Zerene File size 211,900M
Boot disk is SSD
Normal Zerene work disk is NVME SSD
My normal processing time is 30 seconds frame to frame as measured by a stop watch

rjlittlefield
Site Admin
Posts: 23543
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Post by rjlittlefield »

Macro Photog wrote:Alternately, if others think my CPU utilization is too low please let me know along with any suggestions of why.
I think it's too low, but I'd like to look at one of your LastNormalLog.txt files before thinking about suggestions. Can you do a typical run, exit from Zerene Stacker, then send that log to support@zerenesystems.com?

--Rik

Macro Photog
Posts: 92
Joined: Sat Dec 13, 2014 11:45 am

Post by Macro Photog »

Thanks Rik. I'm running a group now. It should be ready sometime tomorrow. Then I'll do a normal run and send it to you.

ChrisR
Site Admin
Posts: 8662
Joined: Sat Mar 14, 2009 3:58 am
Location: Near London, UK

Post by ChrisR »

What is very puzzling to me is ChrisR indicates PMax is processing a 21MP aligned jeg image in under 2 seconds. I have 36MP non-aligned tif images my processing time is approximately 30 seconds. Is there that much difference between jpeg/tif - aligned/non-aligned images?
I know about Task/Process Manager, mostly. That's what produces the graphs.
There's nothing else running which takes significant resources.
Running Jpeg vs Tiff makes no difference, it's the MP which matters.
Alighnment takes a fairly short time but does make a difference, something like 50% extra iirc.
So if my system would be taking 4-5 seconds and yours 30, and your utilisation is so low, I think it's you who has the problem!
My several-year-old laptop would be faster than that.
Last edited by ChrisR on Mon Oct 16, 2017 1:55 pm, edited 1 time in total.
Chris R

Macro Photog
Posts: 92
Joined: Sat Dec 13, 2014 11:45 am

Post by Macro Photog »

I'm glad Rik thinks my process can be improved. Hopefully he will see some adjustments I can make.

Macro Photog
Posts: 92
Joined: Sat Dec 13, 2014 11:45 am

Post by Macro Photog »

This is a follow up to Rik's offer to help me with low utilization and resulting slow processing of my stacks and as it turns out, slow everything else. Rik very kindly offered to do a screen share as other information I sent him and suggested testing did not yield any clues as to what was wrong with the system. For the screen share he sent me a program that gave a more granular look into the system (CPUID Hardware Monitor) to help us along.

Almost immediately he spotted the issue. My processor was running extremely hot (+100 C) and throttling the cores to prevent a thermal melt down. The culprit was the heat sink and fan for the CPU. Though it was not visually apparent, it had become slightly unseated and was providing little to no thermal protection. The HS/fan combo is attached to the motherboard with nylon "pylons" that are splayed open when plastic pins running through the pylon cores are pushed down. The weight of the HS/fan when the system is upright, coupled with the heat of the CPU may have caused the pylons to deform over time. In any event it was providing the CPU with little if any protection. Also, the thermal compound had become dry and hard.

After the call I re-seated the HS/fan and temps came down to acceptable levels. However when running PMax and all cores were running at near 100% there were extremely brief periods when some of them were being throttled to 99%. Let me be clear this was likely in milliseconds and would have worked but I wanted to be able to run PMax full out, no throttling. I purchased an all in one liquid cooling kit and installed it. I am extremely pleased with the result. Running at 100% all cores, no core runs above 54 C. The old processing time (PMax align and stack), image to image was approximately 31 seconds (file size 211MB files). The processing time with liquid cooling is approximately 6 seconds. Faster times may be achieved with some additional tuning in the future.

FYI...the processor is an Intel i7 4770K, Let me know if you want any other system specs.

I want to thank Rik for his time and expertise. Without him I would still be slogging along trying to resolve the issue with Resource Monitor or worse spending a lot of money for a new system to solve the issue.

Post Reply Previous topicNext topic