Zerene configuration with fast second drive

A forum to ask questions, post setups, and generally discuss anything having to do with photomacrography and photomicroscopy.

Moderators: rjlittlefield, ChrisR, Chris S., Pau

PaulFurman
Posts: 595
Joined: Sat Oct 24, 2009 3:14 pm
Location: SF, CA, USA
Contact:

Zerene configuration with fast second drive

Post by PaulFurman »

I got a second small 30GB fast SSD (solid state hard drive; basically a big flash memory card) and I'm wondering what's the best way to take advantage of that for stacking, and even more important: for retouching.

Options > Preferences > New Projects > In specified directory

-seems like the way to go but "With source files if possible" is nice that I can just save right there in the folder that contains the source files (that's the only way I've worked, so not sure). Saved projects seem to not work if renamed or moved.

I'm thinking of starting out downloading the files to the SSD, then move them off to the main library once a month or so. I've used this workflow before and it helps force me to clean up the mess of recent work for archiving. The SSD is only 30 GB and I need space for PS scratch files, etc. Having the files on the fast drive might make it practical to process raw with Lightroom as well.

Another possibility is to install Zerene on the SSD, which seems to maybe not be as helpful for most programs other than initial load up time but I don't know, maybe it would help for the specific tasks of crunching stacks and or retouching. I'm thinking that scrolling through stacks; retrieving cached previews with lots of disk activity might speed up dramatically with either setting that puts the temp files on the SSD. I'm guessing the initial stack and final output stage would speed up a little if the source files are also on the SSD and installing the program on the SSD only matters for initial loading of the program.

I suppose all this isn't too hard for me to test but just checking so I don't reinvent the wheel.

AndrewC
Posts: 1436
Joined: Thu Feb 14, 2008 10:05 am
Location: Belgium
Contact:

Post by AndrewC »

What are you running as an OS ? Back when I was on Vista I used a Ramdrive for my system and cache files which speeded things up a lot when image processing and (I'm pretty sure) on ZS. When I moved on to Win7 the Ramdrive software didn't work anymore, but then Win7 uses all my RAM anyway so I can't be bothered spending the $$'s for an upgrade. I guess Rik can say where ZS sticks data but my advice is to just suck and see - run some big stacks in different configs and time them. Just make sure you have a few gig's of RAM available for temp files, and if you're OS doesn't support that much RAM look at a product like Ramdisk.
rgds, Andrew

"Is that an accurate dictionary ? Charlie Eppes

PaulFurman
Posts: 595
Joined: Sat Oct 24, 2009 3:14 pm
Location: SF, CA, USA
Contact:

Post by PaulFurman »

It's a nice new 'from scratch' assembled desktop with 64 bit Windows 7 quad core i7 2.8 GHz with 12 GB RAM. Much better than the old laptop but you can never be too fast. I like to be able to get other work done while the stacks run, like photoshopping the last stack and Lightroom batch processing, which are all quite demanding. I had to shut those down during stacks with the laptop, so I'm still kind of gun shy from that.

AndrewC
Posts: 1436
Joined: Thu Feb 14, 2008 10:05 am
Location: Belgium
Contact:

Post by AndrewC »

PaulFurman wrote:It's a nice new 'from scratch' assembled desktop with 64 bit Windows 7 quad core i7 2.8 GHz with 12 GB RAM. Much better than the old laptop but you can never be too fast. I like to be able to get other work done while the stacks run, like photoshopping the last stack and Lightroom batch processing, which are all quite demanding. I had to shut those down during stacks with the laptop, so I'm still kind of gun shy from that.
Install some of the toolbox gadgets that show you RAM, CPU, HDD usage etc.

If nothing else, they are fun to watch ! I quite often edit an image in CS4 while stacking another, photographing a third (tethered capture), listening to music, watching a forum and checking email using something pretty similar to yours.
rgds, Andrew

"Is that an accurate dictionary ? Charlie Eppes

DQE
Posts: 1653
Joined: Tue Jul 08, 2008 1:33 pm
Location: near Portland, Maine, USA

Post by DQE »

A way to speed things up even more would be to install multiple SSD drives in a RAID 0 configuration, so that each drive feeds a data stream containing part of the file being loaded.

There is considerable variation in the throughput and performance of the various brands of SSD drives. Up to several hundred megabytes per second sustained throughput can be obtained, as I read assorted test data. For the best performance one should also use a dedicated RAID controller card in one's PC instead of depending mostly on software to run the data stream(s).

Here's a link with a basic explanation of RAID from a Mac perspective. There are of course much more detailed explanations of the various RAID options on the internet.

http://macperformanceguide.com/Storage-RAID.html
-Phil

"Diffraction never sleeps"

lauriek
Posts: 2402
Joined: Sun Nov 25, 2007 6:57 am
Location: South East UK
Contact:

Post by lauriek »

I know this sounds like the opposite of what you want, but I normally configure Zerene to use 3 of my 4 processor cores which allows the system to respond pretty well to other user tasks such as browsing whilst stacking...

DQE
Posts: 1653
Joined: Tue Jul 08, 2008 1:33 pm
Location: near Portland, Maine, USA

Post by DQE »

lauriek wrote:I know this sounds like the opposite of what you want, but I normally configure Zerene to use 3 of my 4 processor cores which allows the system to respond pretty well to other user tasks such as browsing whilst stacking...
Another tactic to use the PC while some background photo crunching task is going on is to lower the priority of the intensive task but let it access all the cores. Then you can merrily share all the cores with the intensive task. I use TaskInfo by Iarsn to do this. I think MS now provides a free utility that will accomplish the same thing, as discussed at the link below. I am comfortable with TaskInfo and haven't used the MS explorer very much.

http://technet.microsoft.com/en-us/sysi ... 96653.aspx

I'm not sure which approach to multi-tasking with an intensive process works best in practice. I suspect either would work OK, but find that simply lowering the CPU priority of the intensive task works pretty transparently.

YMMV.
-Phil

"Diffraction never sleeps"

rjlittlefield
Site Admin
Posts: 23561
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Post by rjlittlefield »

I'm not sure how much I can really add here.

The question of "what works fastest" is best answered by experiment. In general ZS is sensitive to I/O bandwidth but not latency, because all its I/O is streaming. But much of the I/O uses quite a bit of CPU as well. You'll never make it slower by putting the data on a faster disk, but you might not make it as much faster as you expect either.

There are four sources of significant data transfer:
1. Output images. These are big, 12 megabytes per megapixel. They live inside the project directory. If you move the project, they move with it. During retouching, these get read whenever you select an output image as retouching source.
2. Source files. These have to stay wherever they were when the project was created.
3. Cached screen images. These are relatively small, but they get read heavily when you are doing depth navigation during retouching. They live inside the project directory and move with it.
4. Temporary converted tiff files. These get created only if you have selected Options > Preferences > Preprocessing > "Use External TIFF Reader". They live in the java.io.tmpdir location, wherever that happens to be on your system, and I don't know any easy way to change it. If your input is TIFF and you're using the external reader, than these files will get read when you select a source file as retouching source.
Saved projects seem to not work if renamed or moved.
More precisely, saved projects will not work if their source files are renamed or moved. There should be no problem moving the project itself --- that is, the directory containing the .zsj file and the directories of cached and generated images under it. But within the project, source files are referenced by absolute path. Moving source files causes an ugly and cryptic diagnostic that looks for all the world like a programming error, and in some sense it is because that situation really ought to be handled gracefully and it's not.

Moving the project (but not its source images!) to your fastest device seems like a reasonable approach to speed up retouching. I don't have have any unusually fast devices to test with, so I'll be interested to hear what happens.

About timesharing between applications... The last several releases of ZS have automatically dropped the priority of its computational threads to a couple of notches below normal. Before that it could get pretty hoggish. But now I leave mine set on automatic and it timeshares OK. I'm not sure how much more improvement would be made by dropping priority of the entire task. Try it and see I guess.

--Rik

PaulFurman
Posts: 595
Joined: Sat Oct 24, 2009 3:14 pm
Location: SF, CA, USA
Contact:

Post by PaulFurman »

Thanks everyone, lots of helpful input. I'm not particularly an expert at this stuff.

I just did a test; stacking 50 frames with PMax and putting the source files on the the SSD actually took 5 seconds longer, with project files saved in the source folder. Each took about 5-1/2 minutes. :-k
A few more tests (details below) squeezed 5 seconds better than default and boosting available RAM gained another 10 seconds... not a big deal really either way over 5+ minutes. And I tried the stack on the 4-year-old laptop with 3GB RAM, it took 12 minutes; more than twice as long.

I did download some performance gadgets and oddly, the SSD didn't seem to be showing any action during a stack, although it's so tiny, I can hardly see and I'm not familiar with it yet. It's quite possible I haven't configured the thing correctly. Something's funky with the C drive too because it rates only 5.9 out of 7.9 on the Windows Experience Index (the weak link on the machine), other components rate 7.4 (7.1 for the processor). It's a Western Digital 1TB 7200 RPM SATA 6.0Gb/s, possibly I didn't plug it into the right kind of SATA port or something... The SSD is a OCZ Vertex Series 2.5" 30GB SATA II MLC, which is pretty small and 'old' but I guess tried and true - not extravagant.

Settings:
jpegs, no tiff.
Image Saving; 8 bit RGB quality 12 jpeg
Image Caching; both aligned and unaligned cached
Memory Usage; 12286 available, 2048 allocated for ZS 64 bit
('Set Automatically' suggests 9214, so I'll change that later)
Multiprocessing; -use all cores (all eight of them)
New Projects; With source files if possible

Then I tried it with saving projects in a specified directory on the SSD, and the source files on the C: boot drive, with memory still at 2048. That's the same 5 sec slower performance than all on C:

Then I used SSD source files plus a specified project save on the SSD and got the best performance by 5 seconds.

Then I boosted the RAM to the recommended 9214 and gained another 10 seconds. It was interesting to see memory usage on the task manager performance tab continuously rise in the graph, accumulating as the stack proceeded where with the lower setting was forced to level off. Again, this is a stack of 50 frames of 12MP jpegs.

----------------
Next, I tested the retouching speed by scrolling through the stack (12 sec) and touching each frame (3 minutes). That improved by 12 seconds with the source and project files on the C drive :-k

Oh well, it's more than twice faster than the laptop. The SSD is probably not the best as the comparisons confirm but tons better than a conventional disk at certain hypothetical tasks. I tried a benchmark test, here's the hard drive test results:

PassMark performance test comparing C drive with E (SSD):
Image

...the SSD compared against other various computers (including a few with OCZ SSD):
(This Computer at the bottom of each section in green shows my results)
Image

Hmm, looking further, on another test after that, the C drive rates 644 and should be 802:
http://www.harddrivebenchmark.net/high_end_drives.html
-as best I can tell, the SSD is the next one down the list at 801 :-k



Intel Core i7-930 2.8GHz, 12GB DDR3 1600 RAM, 1 TB 7200 RPM SATA 6.0Gb/s

AndrewC
Posts: 1436
Joined: Thu Feb 14, 2008 10:05 am
Location: Belgium
Contact:

Post by AndrewC »

Seeing as how you are doing all the hard work, try installing a RAMdisk (see link below). It will make a big difference to app's that are I/O limited but will not permanently store files - hey, it works on RAM - so a valid test might be to create a temporary project file for ZS, do the hard work and then copy it back to permanent storage ?

http://memory.dataram.com/products-and- ... re/ramdisk

I'll give it a whirl myself sometime soon - it is free for up to 4G which is probably all you need or could spare anyway.
rgds, Andrew

"Is that an accurate dictionary ? Charlie Eppes

ChrisLilley
Posts: 674
Joined: Sat May 01, 2010 6:12 am
Location: Nice, France (I'm British)

Post by ChrisLilley »

rjlittlefield wrote:
Saved projects seem to not work if renamed or moved.
More precisely, saved projects will not work if their source files are renamed or moved. There should be no problem moving the project itself --- that is, the directory containing the .zsj file and the directories of cached and generated images under it. But within the project, source files are referenced by absolute path. Moving source files causes an ugly and cryptic diagnostic that looks for all the world like a programming error, and in some sense it is because that situation really ought to be handled gracefully and it's not.
However, it is very simple to search and replace in the xml project file to indicate the new location of the source images. (I did this recently when re-running some older stacks).

ChrisLilley
Posts: 674
Joined: Sat May 01, 2010 6:12 am
Location: Nice, France (I'm British)

Post by ChrisLilley »

PaulFurman wrote: jpegs, no tiff.
Image Saving; 8 bit RGB quality 12 jpeg
So, both the input and the output files are small. That plus your findings show that ZS is not being disk limited here. It seems to be CPU bound.

PaulFurman
Posts: 595
Joined: Sat Oct 24, 2009 3:14 pm
Location: SF, CA, USA
Contact:

Post by PaulFurman »

When I upped available RAM to 9GB, the windows task manager performance tab graph showed memory usage climbing as the stack proceeded till it (just) used it all up by the end of the stack of 50. So then I guess retouching has it all loaded in RAM, ready to go... or most of it. Probably different for a stack of 100 or 300.

rjlittlefield
Site Admin
Posts: 23561
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Post by rjlittlefield »

PaulFurman wrote:When I upped available RAM to 9GB, the windows task manager performance tab graph showed memory usage climbing as the stack proceeded till it (just) used it all up by the end of the stack of 50. So then I guess retouching has it all loaded in RAM, ready to go... or most of it.
Sadly, no. What you are seeing is just the Java memory manager not bothering to collect garbage because it didn't have to. Most of that 9 GB actually consisted of chunks of memory that had been "cut loose" by the program so that they were no longer reachable and were thus eligible for garbage collection. But the memory manager assumes that it's OK to use whatever memory it was allocated, so it does not spend the extra time that would be needed to move data structures around as would be needed to give the memory back to the operating system.

You might do just as well, or even better, by reducing the ZS memory allocation to about twice the minimum needed for your image size. This will free the rest of your physical memory to be used by the operating system for caching files.

Making better use of very large memory is on the list of future improvements for ZS. Having lots of extra memory is kind of a new phenomenon -- historically the problem has been getting enough!

--Rik

PaulFurman
Posts: 595
Joined: Sat Oct 24, 2009 3:14 pm
Location: SF, CA, USA
Contact:

Post by PaulFurman »

So Andrew's RAMdisk idea might help. I installed it, then when setting up the actual 'partition', I got scared googling some potential issues with not being able to boot if it goes awry (probably from an old version of win/dos)...

Rik, That makes sense because ZS does nearly as well with a stack of 300 as 50 for the most part. Really, I'm doing fine on the new machine... much faster now. Just seeing how to use the SSD thing. It's probably best for photoshop and lightroom cache/scratch... I'll keep playing with it.

Post Reply Previous topicNext topic