How to automate stacking process for time lapses?

A forum to ask questions, post setups, and generally discuss anything having to do with photomacrography and photomicroscopy.

Moderators: rjlittlefield, ChrisR, Chris S., Pau

Starshade
Posts: 133
Joined: Wed May 04, 2011 1:47 am

How to automate stacking process for time lapses?

Post by Starshade »

Hi,

I'm thinking of making time lapses involving z-stacking (one frame of the time lapse composed out of 5-10 exposures). I have an idea what to use as an automatic focus rail and how to program it to do so (in the worst case I have a Nikon remote controlled and programmable table for microscope). I have an idea how to program the camera to take series of shots (with software + usb connection I suppose).

So, as a result I'll have a bunch of files. Any idea how do I automate the stacking. Like picking up 10 files, stacking them, calling "frame1", then picking up the next 10 files, stacking, and saving as "frame2." It's possible to do manually, but if I have 10000 shots...

Would appreciate any advice.

Chris S.
Site Admin
Posts: 4042
Joined: Sun Apr 05, 2009 9:55 pm
Location: Ohio, USA

Post by Chris S. »

Daniel,

The stacking exercise you lay out here sounds pretty easy to do, if you're used to "slabbing" in Zerene Stacker. In your case, I would do slabs of 10 files with zero overlap. While Zerene Stacker does not (yet) have slabbing features per se, it does have a robust batch capability that will handle this situation with ease.

For making the needed batch file, three different approaches come to mind. One, you can make your own batch files by hand; they are simply xml files, so you can write them in any text editor--but I'll warn you that I've found this tedious. Two, you can download the ZereneVS utility that Elf, of this forum, wrote and has kindly shared with the rest of us; ZereneVS automates partial stacks, is primarily command-line driven, and is very powerful--but I've had difficulty using it (my fault, likely). Three, you can use "SlabberJockey," a utility that I wrote for my own use, and have been intending to share with this forum (gratis, as Elf has done). SlabberJockey is much less powerful than Elf's ZereneVS application, but has a graphical user interface and is very simple to use. It will let you create the batch file you need in a few seconds, with a very short learning curve.

PM me if you want a copy of SlabberJockey (this goes for anyone else, as well).

Cheers,

--Chris

--edited for typos and clarity
Last edited by Chris S. on Wed Mar 06, 2013 2:06 am, edited 3 times in total.

Starshade
Posts: 133
Joined: Wed May 04, 2011 1:47 am

Post by Starshade »

Thanks Chris,

I missed that function in Zerene! I'm at the stage of preparation for my experiments and probably start first time lapses in two weeks. Just wanted to make sure it's possible. I'll pm you when I'll have the files to process.

rjlittlefield
Site Admin
Posts: 23562
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Post by rjlittlefield »

Take a look also at "Batch API: Running Zerene Stacker From Another Application". If you're into scripting, it's not difficult to have ZS process each of your stacks as soon as it's shot, versus shooting them all first and then waiting for one long batch job to run.

--Rik

Starshade
Posts: 133
Joined: Wed May 04, 2011 1:47 am

Post by Starshade »

Thanks Rik,

Good thing to know about, although if I use the stackshot automated focus rail, I suppose I won't be able to get around with scripts.

rjlittlefield
Site Admin
Posts: 23562
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Post by rjlittlefield »

Starshade wrote: if I use the stackshot automated focus rail, I suppose I won't be able to get around with scripts.
One idea... Set the StackShot controller to shoot a time-lapse sequence of stacks. Pair the camera with image upload software that drops the images into some folder. Write a script (PHP, Perl, Bash, etc.) to monitor that folder, split the incoming sequence of images into stacks by knowing the number of images per stack, and process accordingly.

--Rik

Starshade
Posts: 133
Joined: Wed May 04, 2011 1:47 am

Post by Starshade »

I came back to this matter a few days ago and I tried both ZereneVS and SlabberJockey. The problem in both cases is that in every approach Zerene has to align around 7000 shots... which turns the images into mess as it's not an ultra-deep stack, but a time-lapse (basically every stacking step "magnifies" the image and shifts it towards one corner in my case). I couldn't make Zerene align selected frames even manually so I suppose automating that process is not easy.

Any ideas how to align and stack each 11 frames separately aside from turning alignment transformations off? Am I missing something? I don't even know if elf made a time lapse as his link is dead...

Just to clarify, my setup included StackShot and 7d where I made a large sequence of images.

rjlittlefield
Site Admin
Posts: 23562
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Post by rjlittlefield »

In concept, it sounds like what you want to do is not process 7000 frames as a bunch of 11-frame slabs. In that formulation, I guarantee that Zerene will try to align all 7000 frames as a single sequence, each against the previous. It would be surprising if this did not result in some significant drift.

Instead, what you want to do is process 640 stacks, each containing 11 frames. In that formulation, Zerene will use one of the frames in each stack as a master, and align the other 10 against that. Assuming there's no drift in your acquisition system, then there won't be any drift across the 640 stacks either.

There are several ways this could be controlled in a batch job. Perhaps the simplest would be to pre-organize the image files as one stack per folder, then construct a Zerene batch job consisting of a list of the folders and a single task such as Align & Stack All (PMax). In processing that batch job, Zerene will process each folder separately, just as if you interactively did a File > New Project, File > Add Files, Align & Stack All (PMax), File > Save Output Image, and File > Close Project.

[Edited to add: see BELOW for how to find free programs like Folder Axe (Windows) and Folder Splitter (Mac) to do this pre-organization automatically.]

Once you have the files pre-organized into folders, it should be easy to construct the batch job interactively, using the panel opened by Batch > Show Batch Dialog. Just click Add Projects or Folders and use click/shift-click or control-A to select all the folders containing stacks. Then in the Stacking Tasks area, select an appropriate Disposition for output images and Add a stacking task. Finally do a Save Batch, Run All Batches, walk away, and check back periodically to see how things are going.

I have no idea how to specify this in SlabberJockey and/or ZereneVS, if it's even possible.

--Rik

Edit: to add links
Last edited by rjlittlefield on Sat Apr 27, 2013 8:12 pm, edited 2 times in total.

Starshade
Posts: 133
Joined: Wed May 04, 2011 1:47 am

Post by Starshade »

Thank you Rik,

That's the route I chose and it seems to be working now.

Chris S.
Site Admin
Posts: 4042
Joined: Sun Apr 05, 2009 9:55 pm
Location: Ohio, USA

Post by Chris S. »

Creating 640 separate folders and moving the appropriate images into them sounds like drudgery. I'd run away screaming.

In a PM to Starshade yesterday night, my first suggestion was to go into Zerene Stacker's preferences and turn off scale changes, X and Y axis shifts, and rotation. Then rerun alignment, then try SlabberJockey. If Starshade's rig and subject are steady (other than whatever changes the subject undergoes during the time lapse), I'd expect Zerene Stacker, using a SlabberJockey-generated batch file, to produce good results with these preferences.

But if these features of Zerene Stacker's alignment process are needed for this group of images, Rik's folder-by-folder approach is probably the best bet for a one-off event. If the need occurs repeatedly, it's probably worth spending some time creating powertools to automate it, which involves a bit of programming.

SlabberJockey is something I created for the specific circumstances I encounter with deep stacks (which are likely the same circumstances other deep-stack users encounter); it was designed for simplicity of use, a short learning curve, and a minimum of coding on my part, at the expense of flexibility. In the current version of SlabberJockey's workflow, alignment is done on all images together, then slabs are assembled of subsets. Performing alignment on each subset, then stacking it, is simply not what SlabberJockey does. That said, changing SlabberJockey to do what Starshade needs is probably an afternoon's work for anyone comfortable writing VBA code. I'm happy to give the SlabberJockey VBA code to anybody who wants to spend the required afternoon--so long as no one asks me to do the work!

Cheers,

--Chris

rjlittlefield
Site Admin
Posts: 23562
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Post by rjlittlefield »

Chris S. wrote:If the need occurs repeatedly, it's probably worth spending some time creating powertools to automate it, which involves a bit of programming.
I agree with that. But for first-time or occasional use, it's often effective to look for some existing tool to do the job.

Just now I tried Googling on "split files into folders". The results point to utility programs for Windows and Mac that look like they'll do just what Starshade needed. I can't speak to whether they actually work, because I haven't tried them.

--Rik

Chris S.
Site Admin
Posts: 4042
Joined: Sun Apr 05, 2009 9:55 pm
Location: Ohio, USA

Post by Chris S. »

Rik's suggested search provides interesting results. From them, the one program I looked at, Folder Axe, seems as if it could separate the folders as Starshade needs. If a solution, even if a bit kludgy, is this easy, I'd go with it in a heartbeat and not look back.

--Chris

Starshade
Posts: 133
Joined: Wed May 04, 2011 1:47 am

Post by Starshade »

Thanks Rik and Chris, everything worked and I've got a nice test time lapse with increased and stable depth of field.

While disabling alignment image transformations would work for me (my setup was very stable and it was essential), I would still want to get nicer proportions...
This time (and this time only!) I created folders with an old piece of software (automatically), but moved files with FAR manager. Took me about 30 minutes of work despite the scary amount of images - I am very fast at keyboarding=) Next time I'll use Folder Axe as indeed it does the job.

Like I said, I will eventually try SlabberJockey for slabbing - so far I was getting around with lots of manual retouching... It is a nice and easy script generator, which does exactly what I asked it to do.

If I continue to do such things for my research I suppose I will have to say good bye to my shutter=) (can't have efsc and continuous image upload to the computer, and I need thousands and probably hundreds of thousands of shots per stack...)

Cheers,
Daniel

Post Reply Previous topicNext topic