Sub-stacks

A forum to ask questions, post setups, and generally discuss anything having to do with photomacrography and photomicroscopy.

Moderators: rjlittlefield, ChrisR, Chris S., Pau

ChrisR
Site Admin
Posts: 8668
Joined: Sat Mar 14, 2009 3:58 am
Location: Near London, UK

Sub-stacks

Post by ChrisR »

Stacking only part ( say 10 images) of a whole (say 100) stack of images can be useful.
I haven't done it an awful lot because it takes (me) so long. I'd like someone else to tell us a better method than my ad-hoc approach, to deliver more potential.

As a starter, this was sharp end of a 15mm (?) ground beetle. ( It has been poted before). Viewed from the top the depth to width ratio is qute high, so there's a lot of images in the stack even for a low magnification. This situation I find leads me to hazy indistinct images, for a bunch of reasons I only partly understand. A lot of it has to do with lighting, I'm sure. At higher magnifications, everything gets harder of course.
The magnification here, is possibly about 4-5x onto most of a 24 x36 sensor. Lens not recorded, it was a year or more ago.
First is a quick orientation pic of the beetle (about 8 images in that stack at f/22, reversed kit lens),
then a Pmax output of the whole closer stack (238 images).
There was plenty wrong, as you can see. Blue, dust...
One of the sub stacks' result is shown, which came from just the (single remaining) antenna, which I think was 80 frames by itself. NB the antenna is "cleaned" of its background, so it can easily be trimmed or masked and dropped onto the remainder of the beetle.

The last image is the result of playing for what felt like long enough. There remain things ( eg artefacts) which would be altered if it were important.
There were about 5 layers in Photoshop, though a couple could have originated from more than one substack.
what I got to when I stopped playing, with 5 layers in Photoshop.
I did try to "work on" the whole image in Photoshop, but one single whole pmax wasn't good enough as a start.

I expect some of you are doing this better, and routinely, and I'd like to hear how :)

Image

Image

Image

Image

Chris S.
Site Admin
Posts: 4042
Joined: Sun Apr 05, 2009 9:55 pm
Location: Ohio, USA

Post by Chris S. »

Chris, I'm doing this a lot, though I call the sub-stacks "slabs." Not sure who coined this term--perhaps Rik? Anyway, I am finding the approach enormously useful, and suspect that in time, slabbing may be the rule, rather than the exception, in my workflow.

Based on some very useful guidance from Rik, I can do this pretty quickly, using an xml file that I call from Zerene Stacker's batch dialog. For different stacks, I edit the file. I suspect that Rik is already typing guidance, and will do a far better job of it than I would, but I'll email you the file I use, which you can edit for your purposes.

--Chris

DQE
Posts: 1653
Joined: Tue Jul 08, 2008 1:33 pm
Location: near Portland, Maine, USA

Post by DQE »

The final photo is to me a very unusual view of this type of creature, and quite interesting.

----------------------

Thanks for the insights into your slab-based workflow. Intuitively, it makes sense to me, although I have little stacking experience. It just seems correct that breaking a project down into sections would be more manageable and perhaps even more understandable and better optimized than trying to do everything in one grand unified stack.

Would it sometimes make sense to adjust the lighting between slabs? I was thinking here that one could provide lighting to see inside the creature's mouth that was only active while sharply imaging the interior. When mostly imaging the exterior, perhaps lighting that optimally but selectively would work.

Just some thoughts. Yet I can also imagine many problems trying to merge images from slabs that were illuminated quite differently. No doubt variable lighting within a stack would have to be carefully and systematically developed, to have much credible viability...
-Phil

"Diffraction never sleeps"

elf
Posts: 1416
Joined: Sun Nov 18, 2007 12:10 pm

Post by elf »


ChrisR
Site Admin
Posts: 8668
Joined: Sat Mar 14, 2009 3:58 am
Location: Near London, UK

Post by ChrisR »

I imagine that's a front end for batch scripts?

I can't get it to open - it shuts immediately.

The difficulties I find with substacks are more about processing the curves etc in each part and then reintegrating the subs together.

elf
Posts: 1416
Joined: Sun Nov 18, 2007 12:10 pm

Post by elf »

ChrisR wrote:I imagine that's a front end for batch scripts?

I can't get it to open - it shuts immediately.

The difficulties I find with substacks are more about processing the curves etc in each part and then reintegrating the subs together.
Yes, it just creates the batch scripts.

Are you running it from the commandline? You also need to provide the initial parameters. I'd make a pretty front end for it, but I think Rik will probably do that inside of Zerene Stacker (hint, hint).

Chris S.
Site Admin
Posts: 4042
Joined: Sun Apr 05, 2009 9:55 pm
Location: Ohio, USA

Post by Chris S. »

By the way, Chris (R.)—very nice image! Sorry for being so breathless about slabbing that I didn't comment on a superb photograph.

I wouldn't have the patience to routinely combine slabs using Photoshop layer masks. While I often use Photoshop layer masks, Zerene Stacker can stack slabs very well, so I let it do most of the heavy lifting.

I’ve been doing slabs one way or another for some time--it's been cumbersome but often worthwhile. As you demonstrated above, having a slab in which an element is delineated cleanly, but all else is a creamy blur, can help a lot when retouching. And slabs can reduce the interference of things in front of other things.

But a large step forward in my slabbing came a couple of weeks ago, when I asked Rik for advice about a stack I was finding difficult. It was, for me, a shallow stack of 259 frames. But by necessity, the individual frames were dark, fuzzy, and confusing, and my naked eye could barely make heads or tails of them. It was not surprising that the resulting stacks had major artifacts and were hard to retouch. Rik suggested that I make PMax slabs of about 20 images, overlapping each slab with the prior slab by ten frames. Then, he suggested, I should stack the resulting slabs using DMap. And he gave me the tips I needed to create an xml file that could be loaded into the Zerene Stacker batch dialog to do this automatically. I found it tedious to create the first xml file, but thereafter have just edited it for each new job.

Wow, has this approach worked. The difficult stack turned out shockingly better right out of the box. But also notable was how much easier other things were. I could quickly test various DMap settings—which unsurprisingly took an order of magnitude less time to run on 25 slabs than on 259 frames. Since I could test more quickly, I did more testing, and found better combinations—some of which I’ll combine using Photoshop layer masks (right now I’m mostly shooting—will leave the finer points of post-processing until the snow flies). And Zerene Stacker’s retouching tools became much easier to use for this very shallow DOF stack (100x). When a feature is spread across 20 frames, it can be hard to retouch in. When it is contained within a single slab, it’s easy.

So for every stack I’ve shot since (about three per day, the shallowest being 94 images), I’ve used this approach. It’s early to say, but I strongly suspect that slabbing may form a key portion of my workflow for anything other than very shallow, low-magnification, stacks. Definitions of “shallow” and “low-magnification” differ from one photomacrographer to another. The lowest magnification I’ve shot during this period was a 5x stack of 461 images. For this stack, I tried both the slabbing regime described here and a straight (no slabbing) stack. My sense was that Zerene Stacker could handle either approach very well. But the human element—me—found the slabbing approach much easier. Retouching was orders of magnitude easier with the slabs—no confusion about which elements are in front and which behind, and no endless searching for which frame to retouch in. Glorious! And with the slabs, I quickly tried several sets of DMap settings, and (as the tutorials say) found that certain settings worked better for various elements of the image. I’ll retouch the best elements together later. Without the slab approach, trying various DMap settings would have been more onerous, and I’d probably have skipped it.

Sorry for the long post. But I suspect this is paradigm-level issue in photomacrography. A frontier. Thanks, Chris R., for bringing it up.

Ed (“elf”) was generous in sharing his superb generator of xml files for Zerene Stacker batch jobs. His work deserves much more attention than it has gotten. I looked—hard—at it before generating my own xml file, but decided that I needed the learning experience of writing such an xml file myself. And I wasn’t sure, on initial inspection, if Ed’s approach would let me generate the overlapping DMap images that I wanted. (It was partly because of my confusion that I decided I needed the learning experience.) But my sense is that Ed’s application either can, or could with minor modifications, generate an xml file such as I’m using, much more easily than I'm doing it. Ed hasn’t gotten his deserved share of kudos for this contribution, and I think it's because most of us (myself included) are not sufficiently advanced to appreciate what he gave us.

And maybe Rik is cooking up enhancements to facilitate slabbing in Zerene Stacker? While I have no inside information, does anybody care to bet against me? :D Anyway, I hope Rik is, because slabbing seems to take focus stacking to a new level.
ChrisR wrote:I imagine that's a front end for batch scripts?

I can't get it to open - it shuts immediately.

The difficulties I find with substacks are more about processing the curves etc in each part and then reintegrating the subs together.
Chris, I can open Ed’s script easily—issue on your machine?

Yes, Ed’s script generates a very nice xml file that can be loaded via the Zerene Stacker batch dialog.

Processing “curves, etc.”—this seems Photoshop specific. Any reason not to stack your slabs with ZS DMap? If you prefer to do it in Photoshop, what’s the problem in processing curves? I use curves layers daily, so this seems like the easy bit. It’s the endless painting in with layers that would kill me.

Cheers,

--Chris (S.)

(Why are so many of us at PMN named "Chris"?)

ChrisR
Site Admin
Posts: 8668
Joined: Sat Mar 14, 2009 3:58 am
Location: Near London, UK

Post by ChrisR »

Chris, I can open Ed’s script easily—issue on your machine?
50% chance of that I expect.
And 50% chance I'm missing something out.
And 30% of something else...

Edit - redownloaded, I can get the help screen.
After lots of guesses about what it means, the program appears to run, but Zerene "doesn't recognise" the .xml file as a batch script.
Hand-holding would help here, with examples. I can't type a 40+ character path name in without an invisible typo, which means doing it all again, and I don't pre-know things like whether to include the < > symbols!
I'll get back to it later.


Any reason not to stack your slabs with ZS DMap? It’s the endless painting in with layers that would kill me.
This is an example of where I was looking for guidance :) so the prompting-post is working! I found that the stacker would take whichever slab it thought had more detail and made that layer "show" even it it happened to be behind something less detailed.
I've always felt that ZS needs a notion of foreground being more important that background, though with some (reverse perspective)lenses it would have to be told which was which.

Not enough overlap between my slabs perhaps?? I used a few frames, from memory of that beetle.

Yes the photshop editing part is easy, usually with cloning and other fakery!
Masking isn't so hard - you can use the auto-select tools, with a small edge-feather, which usually get it about right.

Also the background in this was messy so I replaced it completely

elf
Posts: 1416
Joined: Sun Nov 18, 2007 12:10 pm

Post by elf »

Copy/Paste is your friend :) When you have long paths to enter, navigate to them in Windows Explorer, then copy and paste them into the commandline. I usually create long commandlines by copying the template into notepad, then edit in notepad until it's correct. This makes it easy to retry until it's correct. <something> means replace everything including the <> with the text for 'something'.

ChrisR
Site Admin
Posts: 8668
Joined: Sat Mar 14, 2009 3:58 am
Location: Near London, UK

Post by ChrisR »

Believe me I tried!
Ctrl-X-C-V don't work in the Command window. It's like going back to DOS !

AH but I just tried right-click, something I don't normally do, and that does work :)

rjlittlefield
Site Admin
Posts: 23562
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Post by rjlittlefield »

Chris S. wrote:Chris, I'm doing this a lot, though I call the sub-stacks "slabs." Not sure who coined this term--perhaps Rik? Anyway, I am finding the approach enormously useful, and suspect that in time, slabbing may be the rule, rather than the exception, in my workflow.
As far as I can trace it, the term "slab" as used here started in a post of mine on Apr 21, 2008. That was in the days of TuFuse and CombineZ, but the concepts were the same.

In terms of Zerene Stacker, the idea is to exploit the complementary tradeoffs of PMax and DMap. PMax is good at handling complicated structure with lots of overlaps, but it accumulates noise, alters color and contrast, and those problems get worse with deeper stacks. DMap is not so good with overlaps, but it doesn't have those other problems. So the concept is to use PMax on shallow sub-stacks (slabs) to produce partial outputs, then use DMap to combine the partial outputs. With proper choice of slab depth and DMap parameters, and maybe a bit of luck, almost all of the final pixels can be generated automatically, and where manual retouching is required that will go a lot faster.

In the long run, ZS will get some controls that make this easy to do. But for now the workflow has to be built up from simpler pieces.

A workflow that is tedious but simple to understand goes like this:
  • Launch ZS
  • Load all source images
  • Stack > Align All Frames
  • Select images 1-20
  • Stack Selected (PMax)
  • Select images 11-30
  • Stack Selected (PMax)
  • repeat previous two steps for all other slabs.
  • Select all output images
  • File > Save Output Image(s) to some scratch directory.
  • Exit ZS
  • Re-launch ZS
  • Load the images you just saved
  • Uncheck everything at Options > Preferences > Alignment
  • Stack > Align & Stack All (DMap).
Based on some very useful guidance from Rik, I can do this pretty quickly, using an xml file that I call from Zerene Stacker's batch dialog. For different stacks, I edit the file.
The idea here is avoid having to interactively select, stack selected, select, stack selected, select, stack selected..., by creating a batch file to do those steps unattended.

The GUI is way too tedious to generate such a batch job. However, it's a good way to see what the batch job should look like, so you can build one programmatically. What I suggested to Chris S. was to open his project, select a few source files, open the batch dialog, do an Add Current Project, Add a task to Stack Selected (PMax), close the batch dialog, select some other files, open the batch dialog again, click to select the existing batch, and add another Stack Selected (PMax). Then Save Batch, close the dialog, and in the main menu do a Batch > Save Batch Script. What that produces is a ZereneBatch.xml file that you can read to figure out the pattern. There'll be a lot of repeated information in the OutputImageDispositionCode and Preferences sections. The critical variation is in parts like these:

Code: Select all

        <Tasks length="2">
          <Task>
            <OutputImageDispositionCode ... />
            <Preferences>
            ...
            </Preferences>
            <TaskIndicatorCode value="4" />
            <SelectedInputIndices length="6">
              <SelectedInputIndex value="17" />
              <SelectedInputIndex value="18" />
              <SelectedInputIndex value="19" />
              <SelectedInputIndex value="20" />
              <SelectedInputIndex value="21" />
              <SelectedInputIndex value="22" />
            </SelectedInputIndices>
          </Task>
          <Task>
            <OutputImageDispositionCode ... />
            <Preferences>
            ...
            </Preferences>
            <TaskIndicatorCode value="4" />
            <SelectedInputIndices length="6">
              <SelectedInputIndex value="20" />
              <SelectedInputIndex value="21" />
              <SelectedInputIndex value="22" />
              <SelectedInputIndex value="23" />
              <SelectedInputIndex value="24" />
              <SelectedInputIndex value="25" />
            </SelectedInputIndices>
          </Task>
        </Tasks>
Hopefully this is sort of self-explanatory. As shown here there's a vector of tasks, length 2; each task is TaskIndicatorCode 4, which is Stack Selected (PMax); and each task operates on a vector of SelectedInputIndex values, in this case 6 files per task, with an overlap of 3. Index values start at 0, by the way.

With proper choice of output image disposition and naming template, this shortens the workflow to:
  • Edit batch script
  • Launch ZS
  • Load all source images
  • Load batch script
  • Run batch script
  • Exit ZS
  • Re-launch ZS
  • Load the images created by the previous run
  • Uncheck everything at Options > Preferences > Alignment
  • Stack > Align & Stack All (DMap).
I'm sorry this is still so complicated, but it'll be a while before the process gets simpler.

--Rik

AndrewC
Posts: 1436
Joined: Thu Feb 14, 2008 10:05 am
Location: Belgium
Contact:

Post by AndrewC »

Curious about your choice of sub-stack overlaps, a 50% overlap seems a lot (1-20, 10-30, 20-40, etc). When I do this (and I think of it as "chunking" or "slice'n'dicing" not "slabbing") I tend to only go for about a 25% overlap (1-20, 15-35, 30-50, etc) but I've never done any actual comparisons. It what I picked the first time ! Also, I've never tried combining P and D-max that way, I use/used it as way of creating partial stacks for retouching so something else to try now the nights are getting longer.
rgds, Andrew

"Is that an accurate dictionary ? Charlie Eppes

Harold Gough
Posts: 5786
Joined: Sun Mar 09, 2008 2:17 am
Location: Reading, Berkshire, England

Post by Harold Gough »

Slab away, if it gives us more such excellent final images.

Harold
My images are a medium for sharing some of my experiences: they are not me.

descall
Posts: 178
Joined: Sat Oct 01, 2011 3:12 pm
Location: United Kingdom

Post by descall »

I wonder if there has been any further development with simplifying the slab stacking approach in ZS?

Best wishes,
Des

rjlittlefield
Site Admin
Posts: 23562
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Post by rjlittlefield »

descall wrote:I wonder if there has been any further development with simplifying the slab stacking approach in ZS?
In ZS proper, not yet. But photomacrography member elf has written a program to facilitate the process by creating a batch script that generates the substacks. See Automating Partial stacks in Zerene Stacker for more details.

--Rik

Post Reply Previous topicNext topic