Zerene Stacker now has dust & hot-pixel removal

A forum to ask questions, post setups, and generally discuss anything having to do with photomacrography and photomicroscopy.

Moderators: rjlittlefield, ChrisR, Chris S., Pau

rjlittlefield
Site Admin
Posts: 23964
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Zerene Stacker now has dust & hot-pixel removal

Post by rjlittlefield »

The latest beta release of Zerene Stacker provides a new feature, “Use dust & hot pixels mask”, for addressing the problem of trails caused by dust and hot pixels.

To use it, you construct a mask image, same size and orientation as your source images, that is pure white everywhere, except black where your source images have dust spots or hot pixels. Then at Options > Preferences > Preprocessing, you check the box for “Use dust and hot pixels mask” and browse to find the file. The mask will then be applied in subsequent stacking operations.

Very briefly, what the feature does is to replace blobs of bad pixel values with a smoothly varying average of surrounding good pixel values. This alone provides a good start at hiding the defects, and then the normal stacking process further improves the result by tending to replace the smoothly varying averaged values with unmodified good values from other frames that happen to have different alignments. The result is that instead of a trail, what typically gets produced is an image that is clean except for one spot of lost or degraded detail where the defect lay in the best focused image.

Here is an example. These crops are from a stack that I shot several years ago, at a time when I think I must have had my camera open during a dust storm.

Left: without the mask; Right: with the mask.

Image Image

Left: source frame with dust locations marked in yellow; Right: final mask, black on white

Image Image

Here is another example, this time dealing with hot pixels as a result of long exposure with continuous light.

Left: without the mask; Right: with the mask

Image Image

Left: cropped mask matching the images shown above; Right: mask for whole frame (dots expanded to be easily visible)


Image Image

Note that Zerene Stacker makes no attempt to build an appropriate mask. It just applies whatever mask you give to it.

Building the mask is something that you’ll do outside Zerene Stacker, using tools in your favorite image editor such as Photoshop or GIMP.

For the first example, I pulled a source frame into Photoshop, heavily sharpened it to enhance the dust, added a new layer that would become the mask, and used a small brush to manually “dot over” each spot of dust. For brushing I used a contrasting color (bright yellow) to make it simple to see where I had painted. Then to make the final mask, I used Photoshop’s selection and filling tools to turn the yellow spots into black-on-white.

Initially I tried to construct the dust mask automatically from the source images by a combination of sharpening and levels adjustment to bring out the dust spots. But that approach failed because the dust spots had such wide range of appearance, on varied background, that I was not able to get a clean distinction between dust and no dust. It was much faster and better to just dot over the dust spots by hand.

For the second example, I found that it was tedious and unnecessary to make the mask by hand from a source frame of the stack. Instead, I simply made a very long time exposure of a black scene to make the hot pixels easily distinguishable, then used levels-adjustment in Photoshop to isolate hot pixels, selected those, expanded the selection by a couple of pixels to be sure of covering up the “crosses” that are often produced by JPEG compression, and filled the selection to make the mask. Unlike dust, hot pixels don’t move around or accumulate rapidly, so this mask will be usable for a long time in the future with that same camera.

There are a few caveats…
  • At this time the process is computationally expensive, roughly doubling the time to run a basic PMax stack in my limited testing to date. Probably I can speed that up with more clever coding, but the value at current cost is high enough that I’m comfortable releasing it as a production feature.
  • The mask is applied only during stacking. During retouching, source images will be presented in their original form.
  • Error checking is minimal at this time. If the mask is the wrong size, or the wrong orientation, or the mask file can’t be read, or anything else unexpected happens, then either the function won’t work at all, or it’ll do something weird, or you’ll end up with one of the dreaded “Uh-oh, something went wrong” pop-ups.
  • It's best if the mask image is TIFF, even when you're stacking JPEG's. This avoids possible issues caused by JPEG compression around the edges of the black areas of the mask.
  • The dust & hot pixels mask will be a Pro-only feature, so to trial it with a Student or Personal Edition license, you'll have to click "Yes, start trial" when the notification popup appears.
  • If you haven't received an announcement for this beta even though you're already on the Zerene Stacker beta channel, please don't worry about that. I've bundled a bunch of stuff into the current beta, and I decided to post here before pushing out an in-app announcement because of recent questions about hot pixels.
Feedback solicited, of course.

--Rik

ChrisR
Site Admin
Posts: 8671
Joined: Sat Mar 14, 2009 3:58 am
Location: Near London, UK

Post by ChrisR »

Wonderful, I look forward to trying it!

Well done Rik.
Chris R

Lou Jost
Posts: 6199
Joined: Fri Sep 04, 2015 7:03 am
Location: Ecuador
Contact:

Post by Lou Jost »

Thanks very much Rik, this will be useful to everybody!

Peter M. Macdonald
Posts: 187
Joined: Tue Jan 20, 2009 2:59 pm
Location: Berwickshire, Scotland

Post by Peter M. Macdonald »

Yet another great addition to the tools which you have already put at our disposal. Well done and thank you Rik.

I am aware that Canon has a dust delete feature in many of its cameras. You take an out of focus image, at a large aperture, of something fairly bright, such as a blank area of sky, and then feed the resultant capture into DPPro as a dust file. The software can then apply this information to any other file when asked. Nobody else seems to use this dust information from Canon, so I am not sure if it is something which they have always hidden the details of, or whether the likes of Adobe have considered that it is too much trouble to utilise when other camera manufacturers seem not to offer the same feature.

Beatsy
Posts: 2245
Joined: Fri Jul 05, 2013 3:10 am
Location: Malvern, UK

Post by Beatsy »

Another handy addition to an already indispensable piece of software. Thanks for the continuing updates Rik.

Could I also suggest... :)

Image

Smokedaddy
Posts: 1991
Joined: Sat Oct 07, 2006 10:16 am
Location: Bigfork, Montana
Contact:

Post by Smokedaddy »

Excellent, thanks. No more spot healing brush in PS. 8)

Macro_Cosmos
Posts: 1579
Joined: Mon Jan 15, 2018 9:23 pm
Contact:

Post by Macro_Cosmos »

Beatsy wrote:Another handy addition to an already indispensable piece of software. Thanks for the continuing updates Rik.

Could I also suggest... :)
In Australia, alcohol of all types... even Vodka in many places are completely sold out.

We all know why :cry:

abpho
Posts: 1529
Joined: Wed Aug 17, 2011 7:11 pm
Location: Earth

Post by abpho »

Nice Rik. Thanks.
I'm in Canada! Isn't that weird?

Chris S.
Site Admin
Posts: 4087
Joined: Sun Apr 05, 2009 9:55 pm
Location: Ohio, USA

Post by Chris S. »

Rik,

Delightful to see this added to Zerene Stacker! I've downloaded the beta, and am looking forward to trying out dust and hot-pixel removal.
Peter M. Macdonald wrote:I am aware that Canon has a dust delete feature in many of its cameras. You take an out of focus image, at a large aperture, of something fairly bright, such as a blank area of sky, and then feed the resultant capture into DPPro as a dust file. The software can then apply this information to any other file when asked. Nobody else seems to use this dust information from Canon, so I am not sure if it is something which they have always hidden the details of, or whether the likes of Adobe have considered that it is too much trouble to utilise when other camera manufacturers seem not to offer the same feature.
FYI, Peter, Nikon has a similar feature. However, I've not gotten it to work on the two or three models I've attempted it on. When I try to take the out-of-focus image with my macro lens, the camera seems to think for a while, then eventually tell me to try with a different lens. At that point I've stopped messing with it.

--Chris S.

Scarodactyl
Posts: 1699
Joined: Sat Apr 14, 2018 10:26 am

Post by Scarodactyl »

A very handy feature that I've been getting quite a bit of use from. I had my local camera shop clean my sensor, but they somehow ended up smearing something on it instead. Very vexing on some longer exposures.

RDolz
Posts: 96
Joined: Mon Aug 28, 2017 9:32 am
Location: Valencia (Spain)

Post by RDolz »

Rik: good job and very helpful. =D>

In my case my main problem is the dead and hot pixels. I work with the Fujifilm X-T1, a camera that does not have the ability to map faulty pixels.
So I had an action in photoshop that eliminates the hundreds of pixels that show defects in long exposures. (action that significantly increases post-processing tasks)

Therefore, having a tool, such as Zerene, that integrates these functions as well as folder processing (Batch) is very interesting because it allows you to obtain the best result without the need to apply other processes.

Only if it serves as a help, in my case if you make a long exposure of many seconds with the lens covered, I can see the defective pixels, but others appear when in addition to making long exposures, I also capture thousands of photos (when I do stacking + stitching), that is, when the sensor has been active for a long time and therefore the temperature increases. I have to comment that normally my exposure times are around 1 / 15s, ... they are not very long exposures, but many damaged pixels appear.

Best
Ramón Dolz

JW
Posts: 272
Joined: Wed Feb 23, 2011 7:38 am
Location: New Haven, CT, USA

Post by JW »

Another great feature, but, my friendly admins now keep me from downloading software. Rik, is there a way to side-load the updates?
It's not what you look at that matters, it's what you see - Henry David Thoreau

rjlittlefield
Site Admin
Posts: 23964
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Post by rjlittlefield »

JW wrote:my friendly admins now keep me from downloading software. Rik, is there a way to side-load the updates?
Windows or Mac?

If Mac, then there's really no installation to be done. Just do the download and unpack the .zip on any machine that can, then write the whole application package tree to a thumbdrive and execute it from there.

On Windows, you can try something similar, except in that case it's probably cleanest to install into some ordinary folder like C:\ZereneStacker, instead of letting the installer put it into Program Files by default. Then copy the folder to thumbdrive and move that.

Unfortunately I think that both of these routes are likely to be blocked also, assuming that your admins are good at their jobs.

Other than these approaches, I have no idea how to try working around administrative limits.

--Rik

pierre
Posts: 289
Joined: Mon Jan 04, 2010 12:37 pm
Location: France, Var, Toulon

Zerene Stacker now has dust & hot-pixel removal

Post by pierre »

Many thanks for your care and hardwork Rik !
This is definitively a nice add.

Will try it soon.

Stay safe.
Regards

Pierre

RDolz
Posts: 96
Joined: Mon Aug 28, 2017 9:32 am
Location: Valencia (Spain)

Post by RDolz »

Hello, I downloaded the Beta version to test the new feature, “Use dust & hot pixels mask”. I have tested it to eliminate the hot pixels and, (as could not be otherwise), it works perfectly!!

It took me a while to make the mask, but in the end, with photoshop, it was easy.

The main problem is detecting pixels that are defective in a single image in the stack. But if we use the result of a stacking it is easy to detect the "little snakes" that each faulty pixel generates.

Here I show you the hot pixel mask for my X-T1

Image


and the difference of a portion of an image before and after applying the new feature.

Image

This image has a lot of noise, because it is a black background and I have increased the levels to make the hot pixels look good.

Best
Ramón Dolz

Post Reply Previous topicNext topic