Shooting in RAW

A forum to ask questions, post setups, and generally discuss anything having to do with photomacrography and photomicroscopy.

Moderators: Chris S., Pau, Beatsy, rjlittlefield, ChrisR

CrispyBee
Posts: 999
Joined: Mon Apr 03, 2023 11:17 am

Re: Shooting in RAW

Post by CrispyBee »

ray_parkhurst wrote:
Mon Feb 03, 2025 12:21 pm
It appears not. I'm surprised at the result but the more methodical method you previewed does show that there is significant headroom for whatever reason.
Well the reason is that JPEG is supposed to have good contrast and be 'ready' to print right out of the camera, whereas "pure" linear RAW looks very flat an unappealing. So it's understandable that JPEG might burn some highlights or swallow some shadows, it's just in the interest of having a quick image that does not require editing.

rjlittlefield
Site Admin
Posts: 24220
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Re: Shooting in RAW

Post by rjlittlefield »

ray_parkhurst wrote:
Mon Feb 03, 2025 12:21 pm
I'm surprised at the result but the more methodical method you previewed does show that there is significant headroom for whatever reason. I shot at a ref level just below highlight saturation on the jpg, then at +0.5/1.0/1.5EV. There is a weird blue shift in the jpg, and there is highlight recovery even up to +1.5EV. I think I may be a RAW convert.
I'm happy to hear that you now consider highlight recovery to be a real thing. Seriously -- it is real, and there's no magic involved. The camera just reserves some headroom in the raw data. Data in the headroom space is clipped by the in-camera conversion to JPEG but can be preserved by more careful raw development.

The examples that you've shown suggest that your raw development process can be further improved. If we take your well-exposed JPEG to be a reference image, then your highlight recovery process seems to be compressing the highlights into progressively narrower peaks. There's no reason that has to happen. As long as the brightest highlights still lie entirely inside the headroom area, the histogram from the overexposed-but-recovered raw file can look just like the histogram of the well exposed JPEG.

In my tests, simply adjusting the Exposure slider in Adobe Camera Raw was enough to do that job. Following is my brightest exposure that can be completely recovered, shown first as a default conversion which matches the blown-out JPEG, and again with reduced exposure which matches the well-exposed JPEG.

Image

Image

I'm not sure whether something as simple as pushing the Exposure slider will work given your setup. I can imagine that with a different camera or other software you might have to do some other stuff to preserve contrast in the recovered highlights. But it's definitely doable. There's nothing special about data in that headroom region, except that it gets clipped in default conversion.

--Rik

ray_parkhurst
Posts: 3636
Joined: Sat Nov 20, 2010 10:40 am
Location: Santa Clara, CA, USA
Contact:

Re: Shooting in RAW

Post by ray_parkhurst »

rjlittlefield wrote:
Mon Feb 03, 2025 8:28 pm
The examples that you've shown suggest that your raw development process can be further improved. If we take your well-exposed JPEG to be a reference image, then your highlight recovery process seems to be compressing the highlights into progressively narrower peaks. There's no reason that has to happen. As long as the brightest highlights still lie entirely inside the headroom area, the histogram from the overexposed-but-recovered raw file can look just like the histogram of the well exposed JPEG.
My RAW processing was unusual in that I aggressively adjusted the black level in order to expand the peak near saturation to make the tails in the data more visible. I also tried to carefully match the brightness and peak in the data between the RAW and JPG versions to make any subtle differences easier to see. I would never use these methods in a normal photo. That said, I'm certainly open to and thankful for suggestions on improved RAW processing.

Sym P. le
Posts: 349
Joined: Thu Jul 04, 2019 9:53 pm
Location: BC

Re: Shooting in RAW

Post by Sym P. le »

My take is far more simplistic. The OOC JPEGs are merely algorithm derived images so that the OEM can say their equipments take "nice" pictures. Pure marketing genius. Over saturated clipped schlock is the game.

rjlittlefield
Site Admin
Posts: 24220
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Re: Shooting in RAW

Post by rjlittlefield »

ray_parkhurst wrote:
Mon Feb 03, 2025 8:53 pm
I also tried to carefully match the brightness and peak in the data between the RAW and JPG versions to make any subtle differences easier to see.
I hear your words, but I don't know what they mean.

When I say that "the histogram from the overexposed-but-recovered raw file can look just like the histogram of the well exposed JPEG", the following image shows what I mean. The notation "RAW+1-1.15" describes an image that was physically exposed by 1 stop more, then had the exposure slider set to -1.15 during raw conversion. I wasn't paying any attention to the darks, so they don't match. But I think you'll agree that the histograms of the brights are pretty close.

Image

In contrast, here is what I see for the histograms of your overexposed-but-recovered raw file versus your reference JPEG. To my eye these look quite different from each other, and this is what I was talking about when I wrote that "your highlight recovery process seems to be compressing the highlights into progressively narrower peaks".

Image

It seems clear that I'm thinking about things differently from you, but I'm having trouble resolving the disconnect.

--Rik

wwp347
Posts: 54
Joined: Fri Mar 12, 2021 10:33 am
Location: Douglasville, GA

Re: Shooting in RAW

Post by wwp347 »

When processing a raw file, Adobe Camera Raw and Lightroom both default to Adobe Color profile which is designed to make the mid range look better at the expense of the highlights and shadows. A linear profile makes the entire frame appear dull but highlights and shadows are much better preserved. The mid tones can then be adjusted without losing either extremes. An explanation of linear profiles and benefits can be found here:

https://goodlight.us/linear-profiles.html
Thanks for your time and consideration,,
BudP

ray_parkhurst
Posts: 3636
Joined: Sat Nov 20, 2010 10:40 am
Location: Santa Clara, CA, USA
Contact:

Re: Shooting in RAW

Post by ray_parkhurst »

rjlittlefield wrote:
Mon Feb 03, 2025 10:45 pm
It seems clear that I'm thinking about things differently from you, but I'm having trouble resolving the disconnect.
I just think I didn't describe things well enough. Perhaps just showing you the histograms of the files from the RAW editor might help, or not:

JPG REF
JPG REF.JPG
RAW REF
RAW REF.JPG
RAW +0.5 over-exposure adjusted -0.35EV
RAW +.5 -.35.JPG
You can already see some highlight data loss in the 0.5EV over-exposed file, unlike in your example where even 1EV over-exposure shows little to no loss.

rjlittlefield
Site Admin
Posts: 24220
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Re: Shooting in RAW

Post by rjlittlefield »

ray_parkhurst wrote:
Tue Feb 04, 2025 12:49 pm
You can already see some highlight data loss in the 0.5EV over-exposed file, unlike in your example where even 1EV over-exposure shows little to no loss.
This is very interesting. I don't know your workflow or which raw editor you're using, so I'm not sure how to interpret the results. The spikiness of these last histograms is strange.

I would be interested to take a look at your raw files. If you're up for that also, then please send them to me by wetransfer.com, using email address support@zerenesystems.com . Or if not, that's OK too.

--Rik

ray_parkhurst
Posts: 3636
Joined: Sat Nov 20, 2010 10:40 am
Location: Santa Clara, CA, USA
Contact:

Re: Shooting in RAW

Post by ray_parkhurst »

rjlittlefield wrote:
Tue Feb 04, 2025 1:13 pm
ray_parkhurst wrote:
Tue Feb 04, 2025 12:49 pm
You can already see some highlight data loss in the 0.5EV over-exposed file, unlike in your example where even 1EV over-exposure shows little to no loss.
This is very interesting. I don't know your workflow or which raw editor you're using, so I'm not sure how to interpret the results. The spikiness of these last histograms is strange.

I would be interested to take a look at your raw files. If you're up for that also, then please send them to me by wetransfer.com, using email address support@zerenesystems.com . Or if not, that's OK too.

--Rik
No problem sharing the files, but I was hoping the histograms would be more self-explanatory as to what I'm doing. Perhaps the following ones will make things clearer. These are the JPG REF, RAW REF, and RAW+.5-.35 files without "zooming in" on the highlights. It is the zooming-in that causes the spikiness, but without it the data loss is difficult to see on the full-range scale. On the un-zoomed histograms you can see a little narrowing of the peak, but it's much less obvious.

JPG REF No Zoom
JPG REF No Zoom.JPG
RAW REF No Zoom
RAW REF no Zoom.JPG
RAW +.5-.35 No Zoom
RAW +.5-.35 No Zoom.JPG

rjlittlefield
Site Admin
Posts: 24220
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Re: Shooting in RAW

Post by rjlittlefield »

ray_parkhurst wrote:
Tue Feb 04, 2025 1:34 pm
I was hoping the histograms would be more self-explanatory as to what I'm doing. Perhaps the following ones will make things clearer.
Again, these are very interesting, but now I'm even more confused than before. In these latest histograms, there appear to be no darks. That's not consistent with the black bars on white paper that you showed in an earlier post. At the moment I'm lost.

Send raw files, please...

--Rik

ray_parkhurst
Posts: 3636
Joined: Sat Nov 20, 2010 10:40 am
Location: Santa Clara, CA, USA
Contact:

Re: Shooting in RAW

Post by ray_parkhurst »

rjlittlefield wrote:
Tue Feb 04, 2025 1:54 pm
ray_parkhurst wrote:
Tue Feb 04, 2025 1:34 pm
I was hoping the histograms would be more self-explanatory as to what I'm doing. Perhaps the following ones will make things clearer.
Again, these are very interesting, but now I'm even more confused than before. In these latest histograms, there appear to be no darks. That's not consistent with the black bars on white paper that you showed in an earlier post. At the moment I'm lost.

Send raw files, please...

--Rik
Yes, no darks because I moved the black-level up to "zoom-in" on the highlights.

I have uploaded the two raw files for the RAW REF and RAW +0.5 to DropBox. You (or anyone interested) can download them from here:

https://www.dropbox.com/scl/fo/a6fmh5qv ... olg06&dl=0

Edited to add: here is what the REF JPG looks like:
A7Rm403809_2_2.JPG

rjlittlefield
Site Admin
Posts: 24220
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Re: Shooting in RAW

Post by rjlittlefield »

ray_parkhurst wrote:
Tue Feb 04, 2025 4:29 pm
I have uploaded the two raw files for the RAW REF and RAW +0.5 to DropBox.
Thanks for the upload.

Those were A7Rm403809.ARW (RAW REF) and A7Rm403810.ARW (RAW +0.5).

I pulled the RAW REF into Photoshop using Adobe Camera Raw, with the Exposure slider set to -1.0 to expand the highlights for easier observation.

Then I did the same thing with the RAW +0.5, adjusting the Exposure slider to make the histogram look as much like the RAW REF as I could. Best match occurred with a setting of -1.34 . (ACR only allows 2 digits after the decimal point.)

Here's the result, after raw conversion:

Image

Even flashing between those histograms, all I see is a bit of difference in the stairstepping.

To me those highlights in +0.5 look completely recoverable. So now I'm curious about the files that are more heavily overexposed.

Any chance you can upload the +1.0 and +1.5?

--Rik

ray_parkhurst
Posts: 3636
Joined: Sat Nov 20, 2010 10:40 am
Location: Santa Clara, CA, USA
Contact:

Re: Shooting in RAW

Post by ray_parkhurst »

rjlittlefield wrote:
Tue Feb 04, 2025 5:38 pm
Even flashing between those histograms, all I see is a bit of difference in the stairstepping.

To me those highlights in +0.5 look completely recoverable. So now I'm curious about the files that are more heavily overexposed.

Any chance you can upload the +1.0 and +1.5?

--Rik
Yes, almost completely recoverable. You need to zoom-in as I did to see the detail of the data loss.

I uploaded the other two files to DropBox at same link for your perusal.

rjlittlefield
Site Admin
Posts: 24220
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Re: Shooting in RAW

Post by rjlittlefield »

rjlittlefield wrote:
Tue Feb 04, 2025 5:38 pm
Even flashing between those histograms, all I see is a bit of difference in the stairstepping.

To me those highlights in +0.5 look completely recoverable.
And yet RawDigger disagrees. The histograms from RawDigger clearly indicate that the green channel is severely clipping at +0.5. Here are the graphs:

RAW REF
Image

RAW +0.5
Image

I assume the other two files will be clipped far worse, and I'll be interested to see what ACR does with those.

--Rik

rjlittlefield
Site Admin
Posts: 24220
Joined: Tue Aug 01, 2006 8:34 am
Location: Richland, Washington State, USA
Contact:

Re: Shooting in RAW

Post by rjlittlefield »

rjlittlefield wrote:
Tue Feb 04, 2025 6:38 pm
I assume the other two files will be clipped far worse, and I'll be interested to see what ACR does with those.
Well, this is very interesting!

Indeed the +1.5 is clipped far worse -- for green the entire peak of highlights is clipped off, and for blue most of that same peak is gone.

And yet, ACR still manages to "recover" (reconstruct!) a plausible bell-shaped curve of green and blue values in that region, with appropriate though degraded detail visible when zooming in. (More about that last point after the graphs.)

Image

RawDigger histogram with auto scaling:
Image

RawDigger histogram with manual Y scaling, to make the lower peaks visible:
Image

Now, about that "recovered" (reconstructed!) data in the thoroughly clipped region.

What I've done here is to zoom into an area that in the +1.5 raw file is entirely clipped for blue and green. I've pushed the Exposure slider down far enough to match the REF histogram (after conversion), and I've added a strong levels adjustment to expand the values in this region so we can see detail. The histograms shown are for the selection area shown as the dotted rectangle. RAW REF on top, RAW +1.5 with highlights recovery below that.

Image

What I find telling about this comparison is that appropriate detail is present, but in the overexposed+recovered raw, all the color is gone. The only way I can figure this works is that ACR has used the only available luminosity information, from the unclipped red channel, and then used that to infer green and blue values that at least won't be glaringly out of place.

So, I was definitely wrong when I said there was "no magic". In fact there is magic, in the form of replacing clipped values by other values that are inferred from surrounding valid information. In this particular case ACR has ended up with R=G=B. I expect it will do something different in less severe cases where only one channel is clipped, and/or when the clipping is more localized so that it's reasonable to match color from nearby regions of the image.

At this moment I have no idea how much of the traditional "recovery" of highlights is due to simple headroom versus the magic of inferring plausible values to replace clipped ones.

A matter for further investigation...

--Rik

Post Reply Previous topicNext topic