1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How can I tell whether RAW files have blown highlights?

Discussion in 'Digital Image Editing & Printing' started by ChrisNewman, Sep 25, 2017.

  1. GlennH

    GlennH Well-Known Member

    The problem with attempting to preserve maximum data in an 8-bit JPEG is that the wide gamut you'd use (usually ProPhoto RGB) requires 16-bit colour depth to avoid posterisation when editing. Thus, creating such a JPEG wouldn't often have a purpose. Instead, you'd typically print straight from the raw processor or create a 16-bit TIFF. Of course, a wide gamut wouldn't save a badly overexposed image where all three RGB channels are at or near full saturation, but it does help in highlight recovery. Lightroom uses a version of ProPhoto RGB in its default histogram, and it'll often be the difference between channels clipping or not, which can be seen by soft-proofing smaller colour spaces.

    I wouldn't get too concerned about the 256 levels of an 8-bit JPEG histogram; you'll see the same histogram when editing 16-bit files, too, because it's more practical and intuitive to edit. The presence of the data is more important than its representation in numbers.

    I don't know how Nikon raw software works, but for maximum latitude you don't want a carbon copy of in-camera settings. The histogram on the back of the camera is inherently less forgiving than one representing a bigger colour space, especially if you also skew it with colourful picture modes. To get a bit nearer to the leeway of the raw file, it'd be best to choose the largest colour space possible in the camera (typically Adobe RGB) and use the most neutral picture mode. That's a very inexact method, but it'd probably be a bit more representative.
     
  2. PeteRob

    PeteRob Well-Known Member

    The raw file always has more information than the jpg. In my experience you can always recover highlight detail from a raw file provided that it is not clipped.

    Maybe what you have read applies to shooting at higher ISO values. The sensor output can ony be amplified so much. If you look at a plot of camera dynamic range against ISO it will be max at the base ISO, then decrease a little bit as ISO increases (this is amplification of the signal) and then the fall will steepen to a constant one stop per ISO doubling. When you use an ISO on this slope you basically underexpose the image and overdevelop by remapping the white point. If you are in this range then you can alter the white point quite a lot in raw processing because, as far as the sensor is concerned, the light it records is far away from hardware saturation. It is possible that, to minimise shadow noise, the in camera jpg engine will also sacrifice highlights as ISO increases.
     
  3. EightBitTony

    EightBitTony Well-Known Member

    Unless I'm missing something obvious, you can't. The RAW file is just a set of sensor data. There's no way of deciding what it contains without converting it into an image format *of some kind* and then changing how it's interpreted to see if it fits. You can't just check the sensor data. All you can do is move around the window you're using to view that data (i.e. change the way the numbers are interpreted to create an actual image).

    I can tell pretty quickly by looking at the JPG, and the histogram, and the subject, in Lightroom, whether or not changing the highlights, exposure, and white point sliders will recover all the data - maybe the answer is - experience with any RAW processor will let you work that out over time without having to do it.
     
    Roger Hicks likes this.
  4. ChrisNewman

    ChrisNewman Well-Known Member

    I realize that a RAW file isn’t an image file. But I assume it will include an array of numbers relating to the luminance values of each photosite. I assume each of these numbers will have the same upper limit (perhaps 16383 for a 14-bit file?), that any number less than the maximum will indicate a genuine luminance measurement, and the maximum number will mean that the photosite’s capacity was exceeded, indicating an unknown level of over-exposure, equivalent to the 0-254 exposure range in a JPEG, with 255 indicating over-exposure. I’m hoping to an easy way of displaying the range of these values, in a histogram, or better still showing overexposure according to its position in the potential image.
    I appreciate that reflections from the sun, artificial light, etc, must be accepted as burned out. Thanks for the suggestion of looking for less than 25% of the histogram to be at the right end. But I suspect that could only be a very approximate guideline - a large area of blue sky may be all only slightly overexposed, whilst red berries, or a bright red garment, may represent only a small area of the image, but can lose a lot of saturation if over-exposed.

    Chris
     
  5. GlennH

    GlennH Well-Known Member

    To achieve this, you only need glance at the histogram in a program like Lightroom, which will also show you the clipped areas on the image. Composite RGB values stacked up to the right are depicted in grey and indicate overexposure (i.e. blown-out detail). Overexposure in a single channel causes posterisation and loss of textural detail, which is often recoverable if there is sufficient data in one or both of the remaining channels.
     
  6. El_Sid

    El_Sid Well-Known Member

    In theory I imagine it should be possible to write software that can read the sensor data directly as luminance values, after all that is what the converters do, but I can't say I've ever heard of any such program. I'm not sure it would necessarily be meaningful either as that would not take into account the colour of the light entering each photodiode (as determined by the Bayer filter) nor would it be an accurate representation of the final RGB values of each image pixel since these are calculated not simply from one photosite but from a complex array in order to interpolate the color value of each pixel more accurately.

    I know what you want and why but I'm not sure it's actually going to be that much more accurate than the histogram you already see.
     
  7. ChrisNewman

    ChrisNewman Well-Known Member

    Thanks very much Tony. If it can’t be done, no wonder I haven’t been able to find a method of doing it! It seems I’ll need to rely on my so far rather inexperienced judgement.

    I am surprised if it can’t be done. I assume the sensor data will be in the form of numbers relating to how much charge, representing how many photons, was collected at each photosite. I assume each of these numbers would have the same upper limit, and that if that maximum number is reached it indicates more light was collected than the system was designed to record. So any smaller number should represent a satisfactory reading at the photosite. In that case it shouldn’t be difficult to display how many photosites were over-exposed, and where they where on the sensor.

    Anyway, thanks again for solving my dilemma, although it’s not the answer I was hoping for.


    Chris
     
  8. ChrisNewman

    ChrisNewman Well-Known Member

    I wouldn’t expect any software that displayed the sensor data to be able to show useful information about how close the various colours were to saturation. I just assume that if you could check that none of the photosites had been overloaded, it should be possible to develop that file at a low exposure into an image without any burned-out highlights. But if you knew that photosites over a significant area of the image (rather than just a few specular reflections or light bulbs) had been overloaded, it would be obvious that useful information had been lost, and a less brightly exposed shot would be likely to produce a superior image.

    Chris
     
  9. EightBitTony

    EightBitTony Well-Known Member

    "none of the photosites had been overloaded"

    That's not the problem. Isn't the whole point of 'dynamic range at the sensor's base ISO' that it is about the difference between the darkest point and the brightest point. If you shoot in a situation which exceeds the dynamic range, you have lost either highlights or shadows. If you shoot within the dynamic range you can't have lost any detail?

    The luminance value is a relative number, it's not an absolute 'I was overloaded'.

    My understanding could be totally wrong.
     
  10. Andrew Flannigan

    Andrew Flannigan Well-Known Member

    If you really want to know what's in a RAW file look here http://lclevy.free.fr/cr2/ It's very long and quite boring unless you really need to understand it.
     
  11. ChrisNewman

    ChrisNewman Well-Known Member

    Thanks very much Glenn. I hadn’t appreciated that you can get better results by avoiding the limitations of a JPEG (probably because I haven’t yet got round to quality printing, and most work I’ve done on images has been to stay on the web). I’ll need to remember that for when I get round to reworking some of these shots I’m only wondering whether to preserve at present.

    By the way, I’ve no problem finding histograms, etc, of JPEGs.


    Chris
     
  12. ChrisNewman

    ChrisNewman Well-Known Member

    I should read Andrew Flannigans reference on RAW files. But at present, my understanding is that the dynamic range for the sensor follows from the ISO sensitivity setting. The dark end of the dynamic range of a sensor is set by base noise resulting from the random nature of arrivals if only a few photons are detected, perhaps exacerbated by imperfections in the amplification. At the light end, if you over-expose, either the photosite’s capacity to store charge will be exceeded, or at higher ISO sensitivities, the amplifier will multiply that charge value beyond the maximum the system is designed to count. So if you expose a shot by 1EV less than the sensor’s over-exposure level, you’ve lost a potential 1EV at the top end of the dynamic range, whilst at the bottom, noise prevents you from extracting any more detail from shadows that will be 1EV darker. But if I’m wrong about this, I’d welcome correction.


    Chris
     
  13. PeteRob

    PeteRob Well-Known Member

    They've added some cameras to the .CR2 internal data since I last looked at that link but it won't help your question.

    You are basically right. You have most information (dynamic range) at base ISO. Once you increase ISO by more than a couple of stops you are not gathering information that would saturate the photosites (unless you deliberately overexpose). In post #22 I hypothesised that what you read about recovering blown highlights from raw files was aimed at use of higher ISO values rather than base ISO.

    Back to your rapid screening. I don't know what in camera controls you have but if you set jpeg contrast to minimum that might pull in the histogram at the rhs. By the same account you could set the camera to Adobe RGB to see if that makes a difference. I don't quite "buy" why colour space should have anything to do with whether a photosite is overloaded or not but it might influence how the histogram is drawn. Those changes will make the preview look awful but if your main interest is chimping histograms that matters not.
     
  14. GlennH

    GlennH Well-Known Member

    I don't know if you're appreciating, though, that histograms in a raw convertor do not represent an 8-bit JPEG. Perhaps if the program defaults to camera settings the histogram will look the same with the same distribution of data, but at that point you haven't rendered the image, so you can place the colour in a "bigger box" as it were (e.g. ProPhoto RGB). Given that we cannot see some of the colour in the ProPhoto RGB gamut and that no monitor can fully display it, this is surely as generous an interpretation of a raw file as we could possibly need.

    The 8-bit histogram is meaningless at the raw-conversion point--in Lightroom the 0-255 values are forfeited in favour of percentages. Even if 65,536 possible levels could be plucked preemptively from a raw file, it would be impractical/difficult to interpret or edit them. This is what a 16-bit histogram looks like. It's full of gaps and minuscule representations of data, so you'd have to compress it anyway in the end to make any meaningful changes.
     
  15. EightBitTony

    EightBitTony Well-Known Member

    If the maximum value of luminance was 900, and the value in the RAW file was 900, how would you know that was clipped? You only know that by *looking* at the image. (I don't think that's actually a good model of how RAW files are recorded, but the point still stands).
     
  16. ChrisNewman

    ChrisNewman Well-Known Member

    JPEG colour channel values are reported on a scale of 0-255, so if I see any value from 0 to 254 I think that confirms that it hasn’t been clipped. If the value is shown as 255, there is a small possibility that that represents the true value, but it was probably clipped from a higher value that the format can’t record. Similarly, if a RAW file used a scale of 0-900, I’d assume any value from 0-899 was sound, but assume 900 had been clipped.

    Chris
     
  17. GlennH

    GlennH Well-Known Member

    @ChrisNewman

    I had quick scout about and found something that might be of interest to you: https://www.fastrawviewer.com/about-and-features
    It loads raw file previews quickly, but more significantly for you it gives the pixel count and percentage of overexposed/underexposed areas in the separate RGB channels. As well, you can pull out the histogram and make it as elongated as you like, so effectively it looks like a high-bit histogram and gives you an uncluttered visual. You can turn clipping displays on and off too.
     
  18. PeteRob

    PeteRob Well-Known Member

    Thanks Glenn.

    Fascinating link and only $35 odd if you want to play with histograms. My 2 takeaways from https://www.rawdigger.com/howtouse/beware-histogram for general use (i.e. not subjects where a single colour dominates the highlights like their red background) are:
    • turn the contrast right down in camera and keep the blinkie area small to use the in camera tools
    • use a spot meter and determine how much exposure compensation to give highlights for your camera. The rawdigger tool would help this. Means buying a decent spotmeter though.

    I might get the viewer for assesing raw files. For Canon I use the DPP quick checker before importing the survivors into LR but my Fuji files I have to go through the import to LR process.
     
  19. PeteRob

    PeteRob Well-Known Member

    I had a play. For quickly assessing RAF files Faststone Image Vewer seems better.
     
  20. GlennH

    GlennH Well-Known Member

    Though presumably Faststone doesn't use the raw data to create the histogram. As impressive as that is (unique, I believe), I'm still not sure how much of an advantage it offers over just selecting the image in Lightroom with the Melissa RGB histogram, which should encompass most of the exposure and colour capability of the camera sensor, even if the histogram is derived from the preview and not the raw data. I haven't noticed many nasty clipping surprises after exporting from Lightroom - the histogram usually looks as expected.
     

Share This Page