Abe R. Ration’s lens and camera blog
DPReview is the largest and most pretty digital camera review site in the internet. But how good some of their comparisons really are. Should we take their comparison pictures with grain of salt?
I will be taking advantage of the data DxOMark provides. DxOMark measures camera’s ISO calibration – this makes it easy to see how much light a camera can collect at each ISO setting relative too each other. I will call the DxOMark “ISO” metric DxISO from now on to make it easy to distinguish from the camera’s nominal ISO setting.
Let’s stat with a practical example with four cameras. Three of the cameras are contemporary full frame cameras from different manufacturers, each equipped with image sensor from different manufacturer as well. The fourth item in the comparison is the Micro 4/3 camera with arguably the best output image quality of them, Olympus OM-D E-M1. As the small sensor camera from Olympus has approximately 1/4 of the image sensor surface of a full frame camera, the Olympus is set to ISO 3200 and the full frame cameras to ISO 12800. If the cameras were idealized, this would cause both the small sensor camera and big sensor camera to collect the same amount of light, thus the comparison would give valid results of relative sensor performances per area (i.e. how the different sized sensor would perform if they were of the same size).
I picked the Olympus because Olympus uses quite different ISO calibrations from most other camera manufacturers, thus this serves as a practical sample of equality and possible pitfalls in comparisons.
Doing first comparing
Here is a crop of DPReview comparison image:
The images have been normalized to same output size (“print”) as we’re comparing images not individual pixels.
It would seem like the Olympus equals the Nikon D4s and were superior to the two other full frame cameras relative to the image sensor size.
To see if that is true we need to see if the exposures of the cameras are identical. Let’s inspect the cameras’ exposure parameters first:Olympus 1/50s, f/5,6.
Olympus
1/10s
f/5,6
Nikon
1/50s
f/5,6
Canon
1/40s
f/5,6
Sony
1/50s
f/5,6
To have a proper comparison the exposure parameters should provide equal exposure, and the Olympus should have four times the exposure of the others (as we’re comparing relative, not absolute performances). Looking at the table we might come to the conclusion that the Olympus and the Canon provide a good comparison with each other, while that the other two would be slightly underexposed in comparison making them handicapped relative to the other two.
If only it were this simple
There are two problems we have – one about the true exposure used, and the other about raw image conversion. This article is about former, but one should also consider that even using the same developing parameters on the same raw conversion software the settings “under the hood” may be quite different.
Before we only considered two of the three exposure parameters. If DPReview used a constant ambient light for their test images, we might now happily consider this Olympus to be very slightly better performer than this Canon relative to the sensor size. Regarding the other two we could not make valid comparisons to Olympus or Canon.
DxISO of the Olympus at ISO 3200 is 1900. DxISO of the Nikon, Canon and Sony at ISO 12800 are respectively 9651, 10052, 9109. If the numbers were the same, it would mean from the point of view of this comparison, that all the sensors would collect the same amount of light relative to sensor area. If the number is smaller, the sensor is given more light, is the number is bigger, the sensor is given less light. To consider the imaginary sensor of Olympus camera’s sensor’s quality but full frame size, we need to multiply it’s DxISO at ISO 3200 by four –> this give us 7600.
Before we draw any conclusions, we need to make sure that the raw-data itself is equally saturated. I used Lightroom to check the histograms of the relevant raw files:
Close enough.
Conclusion
DPReview alters the light levels when they do their studio photography tests. This makes objective comparison suboptimal and in extreme cases the results do not represent camera’s relative imaging performances in realistic manner. As they don’t tell what the exposure is in each case the comparison shots are even less useful than they could be. In this example it is clear that the Olympus OM-D E-M1 is given about 32% more light than the Canon 6D in this comparison – this makes it quite impossible to get an objective view on the relevant imagers performance by using DPReview output image comparisons.
Below is a small list of DxISOs of several cameras/sensors for ISO 200. The Olympus in test had DxISO of 122 at this ISO, while the Canon 6D has 153. It is clear that there are even bigger disparencies in DPReview comparison images than the ones presented above.
Note that I assumed that the sensor in the Olympus is four times smaller than the full frame sensors in question. Actually it is only about 3,83 times smaller, thus if one wants to compare the relative performance “per area” of the Olympus more accurately, this would have to be considered. In practise it would make the Olympus perform (“per area”) very slightly worse than it would appear if one simply follows the facts and figures on this page.
