The Ubergizmo Camera IQ Benchmark

The Camera IQ benchmark is a mobile camera Image Quality (IQ) scoring system that gives an excellent indication of image quality (IQ) based on four “pillars” that everyone can interpret: Daylight, Low-light, Ultrawide, and Zoom photography.

See down below, after the detailed description of our benchmark, the COMPLETE TABLE with the smartphone list, and all their Camera IQ scores (direct link to the table at the bottom of this article).

This Image Quality benchmark comes in addition to our Camera HW benchmark, our proprietary algorithm that evaluates the mobile cameras solely based on their hardware’s technical data. To date, we have seen a good correlation between our Camera IQ and Camera HW benchmarks.

Consumers now have an easy and meaningful way to evaluate a phone’s camera performance under real-world conditions. That’s how we review mobile camera quality now.

Consumer Photography research often groups mobile camera users in two categories:

1. Photo producers (photographer, camera user/buyer)
   1. Sub-group: casual photographers (large consumer majority)
   2. Sub-group: serious hobbyist
2. Photo consumers who only watch photos, usually without any context of how or why they were taken.

Our mobile camera scoring system and our editorial reviews are written for the casual Photo Producers, even though serious hobbyists should find our conclusions interesting.

We use different visuals to quickly show the most relevant Camera IQ information such as the score, the phone’s name, and market position when it was introduced (Year/Month).

Why did we create the Camera IQ score?

This scoring system was originally created for internal use. We did it to make better smartphone reviews. From a consumer standpoint, and according to OEMs we talked to, smartphone camera performance is the single most important item when considering a new phone, even ahead of battery life or display quality.

We also have the Camera HW score, which is another proxy for camera performance, based only on camera technical data. It’s great to have such a proxy for just-announced phones, or even for foreign phones we’ve never had our hands on. There is a strong correlation between camera hardware performance and photo quality.

From our editorial review process, we have created the necessary software and algorithms/methodologies to analyze image data and create both cameras scores. It was a lot more work than we bargained for, but it was worth it. Consumers can now search for the best overall camera, the best low-light camera, the best ultrawide camera – and more!

The Camera IQ score pillars

Because numeric scores boil things down, they don’t tell the whole story. To partially remedy this, our IQ score has user-friendly sub-scores, which we consider to be today’s mobile photography “pillars”.

Each covers a use case that everyone is familiar with. By glancing at them, you can understand the nuances much better and choose what’s best for you. When a full camera review is available, you can also read it and see use cases that illustrate the key points of the scores. At the moment those Camera IQ pillars are:

1. Daylight photo (bright-light)
2. Night photo (low-light)
3. Ultrawide photo
4. Zoom

With our algorithm, we generate a score for each pillar before calculating the final Camera IQ score. The scores are absolute, with no upper limits.

Objective camera tests

There are many objective image-quality camera metrics, and that’s exactly what we are looking at. Objective metrics mean that “more” or “less” of them means “better” or “worse”.

Cameras go through a process in which image analysis is based on properties that can be rated as objectively as possible. Rating is separated from personal preferences of “style” of photography (more on styles below). In some cases, software analysis is appropriate, other times, perceptive analysis is preferred.

It is important to point out that any algorithm which computes a score does contain some subjective elements. Image properties that are deemed to be more important may receive a higher consideration, weight, and priority.

In fact, the algorithm is the most important part of any benchmark, as it has more influence on the score than data accuracy fluctuations. For example, and at its simplest, it is like having two algorithms that are “addition” and “multiplication”. Feed them two identical and ultra-accurate pieces of data A and B: A+B and AxB will produce radically different outcomes.

As algorithms get more complex, so is the potential for having vast outcome differences. There is no scientific consensus to rate photo Image Quality, and the photography industry has debated this since photography was invented.

However, you can see the quality of our score algorithm from the quality of its outcome. We have had great feedback from industry-insiders about both the Camera IQ Score principles and its results.

Things such as user interface, filtering styles, and subjective elements are very interesting points but are not be included in the Camera IQ score since they are more related to the camera “user experience” than its image quality. Both are important but are best evaluated separately.

Reality is the anchor. Predictability is satisfaction. Context is everything.

Context Shot, what the scene looks like

Honor View 10 photo

The prospect mobile camera owner (the photo producer) is the first person that Image Quality has to satisfy. We found that there is a very strong correlation between the photographer’s satisfaction and the Camera’s ability to (faithfully) capture what the photographer is looking at. Users want to feel in control.

When it comes to colors, it is important to separate “quality” from “style”. Since quality can mean many things to many people, we use reality as an anchor and always mean image data quality, when talking about “quality”.

Each lens/sensor pair will induce slight color shifts, which is why pro-photographers will use in-scene measurements to color-correct photos later. It is unreasonable to expect consumers to do such things in their daily lives, so we expect cameras to do a great job of capturing what users see with their eyes.

We refer to a good representation of what our eyes see because it is fundamental to have a good scene intelligence. We don’t expect cameras to be perfect, but this matters – a lot. We will often point out if a camera captures what we see, or that another one might not. Some cameras over-filter images to the point that data is lost, and that could upset photographers for good reason.

Night photography

Context photo: this is how the scene looks like as if you were there

When it comes to low-light photos, that’s often where high-end cameras set themselves apart. It is probably the single most desired feature in mobile cameras. As of late, there has been a trend to artificially “brighten” photos using color filters, which is the easy way to try impressing users. But the truth is: with low light photos, brighter is NOT always better.

Context photo: looking at color spectrum detail

Handset A, looking at the same color spectrum detail

Instead, what we’re interested in is how well the scene’s detail, dynamic range, colors, and light intensity have been captured. Any good night photo can be filtered later to the liking of the user, but an over-filtered night photo can irremediably remove shadows, volumes and contrast and shrink your creative options drastically.

Ultrawide photography

Smartphones have traditionally shipped with a single focal length (~24-26mm) because camera modules are too small to accommodate variable focal length lenses. However, being able to switch to ultrawide (~16mm) is an amazing tool that many compact cameras had for decades.

That’s why we consider ultrawide photography to be one of the necessary pillars and mobile camera with support for ultrawide photography will have an advantage.

Zoom photography

Context Shot, we’re going to zoom on the street art

Every single camera can zoom thanks to digital magnification. Sometimes, people like to zoom for framing purpose or just to take a close-up photo. It’s not always possible to move closer to the subject, so zoom can be useful at times. Having fancy algorithms and optical zoom lenses can be incredibly helpful to get sharper zoomed photos. Zoom is also a fundamental aspect that we look at.

NOT included into Camera IQ (for now), and why

To keep a score to the point and easy to interpret, we wanted to keep its core as straightforward as possible. As such, a number of things were intentionally left out of this first version.

1. Flash
2. Ultra-long exposure (Night Mode)
3. Bokeh
4. AF speed
5. Selfie IQ
6. Video IQ (under consideration)

Flash

A camera can be a powerful tool, but for smartphone cameras, the overwhelming majority of users dislike using a flash, and photos are overwhelmingly taken without a flash.  At the moment, we consider that users have a very small interest in flash performance, so it’s not included as one of our pillars.

Ultra-long exposure (Night Mode/Night Sight)

While ultra-long exposure is an amazing tool, we consider it to be outside of what people think low-light photo performance should be. Long exposure can make up for a small aperture, lens quality or sensor size, but only to a point. However, it’s not clear how often users will accept to wait 4 to 5 seconds to take a picture.

We’ll discuss with users, OEMs, and others who want to join the discussion to look at the usage trends. Our Camera IQ score will evolve over time.

Bokeh (out of focus Blur)

Bokeh is great, but it is also quite subjective. At the moment, we have chosen to leave it out of the score because we were not yet satisfied with the Bokeh scoring options that would have hindered more than helped the IQ rating. We’re open to including it in the future if there’s a strong demand for it.

Autofocus speed

It’s true that AF speed can be important in cases such as action/sports photo or with fast-running kids. However, Autofocus is mostly a succeed/fail element. When successful, it has no importance in regards to image quality. If a camera consistently fails AF, the blurry images will affect the IQ score anyway – speed in itself is a user experience factor, but not a great proxy for image quality.

Selfie camera

Selfies are a completely different use case for a mobile camera. With different hardware, software (filters) and expectations, we do not want to lump selfie IQ into the rear camera IQ – it just didn’t make sense to us, and to a lot of people we talked to.

We discussed building a Selfie score, but in some places, people expect to have a lot of filtering and styling, which makes it very tricky to have an objective score. We’ll keep looking for a good way to build this. For now, perhaps the Camera HW score might be a better proxy.

Video

Video IQ is usually related to the performance of the optics and overall image processing capabilities of the handset. However, we agree that things such as EIS (Electronic Image Stabilization) and details about how 4K or super-slow-mo are captured can be of importance to some users. We’re gathering data on that topic, and we are thinking of having a video score later.

Keep an eye out for new scores!

Thanks for your interest in our Camera benchmark. We really try to provide you with insightful and useful information that will help you get the best camera for your needs and money. Keep in touch via Facebook, Instagram, and Youtube for updates.

Happy Photography!

Note that you can re-sort the table by clicking on any of the headers.

Filed in Cellphones >Photo-Video. Read more about Benchmarks, Editorspick, Photography Concepts and Smartphone Reviews.