Xiaomi's annual imaging flagship unexpectedly "stumbles"?
Recently, the well-known professional camera evaluation agency DXOMARK released the imaging comprehensive evaluation score of Xiaomi 14 Ultra. With a score of 144 points, it ranks 13th on the list. Tied with Xiaomi 14 Ultra at 13th place is Huawei's Mate 50 Pro, released in September 2022. Even OPPO's previous flagship Find X6 Pro scored 153 points, ranking 10th.
It is worth noting that Xiaomi 14 Ultra has also become the lowest-ranked current-generation Chinese brand imaging flagship tested by DXOMARK.
(Source: DXOMARK)
After the results were announced, there was intense discussion online. Many netizens believe that even in terms of hardware quality alone, the Xiaomi 14 Ultra should not have ranked 13th. However, some users also pointed out that there are indeed some issues that need improvement with the Xiaomi 14 Ultra.
At one time, DXOMARK, as one of the professional evaluation organizations in the imaging field, was highly regarded by numerous smartphone brands. Its evaluation data often appeared at major flagship model launch events as a quantifiable indicator of imaging capabilities. However, in recent years, models that received high scores from DXOMARK have received mixed user reviews, such as the iPhone 15 Pro, which surprisingly scored 154 points.
(Source: DXOMARK)
This raises the question: are DXOMARK's rating criteria constantly changing, or are this year's imaging flagships truly heading in the wrong direction?
Setting aside imaging and focusing on user experience, Xiaomi falls short
DXOMARK's comments on the Xiaomi 14 Ultra are not too bad. In the camera test, it scored 144 points, with the main judge also acknowledging its continuous zoom capability. Deductions mainly focused on depth of field, portrait mode, which are closely related to computational photography.
As an annual imaging flagship, the hardware of the Xiaomi 14 Ultra is top-notch: the main camera features the Sony LYT-900 1-inch sensor with variable aperture support. For the ultra-wide, mid-range, and telephoto sections, Sony's IMX858 large sensor was chosen. With four similar sensors, it achieves better color consistency during continuous zoom, aligning closely with DXOMARK's assessment.
However, the issue arises in computational photography or perhaps in the photography experience itself, where DXOMARK did not give very high praise. For instance, in bokeh testing, DXOMARK finds Huawei Pura 70 Ultra's blur more natural, or in the preview interface, iPhone 15 Pro Max offers smoother performance.
(Source: DXOMARK)
In fact, as a professional evaluation organization, DXOMARK focuses on optical quality, color accuracy, and hardware fundamentals as well as software tuning even in the field of mobile imaging. In recent years, brands like Xiaomi partnering with Leica, OPPO with Hasselblad, and vivo with Zeiss have all embarked on stylized mobile imaging development. Compared to "true colors," capturing stunning shots on the go better meets the real needs of ordinary consumers.
Upon reviewing the detailed evaluation report, we also identified some issues. After collaborating with Leica, Xiaomi's imaging flagship only offers two modes in photo mode: "Leica Vivid" and "Leica Distinct." Nevertheless, both fall under stylized color grading. DXOMARK mentioned that in certain scenarios, the Xiaomi 14 Ultra has slightly lower exposure compared to its competitors, leading to deductions in this aspect.
(Source: DXOMARK)
Interestingly, DXOMARK has decided not to evaluate the Xiaomi 14 Ultra in Pro mode. Even in this mode, Leica's color tuning will not be involved, and factors like imaging style and exposure control will remain unaffected.
With DXOMARK incorporating more test items, user experience is now included in the image scoring criteria. The Xiaomi 14 Ultra's photo shooting smoothness has been deducted many points. From Xiaomi's perspective, while smooth operation is important, giving a higher weight to optical quality and software algorithm testing in the score calculation may be seen as unfair.
As a result, starting from 2022, most smartphone brands have stopped providing products for testing to DXOMARK. Many flagship camera phones, apart from popular models, have not made it to the list, such as the Xiaomi 14 and the vivo X100.
The diminishing need for "professional" evaluations
Xiaomi, Honor, and Huawei were among the first smartphone brands to send products for DXOMARK testing. In 2017, the Xiaomi Note 3 topped the DXOMARK smartphone camera performance ranking, and Lei Jun excitedly shared this achievement with Mi fans at the later Xiaomi Mix2S launch event.
DXOMARK faced criticism for the first time when the Xiaomi 9CC Pro and Huawei Mate 30 Pro initially received the same score, sharing the top spot. However, with the release of the Mate 30 Pro 5G, which had no sensor changes, it managed to surpass Xiaomi with a higher score. Despite the doubts raised, DXOMARK did not provide much response, leading to ongoing speculation, including rumors of "buying rankings."
At the Xiaomi 12S Ultra launch event, Lei Jun made it clear that they would not be sending the device for DXOMARK testing, signaling the end of the "professional score battle" among brands. By 2024, DXOMARK had received only a few devices for testing, indicating a potential decline in their ability to create rankings.
(Source: Xiaomi)
From a professional perspective, the evaluation criteria of DXOMARK itself are somewhat unpredictable. For instance, does the assessment of mobile imaging prioritize optical quality and image sharpness, or does it emphasize tonal style and user-friendliness? Moreover, why should computational photography be included in professional evaluations? On the other hand, DXOMARK has never disclosed an upper limit benchmark for scores, nor has it established specific standards for a particular test, leading to flagship devices consistently achieving higher scores once they undergo testing.
For example, when the Huawei P20 Pro first received a DXOMARK score, it scored 101 points, the highest at the time. However, the current highest score is held by the Pura 70 Ultra, with a total of 163 points. If this "drama" continues unchecked, it's possible that scores have already surpassed 200 points by now, or even potentially reached 300 points.
(Source: DXOMARK)
The demand for mobile imaging is becoming increasingly diverse, with different user groups having varying preferences and usage scenarios. The scoring systems of professional evaluation agencies are often based on standardized test scenarios, which may not fully cover the various situations encountered in actual use. Some users have higher color accuracy requirements, so overly stylized image tones may not be suitable for these consumers. Therefore, users tend to combine information from multiple sources when making purchasing decisions, rather than blindly trusting the scores given by "professional" organizations, in order to make the most suitable choice for their own needs.
Good reputation is better than high scores
In the development of mobile imaging, it is no longer necessary to rely solely on scores to gain market recognition; satisfying consumers is the key. Looking at sales data, the initial sales of Xiaomi 14 Ultra exceeded 100,000 units, leading the current generation of flagship imaging devices with a very high growth rate, showing outstanding sales performance in the global market. This clearly indicates that Xiaomi's imaging capabilities are recognized by many consumers.
Another device that did not achieve good results in DXOMARK is Samsung's Galaxy S24 Ultra. Despite this, the sales performance of this flagship "super-sized cup" is impressive. Not only did it receive over 6 million pre-orders during the initial sales period, but it also led to a significant increase in Samsung Electronics' stock price, with profits soaring in the second quarter.
(Source: Samsung)
On one hand, these large-sized image sensors are not inferior to any competitors in terms of hardware configuration. Equipped with high-performance sensors and advanced image processing technology, they are more than capable of handling various challenges in daily use. What's more, brands like Xiaomi, Vivo, OPPO, continuously enhance users' actual shooting experience through software updates and optimizations after the release of new models. Although these efforts may not immediately reflect in professional ratings, they significantly improve users' overall experience.
On the other hand, recent market trends indicate that the importance of user reputation is surpassing professional ratings. Various channels such as social media, tech bloggers, and user reviews provide consumers with rich sources of information. Users tend to rely more on real-world feedback to understand a phone's actual performance, rather than solely depending on ratings from professional institutions. Devices like the iPhone 15 and Huawei P60 have succeeded in the market due to the positive reputation they have built among users.
(Source: Apple)
Therefore, the already controversial DXOMARK took nearly half a year after the release of the Xiaomi 14 Ultra to provide a professional rating that fell short of expectations. This rating only represents their perspective from certain "professional" angles, such as poor shooting experience and insufficient highlight suppression. While these issues may indeed exist, they do not have a direct correlation with whether the product's imaging capabilities can be accepted by the public.
In any case, after years of iteration and evolution, the performance of imaging flagships from various brands can be considered quite similar. The differences lie more in stylistic processing or differing concepts, leading to some variations in usage. Users simply need to choose products based on their own needs. As for whether to consider the ratings from professional institutions, Xiao Lei suggests taking a look, but it shouldn't be the sole criterion.
