benchmark
Latest
Facebook's new benchmarking system asks humans to interrogate AIs
Facebook’s AI research (FAIR) lab has taken a new approach to benchmarking: it's putting humans in the loop to help train their natural language processing (NLP) AIs directly and dynamically.
Oppo is the latest to be caught cheating on phone benchmarks
You can add another big name to the list of phone makers found cheating on benchmarks. UL Benchmarks has delisted Oppo's Find X and F7 phones from its 3DMark charts after testing from itself and Tech2 revealed that both devices were artificially ramping up processor performance when they detected the test by name. Oppo acknowledged that it always stepped things up when it detected "games or 3D Benchmarks that required high performance," but claimed that any app would run full bore if you tapped on the screen every few seconds to signal your actions.
Huawei was caught cheating on phone benchmarks (updated)
You'd think smartphone manufacturers would have learned their lesson after being caught cheating multiple times, but that's apparently not the case. UL Benchmarks (aka the Futuremark team) has delisted Huawei's P20 Pro, P20, Nova 3 and Honor Play from 3DMark's charts after internal testing and an AnandTech exposé showed the devices ramping up performance whenever they detected the public versions of benchmarking apps. These were not subtle differences, either, with results up to 47 percent higher than they were with private test variants Huawei couldn't catch.
Uber pursues an investment deal limiting its former CEO (updated)
After plenty of negotiation, Uber is about to get a valuable investment... in return for a few favors. Sources speaking to Bloomberg, Recode and TechCrunch have learned that Uber and SoftBank have finalized a deal that should see SoftBank and an investment group pour $10 billion into the company ($1 billion directly, $9 billion buying existing shares) while limiting the control of former CEO Travis Kalanick. He'll still have some influence, but he won't retake the throne -- and it'll end a nasty legal battle, too.
Galaxy Note 8 and iPhone 8 Plus tie for top spot in camera test
Like it or not, DxOMark is currently the go-to tester for smartphone camera quality. Companies will even base their marketing around its scores. As such, it's a big deal when the outfit declares a new winner... and it just declared two. DxOMark has given Samsung's Galaxy Note 8 an overall score of 94, putting it in a tie for the lead with the iPhone 8 Plus. No, that's not going to trigger endless fan wars, is it? Of course, diving into the scores reveals that the devices reached their scores through different means.
NVIDIA brings benchmarking to VR
If you've ever wanted to make sure that your PC's virtual reality setup is running as well as it possibly can, you now have the software you need. NVIDIA has released its promised FCAT VR tool, which can check for dropped or synthesized frames, warp misses and other signs that a VR title isn't running smoothly. It'll even show you how a VR program runs at different detail levels, so you'll know if you need to tone down the graphics setting to get optimal performance.
The Engadget Podcast Ep 27: American Tune
Managing editor Dana Wollman and reviews editor Cherlynn Low join host Terrence O'Brien to discuss the biggest tech news of the week. First they'll debate OnePlus' benchmark scandal, then try to sort out why the ACLU is partnering with startup incubator Y Combinator and take the "artists" behind the Tinder for baby adoption Kickstarter to task. Finally it's another week of Trump talk as the panel addresses the impact of the recent immigration ban on the tech industry and how the political climate is impacting our social media habits.
OnePlus 3T caught cheating on benchmarks
No, the days of smartphone makers cheating on benchmarks aren't over yet. XDA has discovered that the OnePlus 3T artificially boosts processor clock speeds when running certain benchmark apps, keeping its CPU cores running at higher rates even when there was virtually no processor workload. This has apparently been a staple of HydrogenOS (OnePlus' interface for the Chinese market) for awhile, but only showed up in OxygenOS (the software for other parts of the world) in community builds ahead of the Android 7.0 Nougat upgrade.
Valve uses 'Portal' robots to find out if your PC is VR-ready
In mere months we'll finally have two of the "big three" virtual reality headsets out in the market. While your bank account might be more than capable of writing the HTC Vive and Oculus Rift pre-order checks, can your PC's performance actually cash them? Valve has a test for that. The SteamVR Performance Test is just over two minutes long and takes place in a Portal-inspired non-interactive testing facility. Naturally.
Smartphone chips are about to get better at gaming, Twitter and Netflix
There's been a lot of talk about how phones from certain manufacturers can sense when you're running benchmarking software and will artificially inflate their performance accordingly. It's been a problem for some time now, but the industry appears to be finding its own ways of dealing with the issue. ARM, the outfit that designs the bulk of the smartphone industry's chips, has teamed up with British startup GameBench to make benchmarks not only more honest, but also a lot more understandable.
Samsung lawsuit claims that NVIDIA's benchmarks are misleading
Samsung definitely isn't taking NVIDIA's first patent lawsuit lying down. The Korean tech firm has countersued NVIDIA not just for allegedly infringing on six patents, but for leading buyers astray with benchmarks for the Shield Tablet. NVIDIA is supposedly trying to "confuse customers" by claiming that the slate's Tegra K1 processor outpaces the Exynos 5433 chip in the Galaxy Note 4; regular benchmarks show that's not true, Samsung claims. The suit also accuses PC vendor Velocity Micro of violating two additional patents (for a total of eight), since NVIDIA's graphics cards play a heavy role in its lineup.
Who'd have thought the Power Mac G5 made a good bench?
Like it or loathe it, you have to admit that the design of the Power Mac G5 was a very clever way of getting around the system's legendary thermal issues. It was no surprise that the ol' cheesegrater was kept around for the Mac Pro, at least until last year's solid-state revolution. But what of the numerous G5 chassis that are now lingering in attics, skips and warehouses? If you don't want to gut one to use for your own high-end PC, then Klaus Geiger is more than happy to turn them into furniture. As part of his Benchma[®]c project, two G5 cases and a plank of Walnut is all you need to make a pretty nifty park bench. There's more images down at the source, but you'll have to excuse us, as we're just off to put our collection of Rodrigo Alonso furniture on eBay.
iPhone 6 tops competition in benchmarks, battery life
Apple's annual iPhone update always brings with it a nice bump in specs, and this year is no different. The iPhone 6 and 6 Plus are currently being put through their paces by the benchmarking gurus at AnandTech, and the preliminary results show that Apple's newest pair of smartphones are among the best of the best on several popular graphical tests as well as web browsing battery life. In nearly all the tests -- check out the full rundown to get the whole picture -- the iPhone 6 twins hold the top two smartphone scores, with the exception of a physics benchmark where they only narrowly beat out the iPhone 5s and lag behind the rest of the competition. It's unclear why that particular test came out the way it did, while the iPhone 6 and 6 Plus dominate so handily in the rest, but the testing parameters have been questioned by commenters. It's interesting to note where the iPhone 6 and 6 Plus trade places from test to test. With slightly less screen real estate to worry about, the iPhone 6 has a small advantage and manages to eke out its bigger brother a few times along the way. The two remain very close in nearly every test though, so the slight differences in performance shouldn't factor into your choice either way. [Graph via Anandtech]
AMD wages benchmark war on Intel's tablet chips
For all the success AMD has been having in the console and PC graphics spheres, none of it has really translated into the world of Windows-based tablets and ultraportables. The chip maker insists that's going to change in 2014, however, and it has released a number benchmarks showing that its latest processors have a lead not only over AMD's previous generation, but also over the Intel chips that currently reign supreme in these form factors. We won't bore you with a gabble of numbers when you can check out charts for yourself in the gallery below, but the main curiosity here is probably the 4.5-watt tablet platform, known as "Mullins." This replaces last year's Temash processor, which had impressive gaming skills but failed to catch on in the market. AMD's in-house scores suggest Mullins offers much better performance per watt, with the new A4 Micro-6400T achieving a 15 percent lead over Intel's Bay Trail T (the Atom Z3370) in PCMark 8 -- a lead that could potentially be significant enough to bring the chip into more slim-line (and passively cooled) Windows 8 machines. What AMD doesn't reveal, however, is whether devices equipped with this A4 chip will have comparable battery life to Bay Trail, so it's all academic until actual, commercial devices come around.
Benchmarking the new HTC One: less cheating, better performance
There are actually two "Ones" that launched this week. The star attraction is undoubtedly the HTC One, but let's not forget the brand-new Snapdragon 801 running under its hood: a cutting-edge processor that will also power the Sony Xperia Z2 and the Samsung Galaxy S5, but which happens to have reached the market first in HTC's flagship phone. This chip represents a significant upgrade over the Snapdragon 600 in the old One, promising a hat trick of better all-round performance, more fluid gaming and longer battery life, and these are precisely the claims we're about to explore using a combo of benchmarking apps and real-world tests. At the same time, HTC has suddenly decided to come clean on the issue of benchmark cheating, which makes it a bit easier for us to trust what the numbers are telling us.
Samsung devices no longer boosting benchmark scores after Android 4.4 update
Those artificially inflated benchmark scores Samsung devices were getting last year? They've been patched. According to new tests run by Ars Technica, Samsung devices running Android 4.4 no longer boost CPU speeds during benchmark tests. In July, Exynos variants of the Galaxy S 4 and Note 3 were caught running their CPUs at an unsustainable max speed during benchmark programs, bolstering their total score by as much as 20 percent. Samsung suggested this was normal, claiming that varied processor frequencies were designed to provide an "optimal user experience." The firm never clearly explained if the phenomenon was a feature, fault or foul play, but it's over now: Devices updated to the latest version of Android are apparently running clean.
Which Android phones win at gaming?
A simple kind of happiness reigns in the world of mobile gaming. The app stores are brimming over with four-star ratings; popular titles are making billions of dollars for their creators; and folks on the morning commute seem generally content with what they're playing -- sometimes destroying rows of fruit, sometimes rows of candy. Few of us expect or demand anything deeper on a tablet or smartphone, and surely none of us would be crazy enough to choose our next handset based solely on a criterion as narrow as 3D gaming performance. Right? Well, yes and no. Things certainly get more complicated when you look at the cutting edge -- especially on Android. The industry is pushing the boundaries of what a mobile game can be, what a mobile processor can do and what an Android-based gaming device can look like. And as ambitions escalate, so do the risks. The old nemesis of fragmentation means that certain titles may stutter, or cause excessive battery drain, or fail to run at all, so that those glowing reviews turn into one-star complaints and customers go back to playing it safe. The industry is pushing the boundaries of what a mobile game can be That's why we reckon it's a good time to take stock -- to measure how well some current and older Android devices handle a sample of graphically demanding games. We've got cold, hard numbers to show you, which should help to pinpoint the most future-proofed products. We also have a secondary aim, which is to set a benchmark against which we can judge the next wave of hardware, soon to be announced at Mobile World Congress. Indeed, it's already becoming clear that, from a gaming perspective, smartphones don't always progress in the manner or at the rate that we might expect.
Oppo's next smartphone due in March with Quad HD and 1080p display options
You may cringe at this "Find 7 [are] coming" line, but it's actually an intentional typo, as Oppo's releasing its Find 5 follow-up with two screen resolutions. You see, back in December, the Chinese company teased that its next flagship phone will be using JDI's (not LG's) 5.5-inch 2,560 x 1,440 (538 ppi) panel, which is even sharper than the 6-inch counterpart on the Vivo Xplay 3S. Then to confuse us, earlier this week a Find 7 benchmark showed up on GFXBench with a 1080p display instead, along with a Snapdragon 800 SoC and Android 4.3. We've since checked with our own sources, and we can now confirm that said device will indeed be offered with more than one display option to please everyone: 2K aka Quad HD for the spec chasers, and 1080p for the humble users. We also understand that other specs may differ between the two versions, so if all goes well, we'll take a closer look at the Find 7 variants in Beijing's 798 Art Zone on March 19th.
Google tests the performance limits of D-Wave's quantum computers
We've long known that D-Wave's quantum computers are specialized tools rather than Swiss Army Knives, but just how good are they at their intended tasks? Google has just conducted some benchmarking to find out, and the short answer is that these systems are very good -- but they have definite limits. A current-generation D-Wave 2 is about 35,500 times faster than a generalized problem-solving computer when both are running standard software. However, some of that advantage disappears when a general-purpose computer runs code that simulates quantum computing. While D-Wave's hardware is better at dealing with structured code, it runs neck-and-neck with the "fake" system when tackling random problems. Not that Google is feeling much in the way of buyer's remorse. It believes that further tests could see the D-Wave unit come out ahead, and future quantum machines should make it harder for conventional PCs to catch up.
Major benchmarking service delists Samsung and HTC phones it suspects of cheating
Oh yes, things are really starting to kick off in the arcane world of smartphone benchmarking. First, there came clear evidence of phone makers manipulating scores in apps like AnTuTu and GFXBench, and now a more mainstream benchmarking company, Futuremark, has publicly delisted specific Samsung and HTC phones that it suspects of cheating. Futuremark says that the devices in question -- including the Galaxy Note 3, HTC One and HTC One Mini -- fail to adhere to the fairness policy, which requires that a device treats its 3DMark app just as it would treat any other app, with no tailor-made bursts of performance designed to achieve artificially high scores. Clearly, this bad behavior is just as endemic as we originally feared, so benchmark apps either need to toughen up, as Futuremark appears to be doing, or they need to find entirely new ways of measuring performance.
