Android vs. iPhone in 'flawed' mobile browser performance test
Post edited to clarify that the browser testing is not representative of Safari performance, and included Blaze response to CNET. –Ed.
Blaze Software, a Canadian software company, today released the results of what it calls a "definitive" research effort to discover "which [mobile] browser is really faster from a user's point of view." The study concluded that Android's browser is 52% faster than the iPhone's.
Before you trade in your iPhone for a device powered by Android, The Loop suggests Blaze's study is "flawed." According to its report, Blaze's testing methodology relied on "custom apps, which use the platform's embedded browser. This means WebView (based on Chrome) for Android, and UIWebView (based on Safari) for iPhone."
As we've been hearing from developers of iPhone web apps over the past few weeks, Apple's improvements to the Mobile Safari JavaScript engine and other rendering speedups have not been extended to the internal browser tool used by apps, nor to standalone web apps that are pinned as icons to the home screen. It's not yet completely clear if or when the Safari performance boost will make it to the embedded browser view; John Gruber cites some security-related concerns that may be involved.
The tests don't reflect performance of the official web browsers included with each platform. UIWebView did not include this performance boost; it may be "disingenuous" to conclude Android beat Safari, according to The Loop. Using an embedded browser is not the same as using the official browser where customers spend the most time interacting with websites.
"Obviously someone is looking to make a mountain out of a molehill," Gartner analyst Michael Gartenberg told The Loop. "It's not an apples to apples test." Apple's Natalie Kerris was equally dismissive, speaking to CNET: "Their testing is flawed. They didn't actually test the Safari browser on the iPhone. Instead they only tested their own proprietary app, which uses an embedded Web viewer that doesn't actually take advantage of Safari's Web performance optimizations." Kerris also noted that even without the true Safari match-up, the testing only showed about a second of difference browsing pages.
Blaze's CTO Guy Podjarny admitted to CNET that the testing methodology made an invalid assumption that the embedded browsers would work as fast as Safari: "This test leveraged the embedded browser which is the only available option for iPhone applications. Blaze was under the assumption that Apple would apply the same updates to their embedded browser as they would their regular browser. If this is not the case and according to Apple's response, it's certainly possible the embedded browser might produce different results. If Apple decides to apply their optimizations across their embedded browser as well, then we would be more than willing to create a new report with the new performance results."
Even so, the results of Blaze's research should still disappoint Apple's fans. Apple touted significant web technology performance gains in its latest iOS release. It seems reasonable to expect these gains to appear simultaneously in both the Safari browser and the underlying UIWebView framework used in nearly any app that relies on web technologies like JavaScript.
Blaze's researchers built custom apps to compare the iPhone 4 and Google Nexus S using websites from Fortune 1000 companies. Each site included in the test was loaded multiple times over several days using a Wi-Fi connection. The final results reflect a median benchmark from over 45,000 page loads.
"Android 2.3 was 52% faster than iPhone 4.3, with a median load time of 2.144 seconds vs. iPhone's median load time of 3.254 seconds," Blaze reports on its blog, adding, "Android was faster than iPhone in 84% of the tested websites, and iPhone beat Android in 16% of the races. Android...provided a faster browsing experience 4 times out of 5."
