How an AT&T smartphone comes to life: behind the scenes (part two)

By now, you're probably getting a pretty good idea at what goes into the development of a groundbreaking smartphone. In part one we followed the development of the Motorola Atrix 4G, discussing the process of how AT&T decided upon a design and collaborated with Moto to get the ball rolling. Now, we conclude the series by covering the intense testing process, getting the device prepared for launch and updating the phones after they're released.


Every facet of the phone's development is important, but making sure the device actually does what it's supposed to do is understandably mandatory. AT&T praises itself for having some of the highest standards in testing, though it's impossible for us to verify without being able to compare notes with the other US carriers (Sprint, Verizon and T-Mobile, we await your valued input anytime you wish to divulge this information).

The lab cycle

After all of the collaboration back and forth between AT&T's and Motorola's product teams, doesn't it feel like the Evora should be a polished product by now? Nope, not even close. It was September 2010 when the phone reached the testing labs. Making it to this stage was no small feat in and of itself, but the device still had a long road ahead of it. Think about it this way: AT&T wanted to have the product ready to ship in time for the holidays, and Black Friday was a mere two months away. The LapDock hadn't even been sent to testing yet at this point, which goes to show how much was left on the team's plate before the phone would be ready to hit shelves. We know what it's like to have an impossible deadline, so we can relate.

The carrier wants its test devices to be completely defined, with all of the specs as close to final as possible -- and it requires that the OEM gets its handsets validated by a third party to ensure all of AT&T's quality metrics are met.

After reaching the labs, the Evora had to endure thousands upon thousands of test cases. AT&T has the ability to mimic and simulate virtually any type of network condition or environment that it can possibly think of, and runs the device through all of them to see if it can reproduce any issues. An average test lab contains 1,200 servers and about 100 cell sites, which enables them to simulate specific cities, frequencies and technologies. Need to see how the Evora operates on 1900MHz HSPA+ in Portland, OR? No problem. What about 850MHz UMTS service in Baltimore? Easy as pie. But why would the signal be any different in these cities? Because each individual market typically uses a specific infrastructure provider: AT&T contracts companies such as Ericsson and Alcatel-Lucent to build out the carrier's network, but each one is chosen to do so for a given city or market. The variety of providers adds to the complexity of the network and its operation, and it's up to the testing lab to mimick every possible scenario a customer may face -- no matter how unlikely -- to make sure the phone isn't going to act wonky.

Every simulation's run in a copper cage -- no bigger than your run-of-the-mill walk-in closet -- that's capable of blocking out all outside signals so as to not interfere with the tests (or leave them out in the open to be discovered by curious folks, for that matter). These labs also run tests on firmware updates, radiation, audio quality and any accessory that gets sold in AT&T retail stores. They've set up a bug tracking system that both companies' teams have access to; bugs are assigned a severity level as a method of prioritizing phone issues that need to be resolved, and the process to close out each individual bug is incredibly detailed and closely tracked.

Severity level one is anything that negates service, and absolutely none are acceptable in a final product. These types of issues aren't simply referring to things like dropped calls or network frustrations, though that certainly comes into play in the labs. Anything that prevents service or possible use of the phone is slotted into the top echelon of priority. To give us an example of something that's considered service-negating, Dante showed us one of the Evora units used during testing and had us go into the web browser. Once there, he asked us to try pressing the home key -- and nothing happened. The inability to properly use all of the buttons should be a top-priority fix, naturally.

Severity level two issues are those that are seriously frustrating to the user, and affect their enjoyment of the device: copy and paste doesn't work properly, the phone won't let you switch calls or perhaps the speakerphone doesn't work quite as well as it should. Too many infractions on this level will pause the phone's progress and it can't continue until they're fixed, though one or two penalties might be pushed through if the OEM commits to fixing it as soon as possible. Good to hear that these types of issues are taken care of before reaching final approval; could you imagine owning a phone that wouldn't let you switch or merge calls?

Finally, a level three issue is basically an improvement ticket -- a small bug that AT&T would like fixed in a post-launch maintenance release, and is something to be addressed in future products to ensure it doesn't show up again.

Usability testing

The usability tests for the Evora began in October, which means they were run at the same time the device was still in the lab. Each product goes through a slough of usability tests to help the two companies learn more about how people interact with it. The teams watch multiple people as they use the device to see if they can use it easily and comfortably. Can testers navigate through a certain menu structure quickly enough? Are they tripping up somewhere on the phone? If anything is found that adversely affects the user experience, the teams try to incorporate new things into the phone to make sure it doesn't keep happening. "If we look at incorporating best practices and improving user experience and take out extra steps, the customer becomes satisfied and becomes recommenders," Chris told us. As mentioned earlier, we have a difficult time believing that testers were completely happy with MotoBlur, but at least the UI has improved since the Atrix was launched.

There are some areas of the user experience in which only a few people seem to trip up on. When these issues arise, the team begins to look at other avenues to take care of the concern. For instance, is it something that could be addressed by adding a small blurb into the Quickstart guide that comes with the phone? If not, that particular case -- and others like it -- is worked through in one of the Evora's post-launch maintenance releases and incorporated into any future Motorola phone with similar features.

Lapdock and Webtop

Motorola's unique Webtop environment and accompanying laptop dock were born alongside Evora and given the codename Virgil to differentiate its development. It was going to be a brand new product, a concept that nobody had really tackled head-on before. And not just that, it was the perfect fit for AT&T and Motorola's vision of a groundbreaking device that would completely change the industry. Because it was such an unknown, both companies were going to be taking a huge risk by bringing it to market, because the costs involved in developing the laptop dock and the full Webtop experience were astronomical. How would it be used? What direction should we take the vision? What value will this be to customers? Can you get this thing to do what you need it to without having a PhD?

Since the LapDock was primarily geared toward the business professional, AT&T needed feedback from legit sources to get it right. On September 15th, the carrier invited CIOs from ten different companies to meet together under a Non-disclosure Agreement (NDA). These executives had no idea what they were going to see; it could've been a data card, for all they knew. The unveiling consisted of a few carefully crafted demos, since the Lapdock was so early in its development that not everything was working properly. But the mission was a success: the idea of incorporating Citrix functionality was brought up at the meeting, not to mention a few other additions that were either included in the first-generation models or pushed to later versions (details on those other additions weren't given). Citrix made it into the final first-gen product, but barely made the cut.

The LapDock made its way into the labs near the end of September, not even two weeks after the CIO meeting. Again, the device met AT&T's rigid stability requirements, but it still had plenty of bugs to work out. The lab would find issues and send the trouble tickets over to Motorola, which would then be addressed by spinning new software updates to the labs and repeating the cycle over and over.

Virgil was ready for usability testing in October, but it was still such a secretive process to this point that none of the internal teams even knew about it -- it had only been on an exclusive "need to know" basis so far. So, NDAs were issued to the usability testers, which is a pretty burdensome deal. Why? Not just anyone can be given access to something with such a high level of confidentiality; Dante had to go through a vetting process to determine who would be offered the NDA, and the lucky contestants were required to read and acknowledge it before any more progress could be made on the project.

The testers were faced with a rather tough challenge when Virgil showed up. Here was a completely new product that they'd never seen before, which meant that a full set of test cases had to be thought up. With such a tight deadline to make this work, Dante admits that he wasn't the most popular guy in the world for a while. "But," he said, "we had to do what we had to do."

Prepping for launch

By the time October and November rolled around, the teams knew Evora and Virgil weren't going to be ready in time for the holiday season, but at least a CES deadline was somewhat feasible. Launching a game-changer at the largest consumer electronics show in the world would give the carrier a boatload of publicity, not to mention the momentum AT&T needed going into the device's release (after all, we did deem it to be the best smartphone of the show). It still wasn't going to be easy though: it was red alert from this time forward, and nobody got much sleep the rest of the way -- we can envision several of them asleep at their desks, empty coffee mugs in hand. There were some huge last-minute changes that needed to be made before making the final launch preparations.

Hardware changes and delays

There's a huge reason AT&T doesn't like to make last-minute hardware changes on any of its devices unless it's an absolute must: it involves a hard tradeoff. Even the slightest adjustment usually translates into six to eight weeks, since the OEM has to first manufacture the product and push it through the full test cycle another time. Any hardware or firmware change requires the complete litany of tests all over again, because even the smallest of adjustments can break something else on the device that'd previously worked perfectly fine. Ah, the fun life a software tester leads.

Delays like this become even more costly when you consider how compressed the lifespan of a phone has become. The Atrix is a great example of this, since its sequel launched not even nine months later. Taking six weeks out of a product's nine-month lifecycle to fix issues or make changes means there's that much less time to sell the device before the market changes and the phone becomes obsolete. Thus, lost revenue opportunity -- not hilarity -- ensues. With such a groundbreaking device on the line, further delays were simply unacceptable.

Two key factors can cause delays. First, there's a serious quality issue that prevents the phone from reaching Technical Acceptance (final software certification) and the problem isn't being solved by new builds. Second, the market changes and AT&T sees a need to incorporate a different feature into the phone to make it as successful as possible. "Sometimes we decide [together with our OEM partner] that it just has to be done in order to push the product forward," Dante said.

Of course, tiny wrinkles happen all the time in device launches, and teams are used to dealing with these types of unforeseen circumstances that these obstacles hardly ever affect the timing of the handset's release. Just because the screen protectors designed to fit the Atrix 4G don't function properly, doesn't mean the train stops moving. So many things move along in parallel, and if one item gets completed en route to launch, more resources are reallocated to strengthen the other parts of the phone's development.

But the Evora project had a couple hardware-related concerns that were more significant than mere wrinkles, neither of which we're able to speak on. However, Motorola and AT&T made excellent use of the opportunity to update other components in the phone as well; in such a dynamic market, prices can come down swiftly and market trends can move incredibly fast. One of the big changes AT&T wanted to make was the inclusion of HSPA+ -- the Evora was originally designed without the next-gen tech in the works.

Looking back to the first quarter of last year, Verizon was busy making preparations to launch its LTE network around the same time the Atrix was supposed to come out, and T-Mobile was already rolling out 21Mbps HSPA+ service. Leaving an HSPA+ radio out of the Evora's design seems like a monstrous oversight, right? We know that AT&T had originally planned to simply leapfrog the tech on its way to deploying LTE, but just a week before the Evora was greenlit, Ralph de la Vega declared that his company intended to build out 3.5G sometime in 2011. There was more than sufficient time to switch tracks at this stage in the development process... or so it seemed.

We have a theory on why HSPA+ was left out until the last minute. The phone was originally expected to launch during the 2010 holiday season, ahead of the time that the next-gen tech would blanket the great lands of this nation. However, once AT&T knew the Atrix would be delayed into 2011 and the timeframe for HSPA+ deployment became much more clear, it likely made more sense to add the radio. After all, Motorola was adding a few things to the spec list already, so why not make sure the upcoming flagship device had all of its other components completely up-to-date as well? It's a good thing, too: it's hard to imagine the Atrix, a smartphone marketed as the carrier's top-notch flagship, being released without at least 14.4Mbps connectivity, let alone anything faster -- not this year, certainly.

To the relief of both companies, Motorola was able to refresh the hardware rather fast, delivering the new build to the labs in the middle of November. And the testing cycle started from scratch. It was becoming clear that the device would, worst-case, be ready to announce at CES.

Naming the device

What's in a name? In this industry, everything. When titles like Inspire, Revolution and Transfix reign supreme, it's just way too easy to mock phone names on a regular basis, and we doubt we're going to stop anytime soon. We get it, though -- the value companies take in finding a marketable name that's easy to remember is obviously priceless. Evora is just a codename, of course, so how did it turn into Atrix and why?

In general, the carrier and OEM both have a hand in a phone's name, but AT&T prefers to be in the driver's seat. The team generates a list of possible names to take to their legal department, which performs a trademark search and whittles it down to roughly one-tenth of the size -- and usually it's the worst from the list (all the best ones, we presume, are already taken). If they can't decide on a name, the process repeats until they stumble upon something good.

The Evora was ready to obtain an official name in November, and it had a couple top contenders that had passed legal muster: Glory and Catapult. Dante's team felt that Glory just didn't mesh well with the features they were trying to perpetuate, and Catapult just sounded too... medieval. But everyone loved "Atrix." Since it's a bleeding-edge phone for the tech enthusiast, it should have a futuristic name. Not only that -- starting with the letter A is always good, there aren't too many syllables and the X at the end adds an element of coolness. Making up words can definitely be cool, but it's a gamble. Dante educated us on the danger of diving into the realm of made-up names too often:

One of the nice things about a made-up word is that your message is the assigned value to it... the problem is, if you do it too often, you start sounding like you're coming up with baby talk or gobbledy-gook and it no longer makes sense. There's a fine line there.


We weren't able to glean a lot of information from Dante or Chris on how the pricing of phones is determined. The process is incredibly complex and depends on a number of factors, such as component pricing (which, due to the dynamic industry, is always changing), the carrier's target segment and, as you'd expect, some influence from the OEM. Sometimes pricing can even be affected by either partner's willingness to promote the device (more on that later).

The carrier's decision to begin selling the Atrix 4G at a $200 price point wasn't anything that warranted shock value. The cost of Motorola's LapDock accessory, however, was. We loved the idea of a laptop dock with Webtop built-in, but it certainly wasn't worth $500. AT&T was stuck between a rock and a hard place here: the carrier wasn't oblivious to the fact that the device was highly priced, but a much different business model applies to unsubsidized accessories. Says Dante:

"When you're working with an OEM who's used to carriers using one model and that's how they price, and then they price to us the same way to something we can't apply that model to, it doesn't work out very well."

In other words, without attaching a contract to accessories, AT&T was constrained to keep the LapDock at standard retail pricing, though it was willing to take a hit on profit by bundling the phone and device together in one purchase.

Motorola spared no expense in making the LapDock, which not only included a stellar battery for the form factor, a durable magnesium enclosure and individual metallic keys, but also incurred significant R&D costs associated with a first-gen device and a brand new ecosystem. We're witnessing some validation of that, as the next generation of the LapDock -- namely, the 100 and 500 -- is coming to market at a significantly lower cost.

Controlling leaks

Leaks. They're a fact of life for the tech media, the proliferation of which we participate in daily. Rumors of the latest and greatest (heck, even the blasé) handsets roam free across the internet like a pack of wild buffalo, whether they're true or not. As we learned in our meeting with Chris and Dante, leaks like these frustrate product managers to no end. It stings, but why? According to Chris:

We have an investment. If there's a leak, it can have a detrimental impact on the perception, or when there's a misquote and an assumption is made, then Dante has to work twice or three times as hard to try and convince people that's not what the product was, it never was that, but the decision has already been made, the court of opinion has already been flawed."

Dante went on to explain that each phone AT&T launches has a unique story to tell, since they're meant to reach different segments of the market and hit various groups of people based on what they care about. It's all about projecting the company's vision to the customer. Leaked phones tell no tales, because media organizations report on what components they feature and what they look like, rather than the "story" the carrier's hoping to portray. This especially speaks volumes to any device that isn't considered state of the art and may get negative points for not having the best specs on the market. "They can never set the right expectation," he said.

Since product managers like Chris and Dante are forced to go into damage control mode as a result of harmful leaks, we asked if they ever seed a counter-leak on purpose to resolve the problem. They told us:

We've never personally been involved with a purposeful leak... it's not in our best interest. It's like one thing can go right while 99 things go wrong. We don't know how that makes sense.

The managers went on to say that no matter how something gets leaked, the information goes through the telephone game: no matter what you say, it'll be different once it gets to the other end, and the perception of the truth is altered. Granted, we didn't see a horrible alteration of the Atrix 4G when it was leaked as the Olympus, but that isn't the case with every phone. We'll give kudos to Dante and his team, however, for keeping the LapDock virtually leak-free.

Promoting and marketing the device

This part of the process is when the claims come out to play: "the world's most powerful phone," "the world's fastest phone" and other similar marketing terminology is pushed through AT&T's legal team to make sure the company remains free from the threat of any possible lawsuits.

It's also the time when the product managers determine if the Atrix 4G will be a hero device. Ultimately, a phone is a hero candidate when it either moves the ball forward from a feature standpoint or offers some type of value proposition -- the Impulse 4G is an excellent example of a hero that gets a lot of TV time for this very reason. The carrier will also negotiate with vendors to promote devices, which in turn can lower the price.

"Even though we do a lot of phones, it's not really a cookie cutter process," Dante explains. "New things become important... in general, we have the baseline for each launch. The idea is always to build on that. With each launch we're working with OEMs on new ways to promote it." Samsung and AT&T teamed up to sponsor the most recent Keith Urban US tour, which involves the superstar shooting video with an Infuse 4G. The opportunities to market phones are incredibly diverse, which forces the carrier to get creative.

First Article Inspection

Just as the Atrix spent time getting tested in the labs, pre-production units are seeded to a decent number of field testers to try out in real-life situations. But as these aren't the final units coming off the actual production lines, AT&T wanted to make sure that devices destined to wind up in customers' hands are as good -- if not better -- than what's been tested already. This is where Final Article inspection (FAI) comes in.

To obtain the coveted FAI status, Motorola sent AT&T a few hundred devices from the final Atrix production line to go through the entire suite of tests one last time. Sounds so sentimental, doesn't it? Once the phone passed, the vendor got the green light to begin flooding all of the available distribution channels -- a process that took around two weeks.

Know why AT&T will, more often than not, announce that a device is heading to stores "in the coming weeks?" FAI may occur two weeks before the phone actually gets shipped out; if something goes wrong and it doesn't receive that approval, everything the PR reps promised is no longer true and everyone has egg on their face.

The carrier may elect to announce a phone at CES or CTIA but prohibit the device from being turned on, handled or photographed -- much to our chagrin, of course. This happens because those handsets haven't yet reached FAI and officially aren't finalized. AT&T gets nervous when it comes to negative first impressions and hands-ons because the software may be pre-production quality. We'll offer an example: the Samsung Infuse 4G (shown above) was announced alongside the Atrix 4G at CES 2011, but media wasn't allowed to touch it or take pictures of its back. The phone wouldn't officially launch until May -- roughly four months later -- and was nowhere near achieving FAI. It was definitely not the carrier's style to announce new phones so early, but AT&T's reason for doing so was to further emphasize that it was making some huge leaps to embrace Android. Still, the event sent out mixed signals and the message wasn't clearly received, which ultimately caused much more damage than AT&T had intended.

Preparing all channels

Now that it has the official name and made its way through the proper legal channels, the Atrix is ready to get a final review on the packaging and materials. The box, accessories, Quickstart guide, and so on are all given the thumbs-up here. Which accessories are included in the box is typically related to keeping the device cost as reasonable as possible, while accentuating certain customer experiences at the same time -- such as HDMI cables or stereo headphones, in some cases.

The last few weeks before a device's launch are the most hectic. It typically involves achieving FAI for the final go-ahead, getting all of the marketing materials ready to go, training employees, communicating the proper information to PR reps so they can talk about the device intelligently to the press, distributing review units and shipping out the Atrix to retail, third party and eCom channels. So many elements work together simultaneously to make sure everything is aligned correctly and the launch will be a success.

Launch day and beyond

Team Dante's hard work didn't stop the moment the Atrix 4G was released; far from it, in fact. The device was solid and in great working condition, but there were wrinkles that still needed to be ironed out after its official outing. So now what?

Rapid response

The first day of sales, February 22nd, was all hands on deck and a war room was set up for rapid response (we can't help but picture a standard red rotary phone in the middle of a conference room), so any snafus that show up could be worked out: if the SKU didn't ring up in the register at the corporate stores, for instance, it needed to be resolved immediately.

From that point on -- day of launch and beyond -- Dante was constantly on the lookout for any feedback he could get. Media reviews, social networking (Twitter, for instance) and word of mouth are very important to determining what went wrong and how to correct it in this device and in future models. This is where maintenance releases and bug fix updates become crucial.

Firmware updates

People begin using the phone in ways the product managers never imagined. Feedback on bugs and other issues gets back to the team. The next version of the OS comes out. All of these scenarios occur with every device, which means Dante and Chris need to be on the ball in cranking out updates and maintenance releases. As often is the case, these refreshes take a while to push through. Five moons had passed before the phone was ready to get Gingerbread. The main reason for this is that placing new firmware on existing hardware involves much more than just the click of a button. Chris told us:

Just like in the labs [the first time around], when new software is introduced we have to go through the full gauntlet of tests from this end to that end all over again. New software can always introduce new bugs in areas that were fine before, and the last thing you want to do is take a person who's happy with their phone, and they update it and it doesn't work as well anymore. People think with upgrades that they're getting more. You don't want to break anything. That's why we go through the same rigorous testing in these updates, we don't want to have a negative impact on a customer experience.

There's a fine line between ensuring a good upgrade experience without actually making it worse for the user. But because of this, updates have a tendency to take longer to roll out. Android devices, for instance, appear to be the most difficult. Google announces Gingerbread, OEMs finally get their hands on the source, they push it through their own development cycle to build their own UI onto it (such as MotoBlur, in the case of the Atrix), and then it has to undergo QA (Quality Assurance) tests to ensure the firmware update is good enough to get into AT&T's labs -- and that's when carrier testing can finally begin.

Sometimes, as the managers claimed was the case with the Samsung Captivate, new versions of the OS can take longer to push out on older hardware. According to Chris:

Dealing with devices that are mature in their lifecycle, getting software with new features that are built for newer hardware, means even more grueling regression tests to make sure it doesn't break the phone or any other aspect of the product. Anything that doesn't meet the criteria has to go through the process all over again.

Where's the 4G uploading?

Prior to the Atrix's release, Dante's team already had items on the list for a post-launch maintenance fix. One of these, if you may recall, was the enabling of HSUPA speeds. Here was a device advertised as having 4G capabilities, but only HSDPA -- the download portion of the next-gen network -- actually came enabled on the Atrix when it launched (as well as the HTC Inspire 4G, launched in the same timeframe).

We were left utterly confused as to why this was the case -- was it the biggest oversight in the world, or was AT&T leaving it out on purpose? No matter the answer, the carrier either looks stupid or mean. We asked Dante, who insists it was the latter, but not because the carrier wanted to be a jerk. Instead, it was a result of AT&T's last-minute addition of HSPA+ back in November; additional time was apparently needed to push it out.

We thought, what can we make happen in time? That's how we ended up where we did. We got our MR out as fast as we possibly could to address it, but my rationale was -- as long as you can download your videos at a good clip, you don't sit there and wait while you're uploading something... we wanted it in there. But we had to get the product out there; we didn't want to delay any further.

It's understandable that software changes can take a fair amount of time to implement correctly. But if the uploading capability had been done in concert with HSDPA, how much longer would it have been delayed?


Knowing the journey helps us come to a greater appreciation of the destination. It's not an easy job to come out with a smartphone lineup that pleases everyone, and we certainly got a sense of the pressure Dante's team felt in getting a landmark phone ready and in shipshape condition. The Motorola Atrix 4G wasn't a perfect handset, of course, but it reached its intended goal: it was supposed to be a game-changer, and it's hard to deny that it was. And don't get us wrong, we still -- and always will -- have our concerns about the extraordinary amount of influence carriers hold over phone manufacturers throughout this process, but we cannot refute the fact that Dante and Chris gave us a new perspective on what actually goes into creating a masterpiece.