Beta Testing

November 12th, 2012

You keep using that word. I do not think it means what you think it means.

— Inigo Montoya

The word beta has all but lost its meaning in the modern software industry. Gmail was in “beta” for years. Siri is still “beta” despite having shipped with tens of millions of iOS devices. And the list goes on.

The beta phase of an app is meant to be a time of aggressively finding and fixing bugs prior to the official release of the app. It’s not about cool people getting early access to the hottest new apps. It’s not about haphazardly letting a few people try your app before release. It’s about killing bugs. And we need to get better at that.

Though incredibly important, beta testing should be just one component in a broader strategy to ship bug-free apps. That broader strategy is called Quality Assurance — QA for short. I am in no way a QA expert, and I’m probably still confused about some of the foundational concepts, but I’m doing my best to get up to speed and to better test every app I ship.

The first line of defense in good QA is to have the person writing the code test it on the fly. Seems simple, but it’s easy to get wrapped up in bigger problems and forget to go back and test those seemingly small changes that were made along the way. Given the value of a developer’s time, the amount of testing done at this stage will necessarily fluctuate. It can be cheaper to catch bugs downstream as long as fixing those bugs doesn’t get more complex as time passes from initial coding to when the bug was found. On small projects this usually isn’t a huge issue.

The next line of defense is code-based testing. Due to added cost and complexity, testing in code is not always practical for iOS projects. Even small projects may end up saving money in the long run by taking the time to do some testing in code, but it’s tough to make room for it on a tight deadline/budget. With proper testing procedures, a relatively unskilled tester can more cheaply find many of the bugs and regressions that may or may not have been caught by code-based tests that cost thousands of dollars to create.

If you're part of a bigger team and/or have the time to experiment with the cost effectiveness of code-based testing, several current and former Yelp employees have open sourced a great testing framework for iOS and OS X called GHUnit.

Next up, traditional QA testing. This is where I think most small projects fail and fail hard. Before sending out a beta, there should be at least one person, preferably several who thoroughly test the app. The line between beta testing and QA testing is blurry, but my goal for QA testers is to have a very thorough testing procedures that each tester follows in full, and a device/OS matrix to make sure every important combination is covered at least once. Unfortunately, I have yet to find a simple software solution for maintaining and executing those testing proceedures. Most QA software is aimed at huge enterprises and is way too fiddly and over-designed for a small team building iOS apps.

Once a build has made it through QA, then it’s off to beta testers. And here again, I think most small iOS projects fail hard. Because of the 100 device limit imposed by Apple for testing, most developers only have 20-30 people on their beta list. 20-30 people hammering away at an app would be great, but that’s not how it generally plays out. In my experience very few people on my betas actually take the time to put the app through its paces. Some understandably just don’t have the time to test every build. Some are friends and family members who don’t really use the app much anyway and forget to load the latest builds. Then you have people who just wanted to see the cool new app before it was released and never do any real testing even though they begged for a slot in the beta.

There’s also the matter of giving beta access to friends in the press. It varies from person to person, but most of my friends in the press really don’t want to beta test an app, they just want early access. And that’s actually a good thing since it will give them a better long-term view of the app rather than a shotgun 48 hour review window. That being the case, I’ve decided to only send true release candidates to my friends in the press. Unless they specifically ask to get their hands dirty, that’s really the best for everyone.

Dirty. That’s a great way to describe good beta testing. Finding and killing bugs isn’t particularly glamorous work. Data will be lost. Time will be wasted. Frustrations will be encountered. That’s really the point of all this. Find the crap and make it better.

And the best way to find and kill bugs is not to send betas out to a few friends who haphazardly poke around. An effective beta team consists of people who actually like the app they are testing, and take the time to not only poke around as a normal user would, but to follow a few testing scripts or go through a checklist based on the QA procedures.

That’s all well and good in theory, but much harder to make happen in practice. To that end, I’ve decided to completely revamp my QA and beta testing lists. I’m going to start fresh and only accept testers who have read this post and want to get their hands dirty. Over the next few months App Cubby will be testing updates to Gas Cubby, Timer, Launch Center Pro, and at least one unannounced app. I’m going to need a lot of help in making sure those apps ship with as few bugs as possible.

Good beta testing really is a lot of work, so part of this experiment is going to be to reward testers. For some, just helping out is reward enough, and we deeply appreciate people who take that attitude. For others, getting early access to new apps/features is a great reward. But I’d like to experiment with additional rewards. Over the next few months we’ll be awarding small gift cards to testers who complete testing checklists and bigger gift cards to testers who find reproducible bugs. I’ve also been thinking about other options like creating a “credits” section in each app where hard-core testers are publicly thanked.

If you’ve made it this far and are still interested in beta/QA testing for App Cubby, please email Brock. And let him know you’re serious about helping out, and whether you’re hard-core enough to be considered for the QA team.

david