Linden Lab announces A/B testing

Linden Lab has formally announced that they're performing A/B testing for new Second Life accounts. This is kind of surprising, actually. Not that we think there's anything wrong with A/B testing or that we think it's a bad idea.

It is just that Linden Lab has been performing A/B testing on and off since at least late 2005 that we are aware of, and we find the sudden announcement of it just a little peculiar.

A/B testing is a fairly simple process. One portion of users get the standard 'vanilla' experience. Another group or groups are given an experience that differs in one single respect from the standard. With enough users, you can statistically compare the two groups to see which is the most successful (by whatever criteria you have for success).

Linden Lab's success criteria generally are something like amount of time spent in the first X days, purchases of currency on the Lindex, and that sort of thing.

Linden Lab has used A/B testing to evaluate orientation experiences: Did users of one class of orientation island score statistically significantly better on the success criteria than the ones who went via the usual class of island?

The registration and orientation system that you currently see is the product of long periods of A/B testing, where the current alternatives were shown to be statistically more successful at retaining users according to Linden Lab's success criteria.

What seems to make this different is the possible offering of different viewers as one of the testing variations. That should be interesting, as for A/B testing to be properly successful, you need to limit your changes to the absolute minimum. That suggests that any variant viewers will probably only differ in the most minor respects: a small feature, repositioning of one or two UI elements, or the color scheme, for example.

In short, A/B testing is good if you can afford the resources to do it properly — and since Linden Lab have a tradition of performing ongoing A/B testing, it seems these resources are already likely in place.