I was a panelist in a discussion by Branch Metrics last spring on A/B testing. One of the other members was from Instagram. She described the perfect testing environment - randomly sample 1% of the total audience. Sort that group into experiments where one variable (and that variable only) changes. See what engagement and other metrics look like. Analyze, decide, and continue.
That’s all well and good if you’re Instagram. But what about the rest of us? When you’re working your way up from nothing, starting with a testing sample of 1% is impossible.
I learned about experimental design and testing in my undergraduate days as a research assistant for a quantitative psychology lab. We would develop a hypothesis, create a test around that hypothesis, recruit participants and then analyze the results and hopefully find something interesting.
And then the real world hit. I was at a startup where we were testing email messages. I remember the first test, just the title. It was me saying “lowercase vs title case?” and the rest of the team saying “beach vacation or tax audit?”
A/B testing is the crux of experimentation, but we’re doing something more than that in the world of startups. We’re drawing on a blank canvas, throwing spaghetti on a wall and fishing for insights. And it works.
We experienced this the most while creating and running ads for Live Fantasy TV. We started from nothing, so we tested everything.
The winner? Margaritas on the beach. I can’t tell you why, but I can tell you it gave us a starting place. We could take the image, learn from it and create dozens of similar ads to iterate on the same concept. We were able to win — because we tried everything.
At the end of the day, testing for startups will always be about more than building a perfect experiment. It’s about the same values companies are founded on — risk, intuition and math.