Last month, I got to attend Opticon 2015, the largest conference on optimizing and testing, hosted by Optimizely . Pier 27, nestled between the San Francisco Bay and Coit Tower, was a gorgeous setting for two days of networking, hammock naps, nitro margaritas — oh, and gathering knowledge to take back to work. Dozens of break-out sessions, a keynote from Optimizely’s CEO Dan Siroker, and a fireside chat with Marc Andreessen combined for an inspirational event with a LOT of takeaways to chew on. We sorted through them and distilled them down to seven major lessons.
A/B testing can be a big scary thing if you’re just getting started with optimizing your website. Start with these basics to lay the groundwork, and keep them in mind when things get complicated.
1. Before you start, nail down your most important goals.
It’s pretty important to decide what success is before you start your experiment. Sit down with the stakeholders for your site and have a brainstorming session of what you’re trying to accomplish. Chances are your list will be pretty long, but you need to pare it down and decide on what the 1–3 primary goals are before you conceptualize any experiments. In addition, every goal you set for an experiment needs to be clear and actionable. ‘Make the page better’ or ‘help people find what they’re searching for’ are not clear, actionable goals. A good hypothesis has a variation (the B to your A) and clearly defined metrics.
2. Think Big.
Time and time again at Opticon, we heard how the biggest problem with A/B testing is complexity. There are so many things on any site to test, and it can be tempting to think up a million tests you could do (blue v. orange header! page title variations!)[…]
