Rather than extol everyone on the greatness of A/B Testing, I would like to discuss some important lessons we are learning in the process of implementing the process. It’s always important to remember some simple lessons that can shape the direction your product grows in.

With the technology of real-time analytic data, you can tell your website the desired goal of a specific feature and according to user feedback it can make it so. This is not to be glossed over. You can do some cool stuff if you know what your customers want and be able to deliver it in the wild. While many product segments may see this is trivial, in the tech community this has not always been the case.

This recent breakthrough does carry with it a responsibility to ensure the right kind of quality regarding A/B Testing. This article from Quicksprout highlights some incorrect assumptions that can cause members of the product team to make poor decisions.

One of the easiest mistakes to make is not setting the right goals and expectations around your test. That is easy to set when you are deciding which cola tastes better, but if your lead conversions need to be improved it may not be as simple as testing two different designs.

We talk all the time about managing the customer’s expectations for a product release, but we rarely apply the same logic to our own. If we expect great things, we need to plan for great things in testing of ideas. Only then can we really say that we are ensuring the right kind of quality in our offering.