Optimizely Introduces Stats Engine – and Statistical Analysis Leaps into the Internet Age
– Breakthrough capability aligns statistics with real-world business practice; Companies can now interpret test data more quickly with results that are always valid
SAN FRANCISCO, Jan. 20, 2015 – Optimizely today announced the general availability of Stats Engine, a capability that adds meaningfully to the company’s experience optimization platform by delivering test results that are always valid and freed from the limitations of traditional statistical analysis. Developed in collaboration with statisticians at Stanford University, Optimizely’s Stats Engine relies on innovative approaches in the field of statistical analysis and the company’s own robust algorithms to allow customers to interpret and act on data more quickly and more confidently than ever before.
Businesses today seek the ability to make smart, fast decisions related to their web and mobile experiences. Through A/B and multivariate testing, in addition to other tools, Optimizely delivers real-time data backed by statistics. However, traditional statistical methods – many developed early in the 20th century – burden businesses with limitations that don’t fit today’s world of Big Data and the need to understand and act on it quickly.
“Stats Engine represents a breakthrough in the world of online testing and a break from statistical methods that have been used universally for almost 100 years,” said Pete Koomen, co-founder of Optimizely. “The most important advance is the ability to extract valid, actionable data at any point in time rather than having to wait for an experiment to reach a pre-determined sample size. Stats Engine works the way business does – in real time – and it results in a real reduction of mid-course statistical flukes masquerading as relevant results.”
“Businesses today are generating far more data – and in real time,” said Ramesh Johari, an associate professor of management science and engineering at Stanford who advised Optimizely on the project. “The biggest advantage with Stats Engine is the ability to continuously monitor the rich stream of data created inside a test and get to actionable results more quickly.”
The statistical framework of Optimizely’s Stats Engine is the first in the industry to combine sequential hypothesis testing and multiple testing corrections, and apply them to real-time data sets. In so doing, Optimizely reduces the built-in limitations of traditional statistics that can result in experiment error rates of more than 30 percent.
Sequential hypothesis testing . With classical statistics, teams are advised to look at the results of a running experiment just once – and only then after they’ve reached a pre-set sample size – due to the risk of mistaking statistical oddities for statistical relevance. However, today’s businesses move quickly and watch data closely, and they are especially vulnerable to this error. In response, Stats Engine assumes that a test has an infinite rather than a fixed sample size. It generates an accurate level of confidence after every visitor, so marketing or product management teams can check results frequently and make decisions as soon as results are statistically significant. Importantly, no additional analysis is necessary.
False discovery rate control . Having multiple variations and goals in an experiment increases the likelihood of error, based on random chance alone and due to the challenge of multiple comparisons. Stats Engine accounts for this by adjusting its calculations based on the total number of variations and goals in an experiment. This results in a highly accurate confidence calculation, regardless of the number of goals and variations tested at once.
Optimizely has created a series of informational resources to allow current and prospective customers to learn more about Stats Engine:
- To visit the Stats Engine landing page and discover more about using Stats Engine to experiment fearlessly, click here.
- A video posted on YouTube (https://www.youtube.com/watch?v=APfa7KqhzU8) delivers expert academic and business perspectives on Stats Engine.
- Businesses can start optimizing experiences for web and mobile apps though an easy sign-up process.
About Optimizely
Optimizely is the world’s leading experience optimization platform, providing A/B testing, targeting and multichannel optimization for websites and mobile apps. The platform’s ease of use and speed of deployment empowers organizations to conceive of and run experiments that help them make better data-inspired decisions. Optimizely meets the diverse needs of more than 8,000 customers looking to deliver connected experiences to their audiences across channels. To date, those customers have created and delivered more than 7 billion optimized visitor experiences. To learn more, visit optimizely.com.
