My online stats class last week reviewed Regression, Null Hypothesis Significance Testing (NHST), and the flaws and remedies of using the NHST method to dictate statistical significance. Seeing that Type I and Type II error table definitely brought back some memories of the p-value stats knowledge I learned in my college research class. I remember at the time I had to calculate the t and p-values for several research projects. But the R program introduced in this online stats class totally blew my mind. It is much more powerful and flexible for performing statistical analysis than any programs I used back in school. Though R is popular in academia, I hear that it is used in professional industries more and more. In this online class, we have been taught to tinker with R to make scatterplots, run regressions, and do other fun things.
Andrew Conway, the Princeton professor who teaches my online stats class, mentioned that if students take away just one lesson from this whole course, he hopes that it is the correct notion for the P-value. What is P? Given that the null hypothesis is true, the probability of these, or more extreme data, is P. The P-value is an arbitrary value selected to define statistical significance. P<.05 is considered standard in academia, while the professional world may take a more liberal range of up to P<.1 for research and AB tests conducted in business.
In my digital product development job, we constantly need to check the statistical significance of our product testings and AB tests. Often times we see results clearly trending one way but without reaching statistical significance. This is one of the most confounding situations in business, as I’m sure it is in academia and any other settings as well. While it is very tempting to reap the results without statistical significance, it is extremely important to recognize the randomness embedded in the result and refrain from jumping to premature conclusions based on these numbers.
It is also helpful to recognize the inherent flaws of the NHST method, like how data size could impact results reaching statistical significance. In a business environment, we have developed other methods alongside NHST like diving deep into the consumer behavior funnel to gain a more comprehensive picture. By utilizing a statistical perspective instead of pure gut feelings, being mindful of the system flaws, and gathering more direct user behavior information, we can shrink blind spots in our knowledge, and paint a more holistic picture for our decision making process.
Data and Lore 