Comparing Python packages for A/B test analysis: tea-tasting, Pingouin, statsmodels, and SciPy

Summary

The article compares Python packages for A/B test analysis, including tea-tasting, Pingouin, statsmodels, and SciPy. It outlines the strengths and weaknesses of each package for common experimentation tasks. The author provides code examples and discusses important considerations for A/B testing workflows.

Key Takeaways

1. Tea-tasting is designed specifically for A/B test analysis, targeting experimentation workflows directly with metrics, relative effects, CUPED, power analysis, and concise experiment-style outputs.
2. Pingouin is a user-friendly statistical package focused on convenient inferential statistics in pandas-centric workflows, strong for common tests and effect sizes, but not an A/B-specific framework.
3. Statsmodels is a broad statistical modeling library with strong hypothesis testing, power analysis, and confidence interval utilities, offering building blocks and explicit control for analysts.

Statement Breakdown

• Claimed Facts: 70% of statements the article presents as facts
• Opinions: 20% of statements classified as editorial or subjective
• Claims: 10% of statements surfaced for additional reader evaluation

Credibility & Bias Reasoning

Credibility assessment: The article provides a clear comparison of different Python packages for A/B testing. The author discloses their involvement with one of the packages, which enhances transparency. The analysis is based on established statistical methods and provides code examples.

Bias assessment: Package Preference. The author, being the creator of tea-tasting, may exhibit a slight preference towards it, although they attempt to provide a balanced comparison. The article highlights the strengths of tea-tasting in A/B testing workflows. This potential bias is mitigated by the inclusion of other packages and their respective advantages.

Note: While the author discloses their involvement with tea-tasting, consider potential bias when evaluating the package comparisons.

Credibility flag: Informative, Transparent

Claimed Facts (8)

• This is a factual statement about the article's content.
• This is a standard definition of an A/B test.
• This is a common practice in online A/B testing.
• This is a generally accepted principle in A/B testing analysis.
• This is a technical statement about CUPED methodology.
• This is a statistical necessity when conducting multiple tests.
• This is a practical consideration for efficient data analysis.
• This is a factual statement about Pingouin's features.

Opinions (8)

• This is a subjective assessment of the output format's convenience.
• This is a subjective statement about the importance of efficiency.
• This is a subjective recommendation based on workflow preferences.
• The term 'user-friendly' is subjective.
• This is a subjective recommendation based on analysis speed and type.
• The term 'strong' is subjective.
• Whether being less opinionated is a strength is subjective.
• This is a subjective recommendation based on analyst preference and skill.

Claims (3)

• While this can be problematic, calling it a 'common mistake' without data is a dubious claim.
• The claim of being 'most compact' is subjective and lacks quantifiable evidence.
• This is an appeal to authority without specific evidence.

Key Sources

• Evgeny Ivanov — Author of tea-tasting
• Ron Kohavi — Expert on trustworthy experimentation
• Alex Deng — Expert on delta-method metrics and CUPED