A/B tests work? The basic math is wrong, forget the statistics. And when you get to "diff in diff", the metrics commonly used can be directionally wrong from the actual effect.