# Determining Statistical Significance - Montgomery College Section 4.3 Determining Statistical Significance Statistics: Unlocking the Power of Data Lock5 p-value Using the randomization distribution below to test H0 : = 0 vs Ha : > 0 Which sample statistic shows the most evidence for the alternative hypothesis? r = 0.1, r = 0.3, or r = 0.5 Therefore, which p-value shows the most evidence for the

alternative hypothesis? 0.35, 0.15, or 0.005 Dot Plot Measures from Scrambled Collection 1 -0.6 -0.4 -0.2 Statistics: Unlocking the Power of Data 0.0 r 0.2

0.4 Lock5 0.6 p-value and H0 If the p-value is small, then a statistic as extreme as that observed would be unlikely if the null hypothesis were true, providing significant evidence against H0 The smaller the p-value, the stronger the evidence against the null hypothesis and in favor of the alternative Statistics: Unlocking the Power of Data

Lock5 p-value and H0 The The smaller smaller the thep-value, p-value, The smaller the p-value, the stronger the evidence against H. the

stronger the evidence the stronger the against H . o evidence against Ho. o Statistics: Unlocking the Power of Data Lock5 Question #1 of the Day Does red wine (resveratrol)

promote weight loss? If so, why? Statistics: Unlocking the Power of Data Lock5 Red Wine and Weight Loss Resveratrol, an ingredient in red wine and grapes, has been shown to promote weight loss in rodents. Well look at a study on Grey Mouse Lemurs (a primate). A sample of lemurs had various measurements taken before and after receiving resveratrol supplementation for 4 weeks: body mass resting metabolic rate locomotor activity food intake BioMed Central (2010, June 22). Lemurs lose weight with

life-extending supplement resveratrol. Science Daily. Statistics: Unlocking the Power of Data Lock5 Resveratrol and Weight Loss The different tests resulted in these p-values: Body mass (p-value = 0.007) Resting metabolic rate (p-value = 0.013) Locomotor activity (p-value = 0.49) Food intake (p-value = 0.035) Which test provided the strongest evidence against the null hypothesis and in favor of the alternative? Body mass, because it has the lowest p-value Statistics: Unlocking the Power of Data

Lock5 Formal Decisions A formal hypothesis test has only two possible conclusions: 1. The p-value is small: reject the null hypothesis in favor of the alternative 2. The p-value is not small: do not reject the null hypothesis How small? Statistics: Unlocking the Power of Data Lock5 Significance Level The significance level, , is the threshold below , is the threshold below

which the p-value is deemed small enough to reject the null hypothesis p-value < , is the threshold below => Reject H0 p-value > , is the threshold below => Do not Reject H0 Often , is the threshold below = 0.05, unless otherwise specified (Why 0.05?) Statistics: Unlocking the Power of Data Lock5 Why 0.05? Number of red cards in a row 1 2

3 4 5 Probability if dealing randomly from a normal deck 0.5 0.245 0.118 0.055 (hmmm this is odd....?) 6 0.025 (This would be unlikely just by random chance!) 0.011 (definitely looking suspicious!) 7

0.005 (significant even at = 0.01!) = 0.01!) Statistics: Unlocking the Power of Data Lock5 p-value < , is the threshold below p-value , is the threshold below Results would be rare, if the null were true Results would not be rare, if the null were true Reject H0 Do not reject H0

We have evidence that the alternative is true! Our test is inconclusive Statistics: Unlocking the Power of Data Lock5 Elephant Example H0 : X is an elephant Ha : X is not an elephant Would you conclude, if you get the following data? X walks on two legs Although we can never be certain!

Reject H0; evidence that X is not an elephant X has four legs Do not reject H0; we do not have sufficient evidence to determine whether X is an elephant Statistics: Unlocking the Power of Data Lock5 Never Accept H0 Do not reject H0 is not the same as accept H0! Lack of evidence against H0 is NOT the same as evidence for H0! For the logical fallacy of believing that a hypothesis has been proved to be true, merely because it is not contradicted by the available facts, has no more right to insinuate itself in statistical than in other kinds of scientific reasoning -Sir R. A. Fisher

Statistics: Unlocking the Power of Data Lock5 Conclusions p-value < , is the threshold below Generic conclusion: Reject H0 p-value , is the threshold below H0 Generic conclusion: Do not reject H0 Ha Conclusion in context: We have (strong?) evidence

that [fill in alternative hypothesis] Statistics: Unlocking the Power of Data Conclusion in context: We do not have enough evidence to conclude that [fill in alternative hypothesis] Lock5 Red Wine and Weight Loss In the test to see if mean resting metabolic rate is higher after treatment, the p-value is 0.013. Using = 0.05, 1) 2) Give a formal generic conclusion about H0

Give a conclusion in context The p-value is lower than = 0.05, so we reject H0. There is evidence that mean resting metabolic rate is higher after receiving resveratrol. Statistics: Unlocking the Power of Data Lock5 Red Wine and Weight Loss In the test to see if mean resting metabolic rate is higher after treatment, the p-value is 0.013. Using = 0.05, 1) 2) Give a formal generic conclusion about H0

Give a conclusion in context The p-value is lower than = 0.05, so we reject H0. There is evidence that mean resting metabolic rate is higher after receiving resveratrol. Statistics: Unlocking the Power of Data Lock5 Red Wine and Weight Loss In the test to see if locomotor activity is higher after treatment, the p-value is 0.49. Using = 0.05, 1) 2) Give a formal generic conclusion about H0

Give a conclusion in context The p-value is not lower than = 0.05, so we do not reject H0. The data does not provide sufficient evidence to conclude that locomotor activity is higher after treatment. Statistics: Unlocking the Power of Data Lock5 Red Wine and Weight Loss In the test to see if the mean food intake changes after treatment, the p-value is 0.035. Using = 0.05, 1) 2)

Give a formal generic conclusion about H0 Give a conclusion in context The p-value is lower than = 0.05, so we reject H0. There is evidence that mean food intake is different for mice who take resveratrol. Statistics: Unlocking the Power of Data Lock5 Statistical Significance When the p-value is less than , is the threshold below , the results are statistically significant. If our sample is statistically significant, we have convincing evidence against H0, in favor of Ha Statistics: Unlocking the Power of Data

Lock5 p-value < , is the threshold below p-value , is the threshold below Results would be rare, if the null were true Results would not be rare, if the null were true Reject H0 Do not reject H0 We have evidence that the alternative is true!

Our test is inconclusive Results are statistically significant Statistics: Unlocking the Power of Data Results are not statistically significant Lock5 Resveratrol and Weight Loss Using = 0.05, which of the following results are statistically significant? Body mass (p-value = 0.007) Resting metabolic rate (p-value = 0.013) Locomotor activity (p-value = 0.49) Food intake (p-value = 0.035)

Using = 0.01, which of the following results are statistically significant? Body mass (p-value = 0.007) Resting metabolic rate (p-value = 0.013) Locomotor activity (p-value = 0.49) Food intake (p-value = 0.035) Statistics: Unlocking the Power of Data Lock5 Statistical Conclusions Formal decision of hypothesis test, based on = 0.05 : Informal strength of evidence against H0: Statistics: Unlocking the Power of Data

Lock5 Question #2 of the Day What aspect of sunlight might help protect against multiple sclerosis? Statistics: Unlocking the Power of Data Lock5 Multiple Sclerosis and Sunlight It is believed that sunlight offers some protection against multiple sclerosis, but the reason is unknown Researchers randomly assigned mice to one of: Control (nothing)

Vitamin D Supplements UV Light All mice were injected with proteins known to induce a mouse form of MS, and they observed which mice got MS Seppa, Nathan. Sunlight may cut MS risk by itself, Science News, April 24, 2010 pg 9, reporting on a study appearing March 22, 2010 in the Proceedings of the National Academy of Science. Statistics: Unlocking the Power of Data Lock5 Multiple Sclerosis and Sunlight For each situation below, write down Null and alternative hypotheses Informal description of the strength of evidence against H 0 Are the results statistically significant?

Formal decision about H0, using = 0.05 = 0.05 Conclusion in the context of the question In testing whether UV light provides protection against MS (UV light vs control group), the p-value is 0.002. In testing whether Vitamin D provides protection against MS (Vitamin D vs control group), the p-value is 0.47. Statistics: Unlocking the Power of Data Lock5 Multiple Sclerosis and Sunlight In testing whether UV light provides protection against MS (UV light vs control group), the p-value is 0.002.

H0: pUV pC = 0 Ha: pUV pC < 0 We have strong evidence against H0 The results are statistically significant Reject H0 We have strong evidence that UV light provides

protection against MS, at least in mice. Statistics: Unlocking the Power of Data Lock5 Multiple Sclerosis and Sunlight In testing whether Vitamin D provides protection against MS (Vitamin D vs control group), the p-value is 0.47. H0: pD pC = 0 Ha: pD pC < 0 We have little evidence against H0

The results are not statistically significant Do not reject H0 We cannot conclude anything about Vitamin DPower andof Data MS. Statistics: Unlocking the Lock

5 Hypothesis Tests: Start to Finish! 1. State the hypotheses (defining parameters) 2. Find the observed sample statistic 3. Find the p-value 4. Make a generic decision about H0: Reject H0 or do not reject H0 5. Make a conclusion in context, indicating whether or not we have convincing evidence for Ha and referring back to the question of interest. Statistics: Unlocking the Power of Data Lock5 Hormone Replacement Therapy Until 2002, hormone replacement therapy (HRT),

estrogen and/or progesterone, was commonly prescribed to post-menopausal women. This changed in 2002, when the results of a large clinical trial were published 8506 women were randomized to take HRT, 8102 were randomized to placebo. 166 HRT and 124 placebo women developed invasive breast cancer Does HRT increase risk of breast cancer? Statistics: Unlocking the Power of Data Lock5 Step 1: State Hypotheses Does HRT increase risk of breast cancer? p1 = proportion of women taking HRT who

get invasive breast cancer p2 = proportion of women not taking HRT who get invasive breast cancer H0: p1= p2 Ha: p1> p2 Statistics: Unlocking the Power of Data Lock5 Step 2: Calculate Sample Statistic Does HRT increase risk of breast cancer? H : p = p ; H : p > p 0 1 2 a 1 2 HRT group: 166 of 8506 developed breast cancer

Placebo group: 124 of 8102 developed breast cancer Statistics: Unlocking the Power of Data Lock5 Step 3: Find the p-value Statistics: Unlocking the Power of Data Lock5 Step 4: Make a Generic Conclusion Does HRT increase risk of breast cancer? H : p = p ; H : p > p 0 1 2 a

1 2 p-value = 0.019 Using = 0.01!) = 0.05: Reject H0 Statistics: Unlocking the Power of Data Lock5 Step 5: Make a Conclusion in Context Does HRT increase risk of breast cancer? H : p = p ; H : p > p 0 1 2

a 1 2 p-value = 0.019 Reject H 0 We have convincing evidence that taking hormone replacement therapy does increase risk of breast cancer. (Because of this result, the trial was terminated early and HRT is no longer routinely recommended). Statistics: Unlocking the Power of Data Lock5 Your Turn! Same trial, different variable of interest.

8506 women were randomized to take HRT, 8102 were randomized to placebo. 502 HRT and 458 placebo women developed any kind of cancer. Does hormone replacement therapy cause increased risk of cancer in general? Statistics: Unlocking the Power of Data Lock5 p1 = proportion of women taking HRT who get cancer p2 = proportion of women taking placebo who get cancer H0: p1= p2 Ha: p1> p2

p-value = 0.254 Do not reject H0 We do not have convincing evidence that HRT increases risk of cancer in general. Statistics: Unlocking the Power of Data Lock5 Summary Results are statistically significant if the p-value is less than the significance level, , is the threshold below In making formal decisions, reject H0 if the pvalue is less than , is the threshold below , otherwise do not reject H0 Not rejecting H0 is NOT the same as accepting H0 Hypothesis test conclusions include a generic decision about H0 (reject or do not reject) and a conclusion in context about Ha Statistics: Unlocking the Power of Data

Lock5