The Sampling Distribution of ^β1

Recall: The Two Big Problems with Data

Recall: Distributions of the OLS Estimators

• OLS estimators (^β0 and ^β1) are computed from a finite (specific) sample of data

• Our OLS model contains 2 sources of randomness:

• Modeled randomness: u includes all factors affecting Y other than X

  • different samples will have different values of those other factors (ui)

• Sampling randomness: different samples will generate different OLS estimators
  • Thus, ^β0,^β1 are also random variables, with their own sampling distribution

Recall: Inferential Statistics and Sampling Distributions

Recall: Inference in Econometrics: The Big Picture

Sample→⏟statistical inferencePopulation→⏟causal indentificationUnobserved Parameters

• We want to identify causal relationships between population variables

  • Logically first thing to consider
  • Endogeneity problem

• We'll use sample statistics to infer something about population parameters

  • In practice, we'll only ever have a finite sample distribution of data
  • We don't know the population distribution of data
  • Randomness problem

Two Methods of Statistical Inference

1. Estimation: use our sample data to construct a point estimate of a population parameter and subject it to a hypothesis test

2. Confidence interval: use our sample data to construct a range for the population parameter

• First method is more common, but second is still widely acknowledged

• Both will give you similar results

  • Tradeoff of accuracy vs. precision

• Note statistical inference is different than causal inference!

Estimation and Hypothesis Testing I

• We have already used statistics to estimate a relationship between X and Y

  • OLS estimators ^β0 and ^β1 of the true population β0 and β1

• We want to test if these estimates are statistically significant and they describe the population

  • This is the "bread and butter" of inferential statistics and the purpose of regression

• All modern science is built upon statistical hypothesis testing, so understand it well!

Estimation and Hypothesis Testing II

• Note, we can test a lot of hypotheses about a lot of population parameters, e.g.

  • A population mean μ
    • Example: average height of adults
  • A population proportion p
    • Example: percent of voters who voted for Trump
  • A difference in population means μA−μB
    • Example: difference in average wages of men vs. women
  • A difference in population proportions pA−pB
    • Example: difference in percent of patients reporting symptoms of drug A vs B
  • See all the possibilities in glorious detail in today's Class Notes

• We will focus only on hypotheses about the population regression slope (ˆβ1), i.e. the causal effect¹ of X on Y

¹ With a model this simple, it's almost certainly not causal, but this is the ultimate direction we are heading...

Null and Alternative Hypotheses I

• All scientific inquiries begin with a null hypothesis (H0) that proposes a specific value of a population parameter
  • Notation: add a subscript 0: β1,0 (or μ0, p0, etc)

• We suggest an alternative hypothesis (Ha), often the one we hope to verify
  • Note, can be multiple alternative hypotheses: H1,H2,…,Hn

• Ask: "Does our data (sample) give us sufficient evidence to reject H0 in favor of Ha?"
  • Note: the test is always about H0!
  • See if we have sufficient evidence to reject the status quo

Null and Alternative Hypotheses II

• Null hypothesis assigns a value (or a range) to a population parameter
  • e.g. β1=2 or β1≤20
  • Most common null hypothesis is β1=0 ⟹ X has no effect on Y (no slope for a line)
  • Note: always an equality!

• Alternative hypothesis must mathematically contradict the null hypothesis
  • e.g. β1≠2 or β1>20 or β1≠0
  • Note: always an inequality!

• Alternative hypotheses come in two forms:
  1. One-sided alternative: β1>H0 or β1<H0
  2. Two-sided alternative: β1≠H0
     • Note this means either β1<H0 or β1>H0

Components of a Valid Hypothesis Test

• All statistical hypothesis tests have the following components:

1. A null hypothesis, H0

2. An alternative hypothesis, Ha

3. A test statistic to determine if we reject H0 when the statistic reaches a "critical value"

   • Beyond the critical value is the "rejection region", sufficient evidence to reject H0

4. A conclusion whether or not to reject H0 in favor of Ha

Type I and Type II Errors I

Type I and Type II Errors II

Type I and Type II Errors III

• Depending on context, committing one type of error may be more serious than the other

Type I and Type II Errors IV

• Anglo-American common law presumes defendant is innocent: H0

• Jury judges whether the evidence presented against the defendant is plausible assuming the defendant were in fact innocent

• If highly improbable: sufficient evidence to reject H0 and convict

  • Beyond a "reasonable doubt" that the defendant is innocent

Type I and Type II Errors V

Blackstone, William, 1765-1770, Commentaries on the Laws of England

Type I and Type II Errors VI

Significance Level, α, and Confidence Level 1−α

• The significance level, α, is the probability of a Type I error

α=P(Reject H0|H0 is true)

• The confidence level is defined as (1−α)

  • Specify in advance an α-level (0.10, 0.05, 0.01) with associated confidence level (90%, 95%, 99%)

• The probability of a Type II error is defined as β:

β=P(Don't reject H0|H0 is false)

α and β

Power and p-values

• The statistical power of the test is (1−β): the probability of correctly rejecting H0 when H0 is in fact false (e.g. not convicting an innocent person)

Power=1−β=P(Reject H0|H0 is false)

• The p-value or significance probability is the probability that, given the null hypothesis is true, the test statistic from a random sample will be at least as extreme as the test statistic of our sample

p(δ≥δi|H0 is true)

• where δ represents some test statistic
• δi is the test statistic we observe in our sample
• More on this in a bit

p-Values and Statistical Significance

• After running our test, we need to make a decision between the competing hypotheses

• Compare p-value with pre-determined α (commonly, α=0.05, 95% confidence level)

  • If p<α: statistically significant evidence sufficient to reject H0 in favor of Ha
  • If p≥α: insufficient evidence to reject H0
    • Note this does not mean H0 is true! We merely have failed to reject H0

Hypothesis Testing and the Philosophy of Science I

Hypothesis Testing and the Philosophy of Science I

Hypothesis Testing and p-Values

• Hypothesis testing, confidence intervals, and p-values are probably the hardest thing to understand in statistics

Hypothesis Testing: Which Test? I

• Rigorous course on statistics (ECMG 212 or MATH 112) will spend weeks going through different types of tests:

  • Sample mean; difference of means
  • Proportion; difference of proportions
  • Z-test vs t-test
  • 1 sample vs. 2 samples
  • χ2 test

• See today's class notes page for more

Hypothesis Testing: Which Test? II

There is Only One Test

• Fortunately, some clever statisticians realized "there is only one test" and built a nice R package called infer

1. Calculate a statistic, δi¹, from a sample of data

2. Simulate a world where δ is null (H0)

3. Examine the distribution of δ across the null world

4. Calculate the probability that δi could exist in the null world

5. Decide if δi is statistically significant

¹ δ can stand in for any test-statistic in any hypothesis test! For our purposes, δ is the slope of our regression sample, ˆβ1.

Elements of a Hypothesis Test

Hypothesis Testing with the infer Package I

• R naturally runs the following hypothesis test on any regression as part of lm():

H0:β1=0H1:β1≠0

• infer allows you to run through these steps manually to understand the process:

1. specify() a model

2. hypothesize() the null

3. generate() simulations of the null world

4. calculate() the p-value

5. visualize() with a histogram (optional)

Hypothesis Testing with the infer Package II

Hypothesis Testing with the infer Package II

Hypothesis Testing with the infer Package II

Hypothesis Testing with the infer Package II

Hypothesis Testing with the infer Package II

Hypothesis Testing with the infer Package II

Classical Statistical Inference: Critical Values of Test Statistic

• Test statistic (δ): measures how far what we observed in our sample (^β1) is from what we would expect if the null hypothesis were true (β1=0)

  • Calculated from a sampling distribution of the estimator (i.e. ^β1)
  • In econometrics, we use t-distributions which have n−k−1 degrees of freedom^.red[1]

• Rejection region: if the test statistic reaches a "critical value" of δ, then we reject the null hypothesis

¹ Again, see today's class notes for more on the t-distribution. k is the number of independent variables our model has, in this case, with just one X, k=1. We use two degrees of freedom to calculate ^β0 and ^β1, hence we have n−2 df.

Imagine a Null World, where H0 is True

Comparing the Worlds I

• From that null world where H0:β1=0 is true, we simulate another sample and calculate OLS estimators again

Comparing the Worlds II

• From that null world where H0:β1=0 is true, let's simulate 1,000 samples and calculate slope (^β1) for each

Prepping the infer Pipeline

• Before I show you how to do this, let's first save our estimated slope from our actual sample
  • We'll want this later!

# save as obs_slope
sample_slope <- school_reg_tidy %>% # this is the regression tidied with broom
  filter(term=="str") %>%
  pull(estimate)
# confirm what it is
sample_slope

## [1] -2.279808

The infer Pipeline: Specify

The infer Pipeline: Specify

data %>%
  specify(y ~ x)

CASchool %>%
  specify(testscr ~ str)

• Note nothing happens yet

The infer Pipeline: Hypothesize

The infer Pipeline: Hypothesize

%>% hypothesize(null = "independence")

CASchool %>%
  specify(testscr ~ str) %>%
  hypothesize(null = "independence")

¹ type can be either point (for specific point estimates for a single variable, such as a sample mean, (ˉx), or independence (for hypotheses about two samples or a relationship between variables). See more here.

The infer Pipeline: Generate I

The infer Pipeline: Generate I

%>% generate(reps = n,
             type = "permute")

i %>%
  generate(reps = 1000,
           type = "permute")

¹ Note for spacing on the slide, I saved the previous code as i and pipe it into the remainder.

The infer Pipeline: Generate II

%>% generate(reps = n,
             type = "permute")

The infer Pipeline: Generate III

%>% generate(reps = n,
             type = "permute")

¹ You can do either of these in base R with sample(), which has 3 arguments: a vector to sample from, size (number of obs), and replace equal to TRUE or FALSE. See more for infer here.

The infer Pipeline: Calculate I

The infer Pipeline: Calculate I

%>% calculate(stat = "")

i %>%
  generate(reps = 1000, type = "permute") %>%
  calculate(stat = "slope")

The infer Pipeline: Calculate II

%>% calculate(stat = "")

The infer Pipeline: Get p Value

%>% get_p_value(obs stat = "",
                direction = "both")

simulations %>%
  get_p_value(obs_stat = sample_slope,
              direction = "both")

⁺ Note here I saved the results of our previous code as simulations for spacing.

The infer Pipeline: Get Confidence Interval

%>% get_ci(
  level = 0.95,
  type = "se",
  point_estimate = "")

simulations %>%
  get_confidence_interval(level = 0.95,
                          type = "se",
                          point_estimate = sample_slope)

The infer Pipeline: Visualize I

The infer Pipeline: Visualize I

%>% visualize()

simulations %>%
 visualize()

The infer Pipeline: Visualize II

%>% visualize()

simulations %>%
 visualize(obs_stat = sample_slope)

The infer Pipeline: Visualize p-value

%>% visualize()+
  shade_p_value()

simulations %>%
 visualize(obs_stat = sample_slope)+
  shade_p_value(obs_stat = sample_slope,
                direction = "two_sided")

The infer Pipeline: Visualize Confidence Intervals

%>% visualize()+
  shade_ci()

simulations %>%
 visualize(obs_stat = sample_slope)+
  shade_confidence_interval(ci_values)

The infer Pipeline: Visualize is a Wrapper of ggplot

• infer's visualize() function is just a wrapper function for ggplot()
  • you can take your simulations tibble and just ggplot a normal histogram

simulations %>%
  ggplot(data = .)+
  aes(x = stat)+
  geom_histogram(color="white", fill="indianred")+
  geom_vline(xintercept = sample_slope,
             color = "blue",
             size = 2,
             linetype = "dashed")+
  labs(x = expression(paste("Distribution of ", hat(beta[1]), " under ", H[0], " that ", beta[1]==0)),
       y = "Samples")+
    theme_classic(base_family = "Fira Sans Condensed",
           base_size=20)

What R Does: Classical Statistical Inference I

¹ k is the number of X variables.

What R Does: Classical Statistical Inference II

What R Does: Classical Statistical Inference III

¹ Think of our simulated distribution, the center was 0.

1-Sided vs. 2-Sided p-values I

1-Sided vs. 2-Sided p-values I

Ha:β1≠0

p-value: 2×Prob(t>|ti|)

Hypothesis Tests in Regression Output I

summary(school_reg)

## 
## Call:
## lm(formula = testscr ~ str, data = CASchool)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -47.727 -14.251   0.483  12.822  48.540 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 698.9330     9.4675  73.825  < 2e-16 ***
## str          -2.2798     0.4798  -4.751 2.78e-06 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 18.58 on 418 degrees of freedom
## Multiple R-squared:  0.05124,    Adjusted R-squared:  0.04897 
## F-statistic: 22.58 on 1 and 418 DF,  p-value: 2.783e-06

Hypothesis Tests in Regression Output II

• In broom's tidy() (with confidence intervals)

tidy(school_reg, conf.int=TRUE)

Conclusions

H0:β1=0Ha:βa≠0

• Because the hypothesis test's p-value < α (0.05)...

• We have sufficient evidence to reject H0 in favor of our alternative hypothesis. Our sample suggests that there is a relationship between class size and test scores.

• Using the confidence intervals:

• We are 95% confident that the true marginal effect of class size on test scores is between −3.22 and −1.34.

Hypothesis Testing and Confidence Intervals: Relationship

• Confidence intervals are all two-sided by nature CI0.95=([^β1−2×se(^β1)],[^β1+2×se(^β1]))

• Hypothesis test (t-test) of H0:β1=0 computes a t-value of¹ t=^β1se(^β1)

  and p<0.05 when t≥2

• If a confidence interval contains the H0 value (i.e. 0, for our test), then we fail to reject H0.

¹ Since our null hypothesis is that (\beta_{1,0}=0), the test statistic simplifies to this neat fraction.

Common Misconceptions about p-values

• All of the following are FALSE interpretations of p (and below are reasons why each is wrong)

1. p is the probability that the alternative hypothesis is false

   • We can never prove an alternative hypothesis, only tentatively reject a null hypothesis

2. p is the probability that the null hypothesis is true

   • We're not proving the H0 is false, only saying that it's very unlikely that if H0 were true, we'd obtain a slope as rare as our sample's slope

3. p is the probability that our observed effects were produced purely by random chance

   • p is computed under a specific model (think about our null world) that assumes H0 is true

4. p tells us how significant our finding is

   • p tells us nothing about the size or the real world significance of any effect deemed "statistically significant"
   • it only tells us that the slope is statistically significantly different from 0 (if H0 is β1=0)

Abusing p-Values I

Source: SMBC

Abusing p-Values II

Wasserstein, Ronald L. and Nicole A. Lazar, (2016), "The ASA's Statement on p-Values: Context, Process, and Purpose," The American Statistician 30(2): 129-133

p-value Clarification

• Again, p-value is the probability that, assuming the null hypothesis is true, we obtain (by pure random chance) a test statistic at least as extreme as the one we estimated for our sample

• A low p-value means either (and we can't distinguish which):

  1. H0 is true and a highly improbable event has occurred OR
  2. H0 is false

Significance In Regression Tables