LIS 4273 - Introduction to Statistics & Analytics Using R

Module3.utf8

Set environment

env <- c("dplyr", "ggplot2", "glue")
lapply(env, library, character.only = 1)

Load assigned dataset

boardingScreeners <- c(287,243,237,227,247,264,247,247,251,254,
                       277,303,285,254,280,264,261,292,248,253)
securityViolations <- c(271,261,230,225,236,252,243,247,238,274,
                        256,305,273,234,261,265,241,292,228,252)
secData <- tibble( boardingScreeners,securityViolations )

Create EDA scatter plot

plot(secData)

The plot shows a clear positive relationship: security violations (y) increase as more screening agents (x) are used. One would expect any calculated correlation value to be positive and nearer to one (1) than zero (0).

Calculate Pearson’s Sample correlation coefficient

corPearson <- cor.test(secData$boardingScreeners, secData$securityViolations,
                       method="pearson")
corPearson

## 
##  Pearson's product-moment correlation
## 
## data:  secData$boardingScreeners and secData$securityViolations
## t = 6.5033, df = 18, p-value = 4.088e-06
## alternative hypothesis: true correlation is not equal to 0
## 95 percent confidence interval:
##  0.6276251 0.9339189
## sample estimates:
##       cor 
## 0.8375321

Calculate Spearman’s Rank Coefficient

corSpearman <- cor.test(secData$boardingScreeners, secData$securityViolations,
                        method="spearman")

## Warning in cor.test.default(secData$boardingScreeners,
## secData$securityViolations, : Cannot compute exact p-value with ties

corSpearman

## 
##  Spearman's rank correlation rho
## 
## data:  secData$boardingScreeners and secData$securityViolations
## S = 322.47, p-value = 0.0001096
## alternative hypothesis: true rho is not equal to 0
## sample estimates:
##       rho 
## 0.7575423

Describe the association between boarding screeners and security violations

Correlation measures both the strength of association between two variables and the direction of that relationship. The scale ranges from -1 to +1 and the result is expressed as a decimal value. A value of 0 typically means that there is no relationship.

The Pearson correlation is the most cited measure when continuous and linearly related variables are involved. For the above dataset, the r value is 0.84 which shows a strong positive correlation between screeners and violations.

The Spearman correlation is a non-parametric test. In essence it shows a rank correlation which can be between both continuous or ordinal variables. In the case of continuous data, the raw values are ranked - they become rank-ordered variables. For the dataset, the Spearman correlation comes in at 0.76. Like the Pearson score, it shows a strong and positive association.

While a graph for a Pearson correlation will show a constant rate of change, the Spearman describes a monotonic relationship where the rate of change can vary. Lastly, as the ref points out, when the linearity of the data is in question it may be best to opt for the Spearman correlation.

Ref:
Towards Data Science - Clearly explained: Pearson vs Spearman Correlation Coefficient

Prep report

# prep label
corLabel <- glue("
    Correlation Coeficients
    Pearson: {round(corPearson$estimate,2)}
    Spearman: {round(corSpearman$estimate,2)}
    ")
ggplot(secData, aes(boardingScreeners,securityViolations) ) +
    geom_point() +
    geom_smooth(method="lm", se=0, aes(colour="lm")) +
    geom_smooth(method="loess", se=0, aes(colour="loess")) +
    geom_label(label = corLabel, aes(240,295) ) +
    labs(title = "Airport Security Screening", subtitle = "(1988-1999)",
         x = "Screeners", y = "Violations  ", colour = "Model") +
    theme_classic() + 
    theme( plot.title = element_text(hjust=0.5),
           plot.subtitle = element_text(hjust=0.5),
           axis.title.x = element_text(vjust=-1),
           axis.title.y = element_text(angle = 0, vjust=0.5)
           )

GitHub

Related file(s) can be found at Git Me