Descriptive statistics

Sven Rieger

May 15, 2024

Agenda

Preface

navigate vertical ↓

Preface: Used packages

  • The following packages are used:
descrPkg <- c("merTools",
              "sn",
              "knitr",
              "flextable",
              "psych",
              "lavaan",
              "ggplot2")
  • Install packages when not already installed:
lapply(X = descrPkg,
       FUN = function(x) {
          if( !x %in% rownames(installed.packages()) ) { 
            install.packages(x) }
            }
       )
  • Load (a subset of) the required package(s) into the R session.
library(ggplot2)
library(flextable)

Preface: Cite the packages

Print list of packages and cite them via Pandoc citation.

Show/hide fenced code
```{r}
#| label: write-pkgs
#| code-fold: true
#| code-summary: "Show/hide fenced code"
#| output-location: fragment
#| output: asis

for (i in 1:length(descrPkg)) {
  
  cat(paste0(i, ". ",
             descrPkg[i],
             " [", "v", utils::packageVersion(descrPkg[i]),", @R-", descrPkg[i],
             "]\n"))
}
```
  1. merTools (v0.6.1, Knowles & Frederick, 2024)
  2. sn (v2.1.1, Azzalini, 2023)
  3. knitr (v1.44, Xie, 2023)
  4. flextable (v0.9.4, Gohel & Skintzos, 2024)
  5. psych (v2.3.9, Revelle, 2024)
  6. lavaan (v0.6.16, Rosseel et al., 2023)
  7. ggplot2 (v3.5.0, Wickham et al., 2023)

Preface: Data matrix

Variables (e.g., characterisics), units (e.g., persons) and data (e.g., measurements) are often presented in matrix form. A matrix is a system of \(n \cdot p\) quantities and looks like in the following:

\[ \begin{bmatrix} X_{11} & X_{12} & \cdots & X_{1p} \\ X_{21} & X_{22} & \cdots & X_{2p} \\ \vdots & \vdots & & \vdots \\ X_{n1} & X_{n2} & \cdots & X_{np} \end{bmatrix} \]

  • \(n\) rows; 1 row is also known as a vector or row matrix
  • \(p\) columns; 1 column is also known as a vector or column matrix

Descriptive statistics

navigate vertical ↓

Overview

  • Quantiles (not covered)
  • Measures of central tendency
    • Mean
    • Weighted mean (not covered)
    • Weighted geometric mean (not covered)
    • Median
    • Mode (not covered)
  • Measures of variability
  • Measures of shape
    • Skewness (not covered)
    • Kurtosis (not covered)

Example data set

Consider the following 2 vectors within the example data set

exDat <- data.frame(
  numVec = c(1, 2, 5, 3, 8),
  chrVec = c("low", "med", "low", "high", "high")
)

Absolute Frequencies

Absolute frequencies refer to the numbers of a particular value or category appearing in a variable. It may be abbreviated with \(n_j\) where \(n\) is the number of a specific value/category \(j\).

Example Frequency table
Category \(j\) Absolute Frequency (\(n_j\))
low (\(j=1\)) 2
med (\(j=2\)) 1
high (\(j=3\)) 2
\(\sum\) \(\sum_{j=1}^3n_j=n=5\)

Absolute Frequencies in R

with(exDat,
     table(chrVec)) 
chrVec
high  low  med 
   2    2    1 

An important argument (useNA) and another useful function (addmargins())…

with(exDat,
     table(chrVec, useNA = "always")) |>
  addmargins() 
chrVec
high  low  med <NA>  Sum 
   2    2    1    0    5 
exDat |>
  dplyr::group_by(chrVec) |>
  dplyr::summarise(absFreq = dplyr::n())
# A tibble: 3 × 2
  chrVec absFreq
  <chr>    <int>
1 high         2
2 low          2
3 med          1

Relative Frequencies

Relative frequencies refer to the proportion of a specific value or category relative to the total number of observations (\(n\)).

\[ h_j=\frac{n_j}{n} \]

Example Frequency table
Category \(j\) Absolute Frequency (\(n_j\)) Relative Frequency (\(h_j\))
low (\(j=1\)) 2 0.40
med (\(j=2\)) 1 0.20
high (\(j=3\)) 2 0.40
\(\sum\) \(\sum_{j=1}^3n_j=n=5\) \(\sum_{j=1}^3h_j=1\)

Relative Frequencies in R

with(exDat,
     table(chrVec)/sum(table(exDat$chrVec))) 
chrVec
high  low  med 
 0.4  0.4  0.2 

Another useful function (sprintf()) to force 2 decimal and add %…

with(exDat,
     table(chrVec)/sum(table(chrVec))) |>
     (function(x) sprintf("%.2f%%", x*100))()
[1] "40.00%" "40.00%" "20.00%"
exDat |>
  dplyr::select(chrVec) |>
  dplyr::group_by(chrVec) |>
  dplyr::summarise(absFreq= dplyr::n()) |>
  dplyr::mutate(relFreq = absFreq/sum(absFreq)) 
# A tibble: 3 × 3
  chrVec absFreq relFreq
  <chr>    <int>   <dbl>
1 high         2     0.4
2 low          2     0.4
3 med          1     0.2

Mean

The mean (or arithmetic mean, average) is the sum of a collection of numbers divided by the count of numbers in the collection. The formula is given in Equation 1.

\[ \bar{x} = \frac{1}{n} \sum_{i=1}^n x_i=\frac{x_1+x_2+\dots+x_n}{n} \qquad(1)\]

For example, consider a vector of numbers: \(x = 1, 2, 5, 3, 8\)

\[ \bar{x} = \frac{(1+2+5+3+8)}{5}=3.8 \]

If the underlying data is a sample (i.e., a subset of a population), it is called the sample mean.

How to calculate the mean in R?

with(exDat,
     mean(numVec))
[1] 3.8

A brief note on missing data

In R missing values/data are represented by the symbol NA. Most of the basic functions cannot deal appropriately with missing data.


To demonstrate this we create another example vector (exVec2).

numVec2 <- c(1, 2, 5, 3, 8, NA)
mean(numVec2)
[1] NA

If there is missing data, we are required to set the argument na.rm to TRUE.

mean(numVec2, na.rm = TRUE)
[1] 3.8


Omitting or deleting missing values should–in most scenarios–be avoided altogether (Enders, 2023; Schafer & Graham, 2002)

Median

The median is the value separating the higher half from the lower half of a data sample, a population, or a probability distribution. For a data set, it may be thought of as “the middle” value. The formulas are given in Equation 2.

\[ Mdn = \widetilde{x} = \begin{cases} x_{(n+1)/2} & \:\: \text{if } n \text{ is odd} \\ (x_{n/2} + x_{(n/2)+1}) / 2 & \:\: \text{if } n \text{ is even} \end{cases} \qquad(2)\]

Consider again the vector of numbers: \(x = 1, 2, 5, 3, 8\) with length \(n = 5\). To calculate the median you need to first, order the the vector: \(x = 1, 2, 3, 5, 8\) and then apply the corresponding formula (odd vs. even; here odd):

\[ \widetilde{x}=x_{\frac{(5+1)}{2}}=x_3 = 3 \]

How to calculate the median in R?

with(exDat,
     median(numVec))
[1] 3

Variance

The variance is the expectation of the squared deviation of a random variable from its mean. Usually it is distinguished between the population and the sample variance. The formula of the population variance is given in Equation 3.

\[VAR(X) = \sigma^2 = \frac{1}{N} \sum\limits_{i=1}^N (x_i - \mu)^2 \qquad(3)\]

The formula of the sample variance is given in Equation 4.

\[ VAR(X) = s^2 = \frac{1}{n-1} \sum\limits_{i=1}^n (x_i - \bar{x})^2 \qquad(4)\]

Using again the vector \(x = 1, 2, 5, 3, 8\), the sample variance is calculated as follows:

\[ Var(X) =\frac{1}{4}((1-3.8)^2 + (2-3.8)^2 + (5-3.8)^2 + (3-3.8)^2 + (8-3.8)^2) = 7.7 \]

How to calculate the variance in R?

with(exDat,
     var(numVec))
[1] 7.7

Standard Deviation

The standard deviation is defined as the square root of the variance. Again, it is distinguished between the population and the sample variance. The formula of the population standard deviation is given in Equation 5.

\[ SD(X) = \sigma = \sqrt{\sigma^2} \qquad(5)\]

The formula of the population standard deviation is given in Equation 6.

\[ SD(X) = s = \sqrt{s^2} \qquad(6)\]

Recall the variance calculation from the previous slide, the (sample) variance of the vector is \(7.7\).

\[ SD(X) = \sqrt{7.7}=2.774887 \]

How to calculate the standard deviation in R?

with(exDat,
     sd(numVec))
[1] 2.774887

Range

The range of a vector is the difference between the largest (maximum) and the smallest (minimum) values/observations.

\[ Range(x) = R = x_{max}-x_{min} \qquad(7)\]

How to calculate the range in R?

with(exDat,
     range(numVec))
[1] 1 8

Alternatively, calculate minimum and maximum separately…

with(exDat,{
     c(min(numVec),
       max(numVec))})
[1] 1 8

To compute the range apply Equation 7.

with(exDat,
     max(numVec)-min(numVec))
[1] 7

Exercise

navigate vertical ↓

Put everything together 0

Exercise

Let us calculate several descriptive statistics (e.g., mean, standard deviation, minimum and maximum) for multiple variables. For this exercise, we use a subset of the HSB dataset which is provided in the merTools package (Knowles & Frederick, 2024) (for some details see here):

Put everything together 0

Exercise

Let us calculate several descriptive statistics (e.g., mean, standard deviation, minimum and maximum) for multiple variables. For this exercise, we use a subset of the HSB dataset which is provided in the merTools package (Knowles & Frederick, 2024) (for some details see here):

dat <- merTools::hsb
head(dat, 10)
   schid minority female    ses mathach size schtype meanses
1   1224        0      1 -1.528   5.876  842       0  -0.428
2   1224        0      1 -0.588  19.708  842       0  -0.428
3   1224        0      0 -0.528  20.349  842       0  -0.428
4   1224        0      0 -0.668   8.781  842       0  -0.428
5   1224        0      0 -0.158  17.898  842       0  -0.428
6   1224        0      0  0.022   4.583  842       0  -0.428
7   1224        0      1 -0.618  -2.832  842       0  -0.428
8   1224        0      0 -0.998   0.523  842       0  -0.428
9   1224        0      1 -0.888   1.527  842       0  -0.428
10  1224        0      0 -0.458  21.521  842       0  -0.428

Put everything together I

A flexible approach would be…

1myVar <- c("Math achievement" = "mathach",
           "Gender" = "female",
           "Socioeconomic status" = "ses",
           "Class size" = "size")

2exDescr <- apply(
3  X = dat[,myVar],
4  MARGIN = 2,
5  FUN = function(x) {
6    ret <- c(
             mean(x, na.rm = T),
             sd(x, na.rm = T),
             min(x, na.rm = T),
             max(x, na.rm = T)
             )
7    return(ret)
    })
1
Create a (named) character vector of the variables by using the c() function.
2
Use the apply function to apply a or multiple function(s) on data (here: 4 columns).
3
The input is the dataset with the selected columns of interest (see 1.).
4
MARGIN = 2 indicates that the function should be applied over columns.
5
Create the function that should be applied. Here we calculate the mean(), sd(), min() and max().
6
Create a temporary R object, which should be later returned (here: the vector ret)
7
Return the temporary object and close functions.

Put everything together II

Print the results…

exDescr |>
  print()
       mathach    female           ses      size
[1,] 12.747853 0.5281837  0.0001433542 1056.8618
[2,]  6.878246 0.4992398  0.7793551951  604.1725
[3,] -2.832000 0.0000000 -3.7580000000  100.0000
[4,] 24.993000 1.0000000  2.6920000000 2713.0000


This is a weird format; variables should be in rows not columns. Transpose…

exDescr |>
  t() |>
  print()
                [,1]        [,2]    [,3]     [,4]
mathach 1.274785e+01   6.8782457  -2.832   24.993
female  5.281837e-01   0.4992398   0.000    1.000
ses     1.433542e-04   0.7793552  -3.758    2.692
size    1.056862e+03 604.1724993 100.000 2713.000


Better, but still not really convincing…

Making a table I

1exDescrTab <- exDescr |>
2    t() |>
    as.data.frame() |>
3    (\(d) cbind(names(myVar), d))() |>
4    flextable() |>
5    theme_apa() |>
6    set_header_labels(
      "names(myVar)" = "Variables",
      V1 = "Mean",
      V2 = "SD",
      V3 = "Min",
      V4 = "Max") |>
7    align(part = "body", align = "c") |>
    align(j = 1, part = "all", align = "l") |>
8    add_footer_lines(
      as_paragraph(as_i("Note. "),
                   "This is a footnote.")
      ) |>
    align(align = "left", part = "footer") |>
9    width(j = 1, width = 2, unit = "in") |>
    width(j = 2:5, width = 1, unit = "in")
1
Take the results (here: exDescr object)…
2
…and transpose (i.e., using the t() function) and coerce it to a data.frame object (as.data.frame())
3
Use the so-called lambda (or anonymous) function to bind (using the cbind() function) the variable names as the first column to the dataset.
4
Apply the flextable() function.
5
Use the APA theme (theme_apa()).
6
Rename the column names (set_header_labels()).
7
Center body part of the table (align()).
8
Add a footnote (add_footer_lines) and align it to the left.
9
Change column width (width) to 2 resp. 1 inch.

Making a table II

Print the table.

Code
exDescrTab
Table 1: Descriptive statistics

Variables

Mean

SD

Min

Max

Math achievement

12.75

6.88

-2.83

24.99

Gender

0.53

0.50

0.00

1.00

Socioeconomic status

0.00

0.78

-3.76

2.69

Class size

1,056.86

604.17

100.00

2,713.00

Note. This is a footnote.

Table export

If you want to export the table…

exDescrTab |>
  set_caption(caption = "Table X.\nDescriptive statistics") |>
  save_as_docx(path = "descr-tab.docx")

Descriptive statistics with the psych package

  • Alternatively, it is convenient to use additional R packages such as the psych package (Revelle, 2024) to calculate descriptive statistics

  • Here we use the describe function (with the fast argument set to TRUE) to calculate the descriptive statistics of all variables within the example data set

dat |>
  subset(select = -c(1)) |>
  psych::describe(fast = TRUE) |>
  flextable() |>
  colformat_double(digits = 2)
Table 2: Descriptive statistics with the psych package

vars

n

mean

sd

min

max

range

se

1

7,185.00

0.27

0.45

0.00

1.00

1.00

0.01

2

7,185.00

0.53

0.50

0.00

1.00

1.00

0.01

3

7,185.00

0.00

0.78

-3.76

2.69

6.45

0.01

4

7,185.00

12.75

6.88

-2.83

24.99

27.82

0.08

5

7,185.00

1,056.86

604.17

100.00

2,713.00

2,613.00

7.13

6

7,185.00

0.49

0.50

0.00

1.00

1.00

0.01

7

7,185.00

0.01

0.41

-1.19

0.83

2.02

0.00

Exercise

Style the table according to your ideas/demands and export it to Word.

References

Azzalini, A. (2023). Sn: The skew-normal and related distributions such as the skew-t and the SUN. http://azzalini.stat.unipd.it/SN/
Eid, M., Gollwitzer, M., & Schmitt, M. (2013). Statistik und Forschungsmethoden: Lehrbuch ; mit Online-Materialien (3., korrigierte Auflage). Beltz.
Enders, C. K. (2023). Missing data: An update on the state of the art. Psychological Methods. https://doi.org/10.1037/met0000563
Gohel, D., & Skintzos, P. (2024). Flextable: Functions for tabular reporting. https://ardata-fr.github.io/flextable-book/
Knowles, J. E., & Frederick, C. (2024). merTools: Tools for analyzing mixed effect regression models. https://CRAN.R-project.org/package=merTools
Revelle, W. (2024). Psych: Procedures for psychological, psychometric, and personality research. https://personality-project.org/r/psych/ https://personality-project.org/r/psych-manual.pdf
Rosseel, Y., Jorgensen, T. D., & De Wilde, L. (2023). Lavaan: Latent variable analysis. https://lavaan.ugent.be
Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7(2), 147–177. https://doi.org/10.1037/1082-989X.7.2.147
Wickham, H., Chang, W., Henry, L., Pedersen, T. L., Takahashi, K., Wilke, C., Woo, K., Yutani, H., & Dunnington, D. (2023). ggplot2: Create elegant data visualisations using the grammar of graphics. https://ggplot2.tidyverse.org
Xie, Y. (2023). Knitr: A general-purpose package for dynamic report generation in r. https://yihui.org/knitr/