The apply family, visualized
Loops in R usually get written as apply-style calls, and almost every lab this week uses at least one. They are hard to learn for a specific reason. Two things are invisible. You cannot see what your function is handed on each pass, and you cannot see what shape comes back. This page makes both visible, one step at a time. Its companion, dplyr & the tidyverse, does the same for data frames, and the two pages cross-reference each other wherever they do the same job.
The one idea
Every function on this page does the same three things. They differ only in how they do the first and the last.
Step 2 is yours and it is the easy part. The confusion is always step 1 (what is a "piece"? an element? a row? a group?) and step 3 (do I get a list, a vector, or a matrix back?). Keep those two questions in mind and the whole family collapses into one pattern.
x is a vector of 5 counts, lapply(x, f) hands f a single number, five times. If m is a 4×3 matrix, apply(m, 1, f) hands f a vector of length 3, four times. Nearly every apply bug is a wrong answer to that one question.The functions, one by one
Everything below uses the same four objects, so you can compare like with like. The step-through app in the next section uses them too.
# four sites, surveyed a different number of times each sites <- list(A = c(3,5,4), B = c(0,1), C = c(7,6,9,8), D = c(2)) # the same idea as a matrix: 4 sites (rows) x 3 visits (cols) m <- rbind(c(2,0,1), c(5,4,6), c(0,1,0), c(3,3,2)) # a flat vector plus a parallel factor saying which group each value is in counts <- c(4, 6, 3, 9, 11, 2, 1) habitat <- c("forest","forest","forest","meadow","meadow","edge","edge") # two vectors that line up element by element eff <- c(2, 3, 1, 4) # survey hours at each site tot <- c(12, 20, 6, 25) # animals seen at each site
lapplywhen each answer is a complicated object, such as a model fit, a data frame, or a vector of varying lengthvapplywhen each answer is one number and this is real code (sapplyif you are just poking at the console)applywhen your data is a matrix and you want one answer per row or per columntapplywhen you have a vector plus a grouping factor and want one answer per groupmapply/Mapwhen your function needs an element from two or more vectors at oncereplicatewhen there is nothing to iterate over and you just want to run something n timesReducewhen you want to collapse a whole vector into a single accumulated value
lapply(X, FUN) keep every result, whatever shape it is
Reach for it when each answer is more than a single number, such as a fitted model, a data frame, or a vector whose length you cannot predict. A list is the only container that holds those without mangling them, and lapply is the one function here that never tries to be clever about the result.
lapply(sites, mean) #> $A #> [1] 4 #> #> $B #> [1] 0.5 #> #> $C #> [1] 7.5 #> #> $D #> [1] 2
Four means, but wrapped one per list slot rather than laid out as a vector. That is verbose here and exactly right when the results are models. Day 5 builds the starting values for three MCMC chains this way, because each chain's starting values are themselves a list.
sapply(X, FUN) the same, then tidy the result if it can
Reach for it when you are working interactively, expect one number per element, and want a plain named vector rather than a list to read.
sapply(sites, mean) #> A B C D #> 4.0 0.5 7.5 2.0
Much easier to read, but the tidying is a guess R makes by looking at what came back. When the guess changes, your result type changes with it and nothing warns you. That is trap 1 below, and the reason the next function exists.
vapply(X, FUN, FUN.VALUE) sapply, with the answer's shape promised up front
Reach for it when the code will be run more than once, in a script, a function, or a simulation. The third argument is a template: an example of what one answer looks like. R checks every answer against it and stops if one disagrees.
vapply(sites, mean, numeric(1)) # promise: one number each time #> A B C D #> 4.0 0.5 7.5 2.0 vapply(sites, function(v) if (mean(v) > 3) "high" else "low", character(1)) #> A B C D #> "high" "low" "high" "low"
The templates you will use: numeric(1), character(1), logical(1), and numeric(2) when the answer is genuinely a pair. Typing that extra argument is the entire benefit. It converts a silent shape change into an error on the line that caused it.
apply(m, 1, FUN) one summary per row
Reach for it when your data is a matrix, the rows are your units, sites, individuals, simulated datasets, and you want a single number for each one.
apply(m, 1, max) # the largest count ever seen at each site #> [1] 2 6 1 3
Day 5 uses precisely this line to start its N-mixture model. The most animals you ever counted at a site is a sensible lower bound for how many are really there, so apply(y, 1, max) becomes the initial value for that site's abundance.
apply(m, 2, FUN) one summary per column
Reach for it when the columns are what you want summarised. The commonest case in this course is a matrix of MCMC draws, where every row is one posterior draw, every column is one x value, so a summary down each column is the posterior at that x.
apply(m, 2, mean) # mean count on each visit, across sites #> [1] 2.50 2.00 2.25 # Day 4, on a matrix of posterior draws: apply(mu_draws, 2, quantile, probs = c(.025, .5, .975))
Watch the shape when FUN returns several numbers, as quantile does. You get a matrix whose columns are the per-column answers, not its rows. Trap 2.
tapply(X, INDEX, FUN) one summary per group
Reach for it when you have a vector of values and a second, parallel vector saying which group each value belongs to. This is the "group means" function.
tapply(counts, habitat, mean) #> edge forest meadow #> 1.500000 4.333333 10.000000 tapply(counts, habitat, length) # how many sites in each habitat #> edge forest meadow #> 2 3 2
Two things to notice. The groups come back in sorted order, not the order they appear in your data, which is why the result is named, and why you should index it by name rather than position. And FUN is handed the whole group at once, so anything that summarises a vector works here.
Relatives: aggregate() does this for several columns of a data frame at once; split() hands you the groups themselves instead of a summary.
mapply(FUN, ...) and Map(FUN, ...) walk several vectors in step
Reach for it when your function needs the i-th element of two or more vectors at the same time. lapply and friends only ever hand your function one thing; these hand it one from each.
mapply(function(hours, seen) seen / hours, eff, tot) # animals per hour #> [1] 6.000000 6.666667 6.000000 6.250000
Map is the same function with the simplification switched off. It always returns a list, exactly as lapply does. Use Map when the per-pair answers are complicated, mapply when they are single numbers.
replicate(n, expr) run this n times and collect the answers
Reach for it when there is nothing to iterate over. No input is being split up. The expression is simply evaluated again from scratch n times, drawing fresh random numbers on each pass. This is the engine behind every simulation in Days 1–3.
set.seed(1) replicate(6, mean(rpois(5, lambda = 3))) # six sample means #> [1] 2.8 3.8 2.6 4.2 3.0 2.2 set.seed(1) x <- replicate(2000, mean(rpois(5, 3))) # now it is a sampling distribution mean(x); sd(x) #> [1] 3.0051 #> [1] 0.7881273 <- the standard error of a mean of 5 Poisson(3) draws
The set.seed() is not decoration. Seed every simulation you intend to report, or you cannot reproduce your own numbers. That second block is the whole idea of a sampling distribution, in two lines: repeat the study 2000 times, look at the spread of the answers. It simplifies like sapply, so a block returning two numbers gives you a 2 × n matrix.
Reduce(f, x) collapse a vector to a single value
Reach for it when you want to combine elements pairwise until one value is left: a running total, the intersection of many sets, a series of merges. Unlike everything above, the result is one value rather than one-per-element.
Reduce(`+`, c(3,1,4,1,5)) #> [1] 14 Reduce(`+`, c(3,1,4,1,5), accumulate = TRUE) # show the working #> [1] 3 4 8 9 14
accumulate = TRUE is the version worth knowing. It returns every intermediate value, which is what the app draws when you step through it.
Two companions that are not loops at all
split(x, g) makes the pieces without applying anything, which is useful when you want the groups themselves:
split(counts, habitat) #> $edge #> [1] 2 1 #> #> $forest #> [1] 4 6 3 #> #> $meadow #> [1] 9 11
That is a list, so lapply or sapply takes it straight away, and sapply(split(x, g), mean) is exactly what tapply(x, g, mean) does. Day 3 uses boxplot(split(resid(lm1), Streams$Stream)) to get one box per stream.
do.call(f, list) does not iterate at all. It calls f once, using the list's elements as its arguments, which is how you assemble a pile of pieces into one object:
rows <- lapply(sites, function(v) data.frame(n = length(v), mean = mean(v))) do.call(rbind, rows) # rbind(rows$A, rows$B, rows$C, rows$D) #> n mean #> A 3 4.0 #> B 2 0.5 #> C 4 7.5 #> D 1 2.0
The lapply-then-do.call(rbind, ...) pair is the standard way to build a results table from a loop, and the Day 3 problem set uses it to build its simulated streams and fish.
Watch it run
Pick a function, pick what your function does, then step through. The orange box shows exactly what f receives on the current pass; the right-hand pane shows the result being assembled.
All of them on one screen
A single view for when you know which function you want and just need to recall its shape. Read each row as: what gets split off, what your function sees, what you get back.
| Call | One "piece" is | f receives | You get back |
|---|---|---|---|
lapply(x, f) | one element of x | a single element | always a list, same length as x |
sapply(x, f) | one element of x | a single element | a vector, a matrix, or a list, depending on what f returns |
vapply(x, f, numeric(1)) | one element of x | a single element | exactly the shape you declared, or an error |
apply(m, 1, f) | one row | a vector as long as the row | one result per row |
apply(m, 2, f) | one column | a vector as long as the column | one result per column |
tapply(x, g, f) | all of x sharing a level of g | a vector of that group's values | a named vector, one entry per level |
mapply(f, a, b) | the i-th element of each input | two arguments, in step | simplified like sapply; Map is the same but always returns a list |
replicate(n, expr) | nothing, it re-runs expr | no argument at all | simplified like sapply; the workhorse for simulations |
split(x, g) then lapply | a whole group | the group's values | a list, one entry per level |
do.call(rbind, lst) | — | — | calls rbind once with every list element as an argument |
Reduce(f, x) | the running total and the next element | two arguments | a single accumulated value |
The same job in dplyr
Most of what is on this page has a tidyverse counterpart, and you will see both in this course. The difference is not style but what the object is: the apply family works on vectors, lists and matrices; dplyr works on data frames and nothing else. Choosing between them is mostly a question of what you are holding.
| The job | base R | dplyr |
|---|---|---|
| One summary per group | tapply(d$x, d$g, mean) | d |> group_by(g) |> summarise(m = mean(x)) |
| Group sizes | table(d$g) | count(d, g) |
| Same summary over several columns | sapply(d[c("x","y")], mean) | summarise(across(c(x, y), mean)) |
| Split into groups, keep the pieces | split(d, d$g) | group_split(d, g) |
| Something complicated per group | do.call(rbind, lapply(split(d, d$g), f)) | d |> group_by(g) |> group_modify(f) |
| Element-wise over two vectors | mapply(f, d$x, d$y) | mutate(z = f(x, y)), already vectorised |
| Add a derived column | d$rate <- d$count / d$hours | mutate(rate = count / hours) |
| Keep some rows | d[d$x > 4, ] | filter(x > 4) |
| Stack a list of results into a table | do.call(rbind, lst) | bind_rows(lst) |
| Run something n times | replicate(n, expr) | no dplyr equivalent; use replicate() or purrr::map() |
| Summarise a matrix's columns | apply(m, 2, mean) | convert to a data frame first, or just use colMeans() |
The row that matters most, side by side. This is the same data as the sites list above, written as one row per observation:
base R
sapply(sites, mean) #> A B C D #> 4.0 0.5 7.5 2.0
dplyr
surveys |> group_by(site) |> summarise(mean = mean(count)) #> site mean #> 1 A 4 #> 2 B 0.5 #> 3 C 7.5 #> 4 D 2
Same four numbers. What differs is the result: dplyr returns a data frame carrying the grouping variable as a column, so the next verb can keep working on it. sapply returns a named vector, compact and instantly printable, but you would need as.data.frame() and some renaming before you could join it to anything or hand it to ggplot.
apply() and sapply() dominate there and dplyr barely appears.The full companion page is Data manipulation with dplyr, which walks the verbs the same way this page walks the apply family, and steps through pipelines on the same survey data.
Four traps
1. sapply changes its return type behind your back
This is the single most common apply bug, and it is why code that worked on Tuesday breaks on Wednesday with different data. sapply looks at the results and guesses:
sapply(sites, mean) # every result length 1 → a vector sapply(sites, range) # every result length 2 → a 2 x n MATRIX sapply(sites, function(v) v[v > 2]) # lengths differ → back to a LIST
Nothing warns you. The fix is vapply, which makes you state the shape up front and fails loudly when the data disagree:
vapply(sites, mean, numeric(1)) # promises: one number each time
vapply in anything you will run more than once. The extra argument is the point, not overhead. It turns a silent shape change into an immediate error, at the line that caused it rather than three steps later.2. apply with a multi-value function transposes what you expect
If f returns k values and you apply it down columns, you get a k × ncol matrix, so each original column's answer is a column of the result, not a row. Day 4 uses exactly this:
BCI <- apply(mu_draws, 2, quantile, probs = c(.025, .50, .975)) dim(BCI) # 3 x n, NOT n x 3 — so BCI[1, ] is the lower bound for every column
Read the result with dim() before you index it. If you want it the other way round, t().
3. apply on a data frame silently converts everything
apply is a matrix function. Hand it a data frame with mixed types and R quietly coerces the whole thing, usually to character, because that is the only type everything fits into.
df <- data.frame(site = c("A", "B"), n = c(3, 5)) apply(df, 1, function(r) r["n"]) # "3" and "5" — character, not numeric
For per-column work on a data frame, use sapply or vapply, which iterate over columns and keep each column's own type.
4. An empty input returns something you did not plan for
Filter a dataset down to nothing and sapply hands back an empty list, not an empty numeric vector, and the next line fails somewhere confusing. vapply returns the right empty type. This bites in simulation loops where some replicate legitimately has no data.
Where you'll meet these
These are real lines from this week's labs, so the patterns above are not hypothetical.
| Day | Line | What one piece is |
|---|---|---|
| Day 1 | replicate(reps, { ... }) | nothing, the block re-runs reps times, once per simulated dataset |
| Day 2 | tapply(long$survived, long$class, mean) | all survival values for one passenger class |
| Day 3 | boxplot(split(resid(lm1), Streams$Stream)) | split makes one group of residuals per stream |
| Day 4 | apply(mu_draws, 2, quantile, probs = c(.025,.5,.975)) | one column: every posterior draw at one x value |
| Day 3 | do.call(rbind, lapply(seq_along(nvisit), function(i) data.frame(...))) | lapply makes a list of one data frame per stream; do.call stacks them into one table |
| Day 4 | sapply(kappa_grid, function(k) nll_vbgf(Linf = 100, kappa = k, sigma = 10)) | one grid value of κ, evaluated into one negative log-likelihood, so the result is a vector to plot |
| Day 5 | Ninit <- apply(y1, 1, max) | one row: all visits to a single site, so max is that site's largest count |
| Day 5 | sapply(seq_len(n.draw), function(k) { ... }) | one posterior draw index, used to build a curve per draw |
Check yourself
Shape questions only, because that is where the mistakes live.
1. x is a numeric vector of length 6. What does lapply(x, function(v) v * 2) return?
2. m is a 10 × 4 matrix. What is dim(apply(m, 2, range))?
3. sapply(sites, function(v) v[v > 2]) where the number of values above 2 differs between sites. What comes back?
4. y is a 30 × 3 matrix of counts: 30 sites, 3 visits each. You want each site's maximum count. Which is right?
5. Why prefer vapply(x, f, numeric(1)) over sapply(x, f) in code you will re-run?
FW 536 · built for this course because most of the labs lean on these functions. Back to the course home page.