Discrete distributions

Slide 1: Discrete Probability Distributions

(No text on this slide.)

Slide 2: Binomial Experiments-I

• Binomial experiments are those for which the outcome from each trial is one of only two options (“success” or “failure”). The properties of a binomial experiment are:
• The experiment is repeated for a fixed number of trials, N, where each trial is independent of all the others.
• There are only two possible outcomes for each trial. The outcomes can be classified as a success (S) or a failure (F).
• The probability of success is the same for all trials.
• The random variable X counts the number of successful trials out of N trials.

Slide 3: Binomial Experiments-II

• Why is the following a binomial experiment?
• We randomly sample 500 fish from the population.
• We record whether each animal is mature or immature.
• The random variable X is the number of mature animals.

Slide 4: Binomial Experiments-II(Notation)

• N or n The number of times a trial is repeated
• p = P(S) The probability of a success in a single trial
• q = P(F) The probability of a failure in a single trial, q = 1 − p
• x or k The number of successes on n trials (x = 0, 1, .., n)

Slide 5: Binomial Probabilities-I

• In a binomial experiment, the probability of exactly x successes in n trials is:
• This is a probability mass function, pmf
• The binomial probability therefore involves the probability of x successes and n - x failures multiplied by the number of ways choosing x successes out of n trials.
• n and p are parameters (q = 1 - p). We will use data to estimate the values for unknown parameters.

• # ways you can have x out of n successes/
• E.g. HHT, HTH, THH

P(x) = nCx px qn−x = n! / ((n − x)! x!) · px qn−x

Slide 6: One Binomial Trial: The Bernoulli Distribution

• In a binomial experiment, the probability of exactly x successes in n trials is:
• With n = 1, the Binomial becomes the Bernoulli distribution
• P(x) = px (1 - p)1 – x
• Where x can take the values of 0 or 1
• Bernoulli → probability a single coin flip comes up heads
• Binomial → Probability that 10 coin flips yield 3 heads
• So the Binomial is the distribution for the sum of independent Bernoulli trials

P(x) = nCx px qn−x = n! / ((n − x)! x!) · px qn−x

Slide 7

Concept Question

1. Let X ~ binom(n, p) and Y ~ binom(m, p) be independent. Then X + Y follows:

• (a) binom(n + m, p) (b) binom(nm, p)
• (c) binom(n + m, 2p) (d) other

2. Let X ~ binom(n, p) and Z ~ binom(n, q) be independent. Then X + Z follows:

• (a) binom(n, p + q) (b) binom(n, pq)
• (c) binom(2n, p + q) (d) other

Slide 8

Concept Question

1. Let X ~ binom(n, p) and Y ~ binom(m, p) be independent. Then X + Y follows:

• (a) binom(n + m, p) (b) binom(nm, p)
• (c) binom(n + m, 2p) (d) other

2. Let X ~ binom(n, p) and Z ~ binom(n, q) be independent. Then X + Z follows:

• (a) binom(n, p + q) (b) binom(n, pq)
• (c) binom(2n, p + q) (d) other

1. answer: (a). Each binomial random variable is a sum of independent Bernoulli(p) random variables, so their sum is also a sum of Bernoulli(p) r.v.'s.

2. answer: (d) This is different from problem 1 because we are combining Bernoulli(p) r.v.'s with Bernoulli(q) r.v.'s. This is not one of the named random variables we know about.

Slide 9

• Hints:
• Make a table with X, p(x), and (X - µ )2. Remember µ = E[X]
• Remember that variances of independent random variables add
• Var(X + Y) = Var(X) + Var(Y)

Group Work

1. Prove: if X ~ Bernoulli(p) then Var(X) = p(1 − p).

2. Prove: if X ~ bin(n, p) then Var(X) = n p(1 − p).

Slide 10

Solution

1. For X ~ Bernoulli(p) we use a table. (We know E(X) = p.)

Var(X) = E((X − μ)2) = (1 − p)p2 + p(1 − p)2 = p(1 − p)

2. X ~ bin(n, p) means X is the sum of n independent Bernoulli(p) random variables X1, X2, …, Xn. For independent variables, the variances add. Since Var(Xj) = p(1 − p) we have

Var(X) = Var(X1) + Var(X2) + … + Var(Xn) = np(p − 1).

Slide 11: Binomial Probabilities-II

• Notation:
• We read this as “The random variable X is distributed binomially with parameters n and p”
• Recall: q = 1 - p

X ~ B(n,p); X ~ Bin(n,p); X ~ Binom(n,p)

• Mean : μ = n p
• Variance : σ2 = npq

Slide 12: The Binomial Distribution

n=6; p=0.5

n=6; p=0.9

0 1 2 3 4 5 6

0 1 2 3 4 5 6

This is a probability mass function; PMF

Slide 13: Examples of the Binomial Distribution-I

• We examine 12 animals for the presence of a disease (p = 0.1). What is the probability that:
• We find exactly 2 animals with the disease?
• We find no animals with the disease?
• We find 2 or more animals with the disease?

Group Work

P(x) = nCx px qn−x = n! / ((n − x)! x!) · px qn−x

Slide 14: Examples of the Binomial Distribution-III

• P[X=2]=
• P[X=0]=
• P[X≥2]=1-P[X=0]-P[X=1]=0.3410

(12 choose 2) 0.12 0.910 = 0.2301

(12 choose 0) 0.10 0.912 = 0.2824

• > dbinom(2,12,0.1)
• [1] 0.2301278
• > dbinom(0,12,0.1)
• [1] 0.2824295
• > 1-dbinom(1,12,0.1)-dbinom(0,12,0.1)
• [1] 0.3409977

• > 1-pbinom(1,12,0.1)
• [1] 0.3409977

Slide 15: Sneak peak: Maximum likelihood is just doing this in reverse!!!

• Instead of knowing N and p, we often collect data in which p and sometimes also N are unknown.
• To understand how the world works, we want to determine p
• What is the prevalence of a disease?
• What is the probability of giving birth?

Slide 16

• Instead of knowing N and p, we often collect data, k #successes, to determine p and/or N
• Let’s assume that N is known. Say we conduct N trials, observe k successes, and want to make inference about p, given k and N

This is among the easiest possible likelihood problems

Sneak peak: Maximum likelihood is just doing this in reverse!!!

ℒ(p | k, N) = (N choose k) pk (1 − p)N − k

Slide 17: Sneak peak: Maximum Likelihood is just doing this in reverse!!!

• This equation is exactly what we’ve been working with but the interpretation is very different
• The binomial probability summed over all values of k equals 1
• But now the Likelihood is a function of p with k fixed. k is our observation.
• The range of p must be 0 to 1, but there is no requirement that the integral of the likelihood from 0 to 1 = 1 (as would be the case for a probability)
• Why did I say “integral” and not “sum”

ℒ(p | k, N) = (N choose k) pk (1 − p)N − k

Slide 18: Sneak peak: Maximum likelihood is just doing this in reverse!!!

• This equation is exactly what we’ve been working with but the interpretation is very different
• We usually work with the log-likelihood. Can you find that? → log(a*b) = log(a) + log(b), log(ab)=b*log(a)

Should I show you the derivation?

ℒ(p | k, N) = (N choose k) pk (1 − p)N − k

L(p | k, N) = log(N choose k) + k log(p) + (N − k) log(1 − p)

Slide 19

• How do we find the “maximum likelihood estimate” (MLE) for p?
• Enter Calculus!
• Find the MLE for p

Sneak peak: Maximum likelihood is just doing this in reverse!!!

L(p | k, N) = log(N choose k) + k log(p) + (N − k) log(1 − p)

Slide 20

• Hudson (1971) showed that an approximate 95% CI for a single peaked likelihood function occurs 2 log-likelihood units below the peak.
• Can you see why sample size is so important based on these shapes?

Sneak peak: Maximum likelihood is just doing this in reverse!!!

Slide 21: Back to distributions

• We’ll discuss maximum likelihood later
• Bayesian statistics is very similar - making inference about the parameter given the data - but the inference will be a real probability rather than a point estimate and confidence interval.

Slide 22

The multinomial distribution: more than one kind of success

The multinomial distribution is an extension of the binomial distribution to the case of more than two (we shall assume n) kinds of outcomes, in which a single trial has probability pi of ending in category i. In a total of N trials, we assume that ki of the outcomes end in category i.

Pr{k | N, p} = N! / (∏i=1n ki!) · ∏i=1n pikᵢ

Slide 23: The Poisson Distribution-I

• The Poisson distribution is used to determine the probability that a specific number of occurrences takes place within a given unit of time or space.
• A random variable X is Poisson distributed if:
• The experiment consists of counting the number of times, x, an event occurs in a given interval (the interval can relate to e.g. time, area, volume).
• The probability of the event occurring is the same for each interval.
• The number of occurrences in one interval is independent of the number of occurrences in other intervals.

Slide 24: The Poisson Distribution-I

• In short – Poisson models ”random” count data
• Number of ladybugs or leaves per bush
• Number of prey encountered per unit time
• Number of eggs produced by a fish
• This is different from bounded count data, which is binomial
• # offspring that survive out of N offspring
• #cameras that detect wolves out of N cameras
• You might think that the Poisson could be a good approximation of the binomial if N is large and p is small. You are right!

Slide 25: The Poisson Distribution - pmf

• Various notations:
• Note that x (or k) can be any non-negative integer.

pk(t) = (λt)k / k! · e−λt

P(k events in interval) = e−λ · λk / k!

P(X = x) = μx e−μ / x!

• Mean : μ (rate per interval)
• Variance : μ

Slide 26: Where does the Poisson distribution come from?

In a very small interval of time, either nothing happens or one event happens

• Pr{no event in the next dt} = 1 − λdt
• Pr{1 event in the next dt} = λdt

Slide 27: Where does the Poisson distribution come from?

What we want is the probability of k events in a a longer interval, say time = t

• How could we have no events up to time t + dt?
• Use the definition of the derivative to show that this leads to
• Solve to find the zero term of the Poisson, Pr(no events)

dp0 / dt = −λ p0

p0(t) = exp(−λt)

p0(t + dt) = p0(t)(1 − λdt)

Slide 28: Zero term of the Poisson Distribution

Probability Mass Function:

pk(t) = (λt)k / k! · e−λt

Pr{no event in 0 to t} = p0(t) = e−λt

Slide 29: First term of the Poisson Distribution

• How can you have one event in time 0 to t + dt?
• Either we had no event in 0 to t and one event in t to t + dt or we had one event in 0 to t and no event in t to t + dt
• Continue like this to derive other terms
• Another derivation starts with the Binomial distribution where the average number of successes, Np, is replaced by. and N approaches infinity

Slide 30: The Poisson Distribution - pmf

• Various notations:
• Note that x (or k) can be any non-negative integer.

pk(t) = (λt)k / k! · e−λt

P(k events in interval) = e−λ · λk / k!

P(X = x) = μx e−μ / x!

• Mean : μ (rate per interval)
• Variance : μ

Slide 31: The Poisson Distribution Graphically

μ=2

μ=5

μ=8

Slide 32: An Example of the Poisson Distribution-I

• The mean bycatch of dolphins per month in the Eastern Tropical Pacific is 3
• What is the probability that the catch will be 4 in a given month?
• What is the probability that the catch will be 2 or less in a given month?
• What is the probability that at least one dolphin is killed?

Breakout groups: Try to find analytical solutions AND a solution using built in functions in R

Group Work

P(X = x) = μx e−μ / x!

Slide 33: An Example of the Poisson Distribution-II

• The mean catch of dolphins per month in the Eastern Tropical Pacific is 3.
• What is the probability that the catch will be 4 in a given month?

P[X = 4] = μx e−μ / x! = 34 e−3 / 4! = 0.168

• > dpois(4,3)
• [1] 0.1680314

Slide 34: An Example of the Poisson Distribution-III

• The mean catch of dolphins per month in the Eastern Tropical Pacific is 3.
• What is the probability that the catch will be 4 in a given month?
• What is the probability that the catch will be 2 or fewer in a given month?
• P[X≤2]=P[X=0]+P[X=1]+P[X=2]=

P[X ≤ 2] = e−3 [ 30/0! + 31/1! + 32/2! ] = 0.423

• > dpois(0,3)+dpois(1,3)+dpois(2,3)
• [1] 0.4231901
• > ppois(2,3)
• [1] 0.4231901

Slide 35: An Example of the Poisson Distribution-III

• The mean catch of dolphins per month in the Eastern Tropical Pacific is 3.
• What is the probability that the catch will be 4 in a given month?
• What is the probability that the catch will be 2 or fewer in a given month?
• What is the probability that at least 1 dolphin is killed?

e-3

1 - e-3

• > dpois(0,3)
• [1] 0.04978707
• > 1-dpois(0,3)
• [1] 0.9502129

Slide 36: Example of using Poisson Probabilistic Arguments in Research

constant over the 27-year period. By modeling the expected number of observed attacks as a Poisson process, the probability of zero attacks being seen before 1990 is 0.006 (15).

Slide 37: The Poisson Distribution - Caveat

• The Poisson distribution would seem to be a good distribution to assume for catch data in fisheries. However, fish school. What does this imply regarding the assumptions of the Poisson distribution?
• Variance would be high relative to the mean
• You catch zero or a shit ton!!!
• This is called overdispersion → Onward to negative binomial!

Slide 38: But first: MLE for the Poisson Distribution

• Let’s find the MLE of the Poisson distribution
• Let’s do it for a dataset of n Poisson distributed observations. (very common to assume that t =1 since counts per period is typically desired)

pk(t) = (λt)k / k! · e−λt

L(p | {Xi}) = Σi=1n log( f(Xi | p) )

ℒ(p | {Xi}) = ∏i=1n f(Xi | p)

Slide 39: The Negative Binomial Distribution(1) # failures/trials before fixed number of successes

• We have an experiment with two outcomes: success (with probability p) and failure (with probability q =1-p).
• Let N be the number of trials to achieve k successes, each with probability p
• Number of trials, N, is now the random variable!
• Recall, k, was the R.V. in the binomial case

Slide 40

Try to derive it based on this argument

How do we turn this into “failures”, now k, until r successes? k->r; N-> k+r;

The Negative Binomial Distribution(1) # failures/trials before fixed number of successes

Pr{N = n | k, p} = Probability that the kth success occurs on trial n

Pr{N = n | k, p} = (n−1 choose k−1) pk−1 (1 − p)n − k p = (n−1 choose k−1) pk (1 − p)n − k

f(k; r, p) ≡ Pr(X = k) = (k+r−1 choose r−1) (1 − p)k pr

Now, for the kth success to occur on trial n, we must have k − 1 successes in the first n − 1 trials and a success on the nth trial. The probability of k − 1 successes in n − 1 trials has a binomial distribution with parameters n − 1 and p and the probability of success on the nth trial has probability p and these are independent of each other. We thus conclude

Slide 41: The Negative Binomial Distribution(2) Poisson with a variable rate parameter

• Start with Poisson
• But let’s give lambda a continuous probability (continuous distributions come soon!) and then integrate over all possible values of lambda (this is called “marginalizing”)

Because:

Pr{k events in [0, t] | λ} = (λt)k / k! · e−λt

Pr{k events in [0, t]} = ∫0∞ (λt)k / k! · e−λt · f(λ) dλ

pX(x) = ∫y pX,Y(x, y) dy = ∫y pX|Y(x | y) pY(y) dy

Slide 42: Quick note on Gamma function

• Continuous version of factorial
• For integers, z,
• In general
• Defined (for z > 0):

Γ(z) = (z − 1)!

Γ(z + 1) = z Γ(z)

Γ(z) = ∫0∞ sz−1 e−s ds

Slide 43: The Negative Binomial Distribution(2) Poisson with a variable rate parameter

Common parameterization in ecology with the mean as a parameter:

• We model N overdispersed counts as function of mean, m, and dispersion, k
• (not to be confused with prior uses of k)

Pr{N = n} ≡ pn(m, k) = Γ(k + n) / (n! Γ(k)) · (k / (k + m))k · (m / (k + m))n

Slide 44: The Negative Binomial Distribution(2) Poisson with a variable rate parameter

What is the probability of no events?

This probability can be large if k is small even if m is big

Pr{N = n} ≡ pn(m, k) = Γ(k + n) / (n! Γ(k)) · (k / (k + m))k · (m / (k + m))n

p0(m, k) = (k / (k + m))k

Slide 45: Poisson vs. Negative Binomial

Slide 46: The Geometric Distribution

This is a special case of the negative binomial distribution for which k =1 (i.e. the probability of the number of trials or failures until one success is recorded).

Slide 47: Summary of Discrete Distributions