Skip to content
Signal Theory

Probability for signals: Bayes, random variables and the Gaussian

Conditional probability, Bayes' theorem with proof and an example, mean and variance, the Gaussian distribution and the Q function, sums of independent variables, the central limit theorem.

4 min read 0 comments

The noise of an amplifier, the bits sent over a channel, the arrival times of packets in a network cannot be predicted individually, but they have precise statistical regularities. Describing them requires probability theory. In this lesson we review the essential tools for signals: conditional probability and Bayes’ theorem, random variables, mean and variance, the Gaussian distribution and the central limit theorem.

Probability and conditional probability

A probability assigns each event a number between 0 and 1, with P(\text{certain event}) = 1 and P(A \cup B) = P(A) + P(B) for mutually exclusive events. The conditional probability of A given that B has occurred is

    \[ P(A \mid B) = \frac{P(A \cap B)}{P(B)} \]

Two events are independent if P(A \cap B) = P(A)P(B), that is if knowing that one has occurred does not change the probability of the other.

Bayes’ theorem

Theorem. If B_1, \dots, B_n are mutually exclusive events covering all possible cases,

    \[ P(B_i \mid A) = \frac{P(A \mid B_i)\,P(B_i)}{\sum_k P(A \mid B_k)\,P(B_k)} \]

Proof. From the definition, P(A \cap B_i) = P(A \mid B_i)P(B_i) = P(B_i \mid A)P(A), so P(B_i \mid A) = P(A \mid B_i)P(B_i)/P(A). Since the B_k cover all cases without overlapping, P(A) = \sum_k P(A \cap B_k) = \sum_k P(A \mid B_k)P(B_k) (total probability). ∎

Bayes “reverses” the conditioning: from the observed effect it goes back to the most probable cause. It is the logic of every receiver: having observed a signal, which symbol was sent?

Example. A diagnostic test detects a disease in 99% of sick people and gives a false positive in 5% of healthy people; the disease affects 1% of the population. With a positive test, the probability of being sick is

    \[ \frac{0.99\cdot 0.01}{0.99\cdot 0.01 + 0.05\cdot 0.99} \approx 0.17 \]

only 17%: the healthy are so numerous that their false positives outnumber the true positives. The same trap appears with an alarm triggered by rare events.

Random variables

A random variable X assigns a number to each outcome of an experiment. It is described by the probability density f_X(x), such that

    \[ P(a \lt X \le b) = \int_a^b f_X(x)\,dx,\qquad \int_{-\infty}^{+\infty} f_X(x)\,dx = 1 \]

The most used quantities are the expected value (mean) and the variance:

    \[ \mu = E[X] = \int x\,f_X(x)\,dx,\qquad \sigma^2 = E[(X - \mu)^2] = E[X^2] - \mu^2 \]

For signals they have a direct physical meaning: for a noise voltage, \mu is the DC component, E[X^2] the power (into 1 Ω), \sigma^2 the power of the varying part alone and \sigma the RMS value of the noise.

Example. Quantisation error is uniform between -\Delta/2 and \Delta/2: \mu = 0, \sigma^2 = \Delta^2/12 (lesson on quantisation).

The Gaussian distribution

    \[ f_X(x) = \frac{1}{\sigma\sqrt{2\pi}}\,e^{-(x - \mu)^2/(2\sigma^2)} \]

Gaussian probability density with the areas within one and two standard deviations
The bell curve is centred on the mean μ. The area between μ − σ and μ + σ is about 68% of the total, that between μ − 2σ and μ + 2σ about 95%.

It is determined by mean and variance. Within \pm\sigma of the mean fall 68.3% of the values, within \pm 2\sigma 95.4%, within \pm 3\sigma 99.7%. Tails are computed with the function

    \[ Q(x) = \frac{1}{\sqrt{2\pi}}\int_x^{\infty} e^{-u^2/2}\,du \]

the probability that a standard Gaussian exceeds x: it will be needed for the error probability of transmission systems.

Several variables together

For two variables the joint density f_{XY}(x, y) is used. They are independent if f_{XY} = f_X f_Y. The covariance E[(X - \mu_X)(Y - \mu_Y)] measures how much they vary together, and the correlation coefficient \rho = \mathrm{cov}/(\sigma_X\sigma_Y), between −1 and 1, is its normalised version. Independent implies uncorrelated (\rho = 0); the converse does not hold in general, but it does for jointly Gaussian variables.

Sum of independent variables. If Z = X + Y with X and Y independent:

  • the means add, always;
  • the variances add: \sigma_Z^2 = \sigma_X^2 + \sigma_Y^2, because the cross term 2E[(X - \mu_X)(Y - \mu_Y)] is zero;
  • the density of Z is the convolution of the two densities.

The second property is why the powers of independent noises add, while RMS values combine “in quadrature”: two noises of 3 µV and 4 µV give 5 µV.

The central limit theorem

Theorem (statement). The sum of many independent random variables, with finite variances and none dominating the others, has a distribution that tends to the Gaussian, whatever the distribution of the individual variables.

You can see it already by adding a few uniform variables: the convolution of two rectangles is a triangle, of three a bell curve made of parabolas, and after about ten the difference from the Gaussian is tiny. This is why the Gaussian is everywhere in engineering: thermal noise is the sum of the contributions of billions of electrons, measurement errors are the sum of many small causes. So the standard model of noise in communication systems is Gaussian noise, which we study in the next lesson as a random process.

Related articles

Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted
0
Would love your thoughts, please comment.x
()
x