Maximum Entropy Distributions
Uniform Distribution
Among probability distributions which are nonzero over a finite range of values , the maximum-entropy distribution is the uniform distribution. To show this, we must maximize the entropy,
(D.33) |
with respect to , subject to the constraints
Using the method of Lagrange multipliers for optimization in the presence of constraints [86], we may form the objective function
(D.34) |
and differentiate with respect to (and renormalize by dropping the factor multiplying all terms) to obtain
(D.35) |
Setting this to zero and solving for gives
(D.36) |
(Setting the partial derivative with respect to to zero merely restates the constraint.)
Choosing to satisfy the constraint gives , yielding
(D.37) |
That this solution is a maximum rather than a minimum or inflection point can be verified by ensuring the sign of the second partial derivative is negative for all :
(D.38) |
Since the solution spontaneously satisfied , it is a maximum.
Exponential Distribution
Among probability distributions which are nonzero over a semi-infinite range of values and having a finite mean , the exponential distribution has maximum entropy.
To the previous case, we add the new constraint
(D.39) |
resulting in the objective function
Now the partials with respect to are
and is of the form . The unit-area and finite-mean constraints result in and , yielding
(D.40) |
Gaussian Distribution
The Gaussian distribution has maximum entropy relative to all probability distributions covering the entire real line but having a finite mean and finite variance .
Proceeding as before, we obtain the objective function
and partial derivatives
leading to
(D.41) |
For more on entropy and maximum-entropy distributions, see [48].
Next Section:
Gaussian Mean
Previous Section:
Example: Random Bit String