Bits and Entropy: Using Shannon Information Theory to Define System Performance Limits

1. Introduction: What Is Information Theory?

Claude Elwood Shannon was born in Petoskey, Michigan, in the United States, and grew up in the small town of Gaylord. In 1936, he earned bachelor’s degrees in mathematics and electrical engineering from the University of Michigan. In 1940, he received both a master’s degree in electrical engineering and a doctorate in mathematics from the Massachusetts Institute of Technology. In 1941, he joined Bell Telephone Laboratories for basic research and remained there for about fifteen years. Beginning in 1956, he also taught at MIT, becoming a professor there from 1958 to 1978 and an emeritus professor after retirement. He died on February 24, 2001, in Medford, Massachusetts, at the age of 84, after a long battle with Alzheimer’s disease.

For developers, the most valuable lesson from Shannon is not a specific algorithm but a way of thinking: every engineering system has a theoretical upper limit determined by objective constraints; the art of engineering is to approach that limit at an acceptable cost.

This article answers five questions in turn:

• How is information measured?
• Where are the limits of lossless compression and reliable transmission?
• How is lossy compression calibrated?
• How do these theories appear in modern systems?
• Which “seemingly reasonable” practices actually violate basic information theory?

2. How Is Information Measured?

2.1 Why Shannon’s 1937 master’s thesis still matters

Shannon’s best-known early work came from his master’s thesis, A Symbolic Analysis of Relay and Switching Circuits, written in 1937. He showed that structures in relay switching circuits, such as series and parallel connections, could be fully mapped to boolean algebra operations such as AND, OR, and NOT. This allowed circuit analysis and simplification to be performed using algebraic methods. In 1940, he received the Alfred Noble Award for this work, jointly awarded by several national engineering societies in the United States.

The significance was profound: circuits were no longer merely physical wiring; they became mathematical objects that could be reasoned about precisely using symbols. Today, logic synthesis tools that convert RTL code into gate-level circuits rely on exactly this equivalence. Shannon’s later work can be seen as an extension of the same methodology: first abstract, then quantify, and finally find the limit.

2.2 How much information does a message contain?

Before measuring information, we must ask: what makes a message “new”?

Shannon’s answer is: information is the elimination of uncertainty. The less likely an event is, the more information is delivered when it actually occurs.

The self-information of an event $x$ is defined as:

I(x) = log2(1 / P(x)) = -log2 P(x)

When the base is 2, the unit is the bit, derived from “binary digit.” A fair coin flip has exactly one bit of uncertainty. Rolling a fair die has log2(6) ≈ 2.585 bits of uncertainty. A completely certain message has zero self-information.

2.3 What is entropy?

Self-information describes a single event. To describe an entire source, we need the expected value, which is Shannon entropy:

H(X) = -Σ P(xi) · log2 P(xi)

For a uniformly distributed character set, the entropy of 26 lowercase letters is log2(26) ≈ 4.70 bits. Thirty-two equally likely symbols require exactly 5 bits.

Real text is not uniformly distributed. For verification, here is the calculation basis: first-order entropy estimated from character frequencies, without using contextual relationships. For English in a 27-character alphabet including spaces, common estimates are around 4.1 to 4.3 bits per character. For Chinese, based on the frequencies of about 7,000 common characters, the estimate is around 9.65 bits per character. Because the corpus and character-set size affect the result, values may vary, and the calculation basis should be stated when citing them.

2.4 What are the properties of entropy?

• Non-negativity: Uncertainty cannot be negative.
• Maximum at uniformity: The more uniform the distribution, the higher the entropy. The maximum is log2(n) when all outcomes are equally likely.
• Additivity for independent sources: If two sources are independent, H(X, Y) = H(X) + H(Y).
• Conditioning reduces uncertainty: H(Y|X) ≤ H(Y). The more you know, the less uncertainty remains.
• Concavity: Entropy is a concave function of the probability distribution. The entropy of a mixture of two sources is not lower than the weighted average of their individual entropies. This is the mathematical reason why mixed distributions are harder to compress.

2.5 How do you calculate empirical entropy?

The following code depends only on the Python standard library: math and collections. Note that this is a first-order empirical entropy estimate; shorter samples produce larger bias.

import math
from collections import Counter
def entropy(items) -> float:
    """Calculate the first-order empirical entropy (bits/symbol) of a set of discrete observations."""
    counts = Counter(items)
    total = len(items)
    return -sum(
        (c / total) * math.log2(c / total)
        for c in counts.values()
    )
sample = "ABRACADABRA"          # 11 characters: A×5, B×2, R×2, C×1, D×1
h = entropy(sample)
print(h)                        # ≈ 2.04 bits/character
print(math.log2(len(set(sample))))  # ≈ 2.32: upper bound when 5 observed symbols are equally likely

There are three practical observations:

• Uneven distributions can significantly reduce entropy. More redundancy means more room for compression.
• Entropy is maximized under a uniform distribution. Therefore, compressing already compressed or encrypted data usually provides little additional benefit.
• If adjacent characters are treated as a joint unit and the joint entropy is lower than twice the single-character entropy, this indicates exploitable correlation between characters. Huffman coding takes advantage of this unevenness by assigning shorter codewords to more frequent symbols.

2.6 Common information measures: a quick reference

Measure	Definition	Intuitive meaning	Typical use
Joint entropy H(X,Y)	-Σ p(x,y) log2 p(x,y)	Overall uncertainty of two variables	Multivariate distribution modeling
Conditional entropy `H(Y\	X)`	`-Σ p(x,y) log2 p(y\	x)`	Remaining uncertainty in Y given X	Decision tree splitting criteria
Mutual information I(X;Y)	`H(X) - H(X\	Y)`	Information shared by X and Y	Feature selection, correlation analysis
Relative entropy D(p‖q)	Σ p(x) log2[p(x)/q(x)]	Extra bits needed to describe p using q	Distribution drift detection, model evaluation
Cross-entropy H(p,q)	H(p) + D(p‖q)	Average code length when using q to encode data from p	Loss function for classification tasks

Unless otherwise stated, all logarithms in this article use base 2, and the unit is the bit. In information theory, the concept of “coding” is broader than intuition suggests: any model that maps variables into numerical representations can be viewed as an encoder. The closer the output distribution q is to the true distribution p, the fewer bits are needed to describe the real data.

3. The Three Main Theorems: Where Are the Limits, and How Do We Approach Them?

3.1 What is the lower limit for lossless compression?

The lossless source coding theorem states that for a discrete memoryless source with entropy H(X), the average code length of any lossless encoding cannot be lower than H(X). When the source extension is sufficiently long, codes can always be constructed whose average code length approaches H(X) arbitrarily closely.

The engineering implication is direct: the amount of data after compression cannot be lower than the entropy of the data itself. Blindly compressing high-entropy data wastes CPU resources; real gains come from modeling, discovering structure, and exploiting redundancy. Deduplication in pipelines is essentially the process of mining redundancy across files.

3.2 How is noisy-channel capacity calculated, and how low can the error rate go?

For an additive white Gaussian noise channel with bandwidth B in Hz and signal-to-noise ratio S/N, the capacity is:

C = B · log2(1 + S/N)    unit: bits/second

The theorem has two directions:

• Achievability: If the transmission rate is below C, there exist codes that make the error rate arbitrarily small.
• Converse: If the transmission rate is above C, no code can reduce the error rate to an arbitrarily small value.

Shannon used random coding arguments to prove that good codes exist, but he did not provide an explicit construction. Over the following seventy years, the history of error-correcting codes has been the history of approaching the Shannon limit: Hamming codes in 1950, cyclic codes, convolutional codes, concatenated codes, and later Turbo codes, low-density parity-check codes, and polar codes. Polar codes, introduced by Arıkan in 2009, were the first explicit constructions proven to achieve the Shannon limit on binary-input symmetric channels such as the binary erasure channel.

Three engineering points deserve attention:

• Total rate includes redundancy. After coding, the total bit rate is the original information rate divided by the code rate. A lower code rate provides stronger error correction but consumes more bandwidth.
• Retransmission also consumes capacity. Hybrid automatic repeat request in wireless networks, application-level resumable downloads, and retries all trade redundancy for reliability.
• Rate can be converted into cost. Cross-availability-zone and cross-region transfers in the cloud generate traffic charges. Appropriately improving the compression ratio can directly reduce this expense.

3.3 What is the scale for lossy compression?

When distortion is allowed, the rate-distortion function R(D) gives the minimum code rate required to achieve an average distortion no greater than D. The distortion metric depends on the scenario: mean squared error, structural similarity, or perceptual quality score.

# Dependencies: standard library only
# Assumptions: i.i.d. Gaussian source, mean squared error (MSE) metric, typical high-SNR regime
# D(R) = σ² · 2^(-2R) is the exact rate-distortion operating point in this case,
# i.e., the minimum achievable distortion (a lower bound), not an upper bound.
# R is bits per sample, and σ² is the signal variance.
def distortion_floor(sigma2: float, rate_per_sample: float) -> float:
    return sigma2 * 2 ** (-2 * rate_per_sample)
print(distortion_floor(1.0, 1))  # 0.25
print(distortion_floor(1.0, 2))  # 0.0625: doubling the rate reduces distortion to about 1/4
print(distortion_floor(1.0, 4))  # ≈0.0039

Under MSE, every additional bit per sample reduces the minimum achievable distortion to about one quarter of its previous value. This explains why many media formats achieve much higher overall efficiency with lossy compression than with lossless compression. Human eyes are insensitive to high-frequency details, and human ears are insensitive to faint high-frequency sounds; discarding them has little effect on subjective quality.

The essence of lossy encoder design is to allocate limited bits to the information that is most sensitive to perception, according to the rate-distortion trade-off.

3.4 What is required for perfect secrecy?

Shannon introduced information theory into cryptography and proved that the one-time pad is a necessary condition for perfect secrecy. When the key is completely random, the same length as the plaintext, and never reused, the ciphertext reveals no information about the plaintext.

The lesson is: key space and the quality of the random source are themselves information-theoretic problems. Generating keys with a predictable random number generator effectively lowers the effective entropy of the key and makes brute-force attacks more feasible. In production, validated cryptographically secure random sources should be used, along with tiered storage and periodic rotation.

4. Information Theory Behind Code: From Models to Distributed Systems

4.1 What is the relationship between loss functions and perplexity?

The cross-entropy loss commonly used in classification tasks is a direct implementation of information-theoretic cross-entropy.

Dependencies: numpy.

import math
import numpy as np
def cross_entropy_loss(probs, one_hot_labels):
    # probs: predicted class probabilities, shape (N, C)
    # one_hot_labels: true labels
    return -np.mean(np.sum(one_hot_labels * np.log(probs + 1e-12), axis=1))
def perplexity(loss: float) -> float:
    # Perplexity = the model's effective branching factor per sample
    return math.exp(loss)  # This example uses nats; if loss is in bits, use 2 ** loss
loss = cross_entropy_loss(
    np.array([[0.7, 0.2, 0.1]]),
    np.array([[1.0, 0.0, 0.0]])
)
print(loss)              # ≈0.357 nats (about 0.515 bits)
print(perplexity(loss))  # ≈1.43 = 1/0.7

Pay attention to the logarithm base: np.log produces nats, while np.log2 produces bits. The conversion factor between the two is ln(2) ≈ 0.693; one bit equals approximately 0.693 nats. Perplexity itself is independent of the base. It measures how many equally likely candidate words the model “hesitates” among at each position on average.

Large language models, including Baidu’s ERNIE, demonstrate context understanding and generation in tasks such as question answering, summarization, and code generation. These capabilities also follow the same optimization logic: continuously reducing predictive uncertainty.

This perspective also explains several common issues:

• Class imbalance: The majority class has lower inherent uncertainty. A model that simply predicts the majority class may still have a deceptively low loss. Class weighting, resampling, or switching to metrics such as F1 or AUC may be necessary.
• Label smoothing: This smooths the true distribution and can mitigate overfitting.
• Quantization and caching: Reducing a probability table from 32-bit floating point to 8-bit integer loses about 24 bits of precision per value. For highly skewed distributions, variable-length coding can further compress the representation.

4.2 How do retrieval and recommendation systems measure feature value?

In information retrieval, term frequency-inverse document frequency is naturally connected to information theory. The value of a term as a feature depends on how “rare” it is across the document set; the rarer it is, the stronger its discriminative power.

The information gain in decision trees and random forests is essentially mutual information:

IG(Y, X) = H(Y) - H(Y|X)

For large-scale retrieval systems such as Baidu Search, filtering out the most relevant results from hundreds of billions of web pages involves feature selection, relevance ranking, and result deduplication. Ultimately, these all answer the same question: Which signals best eliminate uncertainty about what result should be returned?

4.3 How can KL divergence be used to detect online distribution drift?

After deployment, the request distribution almost inevitably differs from the test environment. Treat the traffic share of each interface or endpoint as a probability distribution, and use relative entropy to measure the difference between two time windows.

Dependencies: numpy. q must be nonzero everywhere. Here, eps is used for smoothing; in practice, each category should have a minimum sample size.

import numpy as np
def kl_divergence_bits(p, q, eps=1e-12):
    """Calculate relative entropy D(p||q) in bits."""
    p = np.asarray(p, dtype=float)
    q = np.asarray(q, dtype=float)
    p = p / p.sum()
    q = q / q.sum()
    return float(np.sum(p * np.log2((p + eps) / (q + eps))))
baseline = [0.50, 0.30, 0.15, 0.05]   # Baseline traffic share of each endpoint
current  = [0.30, 0.25, 0.35, 0.10]   # Traffic to one endpoint has increased significantly
print(kl_divergence_bits(current, baseline))  # ≈0.241 bit
# If np.log is used instead, the result is about 0.167 nats, which is equivalent.

In observability products such as log analysis and monitoring/alerting in Baidu AI Cloud, baseline comparison and anomaly detection for traffic distributions, error-code distributions, and latency distributions are typical applications of these measures.

Three cautions:

• Relative entropy is asymmetric: D(p‖q) and D(q‖p) have different meanings. Confirm which distribution is the reference before calculating.
• Small fluctuations in low-traffic endpoints can significantly amplify relative entropy. Set a minimum sample-size threshold.
• Batch-processing parameters and compression options in message queues and logging services are ways to find a rate-distortion-like balance between throughput, latency, and storage cost.

5. Common Misconceptions and a Pre-Release Checklist

5.1 Five common mistakes

• Confusing high entropy with useful information. Random noise has maximum entropy but little business value. Value depends on how much information contributes to the target task.
• Treating compression ratio as performance. A higher compression ratio usually increases CPU overhead and latency. Latency-sensitive services should choose algorithms with low compression and decompression overhead.
• Assuming latency can be optimized below the speed of light. Cross-region round-trip latency has a physical lower bound. It can only be mitigated with edge access, edge caching, multi-level caching, and asynchronous design.
• Assuming retransmission is free. Unlimited retries can amplify congestion and reduce overall availability. Backoff, retry limits, and circuit breaking are required.
• Passing off low-distortion reconstruction as original data. Lossy reconstructed data can be used for analytics or display, but it cannot serve as a baseline for exact comparison.

5.2 Pre-release checklist

• Estimate empirical entropy before load testing. Use it to predict whether investing in compression is worthwhile.
• Build a capacity model. Estimate required bandwidth from peak QPS, per-request data volume, and redundancy overhead, then compare it with the channel capacity limit.
• Choose lossless and lossy strategies appropriately. Prefer lossless compression for logs and backups. Prefer lossy compression for images, audio, and video, and choose bit rates based on rate-distortion trade-offs.
• Continuously monitor data distributions. Use measures such as relative entropy to establish baselines, detect traffic skew and model drift, and keep units consistent, whether bits or nats.

6. Conclusion: Use Limit Thinking to Guide Engineering Design

Shannon’s contribution is not only a set of formulas but also a way of looking at engineering problems: first build an abstract model, then quantify uncertainty, then prove theoretical limits, and finally approach them with constructible methods.

Capacity planning, compression strategies, redundancy design, and model evaluation can all find a unified measurement language in the framework of information theory.

The next time you face the question “Can we make it faster or cheaper?”, ask three questions:

• What is the theoretical limit?
• How far is the current solution from that limit?
• Is the benefit of closing the gap worth the cost?

That is the intellectual legacy Shannon left for engineering practice.