Categorical Encoding Explained: Practical Techniques from One-Hot to Target Encoding
A practical guide to categorical encoding for machine learning, covering ordinal, one-hot, binary, Base-N, and target encoding, along with smoothed and unbiased variants. Learn how each method trades off model accuracy, memory, speed, and interpretability, and how to choose the right one based on feature cardinality, category order, and model type—while avoiding common pitfalls like data leakage and train/inference mismatch.
Why Categorical Encoding Deserves Careful Attention
Whether you’re using logistic regression, gradient-boosted tree models, or neural networks, the underlying math operates on numbers. Raw text labels like “red”, “medium”, or “CA” carry no numerical meaning to a model—it only understands numbers and vectors. The process of mapping discrete categories to a numerical representation is categorical encoding.
This step may look basic, but it’s one of the most underrated parts of feature engineering. A well-chosen encoding scheme affects six key dimensions at once: model accuracy, interpretability, degree of overfitting, training speed, memory usage, and the ability to handle rare categories. A mismatched scheme can easily make a model that would otherwise meet the bar fail outright.
To keep things concrete, all examples below use this minimal dataset. The feature columns are color and city, and the prediction target is target:
# Illustrative dataset (the code below is pseudocode for demonstration only and does not depend on any specific library)dataset = [{"color": "red", "city": "NY", "target": 1},{"color": "blue", "city": "LA", "target": 0},{"color": "green", "city": "NY", "target": 1},{"color": "green", "city": "SF", "target": 0},{"color": "blue", "city": "LA", "target": 0},{"color": "red", "city": "NY", "target": 1},]
1. Ordinal Encoding
The most direct approach assigns each category an integer: red→0, blue→1, green→2.
encoding_map = {"red": 0, "blue": 1, "green": 2}X_encoded = [encoding_map[v] for v in X["color"]]
There are two typical use cases. First, gradient-boosted tree models (methods based on ensembles of decision trees) rely on split thresholds, so ordinal encoding is the most natural and efficient representation—paired with the model’s split optimization for unordered categories. Second, when categories have a natural order—small/medium/large, low/medium/high—ordinal encoding preserves that order directly.
Note that forcing ordinal encoding on unordered categories introduces a false ordering assumption like “green(2) > blue(1) > red(0)” that doesn’t really exist. This is especially harmful for linear models and neural networks. For such models, you typically pair it with algorithms that support category-aware splits, or switch to a different encoding.
2. One-Hot Encoding
One-hot encoding creates a separate column for each category, marking 1 on a match and 0 otherwise. For color, this expands into three columns: color_red, color_blue, color_green.
# Illustration: one column per categoryX_encoded = [[1 if c == cat else 0 for cat in categories] for c in X["color"]]
Its biggest advantage is that it introduces no false ordering, and it naturally fits models sensitive to feature independence such as logistic regression and neural networks. It’s also highly interpretable—each column is a clear binary condition.
The cost is dimensional explosion: N categories produce N columns. Facing high-cardinality features with hundreds or thousands of distinct values, the feature matrix becomes extremely sparse, and the memory and compute overhead for training and inference rise significantly. So one-hot encoding is best suited for low-cardinality features (few categories). For medium-cardinality cases where sparsity is acceptable but you still need to compress dimensions, consider combining a sparse matrix with regularization to ease the overhead.
3. Binary Encoding
Binary encoding first maps categories to integer indices, then writes each index in binary and splits each bit into a column. For example, index 5 is 101 in binary, which becomes three columns [1, 0, 1].
# Illustration: convert index to binary and split into digitsdef to_binary_digits(idx, n_bits):return [(idx >> b) & 1 for b in range(n_bits)]
Its value is that it can represent N categories with roughly log2(N) columns, which is very practical when the number of categories is medium-to-high (roughly dozens to hundreds). It preserves distinctiveness between features while saving substantial memory and column count compared to one-hot encoding. Compared with ordinal encoding, it also mitigates the pseudo-ordering problem caused by adjacent integers being assigned close values. That said, the binary digits themselves have no intuitive business meaning, so interpretability is weaker than one-hot. It’s better suited for medium-to-high-cardinality scenarios that prioritize engineering efficiency.
4. Base-N Encoding
Base-N is a generalization of binary encoding, changing the base from 2 to any N. With base=3, index 5 is represented as 12 in base-3, then split into the corresponding number of columns.
# Illustration: split into digits in any basedef to_base_digits(idx, base):digits = []while idx > 0:digits.append(idx % base)idx //= basereturn digits or [0]
By tuning the base, you can flexibly trade off between “number of columns” and “number of distinct values per column”: a larger base means fewer columns but more values per column; a smaller base means the opposite. It’s an adjustable compromise between ordinal encoding (1 column) and binary encoding, suited for cases where you want fine control over encoding compactness.
5. Target Encoding
The earlier methods only look at the input feature itself and never use label information. Target encoding goes a step further: it encodes each category using a target statistic computed for that category. For binary classification, the most direct approach is to take the target mean under that category.
# Illustration: encode using the target mean per categorytarget_mean = group_mean(X["color"], y) # red->1.0, blue->0.0, green->0.5X_encoded = [target_mean[c] for c in X["color"]]
The resulting values are naturally strongly correlated with the target, which often significantly boosts model performance. It’s especially suited to high-cardinality features—no dimensional explosion, and it directly injects information about “what value this category historically tends to take” into the feature.
But target encoding has one fatal problem to watch for: data leakage and overfitting. If you encode a sample using a statistic that includes its own label, the model “peeks at the answer,” showing inflated training performance and a big drop in production. There are two solutions: first, cross-folding (out-of-fold), using statistics from other folds to encode the current fold; second, shrinkage smoothing, covered in the next section.
6. Smoothed and Unbiased Target Encoding
To make target encoding retain information while resisting overfitting, the industry has developed two key improvements—also the most valuable advanced techniques when facing high cardinality and rare categories.
The first is smoothing. The core idea: when a category has very few samples, its target mean isn’t trustworthy and should be pulled toward the global mean; the more samples, the more you trust its own statistic. A classic approach uses sample size for weighting:
# Illustration: shrink by sample size; smaller n stays closer to the global meansmooth = (n_category * mean_category + alpha * mean_global) / (n_category + alpha)
This idea follows the statistical James-Stein shrinkage estimator—by shrinking toward the center in a directed way, it reduces overall estimation error. It’s especially friendly to rare categories, preventing a few samples from skewing the encoded value.
The second is unbiased (stratified/cumulative) target encoding. It no longer uses the current sample’s own label, but instead encodes the current sample using the cumulative statistic of same-category samples that appeared before it, mechanically eliminating self-information leakage. Combining this with multiple random permutations of the sample order and averaging further reduces sensitivity to ordering. This kind of “historical cumulative + unbiased estimation” scheme is usually more robust than naive target encoding when handling data with extremely high category cardinality and uneven sample distributions.
Encoding Method Selection Strategy
For a specific feature, follow this decision path:
- Low cardinality (single digits to a dozen or so categories): Prefer one-hot encoding—simple, no false ordering, interpretable; if categories are inherently ordered, use ordinal encoding.
- Medium cardinality (dozens to hundreds of categories): Binary or Base-N encoding, balancing compactness and distinctiveness; if strongly correlated with the target, consider target encoding.
- High cardinality (thousands or more categories): Prefer smoothed target encoding or unbiased target encoding to avoid dimensional explosion; ordinal encoding is a lightweight choice when training ensemble tree models.
- Categories with a natural order: Use ordinal encoding directly to preserve the order.
- Maximum interpretability: One-hot encoding.
In practice on mature machine learning platforms such as Baidu AI Cloud, feature encoding is typically managed within a reproducible data processing pipeline to ensure that training and online inference use the same encoding mapping, avoiding feature misalignment.
Common Pitfalls and Best Practices
- Train/inference consistency: Category mappings and encoding column order must align exactly between training and deployment, or wrong features will be produced silently.
- Handling unknown categories: If a new category unseen in training appears during inference, have a fallback (map to all-zeros or a dedicated “unknown” bucket).
- Watch for target encoding leakage: Always use cross-folding or unbiased/smoothed schemes, and evaluate real performance on an independent validation set.
- Merging rare categories: For long-tail categories with very few samples, consider grouping them first and then encoding, to reduce noise.
- Memory and speed trade-offs: In high-dimensional sparse scenarios, prefer memory-efficient encodings (binary, target encoding) and make good use of sparse matrix storage.
Summary
Categorical encoding is far more than just “one-hot” and “ordinal.” From the simple and intuitive ordinal and one-hot, to the compact and efficient binary and Base-N, to target encoding that leverages label information along with its smoothed and unbiased improvements, each method is a trade-off between accuracy, memory, speed, and interpretability. True experts don’t just memorize the “strongest encoding”—they understand the assumptions and costs behind each encoding, make appropriate choices based on feature cardinality, category order, sample distribution, and model type, and strictly observe engineering discipline such as train/inference consistency and leakage prevention. Mastering this fundamental skill often delivers more stable performance gains than repeatedly tuning hyperparameters.
