Imbalanced data and unbalanced classification: examples, problems and solutions

Predictive modelling of classification

Classification algorithms are evaluated by their performance. Classification accuracy is a popular measure based on predicted class labels. It is not perfect, but it is a good starting point for many classification tasks, though not for imbalanced ones, as the next sections show.

Instead of class labels, some tasks require predicting the probability of class membership for each instance. This adds a measure of uncertainty to the prediction, which the application or the user can then interpret.

There are four main types of classification tasks you may encounter:

• Binary classification
• Multi-class classification
• Multi-label classification
• Unbalanced (imbalanced) classification

What is imbalanced data?

The number of examples that belong to each class is called the class distribution. Unbalanced classification is a predictive modeling problem in which the number of examples in the training dataset for each class label is not balanced: the class distribution is not equal or close to equal, but biased or skewed.

The imbalance can be extreme. In the public credit card fraud dataset collected by Worldline and the Université Libre de Bruxelles, only 492 of 284,807 transactions are fraud, or 0.17%.

This is why accuracy misleads. On the fraud dataset, a model that labels every transaction as legitimate reaches more than 99.8% accuracy and catches no fraud at all.

The challenges of unbalanced classification

There are three main problems with data in which the classes are unequally distributed:

Problem with the machine. Machine learning algorithms are built to minimize errors. Since the probability of an example belonging to the majority class is very high in an unbalanced dataset, algorithms are more likely to classify new observations into the majority class.

Intrinsic problem. In real life, the cost of a false negative is usually much higher than the cost of a false positive, but standard ML algorithms penalize both with similar weight.

A human problem. Common practices in many industries are often set by experts rather than by empirical research. That is rarely the optimal solution.

How to measure a model on imbalanced data

Choose the metric before the technique, and start from the cost of each kind of error:

• The confusion matrix: how many cases of each class were caught and missed.
• Recall (sensitivity): the share of real minority cases the model finds. Precision: the share of its alarms that are real.
• F1 score, the harmonic mean of the two, when one number is needed.
• The precision-recall curve and its area (average precision). Takaya Saito and Marc Rehmsmeier showed in 2015 that on imbalanced data it is more informative than the ROC curve, which can look good while precision is poor.
• Calibration measures, such as the Brier score, when the probabilities themselves are used.

Real systems need both numbers at once. Before Milarex moved to AI-based X-ray inspection of its salmon products, its system detected 62% of contaminants and still raised enough false alarms to stop the lines for almost two hours a day.

How to handle imbalanced data

There are two different approaches to dealing with unbalanced data: an algorithm-level approach and a data-level approach.

Algorithmic approach: standard ML algorithms penalize false positives and false negatives equally. One way to counter this is to modify the algorithm itself to improve its performance on the minority class, through cost-sensitive learning or recognition-based (one-class) learning. The simplest form is class weights: in scikit-learn, class_weight="balanced" sets each class’s weight inversely proportional to its frequency, so errors on the rare class cost more.

Data-level approach: resampling the data to reduce the effects of class imbalance. It is popular with practitioners because it is flexible and works with any algorithm. The most common techniques are:

• Oversampling increases the number of minority class members in the training dataset. Its advantage is that no information from the original training set is lost, since all observations from the minority and majority classes are kept. On the other hand, it is prone to overfitting.
• Undersampling, as opposed to oversampling, reduces the number of majority samples to balance the class distribution. Because it removes observations from the original dataset, it may discard useful information.
• SMOTE (Synthetic Minority Over-sampling Technique), published by Nitesh Chawla and colleagues in 2002, creates new synthetic minority examples instead of copying existing ones, and its authors combined it with undersampling of the majority class.

Which approach works best in practice

Research since 2020 has made the advice more concrete. A practical order of steps:

• Split the data with stratified cross-validation and resample only the training folds. Resampling before the split puts copies of minority examples into the test fold and inflates the score.
• Start with a strong model and no resampling. In 2022, Yotam Elor and Hadar Averbuch-Elor found that balancing, SMOTE included, helped weak classifiers but not the strong ones they tested: XGBoost, LightGBM and CatBoost.
• Tune the decision threshold instead of using the default of 0.5. The same study found it as effective as balancing, and simpler and cheaper.
• Use class weights where the model supports them: a similar effect to oversampling, without a larger dataset.
• If you resample, check calibration. A 2022 simulation study by Ruben van den Goorbergh and colleagues found that random undersampling, random oversampling and SMOTE made logistic regression overestimate the probability of the minority class without improving discrimination.
• For extreme imbalance, try ensembles in which each model sees all minority examples and a different subset of the majority class, or one-class methods trained on normal examples only.

Examples of unbalanced classification

Many classification problems have severe class imbalance, and a look at domains that are inherently unbalanced makes the ideas and challenges of class imbalance concrete:

• Fraud detection
• Claim prediction
• Loan default prediction
• Churn prediction
• Spam detection
• Anomaly detection
• Outlier detection
• Intrusion detection
• Conversion prediction

Each of these domains is an entire field of research, which shows how multidisciplinary class imbalance is and why a machine learning practitioner needs to recognize and handle it. There is no single solution that improves every model on imbalanced data: the most effective techniques differ with the dataset, so try several and compare them with metrics that match the real cost of errors.