Deep learning in image recognition: the main algorithms explained

This article was first based on “Deep Learning for Computer Vision: A Brief Review” (2018) by A. Voulodimos, N. Doulamis, A. Doulamis and E. Protopapadakis of the Technological Educational Institute of Athens and the National Technical University of Athens. In October 2026 we added what has changed since.

Deep learning lets models made of many layers learn representations of data at many levels of abstraction, loosely imitating how the brain takes in information, and so capture complex structure in large datasets. Deep learning methods have outperformed earlier machine learning techniques in many areas, and image recognition is one of the most visible cases. The main schemes covered by the review are convolutional neural networks, Boltzmann machines (including Deep Belief Networks) and auto-encoders, used in object detection, face recognition, activity recognition and human pose estimation.

This coverage is not exhaustive. Long short-term memory (LSTM) networks, a kind of recurrent neural network, are left out: they serve mainly language modeling, text classification, handwriting and speech recognition, and are used less in vision.

Image classification, detection and segmentation

Image recognition covers several tasks, and the right algorithm depends on which one you need:

• Image classification assigns one label (or several) to a whole image: “cat”, “defective”, “melanoma”. In deep learning this means a network that takes the pixels and outputs a probability for each class.
• Object detection finds every object of interest and draws a bounding box with a label around each one.
• Semantic segmentation labels every pixel with a class, such as road, car or pedestrian.
• Instance segmentation goes one step further and separates individual objects of the same class, giving each its own mask.

Before deep learning, image processing algorithms relied on features designed by people, such as edge detectors and SIFT or HOG descriptors, fed into a classifier such as a support vector machine; HOG with an SVM, for example, became a well-known way to detect pedestrians. Deep learning methods learn these features from the data instead.

Deep learning methods

The review groups the deep models used for image recognition into three families: convolutional neural networks, the “Boltzmann family” of Deep Belief Networks and Deep Boltzmann Machines, and stacked denoising autoencoders.

Convolutional Neural Networks

Convolutional neural networks are trained with error gradients (backpropagation) and achieve very good results in a wide range of pattern recognition tasks.

The CNN architecture uses three specific ideas: local receptive fields, tied weights and spatial subsampling. With local receptive fields, each unit in a convolutional layer receives input from a small neighborhood in the previous layer, so neurons extract elementary visual features such as edges or corners, which later layers combine into higher-order features. Tied weights mean that all units in a feature map share the same weights, because a detector useful in one part of the image is likely useful everywhere. Subsampling, usually done by pooling layers, shrinks the feature maps and makes the network less sensitive to small shifts. A typical network stacks convolution, ReLU and pooling layers and ends with fully connected layers that output the class scores.

This is why CNNs beat ordinary neural networks on images: shared weights cut the number of parameters, the same filter finds a pattern wherever it appears, and the features are learned rather than designed. The breakthrough came in 2012, when AlexNet by Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton won the ImageNet challenge with a top-5 error of 15.3%, against 26.2% for the runner-up. In 2015 Microsoft’s ResNet, with up to 152 layers and residual connections that make such depth trainable, won with 3.57% top-5 error.

Generally, CNNs far exceed traditional machine learning in image recognition. Which CNN is best depends on the job: ResNet variants remain a standard backbone, while Google’s MobileNets, designed for mobile and embedded vision, suit phones and cameras.

Deep Belief and Deep Boltzmann machines

Deep Belief Networks and Deep Boltzmann Machines belong to the Boltzmann family: their building block is the restricted Boltzmann machine (RBM), a generative stochastic neural network. Deep Belief Networks have undirected connections between their top two layers, which form an RBM, and directed connections to the lower layers; Deep Boltzmann Machines have undirected connections between all layers.

Denoising Autoencoders

Stacked denoising autoencoders use autoencoders as their main building block, in the same way that Deep Belief Networks use restricted Boltzmann machines. A denoising autoencoder is trained to reconstruct the original input from a deliberately corrupted copy, which forces it to learn features that capture the structure of the data rather than the noise.

Model comparison

Comparing these models, the review found that CNNs have generally performed better than DBNs on benchmark datasets for image recognition, such as MNIST. Where the input is not visual, DBNs often do better, but accurately estimating joint probabilities is difficult and building a DBN is computationally costly. A major advantage of CNNs is feature learning: their features are learned automatically instead of being hand-crafted.

On the other hand, CNNs rely on labeled training data, while Deep Belief Networks, Boltzmann machines and autoencoders can work unsupervised. Autoencoders can become ineffective if errors are present in the first layers, which may lead the network to reproduce the average of the training data. CNNs, in turn, can be invariant to translation, scaling and rotation, one of their most important assets in tasks such as object detection, because the network recognizes an object even when its pixel values vary significantly.

What changed after 2018: transformers and foundation models

Since the review was written, the biggest shift has been the transformer, the architecture behind large language models. In 2020 Google’s Vision Transformer (ViT) showed that a pure transformer applied to a sequence of 16 by 16 pixel image patches can match the best CNNs on classification when pretrained on enough data. The same year, DETR from Facebook AI made object detection a set prediction task for a transformer, removing hand-designed steps such as anchor boxes and non-maximum suppression.

The second shift is pretraining at scale, so most projects now reuse a pretrained model instead of starting from scratch. OpenAI’s CLIP (2021) learned from 400 million image and text pairs and classified ImageNet zero-shot, from class names alone, as accurately as the original ResNet-50 without using any of its 1.28 million training examples. Meta’s Segment Anything (2023), trained on over 1 billion masks on 11 million images, segments objects in new kinds of images when prompted with a point or a box. Meta’s self-supervised DINOv3 (August 2025) learned from 1.7 billion unlabeled images, and its frozen features beat specialized solutions on several dense prediction tasks.

Should you use a CNN or a transformer? CNNs remain efficient where compute is limited, such as cameras on a production line; transformers and large pretrained models shine with lots of data or tasks that need global context. Both can be fooled by small, deliberate changes to an image, a problem studied as adversarial machine learning.

Object detection

Object detection is the process of detecting instances of semantic objects of a particular class (such as people, planes or birds) in digital images and video, or contaminants on X-ray images of food. A common approach used to be to generate a large set of candidate windows and classify each of them.

Can a CNN do object detection? Yes: the vast majority of deep learning detectors use a CNN variant, with only a few based on other deep models, such as a DBN with a saliency mechanism for remote sensing images. Two-stage detectors of the R-CNN family first propose regions and then classify them; Mask R-CNN (2017) extends Faster R-CNN to predict each object’s mask too. Single-stage detectors such as YOLO (2015) solve detection as one regression problem in a single pass; the first YOLO ran at 45 frames per second, and its successors are still a common choice for real-time detection.

Text in images is a special case. Many OCR systems use CNNs too: the CRNN design (2015) extracts features from a text line with convolutional layers, reads them as a sequence with a recurrent network and transcribes them with a CTC layer.

Face recognition

Face recognition is one of the most active applications of Computer Vision, with great commercial interest. Earlier systems relied on hand-crafted features: an extractor took measurements from an aligned face to build a low-dimensional representation, and a classifier made predictions from it. CNNs changed the field thanks to their ability to learn features and to be invariant to transformations.

Google’s FaceNet and Facebook’s DeepFace are both based on CNNs. DeepFace models the face in 3D, aligns it to a frontal view and passes it through a convolution-pooling-convolution filter, three locally connected layers and two fully connected layers. Trained on four million photos of more than 4,000 people, it reached 97.35% accuracy on the Labeled Faces in the Wild benchmark in 2014, but its representation is hard to interpret, because faces of the same person are not necessarily clustered. FaceNet learns to group the representations of the same person and reached 99.63% on the same benchmark in 2015, with only 128 bytes per face. CNNs are also the core of OpenFace, an open-source tool with slightly lower accuracy, small and fast enough for mobile devices.

What is the best algorithm for face recognition? There is no single winner, but modern systems follow FaceNet’s idea: a deep network turns each face into an embedding, and faces are compared by distance. Losses such as ArcFace (2018), which adds an angular margin between identities, became a widely used way to make embeddings more discriminative. Since 2 February 2025 the EU AI Act bans building face databases by untargeted scraping of the internet or CCTV, and real-time remote biometric identification in public spaces for law enforcement, except in narrow cases.

Activity recognition

Recognizing human activities has received much research attention. In complex event detection, saliency maps were used to detect and locate events, and deep learning then identified the key frames of each event. CNNs have been used to recognize activities in beach volleyball and, from smartphone sensor data, what a person is doing; adding a radius-margin bound as a regularization term improved a deep CNN’s generalization for activity classification.

An increasingly popular strategy is to fuse multimodal features or data: appearance and motion features for group activities in crowded scenes, or video and sensor data for egocentric activity recognition, using two CNNs and long short-term memory. DBNs have been used to recognize activity from video that also contains depth information.

Estimating a human pose

Human pose estimation finds the location of human joints in images, video, depth images or motion capture skeleton data. It is difficult because of the wide range of body shapes and appearances, difficult lighting and cluttered backgrounds. Before deep learning, it was based on detecting body parts, for example with pictorial structures.

Deep learning methods for pose estimation can be holistic or part-based. Holistic methods work on the image globally, without an explicit model of each part and their spatial relations. DeepPose, for example, treats pose estimation as a joint regression problem without a graphical model or part detectors, but holistic methods are inaccurate in high-precision regions, because regressing complex pose vectors directly from images is hard to learn.

Part-based methods detect body parts individually and then add spatial information with a graphical model. Some train a CNN on local and background patches to learn the probability of each part and their spatial relationships; others use many small CNNs, each classifying one body part, followed by a higher-level spatial model that removes outliers and keeps the global pose consistent. The 2018 OpenPose paper describes the first open-source real-time system for multi-person 2D pose detection, covering body, foot, hand and facial keypoints.

How to choose an image recognition algorithm

The review’s conclusions still hold: CNNs learn features automatically and are invariant to many transformations, but depend on labeled data, while DBNs, DBMs and autoencoders can work unsupervised; all are computationally demanding to train. In 2026 the practical choice comes down to a few questions:

• What is the task: one label, boxes or pixel masks? That decides between a classifier, a detector such as YOLO or an R-CNN variant, and a segmentation model.
• How many labeled images do you have? With a few hundred, fine-tune a pretrained backbone; with almost none, try a zero-shot model such as CLIP or Segment Anything.
• Where will it run? On a phone or camera, prefer compact CNNs; on a server with plenty of data, test a vision transformer.
• How will you test it? Keep a test set from real operating conditions, not only from the training source.