The dense (fully connected) block of LeNet-5 receives the output of the convolutional encoder. To bridge the two parts, the four-dimensional tensor output (batch size, channels, height, width) must be flattened into a two-dimensional representation: the first dimension indexes examples in the minibatch, and the second provides a flat vector for each example. LeNet-5's dense block consists of three fully connected layers with $$120$$, $$84$$, and $$10$$ output units, respectively. The final $$10$$-dimensional output layer corresponds to the number of possible output classes in the classification task.

Claude

LeNet-5 architecture

LeNet-5 is a pioneering convolutional neural network consisting of two main parts: (i) a convolutional encoder with two convolutional layers, and (ii) a dense block with three fully connected layers. The input is a handwritten digit image, and the output is a probability distribution over $$10$$ possible classes.

Convolutional encoder: Each convolutional block applies a $$5 	imes 5$$ kernel followed by a sigmoid activation function and a $$2 	imes 2$$ average pooling operation (stride $$2$$). The first convolutional layer produces $$6$$ output channels, and the second produces $$16$$ output channels. Each pooling operation reduces spatial dimensionality by a factor of $$4$$ (halving both height and width). The convolutional block's output shape is (batch size, number of channels, height, width).

Dense block: The four-dimensional output of the convolutional encoder is flattened into a two-dimensional representation where the first dimension indexes examples in the minibatch and the second gives each example's flat vector. Three fully connected layers with $$120$$, $$84$$, and $$10$$ outputs respectively form the dense block, with the $$10$$-dimensional final layer corresponding to the number of output classes.

Weights are initialized using Xavier initialization.

LeNet-5 remains architecturally meaningful today. When evaluated on Fashion-MNIST, LeNet-5 error rates are much closer to those of advanced architectures such as ResNet (Section 8.6) than to those achievable with basic MLPs (Section 5.2). This demonstrates the substantial leap in performance that convolutional architectures introduced over fully connected networks.

LeNet-5 Architecture

Dive into Deep Learning

The convolutional encoder in LeNet-5 consists of two repeated units, each containing three operations: a convolutional layer, a sigmoid activation function, and an average pooling layer. Each convolutional layer uses a $$5 	imes 5$$ kernel. The first convolutional layer produces $$6$$ output channels, while the second produces $$16$$. After each convolution and activation, a $$2 	imes 2$$ average pooling operation with stride $$2$$ halves both the height and width of the representation, reducing the spatial dimensionality by a factor of $$4$$ per pooling step. The convolutional block maps spatially arranged inputs to a progressively increasing number of two-dimensional feature maps while decreasing spatial resolution. Its output has shape (batch size, number of channels, height, width).

Note that while ReLU and max-pooling perform better in modern networks, they had not yet been discovered at the time LeNet was designed.

LeNet-5 Convolutional Block

Passing a single-channel $$ 28 \times 28 $$ image through LeNet-5 produces the following shapes at each layer:

1. Conv2d ($$ 5 \times 5 $$, padding $$ 2 $$, $$ 6 $$ channels): output $$ 1 \times 6 \times 28 \times 28 $$
2. Sigmoid: output $$ 1 \times 6 \times 28 \times 28 $$
3. AvgPool2d ($$ 2 \times 2 $$, stride $$ 2 $$): output $$ 1 \times 6 \times 14 \times 14 $$
4. Conv2d ($$ 5 \times 5 $$, no padding, $$ 16 $$ channels): output $$ 1 \times 16 \times 10 \times 10 $$
5. Sigmoid: output $$ 1 \times 16 \times 10 \times 10 $$
6. AvgPool2d ($$ 2 \times 2 $$, stride $$ 2 $$): output $$ 1 \times 16 \times 5 \times 5 $$
7. Flatten: output $$ 1 \times 400 $$
8. Linear ($$ 120 $$) + Sigmoid: output $$ 1 \times 120 $$
9. Linear ($$ 84 $$) + Sigmoid: output $$ 1 \times 84 $$
10. Linear ($$ 10 $$): output $$ 1 \times 10 $$

The first convolutional layer uses $$ 2 $$ pixels of padding to preserve the spatial dimensions of the $$ 28 \times 28 $$ input, compensating for the reduction that a $$ 5 \times 5 $$ kernel would otherwise cause. The second convolutional layer uses no padding, reducing height and width by $$ 4 $$ pixels each. As the stack deepens, the number of channels increases ($$ 1 \to 6 \to 16 $$) while spatial dimensions shrink, until flattening yields a $$ 400 $$-element vector.

LeNet-5 Layer-by-Layer Shape Trace

Modern reproductions of LeNet-5 replace the original Gaussian activation layer (also known as the Gaussian decoder or Euclidean Radial Basis Function output) with a softmax layer. This substitution greatly simplifies the implementation because the Gaussian decoder is rarely used in contemporary practice. Apart from this change, the reproduced network matches the original LeNet-5 architecture.

Softmax Replacement of Gaussian Activation in LeNet-5

A key factor distinguishing early convolutional networks like LeNet-5 from later, substantially deeper architectures such as ResNet is the availability of computational resources. As hardware capabilities grew—particularly through advances in GPU processing—researchers were able to design and train networks with many more layers, skip connections, and sophisticated modules. Thus, while LeNet-5 and ResNet both belong to the CNN family and share core design principles, the primary driver behind the leap in architectural complexity was the dramatic increase in accessible computation, not a fundamental change in the underlying learning paradigm.

Computational Scale Enabling Architectural Complexity

- Most commonly used to recognize handwritten and machine printed characters, e.g., recognizing the handwritten numbers on bank checks.

LeNet-5 Convolutional Neural Network Applications

The classic LeNet-5 neural network can be modernized to improve performance and reflect contemporary deep learning practices. This modernization primarily involves two architectural updates: replacing the original sigmoid activation functions with Rectified Linear Unit (ReLU) activations, and substituting average pooling operations with max pooling. When applied to classification tasks like Fashion-MNIST, this slightly modernized LeNet provides an efficient, nontrivial baseline model for evaluating training dynamics and optimization algorithms.

Learn Before

Related