I have implemented a Neural Network for classification numbers using PyTorch. The layers use ReLU + Softmax for the classification. For efficiency and better results, I have made use of Glorot Initialization of the weights, 0 initialization of the biases, Shuffling, Minibatching, Data Normalization, BatchNorm, RMSProp and Dropout.
The CNN version uses the same strategy for efficiency and better results and the architecture of Yann Lecun's LeNet-5 (see below).
(https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=726791)