Improved Data Sampling Techniques for Training Neural Networks

For classification problems in which we have equal numbers of samples in each class, we propose and present a novel mini-batch sampling approach to train neural networks using gradient descent. Our proposed approach ensures a uniform distribution of samples from all classes in a mini-batch. We will share results showing that this approach yields faster convergence than the random sampling approach commonly used today. We’ll illustrate our approach using several neural network models trained on commonly used datasets, including a truncated version of ImageNet. We will also present results for large and small mini-batch sizes relative to the number of classes. Comparing these results to a suboptimal sampling approach, we hypothesize that having a uniform distribution of samples from each class in a mini-batch is an optimal sampling approach. Our approach benefits model trainers by achieving higher model accuracy with reduced training time.