The automatic extraction of small objects such as roadside milestones, small traffic signs, and other urban furniture remains a technical challenge. This study focuses on methods of deep learning to detect small urban elements in mobile mapping system (MMS) images. Based on images obtained by an MMS in urban areas, we create an urban element detection (UED) data set containing several kinds of small objects found in a city. A simple feature extraction convolution neural network (CNN) called SlimNet is proposed and combined with an optimized faster R-CNN framework. The resulting deep learning method can automatically extract small objects commonly found in cities, including manhole covers, milestones, and license plates. Experiments on the UED data set show that SlimNet has the highest accuracy compared with other popular networks, including VGG, MobileNet, ResNet, and YOLOv3. The SlimNet model can achieve a mean average precision (AP) that is up to 12.3 higher than that of the lowest ResNet-152 network and can accelerate both training and detection owing to its relative simplicity. Moreover, $k$ -means clustering is used to choose the dimensions of the anchor box for detection. We ran $k$ -means clustering for different numbers of clusters, and the results show that at least four clusters are needed for detection using a small data set such as the UED. We also propose a method to use templates of different scales for anchors to further improve small object detection; this approach improved the AP by 34 in our experiments.
