A Neural Network that Reads Millions of Street Numbers

To read millions of street numbers on buildings photographed for Google StreetView, Google built a neural network that developed reading accuracy comparable to humans assigned to the task. The company uses the images to read house numbers and match them to their geolocation, storing the geolocation of each building in its database. Having the street numbers matched to physical location on a map is always useful, but it is particularly useful in places where street numbers are otherwise unavailable, or in places such as Japan and South Korea, where streets are rarely numbered in chronological order, but in other ways, such as the order in which they were constructed— a system that makes many buildings impossibly hard to find, even for locals.

"Recognizing arbitrary multi-character text in unconstrained natural photographs is a hard problem. In this paper, we address an equally hard sub-problem in this domain viz. recognizing arbitrary multi-digit numbers from Street View imagery. Traditional approaches to solve this problem typically separate out the localization, segmentation, and recognition steps. In this paper we propose a unified approach that integrates these three steps via the use of a deep convolutional neural network that operates directly on the image pixels. We employ the DistBelief implementation of deep neural networks in order to train large, distributed neural networks on high quality images. We find that the performance of this approach increases with the depth of the convolutional network, with the best performance occurring in the deepest architecture we trained, with eleven hidden layers. We evaluate this approach on the publicly available SVHN dataset and achieve over 96% accuracy in recognizing complete street numbers. We show that on a per-digit recognition task, we improve upon the state-of-the-art and achieve 97.84% accuracy. We also evaluate this approach on an even more challenging dataset generated from Street View imagery containing several tens of millions of street number annotations and achieve over 90% accuracy. Our evaluations further indicate that at specific operating thresholds, the performance of the proposed system is comparable to that of human operators. To date, our system has helped us extract close to 100 million physical street numbers from Street View imagery worldwide."

Ian J. Goodfellow, Yaroslav Bulatov, Julian Ibarz, Sacha Arnoud, Vinay Shet, "Multi-digit Number Recognition from Street ViewImagery using Deep Convolutional Neural Networks," arXiv:1312.6082v2.