Detail map of Stanford, California, United States,Edinburgh, Scotland, United Kingdom,Mountain View, California, United States

A: Stanford, California, United States, B: Edinburgh, Scotland, United Kingdom, C: Mountain View, California, United States

A Large Scale Neural Network Connecting 16,000 Computers Appears to Emulate Activity in the Visual Cortex


At the International Conference on Machine Learning held in Edinburgh, Scotland from June 26–July 1, 2012 researchers at Google and Stanford University reported that they developed software modeled on the way biological neurons interact with each other that taught itself to distinguish objects in ­YouTube videos. Although it was most effective recognizing cats and human faces, the system obtained 15.8% accuracy in recognizing 22,000 object categories from ImageNet, or 3,200 items in all, a 70 percent improvement over the previous best-performing software. To do so the scientists connected 16,000 computer processors to create a neural network for machine learning with more than one billion connections. Then they turned the neural network loose on the Internet to learn on its own.

Having been presented with the experimental results before the meeting, on June 25, 2012 John Markoff published an article entitled "How Many Computers to Identify a Cat? 16,000," from which I quote selections"

"Presented with 10 million digital images selected from YouTube videos, what did Google’s brain do? What millions of humans do with YouTube: looked for cats....

"The research is representative of a new generation of computer science that is exploiting the falling cost of computing and the availability of huge clusters of computers in giant data centers. It is leading to significant advances in areas as diverse as machine vision and perception, speech recognition and language translation.

"Although some of the computer science ideas that the researchers are using are not new, the sheer scale of the software simulations is leading to learning systems that were not previously possible. And Google researchers are not alone in exploiting the techniques, which are referred to as “deep learning” models. Last year Microsoft scientists presented research showing that the techniques could be applied equally well to build computer systems to understand human speech....

"The [YouTube] videos were selected randomly and that in itself is an interesting comment on what interests humans in the Internet age. However, the research is also striking. That is because the software-based neural network created by the researchers appeared to closely mirror theories developed by biologists that suggest individual neurons are trained inside the brain to detect significant objects.

"Currently much commercial machine vision technology is done by having humans 'supervise' the learning process by labeling specific features. In the Google research, the machine was given no help in identifying features.

“ 'The idea is that instead of having teams of researchers trying to find out how to find edges, you instead throw a ton of data at the algorithm and you let the data speak and have the software automatically learn from the data,' Dr. Ng said.

“ 'We never told it during the training, ‘This is a cat,’ ' said Dr. Dean, who originally helped Google design the software that lets it easily break programs into many tasks that can be computed simultaneously. 'It basically invented the concept of a cat. We probably have other ones that are side views of cats.'

"The Google brain assembled a dreamlike digital image of a cat by employing a hierarchy of memory locations to successively cull out general features after being exposed to millions of images. The scientists said, however, that it appeared they had developed a cybernetic cousin to what takes place in the brain’s visual cortex.

"Neuroscientists have discussed the possibility of what they call the 'grandmother neuron,' specialized cells in the brain that fire when they are exposed repeatedly or “trained” to recognize a particular face of an individual.

“ 'You learn to identify a friend through repetition,' said Gary Bradski, a neuroscientist at Industrial Perception, in Palo Alto, Calif.

"While the scientists were struck by the parallel emergence of the cat images, as well as human faces and body parts in specific memory regions of their computer model, Dr. Ng said he was cautious about drawing parallels between his software system and biological life.

“ 'A loose and frankly awful analogy is that our numerical parameters correspond to synapses,' said Dr. Ng. He noted that one difference was that despite the immense computing capacity that the scientists used, it was still dwarfed by the number of connections found in the brain.

“ 'It is worth noting that our network is still tiny compared to the human visual cortex, which is a million times larger in terms of the number of neurons and synapses,' the researchers wrote.

"Despite being dwarfed by the immense scale of biological brains, the Google research provides new evidence that existing machine learning algorithms improve greatly as the machines are given access to large pools of data.

“ 'The Stanford/Google paper pushes the envelope on the size and scale of neural networks by an order of magnitude over previous efforts,' said David A. Bader, executive director of high-performance computing at the Georgia Tech College of Computing. He said that rapid increases in computer technology would close the gap within a relatively short period of time: “The scale of modeling the full human visual cortex may be within reach before the end of the decade.”

"Google scientists said that the research project had now moved out of the Google X laboratory and was being pursued in the division that houses the company’s search business and related services. Potential applications include improvements to image search, speech recognition and machine language translation.

"Despite their success, the Google researchers remained cautious about whether they had hit upon the holy grail of machines that can teach themselves.

“ 'It’d be fantastic if it turns out that all we need to do is take current algorithms and run them bigger, but my gut feeling is that we still don’t quite have the right algorithm yet,' said Dr. Ng.

Quoc V. Le,  Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeff DeanAndrew Y. Ng, "Building High-level Features Using Large Scale Upervised Learning," arXiv:1112.6209 [cs.LG] 12 July 2012.  

Timeline Themes