How the World Got Wired · 2012 and After

The Picture Contest

In a different part of academia, a young Chinese-American computer science professor named Fei-Fei Li had spent several years building, with help from many graduate students, a database she called ImageNet.

ImageNet was, on its face, a simple idea. It was a collection of digital photographs — about a million and a half of them — each one carefully labeled by a human being. Each picture was labeled with what was in it. A picture of a golden retriever was labeled golden retriever. A picture of a school bus was labeled school bus. The pictures were grouped into about a thousand categories. The work of labeling them — which had to be done by actual people looking at each photograph — had taken years and cost a fortune. Li and her team had used a Web service called Amazon’s Mechanical Turk, which let them pay small fees to hundreds of thousands of online workers around the world to do the labeling, one picture at a time.

The reason Li built ImageNet was to give computer science researchers a hard, well-defined problem to test their algorithms against. Each year, starting in 2010, she ran a competition. Research groups around the world would write programs that tried to look at a photograph and guess what was in it. The programs would be tested against a portion of the ImageNet database that hadn’t been seen before. Whoever’s program guessed correctly the most often won.

Through 2010 and 2011, the winning programs were classical, hand-engineered computer-vision systems — the kind of program where human researchers wrote, in advance, careful rules for how to detect edges, corners, textures, and so on. These programs were getting better every year, but only slowly. The error rate — the fraction of pictures they got wrong — was somewhere around 25 percent.

In the summer of 2012, two of Hinton’s graduate students at the University of Toronto — Alex Krizhevsky and Ilya Sutskever — decided to enter the ImageNet contest using a neural network instead.

This was, you should know, considered eccentric. Almost no serious computer-vision research at the time used neural networks. The conventional view was that they were too small, too slow, too unstable. Krizhevsky and Sutskever thought otherwise. Hinton had taught them. They had been working, in his lab, on ways to train much larger neural networks than anyone else was using, on the new kind of computer chip — graphics processing units, or GPUs, the same chips used in video games — that turned out to be unusually good at neural network calculations.

Krizhevsky did most of the engineering. He spent several months training a single neural network, with about sixty million parameters — many times larger than any neural network anyone had successfully trained before — on two GPUs in his bedroom. The network was called, simply, AlexNet. It was a convolutional neural network, a kind specifically designed for processing images. It used techniques Hinton and his colleagues had been developing for decades.

In September of 2012, the ImageNet competition results were announced.

AlexNet did not just win. AlexNet won by a margin so enormous that the previous winners looked, in comparison, as if they had not been trying. AlexNet’s error rate was about 15 percent, where the previous year’s winner had been around 26 percent. That is a difference so large that, in the language of academic research, it broke the field’s previous understanding of what was possible.

In a single afternoon, in September of 2012, neural networks went from being a fringe interest to being the most important idea in computer science.

All 24 chapters of How the World Got Wired · Library · All courses