From a bedroom GPU to ChatGPT
In 2012 a neural network trained on two gaming graphics cards in a student’s bedroom crushed an image-recognition contest. Ten years later, chatbots built on the same lineage reached the public.
Through the AI winters, a small group kept working on neural networks, among them Geoffrey Hinton, Yann LeCun and Yoshua Bengio. By the 2000s three ingredients they lacked were arriving: large datasets (like ImageNet, millions of labelled photos), fast parallel hardware in the form of graphics cards built for video games, and better training methods.
2012: AlexNet
In 2012, Hinton’s students Alex Krizhevsky and Ilya Sutskever entered the ImageNet image-recognition challenge with a deep neural network, AlexNet. It won with a top-5 error rate of 15.3%, more than 10 points ahead of the runner-up. It had been trained on two NVIDIA GTX 580 gaming GPUs, in Krizhevsky’s bedroom at his parents’ house.
Yann LeCun called it “an unequivocal turning point in the history of computer vision.” Within a few years, deep learning had taken over speech recognition, translation and much more. In 2018, Hinton, LeCun and Bengio received the Turing Award, computing’s highest honour.
2017: the Transformer
In 2017, eight researchers at Google published “Attention Is All You Need”, introducing the Transformer: an architecture that processes whole sequences at once and learns which parts to pay “attention” to. It trained efficiently on huge amounts of text.
Transformers became the basis of large language models: systems trained on vast text collections to predict the next word, which turned out to be capable of writing, summarising, coding and conversation. ChatGPT, released in November 2022, brought them to the public, followed by other assistants such as Claude.
The long arc
The idea under all of it goes back to the earlier ideas in this trail: Leibniz’s hope that reasoning could be calculation, Turing’s child machine that learns, Rosenblatt’s perceptron, and Weizenbaum’s warning about how readily we see minds in machines. What’s new is mostly scale. The old questions about what the systems understand, what they should decide and who benefits remain open.
Bring it up
“The network that started the modern AI boom was trained on two gaming graphics cards in a student’s bedroom at his parents’ house.”
Follow the thread
- InfluencedIn 1958 the Navy said its neural network would one day be conscious Deep learning is the many-layered descendant of Rosenblatt’s perceptron.
- In contrastAI has gone through “winters” before Neural network researchers kept going through the winters, and were eventually proved right.
- RelatedThe 1960s chatbot people confided in Today’s chatbots revive Weizenbaum’s worries at a far larger scale.
- RelatedThe chess-playing machine that had a man inside Modern AI is trained on huge amounts of data labelled by people, often through platforms like Amazon Mechanical Turk.
