Lesson 2

Neural Networks & Deep Learning

Neural networks are the engine behind today's most impressive AI. In this lesson, you'll learn what they are, how they're structured, and why "deep learning" suddenly took off in the 2010s.

After this lesson, you'll be able to:

  • ✓ Describe the structure of a neural network (input, hidden, output layers)
  • ✓ Explain what makes a neural network "deep"
  • ✓ Identify the three factors that enabled the deep learning breakthrough

What Is a Neural Network?

A neural network is a computing system inspired by the human brain. It's made up of layers of interconnected "neurons" that work together to process information and make predictions.

Every neural network has three types of layers:

The Full Neural Network

Data flows left to right, layer by layer.

Neural network diagram An input layer of 3 nodes connects to two hidden layers of 4 nodes each, which connect to an output layer of 2 nodes. Every node connects to every node in the next layer. INPUT HIDDEN LAYERS OUTPUT

Input Layer

Receives the raw data — numbers, pixels, words, or any information you feed in.

Hidden Layers

The "thinking" happens here. Each layer detects patterns, combining simple features into complex ones.

Output Layer

Produces the final prediction — a category, a number, a probability, or a word.

Analogy: Like an assembly line — raw materials go in, each station transforms them, and a finished product comes out. The input layer loads the raw materials, hidden layers are the workstations, and the output layer is the finished product rolling off the line.

What Makes It "Deep"?

Deep learning is simply a neural network with many hidden layers. The word "deep" refers to the depth (number of layers), not the complexity of thinking.

A simple neural network might have one or two hidden layers. A deep neural network typically has dozens to a few hundred. More layers allow the network to learn increasingly abstract and complex patterns from data.

Why Deep Learning Broke Through in the 2010s

Neural networks have been around since the 1950s. So why did they suddenly start beating humans at image recognition and language tasks in the 2010s? Three ingredients finally came together:

1. More Data (Internet Scale)
The internet produced massive datasets — billions of images, web pages, and documents — giving neural networks the fuel they needed to learn complex patterns.
2. More Compute (GPUs)
Graphics processing units (GPUs), originally built for video games, turned out to be perfect for the math neural networks need. They made training thousands of times faster.
3. Better Algorithms
New activation functions (ReLU), better weight initialization, and techniques like dropout and batch normalization finally made it possible to train much deeper networks without them breaking down.

Without all three, deep learning wouldn't work. Data without compute is too slow. Compute without data has nothing to learn from. And without better algorithms, deep networks couldn't be trained at all.

Training a neural network is a loop that repeats millions of times:

  1. Forward pass: Feed data through the network and get a prediction.
  2. Compare: Measure how far off the prediction is from the correct answer (this measurement is called the loss).
  3. Backpropagation: Work backward through the network, calculating how much each connection (weight) contributed to the error.
  4. Adjust weights: Nudge each weight slightly to reduce the error next time.
  5. Repeat: Do this millions of times with different examples until the network gets good at its task.

It's like adjusting thousands of tiny dials simultaneously — each adjustment makes the predictions a little more accurate. Over millions of rounds, the network goes from random guessing to remarkable accuracy.

Key Takeaway

Neural networks process data through layers — input, hidden, and output. 'Deep' learning means many layers. The breakthrough happened when massive data, powerful GPUs, and better algorithms all came together in the 2010s.

Try This (Optional)

Pick one app or service you'd describe as "AI-powered" — a recommender, a translator, a photo enhancer. Sketch what its input layer (the data going in) and output layer (the prediction coming out) would be. Everything between them is the black box; that's the "deep" part.

Knowledge Check

What makes a neural network "deep"?

Correct! "Deep" refers to the depth of the network — the number of hidden layers. More layers allow it to learn more complex patterns.
Not quite. "Deep" in deep learning refers to the number of hidden layers in the neural network, not the depth of thinking or data.

Which three factors enabled the deep learning breakthrough in the 2010s?

Correct! Internet-scale data, GPU compute power, and algorithmic improvements like better backpropagation all came together to make deep learning practical.
Not quite. The three key ingredients were: more data (from the internet), more compute power (GPUs), and better algorithms (backpropagation improvements).
Previous: What is AI? Next: LLMs & Transformers