The Architecture
of Intelligence

The intellectual foundation for modern AI came together before anyone had used the term. In the 1940s, mathematics, logic, and early neuroscience were converging, and the framework that emerged was initially called "cybernetics," built largely by mathematician Norbert Wiener.

Wiener's core ideas came out of World War II, when he was working on systems to aim anti-aircraft guns at fast-moving bombers. The problem was prediction: human pilots don't move randomly, so past behavior could be modeled statistically to forecast future trajectories. More importantly, he formalized the concept of the "feedback loop," observing that both biological and mechanical systems work by sensing their environment, processing that input, and adjusting behavior accordingly. That architecture is the same "perceive-plan-act" loop that governs modern autonomous AI agents.

In 1943, Warren McCulloch and Walter Pitts published the first mathematical model of an artificial neuron, showing that networks of simple binary threshold devices could perform basic logic operations. It could handle AND/OR logic but was mathematically unable to solve nonlinear problems like XOR or XNOR.

Donald Hebb added something critical in 1949. His principle "neurons that fire together, wire together" introduced neuroplasticity to artificial networks and directed connectionist research toward weighted inputs.

In 1950, Alan Turing reframed the entire question. His Imitation Game replaced "can machines think?" with a behavioral test: if a machine could convince a human interrogator via typed exchange that it was human, that was sufficient. He also described "learning machines," systems that could alter their own rules through inductive processes, essentially what gradient descent does today.

These threads came together at the 1956 Dartmouth Summer Research Project, where the term "Artificial Intelligence" was formally adopted. Frank Rosenblatt introduced the Perceptron in 1957, the first trainable neural network. Then Minsky and Papert demolished it in 1969, proving single-layer networks couldn't compute XOR. Funding collapsed, and neural networks were set aside for more than a decade.

When symbolic AI hit its wall, researchers made a clean break. Instead of writing rules, they built systems to find statistical patterns in raw data. Decision trees, support-vector machines, ensemble methods. This was modern machine learning taking shape.

At the same time, connectionism got a second chance. In 1986, David Rumelhart, Geoffrey Hinton, and Ronald Williams published a paper on applying backpropagation to neural networks. Backpropagation is essentially the chain rule of calculus applied recursively: compute the loss function, then propagate the error gradients backward through the network's hidden layers to adjust weights. This solved the nonlinear limitations that had killed early connectionism.

The "vanishing gradient problem" was brutal: as networks got deeper, error signals tended to either explode or vanish before reaching the early layers. Sepp Hochreiter and Jurgen Schmidhuber solved it for sequential modeling in 1997 with the Long Short-Term Memory (LSTM). Instead of a simple recurrent loop, LSTMs used a memory cell with three learnable gates: input, output, and forget. LSTMs became the backbone of NLP, machine translation, and speech recognition for the next two decades.

The real break came in 2012. Three things converged: massive labeled datasets (ImageNet), GPUs for parallel computation, and better algorithmic design. AlexNet, built by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, beat the second-place ImageNet system by 10.8 percentage points. Deep learning became the dominant AI approach almost overnight.

DeepMind's AlphaGo then mastered the game of Go using a dual-network architecture: a Policy Network to select moves, and a Value Network to estimate winning probability. First trained on human expert games, then refined through millions of self-play iterations.

Before the generative AI wave, NLP was stuck with sequential processing. A critical bridge appeared in 2013 with Word2Vec from Google. Word2Vec replaced sparse one-hot encodings with dense vector spaces, mapping words into coordinate systems based on surrounding context. "King minus man plus woman equals queen" actually worked in the geometry of the embedding space.

The problem was that RNNs and LSTMs still processed words one after another. You couldn't parallelize across a full sequence, so training on large datasets was slow. That constraint broke in 2017 with "Attention Is All You Need," a paper from Google Brain and the University of Toronto that introduced the Transformer.

The Transformer dropped recurrence and convolutions entirely, replacing them with self-attention. Rather than processing sequentially, Transformers process every token in a sequence simultaneously. Running this across multiple parallel sub-spaces ("multi-head attention"), with positional encodings to preserve word order, enabled training to be massively parallelized. That was the infrastructure the LLM era needed.

Google introduced BERT in 2018, and OpenAI launched GPT. GPT-2 in 2019 had 1.5 billion parameters. GPT-3 in 2020 had 175 billion. These models showed something unexpected: just by scaling up parameter count and training data, models started doing things nobody had specifically trained them to do. Few-shot reasoning, translation, code generation. These "emergent abilities" weren't designed; they appeared.

In 2020, Jared Kaplan and colleagues at OpenAI showed that model performance follows a predictable power-law with compute. In 2022, DeepMind's Chinchilla paper upended that. GPT-3 was massively undertrained. The right ratio is roughly 20 tokens of training data per parameter — about 11 times more data than the field had been using.

Scaling up LLMs revealed a problem: training on internet text to predict the next token doesn't automatically produce a system that's safe, truthful, or useful. The statistical model is happy to confidently say false things if false things appear often in training data.

The fix came from a 2017 paper by Paul Christiano and colleagues at OpenAI and DeepMind, introducing what became known as RLHF (Reinforcement Learning from Human Feedback). Human contractors ranked pairs of AI outputs. That feedback trained a "reward model" as a proxy for human judgment. Then the main AI was optimized using Proximal Policy Optimization (PPO) to maximize the reward model's score.

Applied to NLP, RLHF became the core of modern alignment work. By late 2022, OpenAI used it to fine-tune GPT-3 into InstructGPT, teaching the model to follow written instructions rather than just autocomplete text. That alignment work is what made ChatGPT work when it launched in November 2022. Millions of people who had never thought about LLMs suddenly had a coherent conversational AI in their browser.

By 2024 and 2025, the focus shifted toward reasoning. OpenAI's o1 and o3 models, alongside DeepSeek R1, used RL techniques to generate internal chain-of-thought reasoning traces before producing answers. The finding: reasoning behavior emerged from reinforcement learning without explicitly programming it in.

LLMs, even well-aligned ones, have a fundamental limitation. They are static. An LLM is trained on data up to a cutoff date and produces text based on statistical patterns from that training. It can't look things up, run code, interact with systems, or update its knowledge. Human cognition doesn't work that way.

This gap created the agentic AI paradigm: use an LLM as a cognitive core, then wire it to external tools, memory systems, and control loops that let it act on the world. The shift is from generative AI (making content) to agentic AI (taking goal-directed actions in software environments). There's an irony worth noting: this architecture is really a return to the planning goals of symbolic AI, using the Belief-Desire-Intention framework structure. The difference is that the rule-following logic that classical AI hand-coded is now handled by billions of neural parameters.

The ReAct (Reason + Act) framework in 2022 gave agents structure: generate a thought, take a domain-specific action like searching Wikipedia, observe the result, use it to inform the next thought. In 2023, the Toolformer paper from Meta showed LLMs could teach themselves to use external tools through self-supervised learning. This became "function calling."

By 2025 and 2026, the proliferation of specialized AI agents created a fragmentation problem. Anthropic launched the Model Context Protocol (MCP) in late 2024 as a solution: an open-source standard built on JSON-RPC 2.0 that abstracts the integration layer between AI clients and external data sources. By early 2026, MCP had crossed 100 million monthly SDK downloads.

A three-layer protocol stack crystallized in 2026, with governance increasingly moving to bodies like the Linux Foundation's Agentic AI Foundation and the W3C AI Agent Protocol Community Group.