Large Language Models

At this year’s NYU Tandon commencement address Yann LeCun said “Large Language Models do not create anything.” And encouraged students to learn how they work and use them to create new things. He emphasised you will be at a disadvantage if you don’t learn how to use them effectively.

The most valuable commodity in the world is trust. It cannot be manufacured or bought, and can vanish in an instant.

You need to figure out who you can trust. It is the oldest problem in the world.

A Short History of Machine Learning

The origins of ML began in 1805 with Gauss and Legendre inventing the “least squares” method. Given points (x_j, y_j) find a line y = ax + b that “best” fits the data. They proposed minimizing the loss function L(a,b) = \sum_j (ax_j + b - y_j)^2 over a and b.

From 1930–1940 statisical theory advanced with more general linear models in higher dimensions maximum likelihood methods.

(McCulloch and Pitts 1943) gave the first mathematical definition of a neuron.

(Rosenblatt 1958) coined the term perceptron and implemented it in hardware. Given “good” points (x_i,y_i) and “bad” points (x_j',y_j') find a line separating them.

Minsky and Papert (1969)] set the field back by showing a perceptron could not be trained on XOR.

It wasn’t until 1980-1995 that people realized you could solve this with neural networks: layers of perceptron with output of each layer providing input for the next. This is also when reinforcement learning was developed.

From 1995-2012 statistical learning was developed: Expectation Maximization, Hidden Markov Models, decision trees, random forests, … Models were small and understandable.

The deep learning started in 2012 when Moore’s Law made computing power and memory available. No new techniques were invented until transformers (Vaswani et al. 2017) “Attention Is All You Need.” It allowed models to run in parallel instead of sequentially.

Source code for early OpenAI model gpt-2

LLM

A large language model is a function and a context. Given a sequence of characters it returns a sequence of characters. It takes training data consisting of “correct” answers and interpolates.

So called “prompt engineering” is really about context engineering.

The simple prompt “yes” will clue you in to the current context of the LLM you are using at the moment. Use the output to detect what it is not telling you.

Prompt from general to specific. You are looking for a needle in a haystack.

DO NOT TRUST LLMs!

McCulloch, Warren S., and Walter Pitts. 1943. “A Logical Calculus of the Ideas Immanent in Nervous Activity.” The Bulletin of Mathematical Biophysics 5 (4): 115–33. https://doi.org/10.1007/BF02478259.

Minsky, Marvin, and Seymour Papert. 1969. Perceptrons. Cambridge, MA: MIT Press.

Rosenblatt, Frank. 1958. “The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain.” Psychological Review 65 (6): 386–408. https://doi.org/10.1037/h0042519.

Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. “Attention Is All You Need.” In Advances in Neural Information Processing Systems. Vol. 30. Curran Associates, Inc. https://arxiv.org/abs/1706.03762.

People I Trust

A Short History of Machine Learning

LLM