the big blue rectangle

i began with one instinct: take one of the most striking and deeply human sentences in literature, feed it to a machine, and look through a microscope at the shape it takes inside the machine's “brain.” i chose the opening sentence of kafka's the metamorphosis.

as gregor samsa awoke one morning
from uneasy dreams
he found himself transformed
in his bed
into an enormous insect

using the openly available gpt-2 model through hugging face's transformers library, i broke the sentence into tokens: the fragments of text the model processes. these pass through 12 transformer layers, where attention and learned mathematical transformations reshape each token's representation in relation to itself and the text that precedes it. at the output of the final layer, each token is represented by a vector of 768 numbers.

i extracted those final hidden states and mapped their values to shades of blue. in the extracted matrix, each row corresponds to a token; each column, one dimension of its representation. together, they form the big blue rectangle: a view of the numerical form this sentence takes inside a language model.