Machine learning
Looking inside a four-neuron model
I followed a small sentiment model from its predictions to its hidden neurons, then turned neurons off to check what changed.
The question
What can we reasonably say that a hidden neuron does? I used a small model where I could inspect every allowed input and check an explanation against counterexamples.
My part in the project
I reproduced the starter model’s training and wrote an investigation of its activations, output contributions, and neuron ablations. The notebook builds on a team sentiment-model exercise.
What I did
- Reproduced a model with six input words, four hidden neurons, and two output scores. It used 28 training examples and four validation examples.
- Compared activations across positive and negative inputs, then calculated how each neuron contributed to the difference between output scores.
- Set individual activations to zero without retraining, checked all 32 combinations, and looked for examples that challenged an overly broad explanation.
Results & discussion
Turning off neuron 2 changed five of the 32 predictions. Its contribution pushed the model toward a negative answer, but it also activated for positive inputs. Calling it an exclusively negative detector would have missed that distinction.
Limitations
This is a six-word task that ignores word order. The 32 combinations include the training data, and only four examples were held out. The findings describe one seeded toy model, not how a language model understands real sentences.
Investigation by Ammy Lin, building on a team starter-model exercise for AIPI 590. The notebook explains which training results were reproduced.
For the code and full project details, see the project repository.