Machine learning
Learning with fewer connections
Comparing static sparsity, SET, and RigL to study the relationship between neural network capacity and predictive performance.
The question
How much can a neural network’s connectivity be reduced while retaining useful predictive performance?
The approach
- Compared fixed sparse masks, Sparse Evolutionary Training, and gradient-guided RigL on a CIFAR-style ResNet-20.
- Evaluated 90%, 95%, and 98% sparsity under SGD with momentum and Adam.
- Inspected accuracy curves, weight visualizations, and connectivity changes across the training runs.
What this explores
Static sparsity remained a competitive baseline, while RigL generally outperformed SET. Optimizer choice affected the comparison, and the highest sparsity level exposed clear capacity limits.
Scope & limitations
These experiments used 15 training epochs, one model family, and CIFAR-10 without data augmentation. The comparisons describe this experimental setting rather than a universal ranking of sparsity methods.
Team: Ammy Lin, Tonantzin Real Rojas, and Anvita Suresh. Completed for Introduction to Deep Learning at Duke University.
Project summary and attribution are based on the repository documentation.