Natural language processing
Building a spelling corrector
A Python spelling corrector that weighs how common a word is against how likely a particular typo would be.
The question
If a word has a typo, how do we choose the most likely intended word? This project combines word frequencies with an error model for insertions, deletions, and substitutions.
The implementation
- Loaded word counts and character-level error counts to estimate word and edit probabilities.
- Generated vocabulary candidates one insertion, deletion, or substitution away from the input, then ranked them using the language and error models.
- Kept known words and used a deterministic ordering to resolve ties. The repository also includes tests of the corrector.
Results & discussion
The candidate list and the ranking model both matter. A likely correction still has to be in the vocabulary and within the edits the implementation considers.
Limitations
The implementation handles single-character insertions, deletions, and substitutions. It does not include transpositions or sentence context, and it can struggle with rare words and names.
Repository: Ammy Lin. This is an implementation of a standard noisy-channel spelling-correction approach.
For the code and full project details, see the project repository.