Comparing models for mental-health text

The question

Would adding topic or cluster features improve on sentence embeddings alone? We compared those approaches using a public dataset of text labeled with mental-health-related categories.

What we did

• Prepared text and compared LDA topic features, TF-IDF-based clusters, and all-MiniLM-L6-v2 sentence embeddings.
• Trained logistic regression classifiers on each representation and on combinations of features.
• Compared accuracy, macro F1, and per-class results on a stratified test split, including confusion matrices.

Results & discussion

In the saved run, sentence embeddings reached about 70% accuracy and 0.62 macro F1. Adding topic or cluster features changed those results very little. Performance varied considerably across labels, which was easier to see in the per-class metrics than in overall accuracy.

Limitations

These are dataset labels, not clinical diagnoses, and the project is not a diagnostic tool. Some unsupervised features were fitted before the train/test split; a stricter comparison would fit those steps on training data only. Class imbalance and the source of the text also matter when interpreting the results.