Probing Neural Network Generalization using Default Patterns
The 22nd SIGMORPHON workshop on Computational Morphology, Phonology, and Phonetics, 2025
⚠️ We are aware our paper is currently inaccessible through ACL Anthology due to an indexing error. We are in contact with ACL to resolve this issue as soon as possible.
Authors
Brandom Prickett*, Tianyi Niu*, Katya Pertsova
Abstract
Whether neural-net models can learn minority default patterns has been a matter of some controversy. Results based on modeling real human language data are hard to interpret due to complexity. Therefore, we examine the learning of a simple artificial language pattern involving defaults using three computational models: an Encoder-Decoder RNN, a Transformer Encoder, and a Logistic Regression. Overall, we find that the models have the hardest time with minority defaults, but can eventually learn them and apply them to novel words (although not always extend them to completely novel segments or novel CV-sequences). Type frequency has the largest effect on learning in all models, trumping the effect of distribution. We examine the weights of two models to provide further insights into how defaults are represented inside the models.