All that is English may be Hindi: Enhancing Language Identification through Automatic Ranking of Likeliness of Word Borrowing in Social Media

In this paper, we present a set of computational methods to identify the likeliness of a word being borrowed, based on the signals from social media. In terms of Spearman’s correlation values, our methods perform more than two times better ( 0.62) in predicting the borrowing likeliness compared to the best performing baseline ( 0.26) reported in literature. Based on this likeliness estimate we asked annotators to re-annotate the language tags of foreign words in predominantly native contexts. In 88% of cases the annotators felt that the foreign language tag should be replaced by native language tag, thus indicating a huge scope for improvement of automatic language identification systems.