Language Model Adaptation for Language and Dialect Identification of Text

Show full item record



Permalink

http://hdl.handle.net/10138/305333

Citation

Jauhiainen , T S , Linden , B K J & Jauhiainen , H A 2019 , ' Language Model Adaptation for Language and Dialect Identification of Text ' , Natural Language Engineering , vol. 25 , no. 5 , 135132491900038 , pp. 561-583 . https://doi.org/10.1017/S135132491900038X

Title: Language Model Adaptation for Language and Dialect Identification of Text
Author: Jauhiainen, Tommi Sakari; Linden, Bo Krister Johan; Jauhiainen, Heidi Annika
Contributor organization: Department of Digital Humanities
Centre of Excellence in Ancient Near Eastern Empires (ANEE)
Language Technology
Krister Linden / Research Group
Department of Modern Languages 2010-2017
Date: 2019-09
Language: eng
Number of pages: 23
Belongs to series: Natural Language Engineering
ISSN: 1351-3249
DOI: https://doi.org/10.1017/S135132491900038X
URI: http://hdl.handle.net/10138/305333
Abstract: This article describes an unsupervised language model (LM) adaptation approach that can be used to enhance the performance of language identification methods. The approach is applied to a current version of the HeLI language identification method, which is now called HeLI 2.0. We describe the HeLI 2.0 method in detail. The resulting system is evaluated using the datasets from the German dialect identifi- cation and Indo-Aryan language identification shared tasks of the VarDial workshops 2017 and 2018. The new approach with LM adaptation provides considerably higher F1-scores than the basic HeLI or HeLI 2.0 methods or the other systems which participated in the shared tasks. The results indicate that unsu- pervised LM adaptation should be considered as an option in all language identification tasks, especially in those where encountering out-of-domain data is likely.
Subject: 6121 Languages
Peer reviewed: Yes
Rights: cc_by
Usage restriction: openAccess
Self-archived version: publishedVersion


Files in this item

Total number of downloads: Loading...

Files Size Format View
LanguageModelAdaptation.pdf 332.9Kb PDF View/Open

This item appears in the following Collection(s)

Show full item record