As AI companies race to develop more powerful and capable models, a less visible battle is raging in the background, as languages compete for their place within the training data that feeds these models.
The ability of artificial intelligence to understand or interact with a language has become linked to the size of its presence in the data sets from which it learns. While languages like English, Chinese, and Spanish benefit from the abundance of digital content, hundreds of other languages struggle to gain a foothold within this data.
Read also
list of 2 itemsend of list
This is a defect that researchers warn not only affects the quality of translation and the accuracy of answers, but may also redistribute digital influence and economic opportunities, giving some societies an advantage in accessing digital knowledge and services, and leaving others on the margins of the global technical economy.
(pixabay)
Data maps languages into artificial intelligence
The large linguistic models (LLMs) behind apps like ChatGPT, Gemini, and Cloud rely on vast amounts of textual data collected from the Internet, books, research, and other digital sources. Recent studies indicate that some of these models were trained on more than 15 trillion linguistic units (tokens), in one of the largest data collection operations in the history of computing.
But this enormous size does not reflect the linguistic diversity in the world, as a limited group of languages, most notably English, accounts for the largest share of the digital content used in training, while hundreds of other languages receive only a very limited representation.
Abdullah Al-Makki, a researcher in natural language processing at the Canadian University of British Columbia, who specializes in large multilingual linguistic models and artificial intelligence for low-resource languages, and leader of the open-source NileChat project for Arabic dialects, believes that training data sets play a pivotal role in determining which languages become “visible and useful” within the artificial intelligence system, explaining that companies developing models need sufficient amounts of high-quality data in order to be able to support a language reliably.
He points out that training data and evaluation data represent a crucial factor in determining which languages are actually included in the artificial intelligence system.
Speaking to Al Jazeera Net, he said, “Languages that have an abundance of digital texts, a strong presence on the Internet, and advanced data sets tend to get better support in artificial intelligence tools, while languages with limited digital resources remain less visible and less accurate and useful within these systems.”
The digital gap between languages reveals the scale of the problem. According to an analysis of data contained in a recent scientific review on low-resource languages, the World Wide Web archives contain about one billion pages in English, while the presence of the Kashmiri language does not exceed only 60 pages in the “Common Crawl” data that is widely used in training language models.
This disparity means that the ability of models to understand and respond to languages is not only related to the number of speakers, but also to the size of their presence in the data that artificial intelligence systems learn from.
For her part, the Moroccan researcher specializing in artificial intelligence and natural language processing, Omaima Houran, believes that this disparity results not only from the scarcity of data, but also from the weak digital representation of low-resource languages.
She explained in her interview with Al Jazeera Net that the weakness of digital representation is reflected in the performance of models when dealing with them, especially in Arabic dialects, which have only a limited presence in the training data compared to classical Arabic.
This imbalance is not limited to the quality of translation and accuracy of answers, but extends to identifying the languages most capable of benefiting from the current wave of artificial intelligence, versus languages that may face the risk of retreating to the margins of the global digital economy.
From a linguistic gap to an economic gap
The impact of limited training data extends to reshaping societies’ opportunities to benefit from the digital economy. The less present the language is in the data on which the models are trained, the lower the quality of the AI services provided to its speakers.
In this context, Al-Makki warns that the race to develop artificial intelligence may deepen the gap between high-resource languages and low-resource languages, because the former benefit from the abundance of digital texts, development tools, evaluation standards, and commercial incentives, while the latter lacks these components, which may be reflected in opportunities for access to information, education, public services, and participation in the digital economy.
This vision is consistent with the results of a recent study entitled “AI Diffusion in Low Resource Language Countries,” which concluded that countries that rely on low-resource languages record rates of use of artificial intelligence that are about 20% lower than what was expected after accounting for economic and demographic factors, which indicates that limited linguistic support represents an independent obstacle to the adoption of artificial intelligence technologies.
From a broader angle, Mahmoud Al-Hajj, associate professor of natural language processing at Vienna University in Vietnam, believes that this gap is no longer just a technical challenge, but has become an issue of digital justice. Speaking to Al Jazeera Net, he pointed out that the world includes more than 7,000 languages, while artificial intelligence systems only provide good performance for a limited number of them. He adds, “The problem is not the number of speakers, but rather the lack of investment in high-quality digital linguistic resources.”
He adds that linguistic models continuing to favor languages with a strong digital presence creates a recurring cycle, as these languages continually improve, while other languages gradually decline. Over time, this may push some users to rely on dominant languages, such as English, to access knowledge and digital services, which will reduce the production of content in their languages and deepen the data gap in the future.
Al-Hajj asserts that breaking this cycle is possible by investing in high-quality data sets, developing multilingual linguistic techniques, and building partnerships with researchers and local communities, so that communities can interact with artificial intelligence systems that learn from their indigenous cultures and resources, not from translations or interpretations issued by dominant languages.
How are researchers trying to fill the gap?
Despite the widening gap between high-resource languages and low-resource languages, the research community does not view this problem as inevitable, but rather is working to develop new methods that allow training linguistic models even in light of the scarcity of data.
In this context, a recent scientific review on linguistic models for low-resource languages indicates that in recent years, researchers have turned to a group of technical solutions, the most prominent of which is “cross-lingual transfer,” which allows models to take advantage of knowledge gained from data-rich languages to improve their performance in less-represented languages, along with “fine-tuning” and “parameter-efficient tuning” (PEFT) techniques that reduce the need to retrain the entire model, which makes developing models for these Languages are less expensive and more efficient.
The importance of synthetic data is also increasing significantly, as researchers are using the same linguistic models to generate new texts, questions, and dialogues that help compensate for the severe shortage of original data. This is a trend that has begun to achieve promising results in a number of languages that previously suffered from an almost complete absence of digital resources.
Al-Makki believes that these approaches have proven their ability to gradually reduce the gap, noting that his team at the University of British Columbia developed the “NileChat” model for the Egyptian and Moroccan Arabic dialects based on a limited amount of real data, while making extensive use of artificially generated data, which enabled the model to achieve pioneering performance when it was launched despite the limited resources available for these dialects.
Efforts also extend to building new linguistic resources. Al-Makki explains that his team is working through projects such as Alexandria, PALM, and PEARL to collect training and evaluation data directly from local communities, with the aim of providing resources that are more representative of the different Arabic dialects. He stressed that the presence of good evaluation standards is no less important than providing training data, because it allows for measuring the performance of models, discovering their weak points, and improving them systematically.
For her part, Horan believes that addressing the problem requires not only collecting more data, but reconsidering the way it is collected. She says that current models give a structural advantage to dominant languages because they focus on the volume of data, while the goal should be to build curated collections that reflect true linguistic diversity, by involving local communities and native speakers in the development of linguistic resources, so that the quality of representation becomes more important than simply increasing the number of training units.
These efforts are also expanding globally, as Microsoft, in cooperation with research partners and international institutions, has launched initiatives such as “Bring Your Own Language-BYOL,” which relies on data cleansing, generating synthetic data, and fine-tuning models to build custom models for low-resource languages, in addition to the “LINGUA Africa” initiative, which funds the development of datasets and linguistic models for African languages with the aim of enhancing their presence in artificial intelligence applications.
In a similar context, researchers recently presented the AFRILANGTUTOR project, which relies on creating dictionaries and educational data for more than ten low-resource African languages, and then using these resources to train language models capable of providing linguistic education and cultural content in these languages.
The results of the study showed that models trained on this data achieved a noticeable improvement compared to the basic models, an indication that investment in local data can be directly reflected in the quality of artificial intelligence performance.
Can thousands of languages be saved before it is too late?
The debate over language representation in AI is not limited to improving the quality of translation or the accuracy of answers, but extends to the future of linguistic diversity. A recent scientific review of large-scale language models for low-resource languages warns that continued underrepresentation of these languages not only threatens their speakers’ access to AI applications, but may also contribute to a widening digital divide and weaken the presence of these languages in the digital space.
Al-Hajj warns that language is not just a means of communication, but rather a vessel that carries history, culture, and ways of understanding the world. He stresses that societies should be able to interact with artificial intelligence systems that learn from resources produced in their original languages, not from translations or interpretations transmitted through other languages, because this may lead to representing their cultures from an external perspective instead of expressing them in their true voice.
Despite these challenges, researchers believe that this path is still subject to change. In addition to academic initiatives, technology companies and international institutions have begun to invest in building new language resources, enhancing the presence of low-resource languages within the next generation of artificial intelligence systems.
