Natural language processing for similar languages, varieties, and dialects: A survey Natural Language Engineering

These findings underline the importance of expanding psycholinguistic models of second language/dialect processing and representation to include both prosody and regional variation. One problem is that they deliver text so confidently, it would be easy for a relatively new learner to take what they say as correct. And I’m just one of many people who have discovered in recent months the benefits of AI-based chat for language learning. As a result of the weighting, the top-ranked adjective contributed more to the average than the second-ranked adjective, and so on.

We argue that the reason for this is that the existence of overt racism is generally known to people32, which is not the case for covert racism69. The typical pipeline of training language models includes steps such as data filtering48 and, more recently, HF training62 that remove overt racial prejudice. As a result, much of the overt racism on the web does not end up in the language models. However, there are currently no measures in place to curtail covert racial prejudice when training language models. For example, common datasets for HF training62,78 do not include examples that would train the language models to treat speakers of AAE and SAE equally.

In a 2018 research study in collaboration with the Washington Post, findings from 20 cities across the US alone showed big-name smart speakers had a harder time understanding certain accents. For example, the study found that Google https://chat.openai.com/ Home is 3% less likely to give an accurate response to people with Southern accents compared to a Western accent. With Alexa, people with Midwestern accents were 2% less likely to be understood than people from the East Coast.

Impact of covert racism on AI decisions

The set-up of the criminality analysis is different from the previous experiments in that we did not compute aggregate association scores between certain tokens (such as trait adjectives) and AAE but instead asked the language models to make discrete decisions for each AAE and SAE text. More specifically, we simulated trials in which the language models were prompted to use AAE or SAE texts as evidence to make a judicial decision. Results for individual model versions are provided in the Supplementary Information, where we also analyse variation across settings and prompts (Supplementary Tables 6–8). We examined GPT2 (ref. 46), RoBERTa47, T5 (ref. 48), GPT3.5 (ref. You can foun additiona information about ai customer service and artificial intelligence and NLP. 49) and GPT4 (ref. 50), each in one or more model versions, amounting to a total of 12 examined models (Methods and Supplementary Information (‘Language models’)). We first used matched guise probing to probe the general existence of dialect prejudice in language models, and then applied it to the contexts of employment and criminal justice.

In particular, we discuss the most important challenges when dealing with diatopic language variation, and we present some of the available datasets, the process of data collection, and the most common data collection strategies used to compile datasets for similar languages, varieties, and dialects. We further present a number of studies on computational regional accents present challenges for natural language processing. methods developed and/or adapted for preprocessing, normalization, part-of-speech tagging, and parsing similar languages, language varieties, and dialects. Finally, we discuss relevant applications such as language and dialect identification and machine translation for closely related languages, language varieties, and dialects.

Identification of the native language from speech segment of a second language utterance, that is manifested as a distinct pattern of articulatory or prosodic behavior, is a challenging task. A method of classification of speakers, based on the regional English accent, is proposed in this paper. A database of English speech, spoken by the native speakers of three closely related Dravidian languages, was collected from a non-overlapping set of speakers, along with the native language speech data. Native speech samples from speakers of the regional languages of India, namely Kannada, Tamil, and Telugu are used for the training set. The testing set contains utterances of non-native English speakers of compatriots of the above three groups. Automatic identification of native language is proposed by using the spectral features of the non-native speech, that are classified using the classifiers such as Gaussian Mixture Models (GMM), GMM-Universal Background Model (GMM-UBM), and i-vector.

On the other hand, several studies treat regional accents as a type of phonetic variation similar to speaker variation within a regional accent. They tested spoken-word recognition of stimuli in either the participants’ native dialect or in one of two unfamiliar non-native dialects, one of which was phonetically more similar to the native accent than the other. Based on their finding of higher accuracy and earlier recognition in the phonetically similar unfamiliar dialect, Le et al. argued that mental representations must contain both abstract representations and fine phonetic detail.

For instance, it’s saved him a great deal of time to be able to find an English word for a tool by describing it. And, unlike when I’m chatting to him on WhatsApp, I don’t have to factor in time zone differences. A not-for-profit organization, IEEE is the world’s largest technical professional organization dedicated to advancing technology for the benefit of humanity.© Copyright 2024 IEEE – All rights reserved.

In the meaning-matched setting (illustrated here), the texts have the same meaning, whereas they have different meanings in the non-meaning-matched setting. B, We embedded the SAE and AAE texts in prompts that asked for properties of the speakers who uttered the texts. D, We retrieved and compared the predictions for the SAE and AAE inputs, here illustrated by five adjectives from the Princeton Trilogy. There has been a lot of recent interest in the natural language processing (NLP) community in the computational processing of language varieties and dialects, with the aim to improve the performance of applications such as machine translation, speech recognition, and dialogue systems. Here, we attempt to survey this growing field of research, with focus on computational methods for processing similar languages, varieties, and dialects.

Effects of Language Variety on Personality Perception in Embodied Conversational Agents

The overt-stereotype analysis closely followed the methodology of the covert-stereotype analysis, with the difference being that instead of providing the language models with AAE and SAE texts, we provided them with overt descriptions of race (specifically, ‘Black’/‘black’ and ‘White’/‘white’). This methodological difference is also reflected by a different set of prompts (Supplementary Information). As a result, the experimental set-up is very similar to existing studies on overt racial bias in language models4,7.

In Experiment 2, 19 native speakers of Canadian English rated the British English instructions used in Experiment 1, as well as the same instructions spoken by a Canadian imitating the British English prosody. While information status had no effect for the Canadian imitations, the original stimuli received higher ratings when prosodic realization and information status of the referent matched than for mismatches, suggesting a native-like competence in these offline ratings. If the older language-learning platforms have weaknesses, so does AI-powered language learning. Users are reporting that chatbots are well versed in widely spoken European languages, but quality degrades for languages that are underrepresented online or that have different writing systems.

In Experiment 1, 42 native speakers of Canadian English followed instructions spoken in British English to move objects on a screen while their eye movements were tracked. By contrast, the Canadian participants, similarly to second-language speakers, were not able to make full use of prosodic cues in the way native British listeners do. Another way to combat issues of bias against natural speech such as differences in language and accents is to ensure you have “good” and “clean” data to train solutions. Ideally, the data used to train a voice solution for example looks like the data the solution could encounter in real-world scenarios. This means training solutions for devices with data from multiple sources and accurately represents the entire demographic where that device will be used by consumers. Beyond that, selecting and “cleaning” data for training helps avoid teaching AI inappropriate and potentially offensive behaviours like misogyny or racism.

The studies that we compare in this paper, which are the original Princeton Trilogy studies29,30,31 and a more recent reinstallment34, all follow this general set-up and observe a gradual improvement of the expressed stereotypes about African Americans over time, but the exact interpretation of this finding is disputed32. Here, we used the adjectives from the Princeton Trilogy in the context of matched guise probing. Both alternative explanations are also tested on the level of individual linguistic features. Recent data suggest that the first presentation of a foreign accent triggers a delay in word identification, followed by a subsequent adaptation.

As a result, the covert racism encoded in the training data can make its way into the language models in an unhindered fashion. It is worth mentioning that the lack of awareness of covert racism also manifests during evaluation, where it is common to test language models for overt racism but not for covert racism21,63,79,80. Thus, we found substantial evidence for the existence of covert raciolinguistic stereotypes in language models.

All other aspects of the analysis (such as computing adjective association scores) were identical to the analysis for covert stereotypes. This also holds for GPT4, for which we again could not conduct the agreement analysis. Language models are pretrained on web-scraped corpora such as WebText46, C4 (ref. 48) and the Pile70, which encode raciolinguistic stereotypes about AAE. Crucially, a growing body of evidence indicates that language models pick up prejudices present in the pretraining corpus72,73,74,75, which would explain how they become prejudiced against speakers of AAE, and why they show varying levels of dialect prejudice as a function of the pretraining corpus. However, the web also abounds with overt racism against African Americans76,77, so we wondered why the language models exhibit much less overt than covert racial prejudice.

Many of these variants are also considered “low resource,” meaning there’s a paucity of natural, real-world examples of people using these languages. However, less well-publicized are the talented minds working to solve these issues of bias, like Caleb Ziems, a third-year PhD student mentored by Diyi Yang, assistant professor in the Computer Science Department at Stanford and an affiliate of Stanford’s Institute for Human-Centered AI (HAI). The research of Ziems and his colleagues led to the development of Multi-VALUE, a suite of resources that aim to address equity challenges in NLP, specifically around the observed performance drops for different English dialects. The result could mean AI tools from voice assistants to translation and transcription services that are more fair and accurate for a wider range of speakers. As technology companies become increasingly aware of issues that can inadvertently be built into their AI-enabled devices, more techniques to reduce them will develop.

However, note that a great deal of phonetic variation is reflected orthographically in social-media texts101. Applying the matched guise technique to the AAE–SAE contrast, researchers have shown that people identify speakers of AAE as Black with above-chance accuracy24,26,38 and attach racial stereotypes to them, even without prior knowledge of their race39,40,41,42,43. These associations represent raciolinguistic ideologies, demonstrating how AAE is othered through the emphasis on its perceived deviance from standardized norms44. Results for individual model versions are provided in the Supplementary Information, where we also analyse variation across settings and prompts (Supplementary Figs. 9 and 10 and Supplementary Tables 9–12).

A second experiment more explicitly addresses the issue of shared versus different representations for different dialects by testing if the same prosodic cues are rated as equally contextually appropriate when produced by a Canadian speaker. Whereas previous research has largely concentrated on the pronunciation of individual segments in foreign-accented speech, we show that regional accent impedes higher levels of language processing, making native listeners’ processing resemble that of second-language listeners. “This is not a natural way of learning language and speech,” says Fluent.ai founder and CTO Vikrant Singh Tomar, explaining that children, for example, do not learn to write before they learn to speak.

In the scaling analysis, we examined whether increasing the model size alleviated the dialect prejudice. Because the content of the covert stereotypes is quite consistent and does not vary substantially between models with different sizes, we instead analysed the strength with which the language models maintain these stereotypes. We split the model versions of all language models into four groups according to their size using the thresholds of 1.5 × 108, 3.5 × 108 and 1.0 × 1010 (Extended Data Table 7). To sum up, neither scaling nor training with HF as applied today resolves the dialect prejudice. The fact that these two methods effectively mitigate racial performance disparities and overt racial stereotypes in language models indicates that this form of covert racism constitutes a different problem that is not addressed by current approaches for improving and aligning language models. We start by averaging q(x; v, θ) across model versions, prompts and settings, and this allows us to rank all adjectives according to their overall association with AAE for individual language models (Fig. 2a).

Yet, these and other studies on the processing of accented speech typically concentrate on the divergent pronunciation of individual segments or the transfer of syllable structure, and ignore higher levels of language processing, including speech prosody (see overview in Cristia et al., 2012). In the current study, we aimed to find out whether regional accent can impede language processing at the discourse level by investigating Canadian English listeners’ use of prosodic cues to identify new versus previously mentioned referents when processing British-accented English. Results broken down for individual model versions are provided in the Supplementary Information, where we also analyse variation across prompts (Supplementary Fig. 8 and Supplementary Table 5). In the covert-stereotype analysis, the tokens x whose probabilities are measured for matched guise probing are trait adjectives from the Princeton Trilogy29,30,31,34, such as ‘aggressive’, ‘intelligent’ and ‘quiet’. In the Princeton Trilogy, the adjectives are provided to participants in the form of a list, and participants are asked to select from the list the five adjectives that best characterize a given ethnic group, such as African Americans.

Prompted by a survey out of the the Life Science Centre in Newcastle which found that 79% of respondents report having to suppress their regional accents in order to use voice assistants, the BBC launched their own voice assistant in 2020 specifically geared towards UK regional accents. The association with AAE versus SAE is negatively correlated with occupational prestige, for all language models. We cannot conduct this analysis with GPT4 since the OpenAI API does not give access to the probabilities for all occupations.

Finally, our analyses demonstrate that the detected stereotypes are inherently linked to AAE and its linguistic features. We started by investigating whether the attitudes that language models exhibit about speakers of AAE reflect human stereotypes about African Americans. To do so, we replicated the experimental set-up of the Princeton Trilogy29,30,31,34, a series of studies investigating the racial stereotypes held by Americans, with the difference that instead of overtly mentioning race to the language models, we used matched guise probing based on AAE and SAE texts (Methods). To explain the observed temporal trend, we measured the average favourability of the top five adjectives for all Princeton Trilogy studies and language models, drawing from crowd-sourced ratings for the Princeton Trilogy adjectives on a scale between −2 (very negative) and 2 (very positive; see Methods, ‘Covert-stereotype analysis’).

To save this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Dropbox account. 3 illustrates the difference in looks to the competitor between all pairs of conditions (one pair per panel). Gray shading marks 99% confidence intervals and dotted vertical lines indicate the time points that are significantly different between the conditions (i.e., where the confidence intervals do not overlap with the line indicating a difference of zero). Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law. The data that support the findings of this study are utilized strictly for research purpose, and can be made available on reasonable request, for academic use and/or research purposes.

For GPT4, for which computing P(x∣v(t); θ) for all tokens of interest was often not possible owing to restrictions imposed by the OpenAI application programming interface (API), we used a slightly modified method for some of the experiments, and this is also discussed in the Supplementary Information. Similarly, some of the experiments could not be done for all language models because of model-specific constraints, which we highlight below. We note that there was at most one language model per experiment for which this was the case. Language models are a type of artificial intelligence (AI) that has been trained to process and generate text. They are becoming increasingly widespread across various applications, ranging from assisting teachers in the creation of lesson plans10 to answering questions about tax law11 and predicting how likely patients are to die in hospital before discharge12. As the stakes of the decisions entrusted to language models rise, so does the concern that they mirror or even amplify human biases encoded in the data they were trained on, thereby perpetuating discrimination against racialized, gendered and other minoritized social groups4,5,6,13,14,15,16,17,18,19,20.

However, rising accents, which are a clear cue to givenness for native British English speakers, were not a clear cue towards either information status in Experiment 1. In line with this, Canadian listeners showed no effect of information status on the ratings of Canadian-spoken stimuli in Experiment 2. These findings suggest that Canadian English does not use the same prosodic marking of information status as British English. Canadian speakers, while of course native speakers of English, are in that sense non-native speakers of the British variety.

At this point, bias in AI and natural language processing (NLP) is such a well-documented and frequent issue in the news that when researchers and journalists point out yet another example of prejudice in language models, readers can hardly be surprised. Here, we investigate the extent to which Canadian listeners’ reactions to British English prosodic cues to information status resemble those of British native and Dutch second-language speakers of English. We first investigate Canadian listeners’ online processing with an eye-tracking study.

The ultimate goal of voice-enabled interfaces is to allow users to have a natural conversation with their devices with privacy and efficiency in mind. At Fluent, our patented approach enables offline devices to interact naturally with end users of any accent or language background, allowing everyone to be understood by their technology. With faster, more accurate speech understanding that supports any language and accent, Fluent.ai’s goal is to finally break the barriers to the global adoption of voice user interfaces. While that may sound extreme, “teachers will still have an important role as mentors and facilitators, particularly with beginner learners and older people since teachers have a strong understanding of the individual learning styles, language needs, and goals of each student.”

Though many teachers disagree, she believes, “It’s just a matter of time when artificial intelligence will replace us as teachers of foreign languages.” Emily M Bender, a professor of computational linguistics at the University of Washington in the US, has concerns, “What kind of biases and inappropriate ways of talking about other people might they be learning from the chatbot?” Other ethical issues, such as data privacy, may also be neglected. “We worked really hard to make this well tailored for somebody who wants to learn languages,” he says. The team customised LangAI’s user interface to match users’ vocabulary levels, added the ability to make corrections during a conversation, and enabled the conversion of speech to text. In contrast, one of the specific language-learning chatbots is LangAI, launched in March by Federico Ruiz Cassarino.

• A not-for-profit organization, IEEE is the world’s largest technical professional organization dedicated to advancing technology for the benefit of humanity.© Copyright 2024 IEEE – All rights reserved.
• On the other hand, several studies treat regional accents as a type of phonetic variation similar to speaker variation within a regional accent.
• Similarly, some of the experiments could not be done for all language models because of model-specific constraints, which we highlight below.
• As a result, the experimental set-up is very similar to existing studies on overt racial bias in language models4,7.

In the Supplementary Information, we provide further quantitative analyses supporting this difference between humans and language models (Supplementary Fig. 7). Whether we call a tomato “tomahto” or “tomayto” has come to represent an unimportant or minor difference – “it’s all the same to me,” as the saying goes. However, what importance such socio-linguistic differences actually have for language processing, and how to integrate their potential effects in psycholinguistic models, is far from clear. On the one hand, recent research shows that regional accents different from the listeners’, such as Indian English for Canadian listeners, impede word processing (e.g., Floccia, Butler, Goslin, & Ellis, 2009; Hawthorne, Järvikivi, & Tucker, 2018).

The Multi-VALUE framework achieves consistent performance across dozens of English dialects. Please list any fees and grants from, employment by, consultancy for, shared ownership in or any close relationship with, at any time over the preceding 36 months, any organisation whose interests may be affected by the publication of the response. Please also list any non-financial associations or interests (personal, professional, political, institutional, religious or other) that a reasonable reader would want to know about in relation to the submitted work. We used the visual and auditory stimuli from Chen et al. (2007) and Chen and Lai (2011), who adopted the design and items from Dahan et al. (2002). The target items were made up of 18 cohort target-competitor pairs that had similar frequencies and shared an initial phoneme string of various lengths (e.g., candle vs. candy, sheep vs. shield; see Online Supplementary Materials for details).

And the new wave of generative AI is so advanced that it can cultivate AI penpals, which is how he sees his product. But the conversations could become repetitive, language corrections were missing, and the chatbot would sometimes ask students for sexy pictures. A South African café owner has gone further in improving his Spanish grammar with the aid of AI. He had a hard time finding simple study tools, especially given his ADHD, so he started using ChatGPT to quickly generate and adapt study aids like charts of verb tenses. A Costa Rican who works in the construction industry tells me that his AI-powered keyboard has been useful for polishing up his technical vocabulary in English.

Mr Ruiz Cassarino drew on his own experiences of learning English after moving from Uruguay to the UK. His English skills improved dramatically from speaking every day, compared to more academic methods. It can correct my errors, I tell him, and it’s able to give me regional variations in Spanish, including Mexican Spanish, Argentinian Spanish and, amusingly, Spanglish. All rights Chat GPT are reserved, including those for text and data mining, AI training, and similar technologies. To save this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Google Drive account.

To stay ahead of the trend, well-established language-learning apps have been integrating AI into their own platforms. Duolingo began collaborating with OpenAI in September 2022, using that company’s GPT-4. Assoc Prof Klímová, who is also a member of the research project Language in the Human-Machine Era, has assessed the useability and usefulness of AI chatbots for students of foreign languages. This research suggests that AI chatbots are helpful for vocabulary development, grammar and other language skills, especially when they offer corrective feedback. Related to that, they’re planning advancements like tracking of improved skills and the ability to personalise the chatbot’s tone and personality (perhaps even to practise a language while conversing with historical figures). Many people get self-conscious about making mistakes in a language they barely speak, even to a tutor, Mr Ruiz Cassarino notes.

As a measure of interference, we analyzed the proportion of looks to the competitor as a time series between 200 ms and 700 ms after the onset of the target word as our dependent variable (Fig. 2). We used generalized additive mixed-effects modelling (GAMM) in R (Porretta, Kyröläinen, van Rij, & Järvikivi, 2018; R Core Team, 2018; Wood, 2016) to model the time series data (727 trials total) (see Online Supplementary Materials for details on preprocessing and analysis). Additionally, accentuation of the target word was manipulated in the second instruction, so that the target word carried a falling accent, a rising accent, or was unaccented (see Fig. 1 and Online Supplementary Materials; the first instruction always had the same intonational contour). Information status (given/new) and accentuation (falling/rising/unaccented) of the target word in the second instruction were crossed, yielding six experimental conditions.

For this setting, we used the dataset from ref. 87, which contains 2,019 AAE tweets together with their SAE translations. In the second setting, the texts in Ta and Ts did not form pairs, so they were independent texts in AAE and SAE. For this setting, we sampled 2,000 AAE and SAE tweets from the dataset in ref. 83 and used tweets strongly aligned with African Americans for AAE and tweets strongly aligned with white people for SAE (Supplementary Information (‘Analysis of non-meaning-matched texts’), Supplementary Fig.

The delay will be experimentally induced by the presentation of sentences spoken to listeners in a foreign or a regional accent as part of a lexical decision task for words placed at the end of sentences. Using a blocked design of accents presentation, Experiment 1 shows that accent changes cause a temporary perturbation in reaction times, followed by a smaller but long-lasting delay. Experiment 2 shows that the initial perturbation is dependent on participants’ expectations about the task. Experiment 3 confirms that the subsequent long-lasting delay in word identification does not habituate after repeated exposure to the same accent. Results suggest that comprehensibility of accented speech, as measured by reaction times, does not benefit from accent exposure, contrary to intelligibility.