Research16-11-202617 min read

How Europe Tests AI in Its Own Languages

What Blits.ai does in the LLMs4EU project, and what we learned in the first eighteen months. How Europe adapts open language models to its own languages, why testing comes first, how the distributed evaluation centre works, and the lessons from writing, running and building the tests.

Yuri Ihnatov
Yuri Ihnatov
Technical Project Manager
How Europe Tests AI in Its Own Languages

LLMs4EU is a European Union project that adapts open language models to the official languages of the EU and to real work in five sectors. Blits.ai is one of the partners. We work on the part of the project that tests whether the finished systems are good enough to use. This article explains what the project does, how the testing works, and what we learned in the first eighteen months.

A visitor at a village church wants to know its history, so she asks her phone in her own language. About two million people speak that language. The answer she gets depends on a large language model, and on whether anyone has tested that model in her language. For most of Europe's languages, nobody has done that testing yet. LLMs4EU sets out to do it.

Holy Trinity church in Hrastovlje, a village church in Slovenia

Why smaller languages need their own tests

A large language model learns from text. Its makers give it a very large amount of text, and the model learns to predict the next word in a sentence. At that scale, the model can answer questions, summarise documents and hold a conversation.

Most of that text comes from the internet, and most of the internet is in English. So the models work best in English, well in French, German and Spanish, and less well in smaller languages. Irish, Latvian, Maltese and Slovene each have far less text online, so a model has far less to learn from.

Model makers usually publish test scores for English and a few large languages. For smaller languages, there is often no published score. So a user can get a fluent answer and still not know whether it is correct. In many cases, the company that built the model does not know either.

Europe builds its models under its own rules. The makers of a European model train it only on text they have the right to use. They follow EU copyright rules, which let authors and publishers opt out of AI training, and the General Data Protection Regulation. The makers also publish a summary of where the training data came from. LLMs4EU works with organisations that hold text, recordings and records in languages with little text online, and that can share them legally. For many smaller languages, that is where the useful data is.

What LLMs4EU does

LLMs4EU started in March 2025 and runs until early 2028. The EU's Digital Europe Programme funds part of it. The partners come from across the EU. They include universities, national research institutes, supercomputing centres, telecoms companies, an energy company, public bodies and small companies.

ALT-EDIC, the Alliance for Language Technologies, coordinates the project. EU member states set up ALT-EDIC in 2024, so that Europe's languages get the data, tools and testing that the market does not provide on its own. Its offices are in the chateau at Villers-Cotterêts, north of Paris. In 1539, a French king signed an ordinance there that made French the language of law and administration.

The chateau at Villers-Cotterêts, home of ALT-EDIC and the Cité internationale de la langue française

The project takes open language models and adapts them for five areas of work: science, public services, tourism, telecommunications and energy. In each area, organisations that need a result lead the work, and partners that can build it support them. The project aims to cover all 24 official EU languages and a number of regional languages. The project plans to publish its models, data and results openly, so a small company or a public body can use them without asking permission.

Many public services now reach people through text and speech. When a municipality buys an assistant that works well in one language and badly in another, speakers of the second language get a worse service from the same public body. So the project builds open models on European data, adapts them to European tasks, and tests them in public.

LLMs4EU works alongside OpenEuroLLM, another project funded under the Digital Europe Programme. OpenEuroLLM trains new open models on European supercomputers, in all official EU languages, and publishes the weights, the code and the data. LLMs4EU adapts those models and other open models for specific tasks. It also collects domain data, checks the legal status of each dataset, and tests the results. The two projects share partners and data. So European teams train, adapt and test the same open models, and anyone can check the results.

Why testing comes first

A language model writes fluent sentences whether or not it knows the answer. When a model handles a language poorly, it can still write a clear answer that is wrong. So good writing does not show whether the content is correct.

So the project sets its test method in advance, tests against agreed standards, and publishes the results.

How the evaluation centre works

In LLMs4EU, this work belongs to the Distributed Evaluation Centre for LLMs, and Blits.ai is part of it. The centre is distributed because several organisations do the testing. Each one tests the use cases it knows best, and all of them follow one shared set of rules.

The rules come from decades of evaluation campaigns for speech recognition and machine translation. The team that builds a system does not test it. The evaluators write new test questions for each campaign. They keep the questions closed until the test ends, and then they publish them. The people who prepare the training data work apart from the people who write the test, because a system must not see the test in advance. Every system built for a use case runs on the same test material in the same period, so the evaluators can compare the results.

Over the past months, the evaluation group has worked out how the centre will run in practice. The group is working towards a network of specialist centres. Each centre tests the areas it knows, and ALT-EDIC both runs tests and coordinates the network. So an organisation that wants a model tested can ask the coordinator which centre fits, or contact a specialist centre directly. The group will also set criteria for adding new centres over time.

The group also looked at public leaderboards. When test data stays public for a long time, it can end up in the training data of the models it is meant to test. So the group prefers time limited campaigns. The evaluators share the test data only during the campaign, and they publish it when the campaign ends. Later runs on the same data still help developers, but they do not count as official results.

Running every submitted model on central servers takes a lot of maintenance, because software and libraries keep changing. One option is that participants run their own systems on the test inputs and send back the outputs for scoring. The group is working out how to do this while keeping the test data closed until the right moment.

Evaluators will use the shared platform to produce scores. They can check each result before it goes back to the developer. The platform will also hold shared resources: evaluation software, metrics, evaluation plans, guidelines and tools for data formats. A working group for the platform starts meeting in November. Several partners, Blits.ai among them, will present the evaluation tools they already use, so the working group can build on them.

Some new work in the project starts with the test. The team writes the data plan and the evaluation plan first, and then invites developers in the consortium to take part. So several systems can enter the same campaign from the start.

Testing the whole system

At the end of the project, the centre tests the systems that each use case builds. A scientist, a civil servant, a tourist, a call centre supervisor or an engineer will use these systems. So the test covers the whole system, and the language is one part of it.

The assistant the visitor uses at the church does several things. It searches a library of documents and picks the relevant pages. Sometimes it calls other software, for example to look up opening hours. Then it decides whether it found enough to answer. When it did not, the correct answer is that it does not know. Getting a model to give that answer takes deliberate work.

So the test checks several things. It checks that the system found the right page, and that the page it cites exists. It checks that any form the system filled in is correct. It checks that the system said it did not know when the library had no answer. It also checks that the system answered in the language and variety the visitor used. A system with correct grammar can still fail these checks.

Each use case writes an evaluation plan before testing starts. The plan states the metrics, the type and size of the test data, the test conditions and the schedule. The evaluation group reviews each plan, so it can test every use case to the same standard.

Blits.ai builds and runs systems of this kind for banks and large companies. Over several years, we learned what fails when a system moves from a test environment to real customers. In LLMs4EU, we jointly lead the evaluation of the tourism use case. We also put forward our own evaluation platform as one of the tools the centre can use. Our platform checks whether an answer is correct. It also checks whether the answer came back in the language and variety the user used, and whether it stayed in that variety for the whole conversation.

In July, the tourism team asked us to review the design of their assistant. We described how we build our commercial systems, and two of those ideas also apply to their assistant. Every document enters the library through one checkpoint, so nothing gets in without a check. When the system finds nothing relevant, it returns a fixed message that a person wrote in advance, and it does not call the model at all. We do this because a model that is asked to refuse can still choose to answer. So we write the message into the code, and the model never makes that choice.

What we learned from Arabic

Blits.ai learned the language check in the Gulf. A few years ago, a bank there ordered a digital human from us, an on screen avatar that talks to customers. The contract said Arabic, so we delivered Modern Standard Arabic, the formal Arabic of schools and news. It was fluent and correct. Within days, the bank told us it did not sound like its customers. Its customers speak the way people in Riyadh speak, and that is different from the news.

The Riyadh skyline with the Kingdom Centre

So we hired native speakers as testers. We reviewed every step, from the microphone to the spoken answer. We rebuilt our tests around one question: would a person from that city accept this voice as their own. After that work, the answer was yes.

We kept the testing. Our platform now checks every answer for language and dialect. It checks that the system replied in the variety the user spoke, in the register the user expected, and that it kept that variety for the whole conversation. A system can drift back to the formal language after a few turns. That drift shows up in longer conversations, so we test full conversations.

Today we run assistants in Egyptian Arabic, in several Gulf dialects including Saudi and Qatari Arabic, in Libyan Arabic and in Turkish. We test every candidate model on the same recordings, with the same background noise and the same time limits. We also ask a panel of native listeners to judge the voices, because an automatic score counts correct words and cannot tell whether a voice sounds natural to the customer.

One result from this work matters for Europe. On our own Arabic tests, a newer generation of speech models did not always do better on dialect than an older one. So we test every new model on our own data before we use it. A speaker of a smaller European language needs the same check as a customer in Riyadh.

Lessons from the first eighteen months

These lessons come from the consortium meeting in June, from the working meetings since then, and from the evaluation plans the teams have written. They apply to any team that builds or tests language systems.

Writing the test

**Native speakers write the test questions. **Machine translation can help produce training data. In a test, machine translation would measure the translation software, and that is the wrong thing to measure. So native speakers write, translate or correct every test question by hand. They also need to work independently of the team that builds the system. For smaller languages, finding such speakers takes extra effort.

**Partners translate test queries into their own languages. **The team that builds the agentic search for the project's data catalogue wrote a set of test queries in English. Then it asked partners to translate the queries into their own languages. The team now compares the results and the response time in each language, and uses them to choose the model.

**The test data includes speech that mixes languages. **In one speech use case, the broadcasts also contain English words. So the evaluation team builds test data and scoring that handle speech that switches between languages.

**The evaluation team keeps part of the test unseen. **The evaluation team holds back a set of questions that no developer sees before the run. It also adds hard cases on purpose. These include questions from outside the domain, questions with missing information, and questions that test whether the system can be misled.

**Partners make the test harder when scores get too high. **One partner found that his test set of web pages returned very high scores. So he asked the other partners for data to make the test harder, because a test that every system passes no longer shows the differences between systems.

Running the test

**The evaluation team checks the starting point first. **Before it tested adapted models, one evaluation team reran a public benchmark on standard open models. It wanted to see whether it got the published scores. For some model sizes the scores matched, and for others they differed. So the team is now checking settings such as the generation parameters, because it needs a correct starting point before it measures any improvement.

**Teams run a dry run on the data they have. **Partners often need time to share real data, because of contracts and legal checks. So the teams run the full evaluation on data they already have. Then, when the real data arrives, the teams have already tested their pipeline.

**The team freezes the test before the run. **Before a run starts, the team fixes the test package. Then it takes a snapshot of the documents the system searches. Every run gets the same inputs. Once the run starts, nobody edits the outputs by hand. The team runs the full system in a closed test environment, so it tests retrieval, citations and answers together.

**A language model scores first, and people review the hard cases. **A language model scores the answers in a first pass. Then people review the cases where the model is unsure. They also review the results that someone disputes, and the answers that someone flagged. When an answer confidently invents a fact, the answer counts as a failure.

**Teams use different measures at different stages. **Early in development, the team looks at overall precision and recall, to see whether a model is on track. Later, it looks at a breakdown by error type, to see what each small change does. Before a system goes into production for a sensitive task, the team uses a much stricter measure. For removing personal data from customer conversations, a partner developed a measure that heavily penalises a single missed item, because one missed name can reveal who a person is. The same measure also penalises a system that removes too much, because then the text is no longer useful. All of these measures come from the same test set, so developers can choose which one to look at.

**Teams report results per language. **Each team reports its scores per task and per language. So a reader can see how the system performs in each language separately.

**Teams measure speed as well as quality. **Several teams measure how fast a system answers, in tokens per second and in time to the first token. The tourism team also runs a stress test with thousands of synthetic queries. Then it checks the logs, to confirm that the system records resource use correctly for each user.

**Evaluation teams share their test tools. **One evaluation team built a simple interface for its tools. You pick a task and load a dataset, and then you see the results per metric and per sample. The team plans to share the tool with partners, so they can rerun the tests and check the results themselves.

Building the system and its data

**Small refinements can remove content without a warning. **A partner compared tools that turn web pages into clean text. The plain tool with its default settings gave the best result. Each refinement he added removed the tables from the pages, and no tool warned him. He found the problem when he read the outputs side by side.

**Scanning software can drop letters. **Much heritage material exists only as scanned books and old PDF files. Text recognition software often drops accents and special letters, such as č, š and ž. A model cannot restore a letter that the scanner lost, so the project fixes the scanning step itself.

**The training data teaches the system to say it does not know. **The tourism team adds some questions with no answer in the library to the training conversations. So the model learns that it may say it does not know.

**The tourism team uses synthetic data for training and human data for checking. **The team generated a large set of synthetic questions and answers for training. Then it wrote a smaller set by hand for checking. Now linguists review the answers by hand, with a language model as an assistant judge. The team is also building an automated test that follows conversation scenarios step by step.

**Each item in a collection has its own rights. **A large European heritage collection publishes its catalogue records under an open licence. The texts and images that those records describe carry their own rights, and the project cannot use many of them for training. So the project checks the rights item by item. The project's lawyers also noted that anonymised data is not always anonymous in law, because someone can sometimes identify a person by combining it with other data.

**The law can limit what evaluators may see. **In one public service, the public body publishes documents for a short period. After that period, strict legal limits apply. The system owner may keep using the documents, and independent evaluators may not. So the evaluation team agrees with the owner which parts of the system it can test on its own, and the owner tests the rest.

**The project labels its own datasets. **The project built a dashboard that reads the main public catalogues of language data. In many entries, the size field was empty or used different units, such as kilobytes, hours, tokens or words. The domain field was usually free text, or empty. So the project labels its own datasets with size, units and domain.

What comes next

The tourism team has collected its training and test data. It is now fine tuning models together with a retrieval system, and linguists are reviewing the first answers. Other use cases are finishing their evaluation plans and preparing their data. The teams run the first evaluation rounds over the coming months. Larger campaigns follow in 2027. The project ends in early 2028.

For Blits.ai, this is the work we already do, applied across Europe. We build and test conversational systems in languages and dialects where the difference between supported and usable decides whether a product works. We bring the tools we built for that to the evaluation centre. In return, we help shape the rules that European public bodies will use to decide which systems to trust.

When the visitor at the church asks her question, the system she uses will have been tested in her language, and the project will publish the results.

LLMs4EU is funded in part by the European Union under the Digital Europe Programme, Grant Agreement 101198470. Blits.ai is a consortium partner. Views and opinions expressed are those of the author only and do not necessarily reflect those of the European Union. Neither the European Union nor the granting authority can be held responsible for them.

Photos via Wikimedia Commons: Holy Trinity church in Hrastovlje, Slovenia, by Miha Peče (CC BY-SA 4.0, cropped); Cité internationale de la langue française by Renhour48 (CC0); Riyadh skyline by B.alotaby (CC BY-SA 4.0).

Yuri Ihnatov
Yuri Ihnatov
Technical Project Manager
Writes about Prompt Engineering, RAG Systems, Conversational AI, AI Evaluation
Yuri Ihnatov on LinkedIn
Published on 16-11-2026

Related Articles

Benchmarking Conversational AI: Whitepaper
Research25-08-2022

Benchmarking Conversational AI: Whitepaper

Our research into conversational AI performance shows clear gains in chatbot effectiveness across several industries and use cases.

Read More →
Most Enterprise AI Never Leaves the Pilot. Ours Goes Live.
AI Technology13-11-2026

Most Enterprise AI Never Leaves the Pilot. Ours Goes Live.

The line on our home page is a big claim, so here is what sits behind it. What the research says about enterprise AI pilots, the ten ways we see pilots die, and what we do differently so that a use case reaches production and stays there.

Read More →
AI Agency or AI Platform? How Enterprises Should Choose
AI Technology10-11-2026

AI Agency or AI Platform? How Enterprises Should Choose

Hire an AI agency, buy a SaaS tool, build it yourself, or work with a platform? Every enterprise faces that choice, and the data is starting to show which path reaches production. Why custom built AI so often gets abandoned, and the questions that tell you whether you're buying a project or a capability.

Read More →

Stay Updated

New posts on conversational AI, enterprise automation and customer experience, sent to your inbox when they go live

No spam, unsubscribe at any time