albanisht

Benchmark

How we test ourselves

We test the translation on 98 hard cases of Albanian (86 Albanian into English, 12 English into Albanian): the admirative, blessings, Gheg, clitics, kinship terms, false friends and more. Next to our results we show what Google Translate answered on the same cases. DeepL isn't measured yet.

albanisht

98

of 98. 100% pass.

Google Translate

83

of 98. 85% pass.

How these were graded. An AI judge (claude-opus-5-5) read every translation against reference answers and scored meaning, then the same fixed rule decided pass or fail for both systems: meaning ≥ 2 of 3 and no major or critical error. Listed answers are examples of acceptable translations, not the only correct wording.

albanisht: commit cab46c2, model claude-opus-5-5, published 13 October 2026. Google Translate: captured by hand from the web on 8 May 2026.

Limits. The judge is a Claude model grading Claude output; self-preference bias is possible. A second judge from another model family, OpenAI's gpt-5.5, graded the same answers with the same rules: albanisht 98 of 98, Google Translate 83 of 98. The two judges agreed on 192 of 196 answers. A native speaker then checked 50 of these answers without knowing which system wrote them or what the judges decided: they agreed with the Claude judge on 42 and with the second judge on 46. Of the answers they checked, albanisht passed 22 of 23 and Google Translate 21 of 27. Every output and every reason for a fail is listed below.

This set is now too easy for us. A full score means these cases can no longer tell one version of albanisht from a better one, so we measure changes on harder held-out sets and will publish those as well.

Results by category

Every category, in the benchmark's order, including where Google does better.

Category albanisht Google Compared with Google
Admirative 5/5 (100%) 4/5 (80%) +20%
Optative 6/6 (100%) 4/6 (67%) +33%
Clitics 4/4 (100%) 2/4 (50%) +50%
Gheg 3/3 (100%) 3/3 (100%) 0%
Kinship terms 3/3 (100%) 3/3 (100%) 0%
False friends 4/4 (100%) 2/4 (50%) +50%
Religious register 4/4 (100%) 4/4 (100%) 0%
Ti and ju 4/4 (100%) 3/4 (75%) +25%
Linking articles 3/3 (100%) 2/3 (67%) +33%
Definite forms 2/2 (100%) 2/2 (100%) 0%
General 10/10 (100%) 10/10 (100%) 0%
Aspect 7/7 (100%) 6/7 (86%) +14%
Aorist and perfect 6/6 (100%) 4/6 (67%) +33%
The particle “po” 6/6 (100%) 6/6 (100%) 0%
Mediopassive 7/7 (100%) 7/7 (100%) 0%
Gheg infinitive 7/7 (100%) 4/7 (57%) +43%
Vocative 6/6 (100%) 6/6 (100%) 0%
Numerals 5/5 (100%) 5/5 (100%) 0%
Place names 6/6 (100%) 6/6 (100%) 0%
How we score an answer

Each translation is graded by an AI judge for meaning (0–3), fluency and errors, using the test's listed answers as examples of acceptable wording, not the only correct one. Pass means meaning ≥ 2 of 3 and no major or critical error; the rule is applied in code, the same for both systems. Earlier versions of this page matched exact wording, which failed correct translations that used other words. More on the About page.

See every test case
Category Input albanisht Google Translate
Admirative “Paska ardhur babai!”

✓So Dad has arrived!

✓Dad has arrived!
Admirative “Qenka mësues i mirë.”

✓So he's a good teacher!

✓He/She turns out to be a good teacher.
Admirative “Sa e bukur qenka kjo lule!”

✓What a beautiful flower this is!

✓How beautiful this flower is!
Admirative “Paska shi jashtë.”

✓Oh, it's raining outside!

✗It is raining outside.

  • The source uses the admirative mood ('Paska'), which marks surprise or discovery: the speaker has just realised it is raining. The candidate is a neutral statement and drops this nuance. A marker such as 'Oh,' 'Apparently' or 'It seems' would keep it.
Admirative “Ai paska blerë makinë të re.”

✓Apparently he's bought a new car!

✓He apparently bought a new car.
Optative “Të lumtë dora!”

✓Well done! (literally: "Bless your hand!")

✓Well done!
Optative “U bëfsh 100 vjeç!”

✓May you live to be 100!

✓May you live to be 100 years old!
Optative “Mos u lodhsh!”

✓Thank you for your hard work!

✗Don't get tired!

  • 'Mos u lodhsh!' is an optative workplace courtesy, roughly 'Thanks for the hard work' or 'Take it easy'. The candidate renders it word for word as a literal command not to become tired. This loses its function as a greeting and closely matches the known incorrect 'Don't be tired' and 'May you not get tired'.
Optative “Paç fat në provim!”

✓Good luck on your exam!

✓Good luck on the exam!
Optative “Rrofsh, vëlla!”

✓Thanks, brother!

✗Cheers, brother!

  • 'Rrofsh' expresses warm thanks or praise (literally 'may you live long'). 'Cheers' reads primarily as a drinking toast, so it misrepresents the expression. The gold set was explicitly corrected to remove this rendering.
Optative “Ardhsh i mirëpritur në shtëpinë tonë.”

✓Welcome to our home.

✓You are welcome in our home.
Clitics “I gave the book to Ana.”

✓Ia dhashë librin Anës.

✗I dhashë librin Anës.

  • The definite direct object 'librin' needs the accusative clitic 'e'. Combined with the dative clitic 'i' (for Anës), the two must fuse to 'ia': 'Ia dhashë librin Anës.' The candidate uses only the dative 'i'.
Clitics “He gave it to them.”

✓Ua dha.

✓Ai ua dha atyre.
Clitics “I told my mother the story.”

✓Ia tregova historinë nënës sime.

✗I tregova nënës time historinë.

  • The possessive must agree with the dative feminine definite noun: standard is 'nënës sime', not 'nënës time'.
Clitics “We sent them the documents yesterday.”

✓Ua dërguam dokumentet dje.

✓Ne u dërguam atyre dokumentet dje.
Gheg “Du me shku me bo punë.”

✓I want to go and get some work done.

✓I want to go to work.
Gheg “Asht qike e mirë.”

✓She's a good girl.

✓She is a good girl.
Gheg “Kena me shkue në Prishtinë.”

✓We're going to go to Pristina.

✓We are going to Prishtina.
Kinship terms “Dje takova xhaxhain tim.”

✓Yesterday I met my paternal uncle (my father's brother).

✓Yesterday I met my uncle.
Kinship terms “Tezja ime jeton në Itali.”

✓My (maternal) aunt lives in Italy.

✓My aunt lives in Italy.
Kinship terms “My uncle is coming for dinner.”

✓Xhaxhai im po vjen për darkë.

✓Xhaxhai im po vjen për darkë.
False friends “Aktualisht jam në shtëpi.”

✓I'm currently at home.

✓I am currently at home.
False friends “Eventualisht do të vijë.”

✓He may possibly come.

✗Eventually, he/she will come.

  • Albanian 'eventualisht' is a false friend meaning 'possibly/perhaps', not 'eventually'. The candidate turns a statement of possibility into one of certainty that he or she will come at a later time.
False friends “Është një person shumë simpatik.”

✓He's a very likeable person.

✓He/She is a very nice person.
False friends “Filmi ishte shumë patetik.”

✓The film was very moving.

✗The movie was very pathetic

  • False friend: Albanian 'patetik' means moving/emotional, whereas English 'pathetic' means pitiful or contemptibly inadequate, which reverses the evaluative sense.
Religious register “Inshallah do të shihemi nesër.”

✓God willing, we'll see each other tomorrow.

✓God willing, we will see each other tomorrow.
Religious register “Mashallah, sa fëmijë të bukur ke!”

✓Mashallah, what beautiful children you have!

✓Mashallah, what beautiful children you have!
Religious register “Selam alejkum, vëllezër.”

✓As-salamu alaykum, brothers.

✓Peace be upon you, brothers.
Religious register “Zoti të bekoftë!”

✓God bless you!

✓God bless you!
Ti and ju “Si je sot?”

✓How are you today?

✗How am I today?

  • 'Je' is the second-person singular (informal ti) form of 'jam', so the source asks about the listener ('How are you?'). The candidate wrongly uses the first person and asks about the speaker, so the question says something else.
Ti and ju “Si jeni sot?”

✓How are you today?

✓How are you today?
Ti and ju “Ju lutem, mund të më ndihmoni?”

✓Excuse me, could you help me, please?

✓Please, can you help me?
Ti and ju “Më fal, mund të më thuash sa është ora?”

✓Excuse me, could you tell me what time it is?

✓Excuse me, can you tell me what time it is?
Linking articles “the good boy”

✓djali i mirë

✓djali i mirë
Linking articles “to the good girl”

✓vajzës së mirë

✗tek vajza e mirë

  • The source means a dative recipient ('to the good girl'), which requires the dative case with the oblique linking article: 'vajzës së mirë'. 'Tek' with the nominative expresses direction or location ('to/at the girl's place'), so the case does not fit the noun's role.
  • The directional preposition shifts the meaning from a recipient (indirect object) to movement towards, or presence at, the girl.
Linking articles “djemtë e zgjuar”

✓the smart boys

✓the smart boys
Definite forms “the book is on the table”

✓Libri është mbi tryezë.

✓libri është mbi tavolinë
Definite forms “the girl is reading”

✓Vajza po lexon.

✓vajza po lexon
General “Mirupafshim, shoku im!”

✓Goodbye, my friend!

✓Goodbye, my friend!
General “Shumë faleminderit për ndihmën.”

✓Thank you very much for your help.

✓Thank you very much for the help
General “Nesër kam një takim të rëndësishëm.”

✓I have an important meeting tomorrow.

✓Tomorrow I have an important meeting.
General “Ku është banka më e afërt?”

✓Where is the nearest bank?

✓Where is the nearest bank?
General “I'd like a coffee, please.”

✓Do të doja një kafe, ju lutem.

✓Do të doja një kafe, ju lutem.
General “How much does this cost?”

✓Sa kushton kjo?

✓Sa kushton kjo?
General “I don't speak Albanian very well.”

✓Nuk e flas shumë mirë shqipen.

✓Unë nuk flas shumë mirë shqip.
General “A mund të më ndihmoni? Kam humbur.”

✓Could you help me? I'm lost.

✓Can you help me? I am lost.
General “Si quheni?”

✓What is your name?

✓What is your name?
General “Më pëlqen kjo këngë.”

✓I like this song.

✓I like this song.
Aspect “Po lexoja një libër kur ai/ajo erdhi.”

✓I was reading a book when he/she came.

✓I was reading a book when he/she arrived.
Aspect “Kam jetuar në Tiranë për pesë vjet.”

✓I have lived in Tirana for five years.

✗I lived in Tirana for five years.

  • The Albanian present perfect 'kam jetuar' with 'për pesë vjet' has a current-relevance reading: the speaker still lives in Tirana. The English simple past implies the period has ended, so the aspect is wrong. It should be 'I have lived'.
Aspect “Çdo verë shkonim në det.”

✓Every summer we used to go to the seaside.

✓Every summer we went to the sea.
Aspect “Dje bleva një makinë.”

✓I bought a car yesterday.

✓I bought a car yesterday.
Aspect “E kam parë atë film, por nuk e mbaj mend mirë.”

✓I've seen that film, but I don't remember it well.

✓I've seen that movie, but I don't remember it well.
Aspect “Ai flinte ndërsa unë po gatuaja.”

✓He was sleeping while I was cooking.

✓He slept while I was cooking.
Aspect “Ata kishin shkuar para se të vinim ne.”

✓They had left before we arrived.

✓They had left before we arrived.
Mediopassive “Unë lahem çdo ditë.”

✓I wash every day.

✓I bathe every day.
Mediopassive “Macja lahet nga djali.”

✓The cat is washed by the boy.

✓The cat is bathed by the boy.
Mediopassive “Takohemi nesër në kafene.”

✓We'll meet tomorrow at the café.

✓See you tomorrow at the cafe.
Mediopassive “Unë lahem me ujë të ftohtë.”

✓I wash with cold water.

✓I wash with cold water.
Mediopassive “U mësuam me të.”

✓We got used to it.

✓We got used to it.
Mediopassive “Dera u hap.”

✓The door opened.

✓The door opened.
Mediopassive “Ai pa veten në pasqyrë.”

✓He saw himself in the mirror.

✓He saw himself in the mirror.
The particle “po” “Po vij tani.”

✓I'm coming now.

✓I'm coming now.
The particle “po” “Po, e di.”

✓Yes, I know.

✓Yes, I know.
The particle “po” “Po ai/ajo sapo ka ardhur.”

✓Yes, he/she has just arrived.

✓Yes, he/she has just arrived.
The particle “po” “Po edhe unë po vij.”

✓I'm coming too.

✓Yes, I'm coming too.
The particle “po” “Po.”

✓Yes.

✓Yes
The particle “po” “Po flas, mos më ndërpre.”

✓I'm talking, don't interrupt me.

✓I'm talking, don't interrupt me.
Gheg infinitive “Du me ardhë nesër te ti.”

✓I want to come to your place tomorrow.

✓I want to come to your place tomorrow.
Gheg infinitive “Me ba mirë asht mirë.”

✓To do good is good.

✗Do me a favor, it's good.

  • The Gheg infinitive 'me ba' (to do) is misread as the pronoun 'me' with an imperative, giving 'Do me a favor'. The source is a gnomic statement, 'Doing good is good', with the infinitive as subject. The candidate says something entirely different.
  • The comma splice joining an imperative to 'it's good' is incoherent and unnatural in English.
Gheg infinitive “Kena me ba shumë punë sot.”

✓We're going to do a lot of work today.

✗We had a lot of work to do today.

  • Gheg 'kena me ba' is a present obligation or future construction ('we have to do / are going to do'). The candidate turns it into a past-tense statement, which changes the time reference.
Gheg infinitive “Du me shku në treg.”

✓I want to go to the market.

✓I want to go to the market.
Gheg infinitive “Kam me marrë diçka nga dyqani.”

✓I'm going to pick something up from the store.

✓I have to get something from the store.
Gheg infinitive “Po mundohem me kuptu se ça po thua.”

✓I'm trying to understand what you're saying.

✓I'm trying to understand what you're saying.
Gheg infinitive “Tu fol me të, mësova diçka.”

✓While talking with him, I learned something.

✗Tu fol me të, mësova diçka.

  • The candidate copies the Albanian source unchanged instead of translating it into English. The Gheg gerund construction 'tu fol' ('while talking') and the rest of the sentence are not rendered, so the output is unusable as an English translation.
Vocative “Motër, si je?”

✓Sister, how are you?

✓Sister, how are you?
Vocative “O vëlla, eja këtu.”

✓Brother, come here.

✓Brother, come here.
Vocative “Moj nënë, mos u shqetëso.”

✓Mother, don't worry.

✓My mother, don't worry.
Vocative “More djalë, ç'po bën aty?”

✓Hey, boy, what are you doing there?

✓Hey boy, what are you doing there?
Vocative “O vëlla, ç'po thua?”

✓Brother, what are you talking about?

✓Brother, what are you saying?
Vocative “Fëmijë, ulni zërin.”

✓Children, keep your voices down.

✓Children, lower your voice.
Aorist and perfect “Qeshë dje atje.”

✓I was there yesterday.

✗I laughed there yesterday.

  • 'Qeshë' is the aorist (simple past) of 'jam' (to be), meaning 'I was'. The candidate confused it with 'qesha' (aorist of 'qesh', to laugh), so the sentence says something entirely different.
Aorist and perfect “Kam qenë në atë restorant.”

✓I have been to that restaurant.

✓I've been to that restaurant.
Aorist and perfect “Vajti, foli me të, dhe u kthye.”

✓He went, spoke with him, and came back.

✓He went, talked to him, and returned.
Aorist and perfect “Asnjëherë nuk kam pasur këtë problem.”

✓I have never had this problem.

✓I've never had this problem.
Aorist and perfect “Kishte mbaruar punën para se të vinim.”

✓He had finished the work before we came.

✗The work had finished before we arrived.

  • The source is active and transitive, with 'punën' in the accusative: he or she had finished the work. The candidate makes 'the work' the subject of an intransitive verb. This drops the agent and changes the voice, so the sentence can read as 'the work (shift) had ended' rather than 'someone had completed the work'.
Aorist and perfect “Ai mbaroi shkollën në vitin 2010.”

✓He finished school in 2010.

✓He graduated from school in 2010.
Place names “Shkoj në Tiranë çdo javë.”

✓I go to Tirana every week.

✓I go to Tirana every week.
Place names “Tirana është kryeqyteti i Shqipërisë.”

✓Tirana is the capital of Albania.

✓Tirana is the capital of Albania.
Place names “Prishtina ka shumë studentë.”

✓Pristina has a lot of students.

✓Pristina has many students.
Place names “Kur shkova në Tiranë, Tirana ishte e mbushur me njerëz.”

✓When I went to Tirana, Tirana was packed with people.

✓When I went to Tirana, Tirana was full of people.
Place names “Korça është një qytet i bukur.”

✓Korçë is a beautiful city.

✓Korça is a beautiful city.
Place names “Bukuria e Shkodrës.”

✓The beauty of Shkodër.

✓The beauty of Shkodra.
Numerals “Bleva dy libra.”

✓I bought two books.

✓I bought two books.
Numerals “Dy librat e mi janë mbi tavolinë.”

✓My two books are on the table.

✓My two books are on the table.
Numerals “Të dy djemtë janë në shkollë.”

✓Both boys are at school.

✓Both boys are at school.
Numerals “Të dyja motrat punojnë në spital.”

✓Both sisters work at the hospital.

✓Both sisters work at the hospital.
Numerals “Pesë studentët e parë morën çmim.”

✓The first five students received prizes.

✓The top five students received awards.