We tested Pangram, the controversial AI detector: can it really be trusted?

Since September 21, 2026, the Pangram affair has rocked the French literary season. That day, Canadian-Haitian writer Thélyson Orélien received the Fnac prize for the best novel in France for “That Was It or Die.” In the wake of the award, an anonymous account named “Balance ton Claude” publicly accused him of writing his book with the help of an AI.The prose of Thélyson Orélien does indeed contain an enormous number of disturbing elements, which tend to recall the way generative AI, such as ChatGPT or Claude, constructs its sentences.

Have you heard about this novel? It is the “literary autumn phenomenon.” It is the winner of the Fnac Prize, the Méduse Prize, and the Première Plume Prize, is a strong contender for the Prix Renaudot, and is shortlisted for the Goncourt, Femina, Médicis, and Décembre prizes. Oh yes,… pic.twitter.com/9evcXrPyFD

— Show us your Claude (@Pangramed) September 21, 2026

To substantiate its accusations, the X account relies on Pangram, software designed to detect AI-generated texts with nearly 100% reliability. The tool’s verdict is unequivocal. The novel was supposedly written by a machine, not by a human mind. The author firmly denies the accusations leveled against him.

We submitted four pieces written by Thélyson Orélien in 2014 to Pangram (the only ones we managed to find online; references below). For each, Pangram displays an AI score of… 0%.

Pangram’s false-positive rate, which is well documented, is extremely… pic.twitter.com/BSbhXAMVqx

— Show us your Claude (@Pangramed) September 22, 2026

On social media, the controversy quickly gained momentum and was even accompanied by new allegations of plagiarism.Amid the uproar, the Montreal publishing house, which published the novel in Canada, chose to suspend “the promotional activities surrounding the book”. The publisher says it is unable “at this stage to refute or confirm the allegations the author is facing”. Researchers and artificial intelligence experts nevertheless urge caution and stress that detectors are not infallible.

“Unfortunately, these systems cannot be relied upon 100% because there are false positives and false negatives”, explains Dominic Martin, a UQAM professor specializing in AI ethics, in Le Devoir.

The case revives the debate on the reliability of detection software for artificial intelligence. Increasingly used by publishing houses, academic journals and educational institutions, these tools cannot claim to uncover every trace of AI in a text.

How does Pangram work?

Founded two years ago by Max Spero and Bradley Emi, two former Google and Tesla employees, Pangram is a young New York-based startup. The tool’s name refers to the “pangram,” that is, a sentence that uses every letter of the alphabet.

As the startup explains, the tool was trained on tens of millions of texts. Some were written by humans, while others were generated by AIs such as ChatGPT or Claude, and they often concern the same topics.By comparing these two mountains of data, the software learns to identify the similarities characteristic of AI-generated writing, whether in terms of vocabulary, sentence structure, punctuation, or certain words and phrases.

“Let us take a concrete example: I have a 500-word essay on Moby Dick, written by a fifth-year student. I then ask Claude, ‘Write me, too, a 500-word essay on the same subject, in the same style.’ This gives us two texts: one written by a human, the other generated by an AI. We then train our large machine-learning model to compare these two documents, so that it learns to distinguish the one written by a human from the one produced by the AI,”, explains Max Spero to The Atlantic.

If you are accustomed to conversing with a generative AI, you have inevitably noticed that AI produces texts riddled with tics. This is the case, for example, of the famous em dash, a punctuation mark used extensively by AI. We can also cite the excessive use of three-part rhythms, formulations contrasting two elements, or forced metaphors.

“When you write a text, every word and every sentence is a decision that makes you follow a precise path along an immense decision tree. The longer the text becomes, the wider this tree grows.Large language models tend to repeat the same choices”, explains Max Spero.

Pangram is based on a “mixture of experts” model. The tool relies in fact on several specialized sub-networks that work together to analyze content. To distinguish AI content from human content, it analyzes a text passage by passage. It then summarizes its verdict in the form of a percentage accompanied by a confidence score.

False positives and false negatives

The company claims a false-positive rate that is extremely low. For English-language texts, the rate does not exceed 0.0041%, a figure measured over more than one million human-written texts. For French-language writing, the false-positive rate is 0.0026%. The risk of human text being labeled as AI is extremely low, according to the figures put forward by Pangram.

However, the false-negative rate exceeds 1.75% for French. AI text is therefore more likely to pass as human content than the reverse.The company explains that it designed its tool to minimize these errors so that it can be used “in real-world contexts” such as education or publishing.

Also read: Anthropic details the labeling of texts generated by Claude and promises a detector accessible to everyone

No, Pangram is not infallible

Like several other media outlets, we put Pangram to the test with a mountain of text.Our tests, conducted with the Pangram 4 model, seriously temper the promises of the American startup. We began by feeding the AI texts written by humans, either by us or by others. For safety, we chose texts predating the advent of artificial intelligence, such as novels, editorials, newspapers, or legal texts. When faced with human writing, Pangram hits the mark. The tool correctly identified every time that a human was behind the text. We could not catch the tool out.

Next, we gave Pangram a large number of 100% AI-generated texts, produced by chatbots such as Claude, Grok, ChatGPT, or Gemini. Here too, the tool proved particularly effective at detecting all AI-generated content. Pangram often assigned a score of 100% to AI content.

Finally, we gave Pangram hybrid texts, meaning texts written by AI but heavily edited by humans. This is where the detector made the most mistakes. Edits, sometimes limited to connectors or a few keywords, were enough to fool Pangram. By reworking some texts generated by Claude and ChatGPT, we managed to stay under the radar and obtain a 100% human rating.

To fool the detector, it is also enough to give the generative AI tasked with writing the text sourced data. This is what our colleagues at Journal du Net discovered by putting Pangram to the test.If you ask Claude or ChatGPT to draft a text based on an army of sources, the output will often be capable of fooling Pangram. According to our tests, the detector generally falls for it. By combining this trick with edits, even token ones, made by a human, the text consistently slips past the detector’s scrutiny. Pangram has thus been fooled on several occasions.

An independent study from the Vrije Universiteit Brussel confirms Pangram’s limitations. On “hybrid” scientific articles, which combine passages written by humans with AI-generated text, Pangram tends to underestimate or overestimate the use of artificial intelligence. Once a text mixes machine-written sections with text that came from a human brain, Pangram is therefore likely to get its wires crossed.

Pangram is not a “final arbiter”

Max Spero, the head of Pangram, readily acknowledges that the AI detector is not an infallible tool whose results cannot be challenged. For Max Spero, his tool “should never be the final arbiter”, but only the starting point for a more thorough investigation. The detector should rather serve as a “starting point for initiating the discussion on the use of AI”.

In an interview with The Atlantic in May 2026, the tool’s co-founder adds that the tool must regularly adapt to the latest developments in AI.In practice, the detector constantly tries to make up for the gap it has fallen behind on the latest AI models. The most recent models, such as Claude Opus 5.5 or GPT-6 Sol, may theoretically be better equipped to fool Pangram.

“We retrain our model from scratch every three to six weeks. This pace is dictated by how quickly AI is evolving,” says the head of Pangram.

Nevertheless, Pangram remains one of the best-performing detectors on the market, well ahead of competitors such as GPTZero or Copyleaks. A test conducted in July 2026 by Epoch AI confirms this. Researchers submitted 500 excerpts of human-written texts (fiction, scientific articles, blog posts) to three different detectors. Of the 495 texts analyzed, only Originality.ai was wrong, falsely accusing 19 of them of having been generated by AI. Pangram, for its part, made virtually no mistakes. Nevertheless, no detector can currently distinguish between a 100% AI text, a edited text combining human and machine input, and a 100% human text. However effective a tool may be, it can be fooled by edits, rephrasing, carefully calibrated prompts, or newly released AI models.

👉🏻 Follow the latest tech news in real time: add 01net to your Google feed and subscribe to our WhatsApp channel.