Google keeps advertising its translation technology as a miracle worker that functions off the shelf without any human intervention, as shown by its recent ads and demos for live translation service. Perhaps somewhat deceptively, Google itself does not believe these claims, as we were able to identify a Google-authored document that warns against relying on Google Translate. A good litmus test for if Google Translate works reliably enough to use for anyone’s business use case, is whether Google’s core business—advertising—finds that translate works well enough to rely on. The answer seems to be a resounding “NO.”
That document is the system level-instructions for the Google Ads AI assistant. To find it, we did an experiment you can easily reproduce below on another Google service, that being Google Ads. To provide a bit of context, Translate and Ads are difference services run by different people, but with insider access. From what we can tell, the Ads team is not rewarded for Translate’s performance, so if they can get ahead by throwing Google Translate under the bus, there’s a good chance they will do just that.
Background Context. Google has added its Gemini AI model into several of its other services, such as Google Ads, and functions by looking up answers in its database of pre-written responses and sending the answers to the user. If it doesn’t have reliable information, it simply says sorry. More importantly, Gemini can call the Google Translate service, or do translation itself, or answer in a foreign language based solely on its own knowledge.
The Google Ads team explicitly hard coded Gemini to avoid two things it will usually do:
- Avoid calling the Google Translate service to translate its knowledge base info
- Avoid using Gemini to translate its knowledge base information
If asked to respond in a different language (here, Chinese), the Google Ads assistant refuses to answer except in English, which is only explainable in Gemini as a result of system level instructions. Looking at the AI’s thinking log, it was running searches in its English database due to the account configuration, looked up what to do if it may need to translate, and ran into an instruction that is broadly designed to prevent the AI from using Google Translate or Gemini translation to transform the English database results to a foreign language.
There is no doubt that Google’s executives were aware that the basic functionality of the crown jewel of its product lineup doesn’t work—Sundar Pichai no doubt believes at a strategic level, Google Translate is unreliable.
The collision between a Chinese-language prompt and its instruction results in bizarre bug: Gemini tries to “prove” it understands Chinese by translating the question into English! This is important because Google Gemini is actually the highest performing AI for translation tasks: if Google’s tech doesn’t work, nobody’s tech works.
There may be some reasonable doubt that another reasonable explanation for Google’s decision exists. If not because their translation technology is unreliable, why would the Google Ads business unit go to all the trouble of aggressively blocking the Google Ads AI assistant from using Google’s own translation technology on Google’s documentation? It’s not to prevent the AI from jumping directly to answering in a foreign language without translating it, which can cause AI to become unstable.
That’s because Google could have required the assistant to go through several translation steps to validate its response. They must have certainly tried to get automated translation working fully reliably, but simply failed.
Apparently, what Google plans is to manually translate all of its documentation into supported languages. If there isn’t documentation in a supported language, Google thinks it’s a better decision to tell a customer, “No, I can’t help you, sorry” rather than use its Google Translate service. Google Ads’ AI assistant only became available very recently, so other languages likely aren’t supported.
Prognosis. Google has internal documents that admit Google Translate, and Gemini translate, simply do not work as advertised: they are unreliable and will distort facts, damaging business value.
Realities of AI and Translation
Google’s AI behavior also reflects an important consensus about AI assistants, which is they can’t be relied on to answer questions accurately, they need to read from a carefully built database of canned answers. Google’s internal documents show, it does not think its translation technology can reliably be used to answer even simple questions like “What is an impression.”
Above, we said that Google’s translation technology is still best in industry. A leaderboard in OpenRouter shows that among translation models, it beats the AI leader for every other category: Anthropic’s Claude.
Nonetheless, “best” currently has the reliability of robot plumbers. In tests several years ago for law firms handling Chinese tech company semiconductor import matters, we quantified Google Translate’s accuracy for ordinary business at 24%. That is, 75%+ of the time, it simply made words up or said the opposite of what the documents intended to say. This wasn’t based on anyone’s personal opinion, instead it was based on deep linguistic analysis. Attorney compliance interviews confirmed the same. Worse, it seems these companies suffered millions of dollars in regulatory penalties due to relying on Google Translate.
Unlike robot plumbers, who will create a highly visible mess and possibly turn your living room into a pond, these dangers are invisible unless you’re very sophisticated in data analysis, like Google, and know the dangers of using Google Translate.
Using AI Reliably
In my own experience for clients, I was able to help configure machine translation systems to handle problems like, translate 5 million words of repetitive business records at very low cost. In the business records case, 90% of the content is the same 300 words repeated over and over very formulaically, and two days of work to make sure those 300 words are configured correctly when machine translating can cut the labor requirement from 4 years to 2 days. Accuracy jumps from 25% to 98% because you told the AI how to handle the question and the client pays very little.
That company, hit with a denial order causing a multi-billion dollar plant to be shut down for two years, prevailed in the litigation using those documents, proving its narrative coherently.
Technology companies selling AI products often try to sell a false dream of “AI for Dummies”—that you can simply input a prompt like:
- “Code a business accounting system”
- “Draft an outsourced manufacturing contract”
- “Translate this to Chinese”
As Google’s own use of its technology reveals, they do not believe the sales pitch, and nor should you. In programming, we’re hearing that coding AI is being used extremely widely, but primarily by people who are already coding experts using very specialized programs that are already configured by experts to follow a series of steps to arrive at a useful solution. For translation and other professional fields, value-adding use of AI tools is likely to follow a similar pattern, with knowledgeable people being put in control of the technology.
In the broader industry, we can still see that professionals in law, accounting, and consulting are not fully relying on the technology either. For example, an English-speaking corporate lawyer in Beijing I used to work with in New York, charges $1,500/hour, and a similarly experienced and skilled local lawyer who speaks Chinese, uses flat fees and has an annual income 10% of that (i.e., $200k vs $2M).
Given that cost of living in China is so much lower than in New York, this is the purchasing power parity equivalent of a lawyer charging $5,000/hour, more than even the most expensive US lawyers charge, and in a country where most people earn just several hundred bucks a month. There are articles all over the internet by lawyers expressing shock and outrage at this reality, in part because the only thing special about these lawyers is their parents had enough cash on hand to send them abroad to learn English. They’re not exceptional lawyers by local standards at all. But they haven’t been threatened by automated translation tools.
Translation AI can be Used—Strategically
AI in translation can actually be used strategically, but not in the sense of clicking a button on Google Translate and then asking someone to please clean it up for half the cost. The result is that a translator is starting from a 20% correct text, and maybe will revise it to be 60% correct. Incidentally, this was the root cause of the controversy over how the Squid Games was translated.
Instead, as noted above, for large scale use cases, a translator expert in the domain can configure an AI translation tool to produce much more correct outputs. In less structured contexts, going from 20% to 90% accuracy with the automation itself is achievable.
In practice, this is simply not how the translation industry works. Generally, translators are not experts in any domain. Among translation master’s students in China, I counted out of every 1,000, fewer than 10 had any expertise outside of literature (i.e. engineering, law, finance). Second, even if they are, then they generally know next to nothing about basic technologies. A translator on Quora working for a tech company proudly posted publicly that she doesn’t know what a server is. I’ve had students ask what a URL is.
As a result, translation technology interactions with translators tend to involved poorly qualified translators picking between which option they like better, or editing over an AI output. Success is measured by BLEU scores against a human who doesn’t understand what they’re doing. This isn’t confined to translation, Google also pulled its healthcare AI overviews after The Guardian exposed how the company was relying on people with no healthcare experience to train AI more cheaply than qualified doctors.
We can interpret Google’s strategy as to match up the AI’s opinion on expert domains, with what laypeople will think, so that laypersons are persuaded. They understand user engagement isn’t product effectiveness. We can expect Sundar Pichai to keep using human doctors and human interpreters in the future.
Despite so much misleading advertising, the technology internals are impressive. In the future, we can expect translators using Google Gemini to do the work of 100 translators. The $1,500/hour lawyer in Beijing can be replaced by a $150/hour lawyer and translator combo. A $1,200,000 legal bill will be trimmed down to $100,000. People who couldn’t afford genuine bilingual attorneys in foreign jurisdictions and who were relying on unlicensed “legal consultants,” but getting scammed by local suppliers, will have access to highly qualified service.
So, Google’s own documents recognize that Google Translate does not work as advertised and shouldn’t be relied on, but plan on translation experts finding a revolutionary use for the technology in the near future anyway.
