I tried the Sogou grammar tool and it was a disaster

“Shall I compare thee to a summer’s day? You are cuter and more restrained.”

Shakespeare didn’t write that—maybe because he didn’t have help from an artificial intelligence-powered proofreading tool made by Chinese search engine Sogou.

The company recently rolled out the Sogou AI Writing Assistant, which it says uses “optimized AI algorithms” for its grammar- and spell-checking features. AI also powers its suggested replacements for terms it identifies as “Chinglish,” an English writing or speaking style influenced by the Chinese language and criticized in China for not being “native.”

Sogou said in a statement that the tool is accurate 96.83% of the time. When I saw that, I felt concerned about my editor’s job security.

Its performance reflects how Chinese tech companies treat artificial intelligence as the technological panacea. It is promoted as a “secret ingredient,” over-hyped for its ability to solve problems beyond its capabilities.

I tested the tool using an article I wrote last week about Tiktok CEO Kevin Mayer’s resignation from Bytedance (which had already been edited), and it turned out that Sogou’s AI is so smart that it corrected Bytedance, the company Mayer worked for, to “Bidens.” (Which Bidens? Which Biden?)

The AI algorithms identified the word “Bytedance” as a typo so it suggested an alternative which is a type of daisy, also known as “beggar-ticks.” It also happens to be the surname of a pretty well-known American. 

Let’s try another TechNode story about Chinese electric vehicle maker Xpeng’s US initial public offering (IPO) last week.

This time perhaps the tool didn’t find the term “Xpeng” in its corpus, which the company says contains more than 100,000 English sentences (equal to about 31 books) and upwards of 10,000 pieces of “high-quality human correction data,” so it replaced it with “Peng,” a common Chinese last name. 

What a smartypants.

The tool also changed the phrase “backed by,” which originally expressed that Xpeng had received funds from Alibaba and Xiaomi, to “with the support of,” creating the implication that Alibaba and Xiaomi may have helped it with the IPO.

I then input Shakespeare’s Sonnet 18 into the assistant. 

A Sogou spokesperson declined to comment on my findings but said that the accuracy mentioned in the press release was calculated based on “real-world data from users.”

When the product first launched, local media compared it (in Chinese) to Grammarly—the popular grammar checker software run by a company based in San Francisco, Calif.

To compare, I checked some TechNode sentences with Grammarly. And, to be fair, it also gave me some unreliable suggestions. You still have to review all of its suggestions yourself and make decisions on whether to use Grammarly’s corrections. But overall, it didn’t try to dig into the meaning of sentences, an area that current AI technology is really bad at. 

Here is how Grammarly corrects Sonnet 18.

Grammarly seems to have an issue with the old-fashioned expression of “ow’st,” but other than that it gives Shakespeare a score of 98 out of 100. Well done, William.

According to a Grammarly blog post, it also uses AI for its grammar corrections. Specifically, it uses machine learning, which enables systems to learn and perform tasks by feeding it with plenty of data, and a variety of natural language processing (NLP) approaches, which teach machines to understand and process human language.

Software engineer and tech analyst Ben Dickson wrote in an article published in 2019 that Grammarly’s success was “largely due to its focus on a narrow application of AI NLP: grammar assistance,” because it knows “the limits of current AI technologies.“

According to Dickson, currently, the deep-learning and neural network technologies are terrible at solving “problems that require common sense.”

“When dealing with the meaning of sentences and the semantic relations of words, deep learning algorithms fail miserably,” he wrote.

Accordingly, Sogou’s AI writing assistant fails here. We can see there is a pattern with the tool where it attempts to interpret the author’s meaning and rewrite the sentence using its own logic—which sounds like something that AI could do but is actually just not good at. 

As a result, China has AI teachers, AI doctors, and AI poets—one of which wrote this verse (which disgusted an actual Chinese poet): 

“The rain is blowing through the sea/ A bird in the sky/ A night of light and calm/ Sunlight/ Now in the sky/ Cool heart/ The savage north wind/ When I found a new world.“

I think my editor is safe for now.

Update: This article was edited on Nov. 18 to restore a paragraph removed owing to good, old-fashioned human error. Touché, Sogou AI Writing Assistant.