Understanding the real limits of current AI technology

Junko Yoshida

February 18, 2019

Junko YoshidaFebruary 18, 2019

Go ahead. Ask Siri on your iPhone: “Tell me good Italian restaurants nearby.” Instantly, you get a list. Great.

So, ask again. This time, “How about bad Italian restaurants nearby?” A list pops up, but most of the bad restaurants are also good restaurants on the previous list.

No one with a minimal grasp of technology should be surprised. Siri — on your so-called smartphone — doesn’t know good from bad.


In fact, modern AI (i.e. Siri, IBM’s Watson, etc.) is not capable of “reading” (a sentence, situation, an expression) or “understanding” same. AI, however, is great at pretending as though it understands what someone just asked, by doing a lot of “searching” and “optimizing.”

This is one point made emphatically by Noriko Arai, a mathematician at the National Institute of Informatics in Japan, in a recent book entitled “AI vs. Children Who Can’t Read Textbooks.”

You might recall her name from a 2017 TED talk. Arai gained her notoriety by launching the “Can a Robot Get into the University of Tokyo?” project in 2011. I’d heard of the project but paid little attention until I read her book.

Noriko Arai

Arai explains that passing the exam to enter Todai (short for the University of Tokyo in Japanese) turns to be very ambitious challenge for a machine. Unlike Jeopardy! — always framed as factoid questions — the entrance exams for Todai consists of seven sections, including math, English, science, and a 600-word essay. Each section poses distinct challenges for researchers translating natural language into machine-readable formula.

In the end, after several years of R&D by roughly 100 people — including PhD students and corporate researchers — Arai’s Todai Robot flunked the Todai exam. However, the robot did better than 80% of the students who took the test.

Definitely junior-college material.

There are many ways to interpret the Todai Robot Project, and many implications. However, here’s where I find the genius in Arai’s project:

    1. Arai framed her big AI research project in a simple, easy-to-understand catch phrase that got everyone’s attention: “Can AI Ace the Entrance Exam of a Top University?” Who wouldn’t want to know?

    2. Arai assessed where Japan went wrong with its now infamous “Fifth Generation Computer System.”

    3. Succinctly lays out the basic limitations AI faces today.

    4. She compares a Todai Robot, that can search and optimize but can’t really understand the meaning of questions, to Japanese kids who similarly cram information into their heads before taking tests, but on a very superficial level, not understanding questions they said they’d read.

Fifth Generation Computer Project
You might have heard of Japan’s Fifth Generation Computer System, a massive government-industry collaboration launched in 1982. The goal was a computer “using massively parallel computing and processing” to provide a platform for future developments in “artificial intelligence.” Reading through what was stated then, I know I’m not the only one feeling a twinge of “Déjà vu.”

The plan was to build a high-speed parallel computing machine around the concepts of logic programming. The planners believed that logic reasoning would help Japan design AI applications such as automatic diagnostic systems or machine translation. More than 5 billion yen was poured into the project.

Japan bet its national reputation on this effort. In those days, Japan was often viewed as a heavy borrower of knowledge from abroad while contributing little of its own ingenuity to the outside world. Japan hoped to deflect the stereotype and build a new reputation.

The project was an abject failure. There is no fifth-generation computer to speak of today. Yet, no official reports exist in Japan today analyzing what went haywire, according to Arai. She suspects that this is one reason why Japan has steered clear of big AI projects ever since.

Curiously, Arai points out that the U.S. corporations have all learned from Japan’s failure.

Big companies like IBM and Google “quickly abandoned the idea of developing AI around logic programming. They shifted their efforts to developing a statistical method in designing AI for Google translation, or IBM’s Watson,” she explained in her book. Modern AI thrives on the power of statistics and probability.

>> Continue reading page two of this article on our sister site, EE Times: "The Functional Illiteracy of AI."

Loading comments...