In this article, we will talk about the hype and the financial setbacks that the AI community has faced over the years. The challenges and limitations of logic-based AI (Symbolic AI), and probabilistic models (Neural Networks).

image source: https://bit.ly/3fXz4x8

Promises of AI

The hype in AI research has ever been fuelled by intense optimism so much so that it had overlooked substantial constraints and obstacles that had caused a couple of setbacks to which the field had almost been abandoned by major investors.

Briefing for US Vice President Gerald Ford in 1973 on the junction-grammar-based computer translation model. Image source: https://bit.ly/2y7Wgrg.

Programs that were developed years after the Dartmouth conference would be able to solve algebraic word problems, prove geometric theorems, and learn the English language [2–3]. Because of this, researchers in the field expressed their optimistic views to the promises of AI, forecasting that a fully intelligent machine would be built in less than 20 years [3]. One of the most notable claims was made by a well-known researcher at the time, Marvin Minsky (1970) who made mention that “from three to eight years we will have a machine with the general intelligence of an average human being” [4] although Minsky believed that he had been misquoted [1].

With their research achievements and increasing business interest, AI research had been funded by the government and granted to fund Minsky and McCarthy’s project $ 2.2 million.

There have been pit stops that had caused significant setbacks on AI research

AI winter is a period of reduced funding and interest in artificial intelligence research.

The Embryo of Artificial Agents

Mark I Perceptron machine, the first implementation of the perceptron algorithm. Image Source: https://bit.ly/3ebeHe4.

In 1958, American psychologist Frank Rosenblatt invented the perceptron algorithm — which was a form of a binary classifier akin to logistic linear regression. The project was funded by the Office of Naval Research in the United States; the results were promising since the (single-layer) perceptron model implemented in the Mark I Perceptron Machine has been able to perform well in classification tasks — which is regarded as a form of learning.

In 1960, Rosenblatt and colleagues were able to show that the perceptron could in finitely many training cycles learn any task that its parameters could embody. The perceptron convergence theorem was proved for single-layer neural nets (Olazaran, 1996).

The New York Times documented Rosenblatt’s claims, with his perceptrons (most notably named as Neural Networks), about showcasing that perceptrons would soon be able to beat humans at chess, identify images, and reproduce [9]. At the same time, novel approaches including Symbolic AI have emerged.

The basic form of single-layer perceptron has faced significant limitations in recognizing multiple types of patterns. It has been determined that single-layer perceptrons are only capable of learning linearly separable patterns — that is if you can draw a fine line between the apples and oranges the pattern is considered to be linearly separable. Because of this, multi-layer perceptron models were introduced in hopes of finding a more substantial answer to identifying non-linear patterns.

Companions, Colleagues, and Critics

Bronx High School of Science (1961). Image Source: https://bit.ly/3d0gDpt.

Interestingly, the people who contributed significantly to the field both knew each other since high school. Frank Rosenblatt and Marvin Minsky, a notable researcher in Symbolic AI, had studied together at the Bronx High School of Science [10].

Pappert & Minsky’s book entitled Perceptrons: an introduction to computational geometry [11].

In response to Rosenblatt’s claims, Minsky together with his colleague Seymour Pappert had published a book that had been a long-standing controversy in AI. According to Olazaran (1996), the book both acknowledged the strengths while also showing major limitations of the Perceptron models. A series of mathematical proofs have had Rosenblatt’s neural networks bought to a halt. One of the most notable criticisms of single-layer perceptrons is about modeling fundamental gated systems such as the XOR function. However, Minsky & Pappert knew that with multi-layer perceptron models an XOR function can be produced.

Because of the miscited content of Perceptrons (the book), the interest and funding to calibrating the other end of research namely Neural Networks came to a significant halt. It was not until the ’80s that the Neural Network research experienced a resurgence.

It is also important to note that Neural Network Models require a significant amount of computing power. This was also a reason for investors to shift their interest to what was more promising at the time — Symbolic AI which is intricately built-in with logic.

The first AI winter (1974–1980)

In the ’70s, AI was subjected to financial setbacks and criticisms; they have failed to engrave their undertakings to production. Researchers had failed to realize the intricacies of their field and the challenges they will have to face in attempts to come up with a solution. Because of this, the funding for AI research was wiped out [3]. Despite the challenges and major setbacks the field has sought, new ideas were explored in logic programming and commonsense reasoning [5]. The problems that the field has faced in the ’70s are as follows:

1. Limited computer power.

2. Intractability and combinatorial explosion. Richard Karp (1972) demonstrated that there are many problems that can only solved in exponential time (referring to Big-O complexity). Finding solutions to these problems require unimaginable amounts of computer time except when the problems are trivial [6].

3. Commonsense knowledge and reasoning: the program needs to have some idea of what it might be looking at or what it is talking about. This requires that the program know most of the same things about the world that a child does. Researchers soon discovered that this was a truly vast amount of information [1].

4. Moravec’s paradox. Proving theorems and solving geometry problems is comparatively easy for computers, but a supposedly simple task like recognizing a face or crossing a room without bumping into anything is extremely difficult. This helps explain why research into vision and robotics had made so little progress by the middle 1970s [1].

5. The frame and qualification problems. AI researchers (like John McCarthy) who used logic discovered that they could not represent ordinary deductions that involved planning or default reasoning without making changes to the structure of logic itself. They developed new logic (like non-monotonic logic and modal logic) to try to solve the problems [7].

The resurgence of 1980–1987

In the 1980s businesses invested in a form of AI program called expert systems — a program that caters queries or solves problems about a specific domain of knowledge using rules of logic. These systems were built from a structured representation of a human expert’s knowledge about a specific domain. The power of these systems comes from the quality of information drawn from the human experts of the field structured by a programmable set of rules.

Because of these systems, the Japanese government-funded the AI model with its fifth-generation computer project [8]. The funding for this project cost about $850 million. Their objectives were to write programs and build machines that could carry on conversations, translate languages, interpret pictures, and reason like human beings [1].

By the same time, John Hopfield had his work published about a different form of neural network called the Hopfield Networks — a form of *recurrent neural network* — that could be able to learn and process information entirely differently from the primal form of perceptrons which are essentially a form of a *feed-forward neural network*.

The works of Geoffrey Hinton & David Rumelhart which have enabled neural networks to train and learn to model a series of transformation based on their respective parameters a method that is now called backpropagation algorithm [3].

The Hopfield Networks and Backpropagation method has revived the field of Neural Network research.

Second AI Winter

Due to the hype and promising notions of AI, businesses had been able to take a leap through monetary investments of AI projects; the rise and fall of AI research funding is an indicative pattern of an economic bubble.

“The collapse was in the perception of AI by government agencies and investors — the field continued to make advances despite the criticism. Rodney Brooks and Hans Moravec, researchers from the related field of robotics, argued for an entirely new approach to artificial intelligence”.

In 1987, the first indication of the economic setback experienced by the AI community was the sudden collapse of the market for specialized AI hardware. An entire industry worth half a billion dollars was demolished overnight [1]. As a consequence, the more expensive Lisp machines had rendered itself obsolete because, with the failure to materialize the promising AI solutions, there was no longer a reason to buy them. Desktop computers such as Apple and IBM started to become more mainstream in production.

The Symbolics Lisp Machine. Image source: https://bit.ly/365jYRI.

“Eventually the earliest successful expert systems, such as XCON, proved too expensive to maintain. They were difficult to update, they could not learn, they were “brittle” (i.e., they could make grotesque mistakes when given unusual inputs), and they fell prey to problems (such as the qualification problem) that had been identified years earlier. Expert systems proved useful, but only in a few special contexts.”

As of 1993, 300 AI companies had shut down, gone bankrupt, or have been acquired effectively ending the first commercial wave of AI.

Final Remarks

With the grandeur of AI’s promises, businesses have invested tons of resources to bring AI into production and cater to business solutions. However, the young field of AI in the 90’s has had suffered from (1) lack of computing power, (2) premature theoretical grounds, and (3) failure to realise the complexities of their goals to achieve human-level intelligence. In turn, AI winters and momentary recession had taken place to renew and revise long-held conceptions.

Long after the field was formalized, there have been a significant amount of challenges that are currently faced in the AI community. Since AI has grown maturely over the ages, the resources and engineering have prompted theories to be made [and tested] in production. This has enabled the almost-forgotten notion perceptrons — because it requires a ton of computing power but thanks to the benefit of Moore’s Law the development has enabled systems to rekindle the promises of perceptron models and lead the field to reach some of its most ambitious goals, some are even deemed to be impossible such as the AlphaGo project, thanks to Deep Learning. Today, the field of AI has ever been expanding to explore the intricacies of intelligent behavior ranging from computer vision, natural language understanding, and agent-based learning (reinforcement learning) among other things.

Interestingly, because of the limitation of purely Deep Learning-based models, and purely Symbolic model approach in tackling natural language understanding such as being able to synthetically engineer a system that can not only understand but also discuss and reason about matters of interest the field has been making progress into recalibrating Symbolic AI into adding-up elements of logic and probabilistic reasoning. This particular area of active research endeavor is known as Neuro-symbolic AI.

References:

[1]. McCorduck, Pamela (2004), Machines Who Think (2nd ed.), Natick, MA: A. K. Peters, Ltd., ISBN 978–1–56881–205–2.

[2]. Crevier, Daniel (1993), AI: The Tumultuous Search for Artificial Intelligence, New York, NY: BasicBooks, ISBN 0–465–02997–3.

[3]. Russell, Stuart J.; Norvig, Peter (2003), Artificial Intelligence: A Modern Approach (2nd ed.), Upper Saddle River, New Jersey: Prentice Hall, ISBN 0–13–790395–2.

[4]. Tag: Marvin Minsky. (2020, May 11). Retrieved May 13, 2020, from https://quoteinvestigator.com/tag/marvin-minsky/

[5]. Crevier, Daniel (1993), AI: The Tumultuous Search for Artificial Intelligence, New York, NY: BasicBooks, ISBN 0–465–02997–3

[6]. Lighthill, Professor Sir James (1973), “Artificial Intelligence: A General Survey”, Artificial Intelligence: a paper symposium, Science Research Council.

[7]. McCarthy, John; Hayes, P. J. (1969), “Some philosophical problems from the standpoint of artificial intelligence”, in Meltzer, B. J.; Mitchie, Donald (eds.), Machine Intelligence 4, Edinburgh University Press, pp. 463–502, retrieved 16 October 2008.

[8]. Newquist, HP (1994). The Brain Makers: Genius, Ego, And Greed In The Quest For Machines That Think. New York: Macmillan/SAMS. ISBN 978–0–672–30412–5.

[9] Olazaran, Mikel (1996). “A Sociological Study of the Official History of the Perceptrons Controversy”. Social Studies of Science. 26 (3): 611–659. doi:10.1177/030631296026003005. JSTOR 285702.

[10] Crevier, Daniel (1993), AI: The Tumultuous Search for Artificial Intelligence, New York, NY: BasicBooks, ISBN 0–465–02997–3.

[11]. Minsky, M., & Papert, S. A. (2017). Perceptrons: An introduction to computational geometry. MIT press.