Checkers was one of the first non trivial games where machines were...
Here Wiener is referring to our modern understanding of Game theory...
How much of this has now changed since this paper was published? It...
Samuel Butler was a 19th century English novelist, essayist and cri...
Even tho chess remains an unsolved game, checkers has indeed been "...
The paradox that Wiener describes is the famous Barber paradox by ...
In games like chess, "end game" is the stage of the game when few p...
Ataxia is a term used for a group of neurological conditions that a...
It took roughly 23 years from Wiener's writing for a program to pla...
Some Moral and Technical
Consequences of Automation
As machines learn they may develop unforeseen
strategies at rates that baffle their programmers.
Norbert Wiener
Some 13 years ago, a book of mine
was published by the name of Cyber-
netics. In it I discussed the problems
of control and communication in the
living organism and the machine. I
made a considerable number of predic-
tions about the development of con-
trolled machines and about the
corresponding techniques of autom-
atization, which I foresaw as having
important consequences affecting the
society of the future. Now, 13 years
later, it seems appropriate to take stock
of the present position with respect to
both cybernetic technique and the social
consequences of this technique.
Before commencing on the detail
of these matters, I should like to men-
tion a certain attitude of the man in
the street toward cybernetics and au-
tomatization. This attitude needs a
critical discussion, and in my opinion
it should be rejected in its entirety.
This is the assumption that machines
cannot possess any degree of originali-
ty. This frequently takes the form of a
statement that nothing can come out
of the machine which has not been
put into it. This is often interpreted as
asserting that a machine which man
has made must remain continually sub-
ject to man, so that its operation is at
any time open to human interference
and to a change in policy. On the basis
of such an attitude, many people have
pooh-poohed the dangers of machine
techniques, and they have flatly con-
tradicted the early predictions of
Samuel Butler that the machine might
take over the control of mankind.
It is true that in the time of Samuel
Butler the available machines were
far less hazardous than machines are
today, for they involved only power,
not a certain degree of thinking and
communication. However, the machine
6 MAY I960
techniques of the present day have in-
vaded the latter fields as well, so that
the actual machine of today is very
different from the image that Butler
held, and we cannot transfer to these
new devices the assumptions which
seemed axiomatic a generation ago. I
find myself facing a public which has
formed its attitude toward the machine
on the basis of an imperfect under-
standing of the structure and mode of
operation of modern machines.
It is my thesis that machines can and
do transcend some of the limitations
of their designers, and that in doing so
they may be both effective and danger-
ous.
It may well be that in principle
we cannot make any machine the
elements of whose behavior we cannot
comprehend sooner or later. This does
not mean in any way that we shall be
able to comprehend these elements in
substantially less time than the time
required for operation of the machine,
or even within any given number of
years or generations.
As is now generally admitted, over
a limited range of operation, machines
act far more rapidly than human
beings and are far more precise in
performing the details of their opera-
tions.
This being the case, even when
machines do not in any way transcend
man's intelligence, they very well may,
and often do, transcend man in the
performance of tasks. An intelligent
understanding of their mode of per-
formance may be delayed until long
after the task which they have been
set has been completed.
This means that though machines
are theoretically subject to human
criticism, such criticism may be in-
effective until long after it is relevant.
To be effective in warding off disastrous
consequences, our understanding of
our man-made machines should in gen-
eral develop pari passu with the per-
formance of the machine. By the very
slowness of our human actions, our
effective control of our machines may
be nullified. By the time we are able
to react to information conveyed by
our senses and stop the car we are
driving, it may already have run head
on into a wall.
Game-Playing
I shall come back to this point later
in this article. For the present, let
me discuss the technique of machines
for a very specific purpose: that of
playing games. In this matter I shall
deal more particularly with the game
of checkers, for which the Internation-
al Business Machines Corporation has
developed very effective game-playing
machines.
Let me say once for all that we are
not concerned here with the machines
which operate on a perfect closed
theory of the game they play. The
game theory of von Neumann and
Morgenstern may be suggestive as to
the operation of actual game-playing
machines, but it does not actually de-
scribe them.
In a game as complicated as check-
ers,
if each player tries to choose his
play in view of the best move his
opponent can make, against the best
response he can give, against the best
response his opponent can give, and so
on, he will have taken upon himself
an impossible task. Not only is this
humanly impossible but there is ac-
tually no reason to suppose that it is
the best policy against the opponent
by whom he is faced, whose limitations
are equal to his own.
The von Neumann theory of games
bears no very close relation to the
theory by which game-playing ma-
chines operate. The latter corresponds
much more closely to the methods of
play used by expert but limited human
chess players against other chess
players. Such players depend on cer-
tain strategic evaluations, which are
in essence not complete. While the
von Neumann type of play is valid
for games like ticktacktoe, with a com-
plete theory, the very interest of chess
and checkers lies in the fact that they
The author is professor of mathematics at
Massachusetts Institute of Technology, Cam-
bridge. This article is adapted from a lecture he
delivered 27 December 1959 before the Com-
mittee on Science in the Promotion of Human
Welfare, at the Chicago meeting of the AAAS.
1355