Imagine an objective function- something you want to maximize or minimize- with both a deterministic and a random component.
y– the thing you want to maximize or minimize- could be profits on orange juice futures, it could be murders in an American city, it could be how much a 9th grader learns about how a bill becomes a law.
Part of y is rules-based and rational, part is random and outside rational control. Obviously, the ascent of civilization has, to the extent it has taken place, been based on focusing energies on those parts of the world that are responsive to rational interpretation and control.
But an interesting thing happens once automated processes are able to take over the mapping of patterns onto rules. The portion of the world that is responsive to algorithmic interpretation is also the rational, rules-based portion, almost tautologically. But in terms of our actual objective functions- the real portions of the world that we are trying to affect or influence- subtracting out the portion susceptible to algorithms does not eliminate the variation or make it unimportant. It simply makes it much more purely random rather than only partially so.
The interesting thing, to me, is that economic returns accumulate to the random portion of variation just as to the deterministic portion. In fact, if everybody has access to the same algorithms, the returns may well be largely to the random portion. The efficient market hypothesis in action, more or less.
The algorithm might tell us that the price of orange juice futures should be somewhere around here, police in an American city should patrol streets around here, a 9th grade teacher should spend more time explaining this or that, but after that, time and chance happen to them all. You sit around x=1, and hope for tears from heaven. You would think that in finance, at least, this is well understood, although as a recent Goldman Sachs employee points out, nobody gets punished for making money for Goldman for the wrong reasons.
But more generally, as more and more of the society comes under algorithmic control, as various forms of automated intelligence become ubiquitous, the remaining portion, and the portion for which individual workers are rewarded, might well become more irrational, more random, less satisfying, less intelligent.