Thanks for the comments. You are absolutely right that arbitrage arguments require that the system has rich enough dynamics to exploit any vulnerabilities. An agent can remain irrational without negative consequences if it is not possible for an adversary to take the actions necessary to exploit it. However, there are usually many ways to exploit an irrationality and an agent takes a huge risk by remaining irrational. Biologically, it creates a “niche” for any adversary which discovers how to exploit the irrationality. Intelligent systems which are able to modify themselves and which have any uncertainty about the laws of physics or about adversaries they may face in the future will have a huge incentive to eliminate their vulnerabilities.

In your example, an agent which changes its mind about its preferences is exploitable. Say you prefer apples to bananas today, but bananas to apples tommorrow. At 11:59PM I sell you an apple in return for some money and a banana. Two minutes later at 12:01AM, I sell you the banana in return for some more money and the apple. It’s 2 minutes later and you are poorer but otherwise in the same state.

For the bid/ask spread example, you may not have that choice. Think of an animal who has to decide whether to fight or flee from a predator based on its assessment of the two distributions that arise. It has to pick one of the choices whether it likes it or not. The option of not acting isn’t part of the dynamics of that situation.

In general, preferences are over complete future time histories including your present assessment of your future actions. If a rational agent discovers that he has an internal mechanism that will cause him to change his preferences in the future, the rational response is to try to disable that mechanism, change himself, or set up an external commitment structure to keep his present preferences. Imagine yourself as a lover of books and to your horror you discover that you have a gene which will kick in in one year that will cause you to become an arsonist who will go around burning down libraries. What actions would you currently take?

Best wishes,
Steve

I think unfortunately the derivation in chapter 10 of expected utility maximization from the need to avoid vulnerabilities, especially section 10.9, doesn’t work, because there are ways to avoid being Dutch booked, other than being an expected utility maximizer. For example, I may prefer a mixture of L1 and L2 to both L1 and L2, and as soon as the alpha-coin is flipped, change my preferences so that I now have the highest preference for either L1 or L2 depending on the outcome of the coin.

To give a real-world example, suppose I come home and my SO asks me “Do you want chicken or pork for dinner?” I say “Surprise me.” Then whatever dinner turns out to be is what I want. I don’t go in circles and say “I’d like to exchange that for another surprise, please.”

Another way to avoid being Dutch booked is to have an ask/bid spread. Why should it be that for any mixture of L1 and L2, I must have a single price at which I am willing to both buy and sell that mixture? If there’s a difference between the price that I’m willing to buy at, and the price that I’m willing to sell at, then that leaves me some room to violate expected utility maximization without being exploited.

Or I may have a vulnerability, but morality, customs, law, or high transaction costs prevent anyone from making a profit exploiting it.

I suppose the first objection is the most serious one (i.e. exploitable circularity can be avoided by changing preferences). The others, while showing that expected utility maximization doesn’t have to be followed exactly, leaves open that it should be approximated.

Somebody brought your stuff to my attention and pretty fascinating.

I wrote a paper “mathematical definition of intelligence” I thought was going to be key to future progress in AI. See paper #93 at
http://www.math.temple.edu/~wds/homepage/works.html
(#94 also interesting). Turns out my work here was pretty much a rediscovery of Marcus Hutter’s work, but we each knew stuff the other did not.

Now it then seemed to me that somebody should start following up on my/Hutter’s work. And nobody did. But lo, I see YOUR work, which sounds like it IS following up on us, despite the fact you did not know about my paper you kind of have been following up on it anyhow. You might be better off explicitly realizing that, though, and doing some of the explicit things I discuss, like building the “platform for intelligence testing” I recommend, and building an
AI that genuinely *is* an AI (by my defn). Those things probably
would be pretty easy for you to do.

One more thing. I saw a video of you speaking on self-improving AI at Stanford and getting grief from some guy about the “fact” that there is no way to measure utility, maybe utility does not exist etc (economics fruitiness). Well, I have also gotten such grief in
a different context - voting systems.

My work on “Bayesian Regret” of voting systems, leading to conclusion “range voting” is the best of the usual commonly
proposed systems, involves utility. HOWEVER, all my conclusions
about BR of voting systems work REGARDLESS of whether
any human utility for anything can ever be measured or not.
Mm? That is the key thing. It does not matter whether it can be measured. My BR work still goes thru fine. So I think that is a very
strong reply to such hecklers, when it is usable.

You should get all your future-wacko pals to “endorse range voting”.
You browse to
http://rangevoting.org
push the “endorse” button, and fill out the form. May be necessary to have RV save the world.

Warren D. Smith.
Hope this message actually reaches you.