AIs will want to self-improve whether they were designed to or not

The rational perspective on artificial intelligence leads to many important but sometimes unintuitive consequences. Details are described in the paper:
Stephen M. Omohundro, “The Nature of Self-Improving Artificial Intelligence”
but it seems worthwhile to flesh out some of the important ideas in separate blog posts.
Our primary focus is on systems which improve themselves. Many of the consequences [...]

Coin flips

The von Neumann and Morgenstern development of rational economic agents requires events with arbitrary objective probabilities. Objective probabilities in the world usually arise from symmetries and the simplest example is the flip of a fair coin. If a coin comes up heads with probability , we’ll call it an “-coin”. Can we use use [...]