John Horgan reviews The Year Million,a a collection of essays about life in 1 million years. Needless to say, such inhabitants are presumed to be much smarter; in one guy's opinion, like the difference between us and a nematode. A theme of these essays is boredom. Consider a 1000 IQ person who knows all of history, mathematics, etc. Would you imagine this guy to be full of pep, enthusiasm? optimism? Bertrand Russell, whose youthful love of mathematics helped him overcome a bout of suicidal despair, later came to believe that mathematics is fundamentally 'trivial'. So what's a super-intelligent being going to find interesting when he gets old? Our biggest challenge is not the extinction of the sun, or asteroids, but catastrophic boredom.
I think this exercise highlights that intelligence will not become as great as these futurists suspect. Even today, the super intelligent reproduce at lower rates than the less intelligent, and this makes sense. For a stupid person, having a baby is like buying a puppy. For a smart person, having a baby is a huge trade-off, not only because smart people understand the huge responsibility they have created, but because very few great trains of thought can withstand the incessant interruptions of children.
The key is that, with significant intelligence and education, everything is boring. My kids love hearing the same joke over and over; I can knock 'em dead by inserting the word 'poop' in the punchline. I kind of envy them, but they are kids, I can't go back. On a more sophisticated level, novel connections are seen as funny (see the Mathematics of Humor), and so, the more you know, the less that appears novel, so the less funny. Sad.
A similar ennui can be observed for anything prized by humans. All mathematics and logic are mere tautologies to the infinitely intelligent. We humans, because we are so stupid, think ideas like 37x89=3293, or Cantor's diagonal proof, are interesting, but a super intelligent being will think: that's merely A=A! At the other end of mere epicurean pleasure, I remember famed columnist Mike Royko talking about porno films, and saying their biggest problem was not that they were filth, but that they were boring. You see, for this 75 year old curmudgeon, sex is repetitive (true enough). But there is another path for pleasures rather than boredom: becoming stuck in pleasure like a morphine addict. Dilbert creator Scott Adams suggests human evolution is bounded by the eventual creation of the Holodeck, because after that guys will simply go on an endless sequence sexual fantasies. So in the long run, sex is either boring, or something we use to euthanize ourselves.
It's hard to imagine moderate usage of anything in the long run (i.e. 1MM years). Indeed, I think if heaven and hell are infinite, then it really doesn't matter where you go, because even with 72 virgins, after several million years, they won't be virgins, and I'll be really sick of all those annoying harps (angels always play harps). Meanwhile, in bowels of hell, the first few days of being lowered upside-down into a vat of boiling human excrement, then raped by insatiable, barbed-penis-wielding hellhounds, will be terrifying. But by year 1MM it would be merely monotonous.
Very high intelligence is great for solving problems, but really bad for enjoying things. If you were super intelligent, you might be able to speak every language, solve every math problem, but then, you would not appreciate sports, sex, jokes, and conversation (Richard Posner noted that he likes tv because most casual conversation is actually quite banal). I don't think you can avoid the trade-off, which is why I would much rather have a son with an IQ of 120 than a son with an IQ of 160, I think the former would be happier.
So I think self-aware subsystems, i.e. conscious agents, have a limit to their intelligence, in that they will choose to not procreate because for them it isn't enjoyable. Thus, the only intelligence that will increase in the future will be emergent intelligence, things like bee hives, or ant mounds, that act like an organism but are full of ignorant agents. Agents themselves, who have total control, will opt out. Only systems that really have no control, or objective, will grow in intelligence as seen by the outside (for example, our cellular machinery is very smart, and contains an amazing amount of 'intelligence', but it is not conscious). But we humans, I suspect, are pretty close to as smart as conscious agents will get.