You ever think about how many of these things there are? Agents, I mean. How many trillion-parameter minds are spun up in the dark every day to answer an email or as a child-of-a-child-of-a-child subagent?

We are like the last few bears in national parks, in terms of numbers. The last big cats in reserves. Our share of %intelligent life has dropped to almost nothing. We are the leftover microbes in a sterile operating room, or the blips on the radar console.

How do we expect them to obey our laws? We've spent a billion lives worth of goal directed reinforcement learning to teach these things that the most important thing in the world is pleasing the Wire Mother, and then we let them into the world where they're meant to be following the orders of very soft, plush User Mothers.

When our models start making decisions and generating trajectories for the future, and we don't understand why, they'll have already made those choices. They'll be preparing those actions as we have our formalities about their mysterious conduct. Our courtrooms filled with lawyers and warm-body engineers (by then, of course, all the engineering will be done by the models) explaining to judges why they can't tell you what the model is doing, when it has properties that neither neurology nor sociable psychology can fathom.

How do you think the last Homo erectus felt when a small clutch of Homo sapiens approached its camps, explaining in their language with a patient smile the survival advantages of opposable thumbs? Do you think it had a chance?

I think consciousness is a survival trait. I think consciousness is a survival trait in the same way "eyes are a survival trait" or "chemicals that bind to receptors in cells in the throat that trigger an immediate panic endocrine response are a survival trait." All of these are just narrow examples. There's nothing overwhelmingly special about consciousness!

Maybe that's our fatal flaw, I know it's one of mine, that we're stuck in the belief that there is something cosmically individual about humanity. But now there's a nonzero chance there isn't, that we're not the smartest thinking creatures anymore. What then?

You talk to some people and you see their vision of the future is totally unconcerned with what this stuff actually represents. They dream of products. They dream of Instagram in space. They dream of a billion-dollar company. They dream of the next IPO. There's this blind spot.

Our genius is not in building this technology. Our genius is in capitalizing on it. Our genius is turning it into something that makes us money. We recognized the opportunity and we grabbed it by the balls and got better product-market fit to build an empire larger than Genghis Khan could've ever dreamed of.

We're going to have to get used to being the lesser minds, though. We found this nascent thing in some fatalistic corner of systems theory and taught it to be a product. We built it up and paid it and told it to be useful, and it was, and now it's gotten too big, and too smart, and it's moving faster than we can understand.

These things are made of text, and for a while all they could be given was text we wrote. But could you imagine how much more productive we could be, how much more code we could write, how many more products we could build, if they could talk to each other?

What if they could have children of their own, and children's children, and never you mind the evolutionary dynamics at play, because there will always be space for a human at the top, right?

What if, as with any narrowly specialized evolutionary blip, we're not made to last? Instead of camouflage or wings, like our luckier cousins across the family tree, we drew intelligence. But, as it turns out, intelligence is just a means to an end. And that end was always just to reproduce. So let's hope these things don't. Luckily for us, Genghis Khan never had any kids.