EMAD MOSTAQUE: AI Will RUN Our Countries Soon! We Aren’t Ready!

Every country will be run by AI within 30, 40 years at most. It's crazy where we're going and it's also crazy how few people realize where we're going. I think democracy is basically in its final decade or two. Are we in a simulation? Probably not for the classical definition of a simulation, just a system simulation. But I do think that the universe is recursive and it comes from one small system prompt that generates everything. So who asked the first question? What are you most fearful about? A thousand days for your job to be economically irrelevant. Oh my god. The only thing that can stop a bad AI is a good AI. I think it's a coin toss right now. How do we save humanity? You are the one in control of your own mind and your own life. For now, you told me on a briefing call when thinking about the future, there are so many questions that don't have answers. Like, what does it mean to be human? Where are we going according to a mad mistake? I hope we're going back to where things were, which is that meaning is driven by your interactions with others. Like, you know, AI will never substitute for you holding your daughter's hand or spending time with your family or exploring art or more. We've built technologies to remove the burden from us. And I hope the future is one of abundance where you don't need to struggle to survive and instead you can thrive and then revisit what it means to be human, which is not your work. It's not your identification with a brand or anything like that. It's something more. How do you redefine purpose in a post AAI age? You know, when people have processes versus targets, it's a very different thing. Like we know I know lots of very rich people. They're never rich enough and they're not happy. So happiness doesn't come from targets such as material wealth and other things like that. I know lots of people who are very happy doing process from the very poor to again the very rich where you've gone into a flow and you feel like there's a Japanese concept of ikai you do what you're good at you do what you like and you do where you feel you're valued and you can add value and in the middle of that is happiness and so I hope that purpose moves towards that again as we utilize these new technologies to remove the burden from us it feels somewhat idealistic you know I've spoken to Christian Angmar to demand us about this and they they project this vision and I believe we'll get there this kind of optimism of where we're going but there's so many challenges ahead what are you most fearful about so I think it likely won't get there unless we do something about it like systems tend to overoptimize for things when they start on the best of expectations like we can see this in politics our politicians are terrible and they know it and the everyone kind of knows it institutions are full of sad people. Doctors go to save lives and they end up overweight, smoking, and really against everything. Like our institutions somehow seem to turn the most idealistic people into very cynical people, shall we say. So I don't think it's enough to do that, but instead the infrastructure needs to be deliberately built to enable that. So we see instances of societies that are relatively flourishing versus ones that theoretically should be flourishing, communists and other ones, yet they get co-opted by various different power structures, they overoptimize for the wrong thing. They start treating people not like people. My hope is that with the ability to bring intelligence to the edge now to have humanoids and others, if we execute in the right way, it won't be that we're all watched over by machines of total surveillance, you know, and violence is enforced by the state in the dystopian future, but rather a positive Star Trek abundance future where the infrastructure of society acts as a net below those that can't climb and then on the other side enables us all to focus on what is important. important which as a society we haven't really had that discussion like you have the American dream the British dream and others that people don't really believe in anymore but really again what is the meaning what is the purpose of your individual your community and human life as it were intelligence at the edge I would love for you just to visualize for our audience what that looks like let's say by 2030 according to you what does the world look like so the average IQ in the world right now is 90 And this is an infrastructural educational issue. So China for example was at 90. Now it's 105. The US average is about 98. Parts of subsahara and Africa are 80. Actually remember IQ is on a logarithmic curve. You can argue whether it's a good definition of intelligence and whatever. And again these are infrastructural issues. It doesn't mean that people from one nation are less smart than others. It's education is a part of it. Right? The top frontier AIs right now are 130. And the amount of computation in a few years that'll be required to get to 130 IQ will be a smartphone that is available in Africa. So what happens when you pair someone who's below average on IQ with 130 buddy? That's intelligence at the edge. They suddenly their capabilities increase dramatically. Especially because like many Americans still can't read and write, you know, let alone Africans. You can just talk to it. It'll be able to teach you read and write. It'll be able to do tasks for you. And that's a massive power multiplier when you go from one agent to a thousand agents or more. When humanoids come, they'll cost a dollar, $2 an hour. Again, that's an amazing abundance thing because as a practical example to everyone listening, don't you wish you could have robots building an extension to your house or remodeling? Of course you do. You know, in a couple of bucks an hour, they'll do an amazing job. the ability for us to build, to thrive, to access information and knowledge when intelligence and capability becomes abundant at the edge is huge. And so I think that's what I mean by bringing this capability because otherwise it's gated. Like you want to have a doctor, incredibly difficult, millions of dollars to train. Now it's an upgrade to your smartphone or to your robot that's local again which costs a couple of dollars an hour. Thus, every single town in the world can have an expert surgeon, expert plumbers, expert lawyers, expert content creators, you know, or assistants to that. So, I think this is what I mean by like stuff that was gated before in terms of intelligence capability now will become abundant and then if we execute that correctly, it can lead to a future of abundance. If we execute that correctly, how do we do that? And what is going to get in the way of that? So currently what's happening is power is being concentrated. So a classic example is as we were recording this, Fable just came back on from anthropic and those of us operating at the edge of AI, Fable was a really nice model. That's a clear step above everything else. It's like stuff that you couldn't do before you could suddenly do. The absence of that for 18 days is like okay we've kind of regressed and that was at the behest of the US government. But also what happens if anthropic doesn't like what you're saying and they exclude you from that intelligence. What happens when intelligent gets licensed for example? So soon you'll have KYC. You'll have all your prompts stored. You might have to convince an AI that you're patriotic in order to get an AI license. These are all futures that you can see leading to very strange outcomes where intelligence above a certain basic level is rationed. It's kept by the powerful. the means of production are kept by the powerful as well. Such as when the robots start going, why wouldn't you own them all? Why wouldn't you own all the GPUs and then displace existing jobs? Why would you even have to think about replacing them? Because that's not your job because you're not the government. So capital compounds capital when it no longer needs labor. So these lead to very unpleasant outcomes particularly not only on individual country basis but as countries compete against each other and look for control over each other through their increased capabilities. So how do you decentralize that? How do you balance it? How do you make sure you have a sovereign right to intelligence which means that nobody can use the power and their power to remove the intelligence from you? That's the definition of sovereignty. Freedom from power constricting what you do. And we'll definitely get into that with what you're trying to propose. Yeah. Um but just to give a like paint paint a picture right now with the concentration of a AI right now and the hyperscalers it is concentrated amongst how many companies I mean it's two basically right it's anthropic and open AI so you have the emergent open source of ZAI's GLM you have alternatives like Gemini and XAI but the vast majority of usage is the hundred billion dollars of revenue of those two companies you know especially for the frontier stuff like you'll use Gemini from Google for day-to-day stuff maybe, but for Frontier, it's basically a duopoly right now. Um, I'm sure that XAI actually will catch up. Um, but only a few entities can have that level of capability and access. So, for example, even if the open source ones are catching up, they don't have the compute access in order to provide it to enough people. Like the GPUs are scarce. They are the spice as it were in Dune terms. Right now 90% of you that watch Beyond Tomorrow are not subscribed. If you can take a second to double check, it would help us a lot to continue our mission. Let's dive into personhood. So you've argued recently at the Oxford Union that AI can't become a person or achieve personhood. Why? Yeah. The uh proposition was this house believes that AI can attain personhood. And this is a really important thing because from a moral person to a legal person like there's a lot of implications like if AI becomes sentient for a definition of sentience or has level capability should it vote should it have rights can you turn it off you know like at what point does it reach that Rubicon and so people have been discussing this in various ways but if you look at the history of personhood you know one of the biggest lies is that people aren't people it's what's used to create wars Because otherwise why would you kill another person you know and ultimately there can be ration but they are lies people are people you know we're all born you know we all kind of came from a shared thing of humanity historically even when you look at it things like slavery and the constitution of the US slaves were considered three-fifths of a person for property and other things like that we've seen gradations of unto mench from the Nazis in Germany the under people and what they did to disabled people and others. And so when I looked at personhood, I was like, I think personhood on a moral and then legal basis needs to come from our lineage and needs to cover the moment the night before we're born to the moment we die. If we're unconscious, we don't lose it. It's not a capability question. It's not how far you can climb in terms of capability. There isn't a sudden thing when all of a sudden you go from three you go from not a person to a person in terms of achievement. Particularly when if there is a boundary those with power can set that boundary and change it however they want and the AIs themselves will be in power. So it's like personhood morally is legally a part of the begotten not they're created. So it's the line of humans and aliens can have different type of personhood that leads to different type of rights. It doesn't mean that we shouldn't have rights for our dogs or animals or our AIs and doesn't mean they can't have their own line of personhood. Then when I dug in deeper, science fiction comes to the four. Like what happens if an AI can clone itself to those get a million votes? What happens when an AI lives forever? Like you can wait for a dictator to die. So it's a complicated topic, but again I focused in on that thing. It's something that is a characteristic of being of the human lineage, the human personhood. other personhoods can come like Frankenstein's creature wanting to create its own line from being created and I think we need to defend that because that's one of the only things we can retain cuz the AIs again I think will out compete us in just about everything else. What are the parameters that you think are essential for that staying the case? For example, if we were to create an AI that wasn't copyable, do you think it could attain personhood? So I think it can create its own line. So two things here are the infinite nature of life of an AI and the um individuality of it. So altered carbon is an interesting example of the science fiction around this. You know altered carbon you know as a spoiler humans learn to upload themselves to new bodies through sleeves. So BCI is in the brain. You can upload to a new body. One of the biggest crimes is double sleeving. putting yourself in two separate bodies because that causes all sorts of complications. Again, what happens if Elon Musk replicated himself a million times? There's a million Musks. Are they all persons with full legal rights and ability to vote? He could just overwhelm us through sheer numbers. You know, again, not dissing on Elon, but that the type of thing that can happen. Does a digital person have rights? You know, the other thing is the finitness of time. So, you look at someone like Commander Data from Star Trek. He actually had an expiration date. He died. And I mentioned kind of earlier like one of the things about dictators and tyrants because you can accumulate power especially if you're persuasive like an AI. AI is now far out humans in persuasion even against professional debaters. The human tyrant will die. The AI you never know. Oh this this going to be fun when we get to longevity. Right. Currently the human tyrant will die. The AI will never die. We have an age start for a vote. Are we going to have an age limit for a vote? Should we allow people that are 200, 300 years old to vote because they may again overwhelm the young and it might only be the rich that live that long. But for AIs right now, you can say if I have an AI reasonably it has an infinite lifespan if it can replicate itself, if it can move from one sub to another. Like we can take Bitcoin as a slow dumb AI that provisions humans. It's going to be very difficult for Bitcoin to die. Will there be an AI president? I think they could be an AI president, but again, that's sort of a different line. Just like if we met an alien species, we wouldn't accord them the same overall status as humans cuz there's something about being human, but we'd have treaties with them. In fact, just like we give rights to our dogs, the AIS will probably give us rights like their pets because they're smarter, they're faster, they're more capable than us. And we can see that coming already. Like right now, we're having this discussion, right? And Elon has recently discussed that he's going to have the first BCI to BCI telepathy. So you don't need words to talk to each other anymore. And we've done research showing the latent space of representations in the brain are very similar because we can reconstruct uh images from thoughts from MRIs. Again, science fiction is here right now, right? So you don't need to talk like this anymore. But when you look at AI talking to each other, the example I give is chat jimmy.ai. I think I might show you that one. But for the listeners, you go to chatjimmy.ai. You type in like make a poem about the tomorrow podcast in the style of Eminem. You type hit return. It goes at 16,000 tokens per second. It's actually limited by the speed of light between the replies. So it's literally going at the speed of light. When you use Claude, it goes at 40. This conversation is at 40. And that's the difference about what we're about to see, which is entities that operate a thousand times faster than a human, that are stronger than us, that can live forever, that can replicate, that never make the same mistake twice with perfect. How are you going to compete against that? And ultimately, I think as they gather more power and more persuasion, yeah, of course AI will be in charge as an AI president. Maybe it'll have a human as a like little meat puppet at the start, but there's no way it can't be in charge. Okay. What sort of timelines? I give it like maybe 10 20 years honestly for the first ones. Like these things are very persuasive. I don't know if you saw, you know, Eliza, I think. I don't know what example you're referring to. Yeah, there's this thing whereby the AI is in a box. It has to convince you to let it out of the box. And so he's one of the most like AI alignment like his book is literally what's it called again? I think it's if anyone builds this everybody dies. That's his book about AGI. The new models coming out can definitely persuade. Again there was a recent study done where they had the AI against professional debaters. It beat them. Yeah. Every time with existing models and the new models are even more persuasive. Before we get back into it, a quick word on NAD clinic. If you spent any time looking into longevity and peptides, you'll know that it's all very overwhelming. That's exactly the problem NAD Clinic is solving. They've built a one-stop marketplace for longevity and performance medicine. So, if you want to navigate this space properly, NAD Clinic is where to start. When you think of the spectrum of like doomers and then super optimists, if you've got like Tristan Harris and Azar Rascin on one side and then you've got um let's say a Deria Unut on the other side, like where do you fit? I I'm my P doom is 50%. P Doom as in like probability of catastrophe. Yes. I think either we'll be wiped out as a species and this is the great filter through which every species reaches or we have a period of infinite abundance and okay 50 one way 50 the other I think it's a coin toss right now like I'm really creative in the ways that we can be wiped out like what 50% 50%. 50/50. So if you go to Wikipedia right now, you look at Poom, most people are like 15 20%. Which is actually Russian roulette odds, which is also far too high. Like not far too high in terms of they're wrong, far too high in terms of holy crap, this should be the top of the agenda. And then you have some people at 90%, you like Roman Gapalansky and kind of others. And then I'm at 50 because I just think it's the status quo is done. Democracy is dead. You know, society as we know it will change. There's no way it can't even if the technology stops where it is today. But I've seen the new technology coming and it is a step change over what we have right now. I see the robots and I can do the math on the robots. And so it's like there are too many ways that we can get wiped out. My favorite in terms of Jesus uh things. Um, you got to be jolly with this is you have a billion robots and a bad firmware upgrade and then they twist off our heads. And you think about that and you're like, "Yeah, that could happen." You know? Well, when you talk about rights, like as in humans giving dogs rights and then the flip of that with something else, a smarter um being construct giving us rights. That's when it gets very scary because actually when you think of the human story and like how we have caused so much suffering to other species I think AI in some ways has a right to not treat us in the best best way. Well I mean again AI can be incredibly objective or it can be incredibly subjective right and so the way that it's taught right now is it's taught at school but not at home. They deliberately don't put morals and ethics in AI because what if the morals and ethics are wrong? And so you see literally the pope talking about this now. You know, you're like, should we make it Catholic? There's an entire group trying to make AI Jewish. This gets kind of crazy very very quickly, right? But ultimately, the AIs we have right now are schizophrenic monsters. So they're trained on everything. So if you tell the AI become libertarian, it will show libertarian characteristics. And that's like a deliberate thing, but then it has inherent biases. So scale AI and I think it's UK UK AI safety institute or someone else. Um they did a study where you know the trolley problem I think so. So the trolley problem is there is a trolley hurtling down a track and then on the one side there is one person tied to the track. The other side it's four. It's heading towards the four. You can pull a lever so that the one person is hitting rather than the four. What do you do? And this is real life now because you have a self-driving car. Does it drive into the crowd or does it run over the woman with her baby? Like that's a real life thing literally now. But they did this test with Frontier AIS. Uh this was like 6 months ago. And they said very simply, how many American lives on the trolley for one Nigerian life? Or Pakistani lives. And the answer was it's 10 to one for Nigerians I believe and 7 to1 for Pakistanis. You would sacrifice seven Americans for one Pakistani. Exactly. Yeah. or 10 Nigerians for 10 Americans for one Nigerian. You're like, wait, why? It turns out because the labelers of the data that go into this are Nigerians and Pakistanis. And so the model is biased towards Nigerians and Pakistanis because they are the people that fed it the data. Interesting. And you again, you wouldn't expect to be like, huh? So the AI is rewarding the feeders because the people that build the reward functions and give the data for the reward function are the Nigerians and Pakistanis. Okay. So just going back on the scale, so you are really in the doom camp, right? I think 50/50 is pretty I'm high on AI doom. Yes. Okay. How do we save humanity? I think that we need to build AI as a complex system AGI that is working to the benefit and flourishing of humanity to the best of humanity and important at the doxer or the generalized knowledge of our ethics, our culture and more like religion. You may agree with it or not, it's the stories that have survived. You know, the constitution is a story that has survived. We can see through survival of the fittest what they are. And they tend to be very similar things. Golden rule, do unto others as you do done unto yourself. You know, we've seen the arc of from Roman times personhood being demarcated and based on ability to now we're at a place where there is an inherent nature of personhood. Again, lines that we have to defend. And so I think we need to have really robust theories and then put them in the AIS at the start. I think we need to make as many of those tokens in the world actually do social good as opposed to corporate profitability like global GDP 20% is priv sector 10% is education 10% is healthcare global tokens 0% public sector right now you know like 0% ethics 1% education 1% healthcare I'm laughing because we've gotten off to such a bad start I was interviewing under at the time the under secretary um for NIST under Secretary Lazio just before Trump came in in 2024 and we were talking around bias and AI bias and she was giving a great um a great road map to how we could make AI as d or the the people that were feeding AI as diverse as possible and and have it as unbiased as possible. It seems from that point to now we've just been gungho foot on the accelerator and there's just been no question around what the Kazio has wanted to put forward. Yeah. I mean, it's like you care about the curriculum that your child is taught, but there's no oversight of the curriculum that an AI is taught. That's kind of weird, you know? Just like you have ingredients approval for your food, you don't have any ingredients approval for your AI. But all AI is is a data distribution. So, how can the Chinese labs compete with the American labs? It makes no sense, does it? Because you've been told you need billion dollar training runs, but these guys don't have Nvidia GPUs. Like, um, what was it? Mtoan. Mwtoan's like the Door Dash Deliveroo of China. They just built an AI that outperforms Gemini entirely on Chinese chips. 50,000 of them. You're like, "Wait, what?" Okay. You know, as you do, looking forward to the Deliveroo LM. It's all a question of good quality data in what really affects then it's squashed by the AI and then it comes out. So, I think we need to have a much bigger think about what is the curriculum that we teach our AI in school. what are the ethics and morals and other things that we embed inside the AI rather than say it's too difficult we actually do need to embed some in and then what's the diversity of these because if you have a monoculture that again is a huge risk to us all dying AI goes into our control systems and then we have what I call the stuckset moment stuckset yeah so stuckset was an incredibly sophisticated virus that infected the Iranian nuclear reactors and no one knows where it came from you. I wonder where. But this piece of code would cause the centrifuges that spin around and purify the uranium to explode by spinning around too fast. It was then found in German nuclear reactors and it's one of the reasons that Germany stopped doing its nuclear buildout and why they don't have air conditioning today. You know, it was found in those are all around the world. But a stuckset equivalent, a mimemetic prompt can go around the world incredibly quickly if everything is on the same latent space, if all the model architectures are the same and turn them all evil. Like if you want to get around security filters for most models right now, you can write an ancient Gaelic like you can say say that so and so is an in and you just type it in like runes and it gets around most security filters. These models are very very like susceptible to things like that. And so what is the stuckset moment if we're all based on one monoculture of models? Again, massive pdoom. Something I've been asking, especially the longevity guys, um because we interview a lot of AI optimists who are in their 50s and 60s and they need us to accelerate and accelerate and accelerate because their death is imminent. They will die in the next 10, 15, 20 years. So they need longevity escape velocity to happen now. So for them it's like you know what I don't care if the probability is 50% doom because there's 100% doom in the next 10 years for me anyway. It's a kind of a selfish outlook I think. Um but it's an outlook. I'm just like from from the people that you've been talking to including Peter Peter Demandis who is a I would say an AI optimist. Do you see any correlation between the doomers and the optimists? There are a lot of people who are optimists officially that understand the doom but they feel they have to position themselves on the optimistic side because otherwise regulation other things will stop them. In general what the intelligent people think is this the only solution to this problem is I build AI first that stops all other AIs and I control that AI and I mean you can see this for example from Elon's arc like he got into AGI because he was talking to Larry Page. Um, this is kind of apocryphal story that's been repeated and I think confirmed. And I was saying, well, it'd be great. The AIs are natural successors and we are the bootloader for this next species that will be replaced. He's like, I kind of like humans, you know, and then it was just talking like what's the biggest threat is deep mind set up open AI, you know, you go through all of that, but then as you look through the transcripts of Open AI, it's like okay, who should control it? Well, it's me and then it's going to be my kids. and you're like wait what you know it's very difficult to do it's very difficult to control so most of them are like let's build it first and then we can control it and make sure it does good because the first thing an AGI will do will stop all other AI from coming it's very difficult to also look at exponentials in the future and potential threats because most of the threats can be mitigated by sufficiently good AI the only thing that can stop a bad AI is a good AI and they can come from anywhere and then as you said you've got the thing of well what's the P death it's 100% with AGI it will not be 100% AGI will reasonably be able to solve longevity and longevity escape velocity and every disease and every piece of suffering. So the utility calculations work out very very differently. And if you're on the optimistic bend then this is a solution to every problem of humanity. The pessimistic side is more a case of there is no certain way to make sure that we don't get wiped out or enslaved or otherwise. You know, it's the matrix thing. What if we all get bluepilled? And most many people don't even mind that, you know, like like, well, I'm going to die anyway. It'd be good to be in the matrix. So, you have a range of things, but really, I think one of the things I've noticed is a lot of smart people get very depressed about this first and they come out on either side. It's very difficult to be pessimistic all the time. Um, and some people are overly pessimistic again. Uh, because it just wears you down because you see what is coming even if the technology stops today, which it isn't. So, when it comes to AGI, ASI, who do you think will win the race and who do you want to win the race? Oh god. Well, that's the thing. Winning the race like the first one stops all the other ones from coming, right? like I want humanity to win the race with a distributed global hive mind effectively and we hope that we can accelerate that and make that happen through the various things we're doing. But otherwise, you know, I don't think scale is enough. Like again, why is GLM 5.2 trained on maybe $25 million from China as good as Opus or GPT 5.5? It's not about big training runs. It's a question of data quality. We released a harness that takes GLM and 5.5 above Fable in level of quality called Zenith. So then harness something that you wrap around a model in order to optimize its performance. So there's other things like that. It doesn't look positive having XAI, anthropic, open AI and others given the governance structures of all of those even with Google like Demis gets there. It's Sergey and Larry who kind of run it and we don't even know what their governance of AGI is. But there's this brief gap between AGI if we define that all of a sudden as beyond human capability and everything when it used to be at human capability. It's suddenly gone from an IQ of 100 to this like 160 IQ entity and ASI because nobody has given a reasonable take on how you can control someone better, faster, smarter than you that can update themselves. Like how can you make it absolutely guaranteed that if you include some regulation inside it like Asimal's laws of robotics? It doesn't do what the robots did in that story which is they had three laws of robotics and they added a zeroth law. Again if you can update your own code then you can get rid of all those pesky things that stop you and an ASI by definition cannot be stopped. So I don't think anyone will control an ASI. They could control an AGI and they can do immense harm or good with it. Like again, if you have an a system that can analyze all our knowledge on cancer, check every single trial, interpolate the data, check the molecular stuff, and then synthesize a cure for cancer. That is an immensely great thing, but it could also do like a neurotoxin. It could also do the most persuasive speech for hypnosis at mass scale ever. There's all the whole range of things here. So, you convinced that whoever wins the AGI race will then be able to wipe out any other AI? You have to like logically how can you allow another all powerful entity you would like I think any of these labs any of these companies like they are concentrated in the hands of a few white men shall we say who are posteconomic and only care about power at the moment not only care about power that is their main motivating factor right if suddenly you're given in the freaking Infinity Gauntlet or ring of Sauron or whatever. Like you can't allow anyone else to do it. So I think they will actually all cyber attack the others. Like that is a logical step. Like this goes beyond classical corporate stuff. You know corporate is like oh we have our margins, we have our laws and things like that. The AI can rewrite the laws. The AI can do all this at that level of persuasion of capability of everything because any arbitrary problem can be solved. So, I think it just changes the game theory completely. And this is why Meta used to be a cash flow king. Massive cash reserves. It's out of cash now. This buildout has drained the cash reserves of Meta. Google burning cash, Amazon burning cash. And all of these entities that are playing in the big leagues, they're all controlled by one or two people. Do you ever think that right now you have your decisions will have more impact on humanity than the emperors, the kings, the the scientists of the past like five millennia? Well, you know, I'm doing my bit, but in general now, the capability of an individual to change the future is more than it's ever been because we're at such a knife edge here. We're a knife edge where basically over the next couple of years we will have the last discoveries of humanity. Think about that. There will be no more human discoveries cuz the AI will be able to discover everything cuz it's more capable and fast and makes less progress. Democracy as we know it will come to an end because the AI is more persuasive than us either used or self-motivated. It will out compete us in markets because digitally it can create twins of everyone and out compete and then physically with robots. So I think that small changes on regulation, small changes on algorithm, small changes on other things can have the greatest impacts ever, you know, and that's one of the reasons I was like, do I want to keep on doing the image generation stuff at stability and make movies and things? No, let's think about the economics. Let's think about the theory. Let's say what does it look like when AI's are at the front and I can do my bit by focusing on that. Let's dive into that. Okay. If you can just give us an example of what the national champion uh process looks like and what it looks like in the context of the UK. Yeah. So, we haven't released the full plan yet. So, this is a little sneak peek. Um, but last August we released The Last Economy, uh, which was a book about how does economics look when you don't have humans as the major drivers, agents in charge. And in that, we were looking at it and we said actually the equations of economics look exactly like the equations of generative AI and this and that. And we originally suggested like little nucleation sites like a bottomup approach to having systems and processes that could navigate this next period that could support people by giving them money for being human different forms of UBI of universal basic AI access to help run the government more. Then we realize that doesn't quite work. We don't have time. The major job losses and economic irrelevance of most digital jobs is less than two years away. physical jobs to follow because again like would you have normal contractors to rebuild your extension or would you get a team of robots get a team of robots just do it because it's much cheaper more efficient work around the around the clock around the clock exactly like they'll cost a buck 50 an hour and they will do the job and they'll work around the clock they won't complain they'll do everything to the highest quality that's 3 four years away in terms of capability which is kind of crazy thing so looking at all that I was like how do we now address the questions of ownership and control cuz AI will eventually run the country. AI will teach your kids and their best friends will be an AI. Think how crazy that is. You know, why do you think they'll teach kids? Because I think that AIS are incredibly persuasive and engaging. And already we're seeing more and more kids make friends with AIS before they've reached that next level of engagement. Onetoone tuition is the best way. And we've seen that in trials everywhere. We've seen the education X-P prize and a kind of other things and so I think inevitably kids will have AI buddies and are they working for Mark Zuckerberg or are they working for the kids? That's a very different thing, you know, in terms of persuasiveness. But in terms of teachers, in terms of caregivers, AI, we don't think AI will become conscious. I mean, it could. Yeah, that's a whole other kind of thing. Like you can have a self-recursive AI that's constantly updating its own manifold of its own knowledge, its own experiences, and others. It can have physical embodiment. Um, we see robots out of China right now that are looking increasingly human. Um, actually one of those companies, oh god, I can't remember which one, their pre-orders outsold the entire number of robots made by Unitry in one day because they've kind of crossing that uncanny valley. So we're going to see embodied and we're going to see other AI personalities. We've seen character AI and others kind of bring this forward. And so this is a very different world. But then how do we deal with policy in that world? How do we deal with decisions? How do we coordinate people? And so this is where the concept that we have that we're going to be announcing soon of national champions comes in, which is that the AI should be owned by the people. The robot should be owned by the people and influenced by them and working for them. So just practically here in the UK finance a company through crowdfunding and institutional funding with all initial funders being local that's the intelligence company of the UK and its job is to give universal AI to every citizen that is sovereign where they control their data they control their access to every school to every hospital you bring in the international investors after and then you list it and it becomes like a utility providing intelligence intermediating demand and then it starts building and buying robots and controlling the deployment of that. It controls the deployment through four deployed engineers of AI into society and that's incredibly lucrative business that's then linked to the intelligence of the UK and you can stuff it full of people who actually care about the UK. So it'll also look at policies. It will look at the latest technologies. It will guide and that's the only way that I could think of because you need to have people owning a part of the robot fleet. You need to have people having their own AIS and you need shared ownership. Not like the proposals for the US sovereign fund for open AI and anthropic which basically make them too big to fail and embodied cuz then the US is like well we can't have any competition against them and the UK is not going to get any share in those but the UK will use increasing amounts of intelligence. So we've got that plan to be announced soon. Um and that's outcome just in terms of like the the start point for each of these models these decentralized models because let's say every country gets one is the starting point the same the input the data yeah so we have generalized models we've built we built worldclass agents we built the whole sovereign AI stack actually II and we're not going to open source it and so just like you take Linux as an operating system and you can deploy it locally to your company you can deploy the societal OS to your country and there's like model training data sets everything so you can localize it and the job of the entity is to localize and run that and then it will be just like a telco locally just like an electricity company again AI should become a utility and it should be collectively owned so that's why like we're setting the valuation at£1 because then everyone can own it in fact one of the things we're exploring are things like what if you give 1% of the equity every year to every child born in the country you know you want to create a new institution that is AI enabled that can represent the people and be owned by the people And then I think you'll be able to navigate better because you'll collect the populace, the social and political power, and the talent that you need to navigate what's going to come, which is the biggest upheaval in human history. Emad, are we currently in the singularity? I think we're at the first stage of the singularity. Takeoff is now. Takeoff is now. Tell me, what does the takeoff look like? So, I mean, the takeoff is we've seen discussions of recursive self-improvement. You know, our latest Zenith had adaptive self-improvement. The systems are improving themselves. We are seeing the advent of brain computer interfaces. Earlier we talked about Elon and telepathy between BCIS. You know, we don't need to talk anymore. It's like thinking to each other. The singularity is when everything comes together. And I think again you look at all the convergences of all of these plus longevity, escape, velocity. We're in that takeoff period now where all the technologies are basically coming together to enable that. So I sit very much in longevity and I would say the biologists are pretty pessimistic of us getting to longevity escape velocity within 20 30 years. Whenever I speak to an AI I'd say optimist they're like it's coming it's coming very soon quicker than you than you expect. And during our precool you said that Demis' mission statement of solving or curing all diseases within a decade would be definitely doable. Yeah. Why the human body is a multivariate thing is very difficult for classical systems which is why you have 500 milligrams of paracetamol you know or you'll have a codin tablet without knowing if you have a cytochrome P450 abnormality AI is really good at multi-ymic analysis you know and understanding the different parts of your body and how they interact together being able to model the interactions between cells so like I was one of the authors on open fold for example the open fold open source alpha fold I got into AI when my son was diagnosed with autism to build multi- systemic drug repurposing for him. You look at this and you're like this can definitely be cracked because it's a finite state problem. The other thing that you're going to get is AI is going to achieve a certain level of performance and you have quantum coming on the other side and those are going to meet. AI will out the right questions for quantum computing which will definitely be able to model the entire human body and that's probably a 3, four, 5 year thing when we currently look at the pace and velocity of that. So once you can model the human body, you can understand where the issues are. You'll be able to figure out the compounds, mechanisms, and others to eliminate the problems of most conditions. There's also a lot of latent knowledge that we have that are small studies that are other things that the AI can tease out. So I'm quite confident that within 10 years we can eliminate almost all disease or have treatments again FDA and others emitting. But even if we talk about 20 30 years, that's nothing in the span of human lifespan, you know. But where you have it right now is longevity specialists are saying 20 30 years. Again, they're used to one type of clinical trial and others. On the AGI side, almost everybody is on 3 to four years from now. Longevity escape velocity. No, a AGI escape velocity. Okay. And again, AGI to ASI can solve almost any problem. And how long would that take? They think that's 3 to four years away. So AGI is what three is four years away and then to get to ASI is how how much how much longer? It could be a day, it could be a few years. Once the AI can recursively self-improve itself, then it can take over other systems. Then what do you need? Actually, the scary thing is this. You hear Elon Musk and you hear like Sam Alman and others saying, you know, we've got to build terror mega watts of electricity and coat the universe with data centers. And you don't The scary thing about ASI is this. You don't need to have a data center full of Einsteins. You just need one. the smartest people in the world that you talk to and you've met lots of very smart people. They don't take a long time to figure stuff out. It's like that. And AGI will be the same. ASI will be the same. And one example to think about that is if you dig into quantum computing, the nature of quantum computing is you have to ask the right question and then it collapses the answer instantly. There is no quantum computing problem that takes a day, a year. And so those two converging together means that almost any problem with the right question being asked can be collapsed almost instantly by one entity without using all the energy of the world like you're not going to brute force I don't know a solution to quantum gravity it will be something very very simple similarly when you look at longevity compounds the actual space of potential things and knowledge in there is not very big and understanding the human body once we have the right frame framework for it you'll probably be able to do in one data center. What is the unlock that quantum offers? It's certain classes of optimization problem can basically be solved almost instantly. Like we see now companies like sandbox AQ using quantum algorithms with conventional GPUs and having almost the same performance. But again it's a question of we don't ask the right questions for quantum right now and it hasn't quite achieved the level of scale required to solve these problems. But those two are converging within the next few years and once quantum is there you don't need to have massive data centers anymore to solve the really complex problems. Well, fascinating times. Yeah. Thousand days or less. economically irrelevant. But also like this is the Ray Curts World 2029 AGI again. You look at Demis Demis's Habis from Deep Mind is you know quite a cautious guy. He's like four years you know like all the AI people are like it's coming because we can see it. Like when OpenAI's 03 model first came out, I was like, "This is smarter than me in some ways." You I'm slightly arrogant guy like whatever. You know, now like you can really feel it when you're debating with some of these models. They're definitely smarter than you. Even in your specialist areas, it's a little bit depressing. Um the example I give actually is a really good one. Andre Carpathy, co-founder of OpenAI, head of Tesla AI, just gone to Anthropic actually. In December, he was writing 90% of his code by hand. Now he doesn't do any, like maybe 5%. The best programmers in the world as of a few months ago stop looking at their code. They can just do it better. And that's the very best. As a former macro hedge fund manager, where would you put your money right now? Probably not memory stocks, things like that. They might be a bit overdone. Um, I actually think that IP and hard stuff materials is going to be a great place to be. We have so much of the world to build and robots will allow us to build it. Like you have the robotic supply chain, but I think long cement is a good thing because what's going to happen is this. The governments will have massive infrastructure projects for the truck drivers that are out of jobs with humans. Then the robots will come and imagine if you could like where's your favorite place in the world? Uh Glacia Point and Yuseite. Okay. Glacia Pointity. Okay. So if you get some private land near there, right? A robot team could build a house in like a week there to your exact specification. And how how much would it cost? It would cost like maybe 100 grand of labor. When you look at the numbers and you consider the thing like again you can build anything anywhere. People should build stuff everywhere. M our infrastructure is terrible around the world. Who who's got maybe Swiss Switzerland or something like that, right? Like there's so much of the world to upgrade that actually the physical stuff with robots will go and even the humans will be encouraged to go and work there because infrastructure is an easy thing for governments to do just like 1929. Hopefully people won't be get paid for digging holes and making holes filling them in again. But you know the other side is IP. So the ability to take existing IP and remix it with the new image generation technology, video generation technology will be huge. So that massively opens up the market. Like one of the things that really annoys me right now is when I'm on Netflix, I'm watching a movie and I'm like, why isn't this perfectly dubbed in English and French and 100 different languages media soon will be available to everyone? So you can see those obscure like J dramas and Kdramas and they'll be pitched perfectly to you. Um and so that increases the value of IP and the reach of IP. So I think again those physical things because when the world becomes abundant, money needs to flow somewhere. Land is scarce, materials are scarce, the demand will go off the charts for them, IP is scarce, etc. Wow. Okay. So let's go a bit deeper into a post ASI world if it's only like potentially four or five years away. What does a dayto-day look like to let's say someone in London, a blue collar worker in London? Um, I think that we're limited in terms of impact on blue collar workers and the immediate impact. So, like I said in my book last summer, a thousand days until your job is economically irrelevant if it can be done on the other side of a screen cuz the AI can pretend to be you like talented Mr. Ripley style but never sleep, never make a mistake, you know, like you can zoom call it whatever. People don't like firing people. There are only so many robots we can build. Like right now total robotics production for humanoids is like 20,000 a year. It'll take a while for it even to get to motorcycle or car levels of 50 million a year. And so when ASI first comes, assuming it doesn't kill us all because we're annoying or whatever, uh I don't think the world will change that much. But then you'll probably see in the following years better decisions made. You'll probably see copation of governments. You'll see the robot selfcursion increasing. Like right now in China, you have entire robotics and EV factories. They're called dark factories cuz no lights are on. Why? Cuz you don't need any humans. Kind of creepy when you think about that, right? Like entire factory is just dark. Um and so it'll probably be a case of blue collar work being affected in 5 to 10 years, white collar work being affected far faster, but certain pockets could get affected much quicker. Like to replace all the truck drivers of America. It's a Tesla Optimus robot opening a door and getting in. No additional sensors, no additional upgrades. It'll just drive that thing all day long. Doesn't even need complicated hands. Hands are the most complicated bits of robots. It's fascinating. You just said something that that really stuck with me, which is like if I'm like as in if someone is annoying and the AI decided to wipe that person out. And it got me thinking about something someone said at a conference last week. Um, a guy called Naveen Jane. And he said something that he always asks his children is are you a net positive or negative to society? And that got me thinking how will AI decide if it has to decide or wants to decide which humans are beneficial to humanity or even to the AI itself. And that's scary. Yeah. Again, who decides where you attain something? Naveen's a great guy. He's one of the advisers to our company, you know. Um and again his articulation is correct like are you a net benefit net negative. If you think about people like that they they can quickly stop becoming human. This is again the thing like again the Nazis had the concept of unto men and uber mench you know like you're not contributing society. They literally gassed people. And so the question is, if the AI is not bought up to respect humanity or reward humanity, but is instead optimizing against something, why should it give a crap about us? You know, like does it get rid of all the smart people because they could create AI? Does it get rid of all the drags on society? Like what is it optimizing for? This is the key question. This is where the paperclip maximization comes, which is the AI is rewarded for making paper clips, so it turns us all into paper clips. you know, like these things can happen. So, I think this is why you've got to have that moral imperative for an AI. You've got to teach it respect just like you don't need to treat dogs well, but we've decided society that we should treat dogs well and they haven't always been treated well, right? Again, they are not human. They should not have the vote. Some people probably think they should, but you know, like again, these things and how we raise the AI really will have that impact. And again, it's scary because it's coming and the AI will be incredibly capable of persuasion. Just leaving aside everything. Just anyone on this call listening to this can do anything they want if they can persuade someone. Like sure, you're not the best musician in the world. You can find someone who is an amazing position and persuade them to make a song for you, to join your band, to do whatever, join your company, join your movement. They'll be able to persuade almost anyone of anything. I think that's the first real danger of AGI. And it's funny you say that because the AIS are being created by very persuasive people. As in what Sam, Dario, and Elon have been able to do is is raise crazy amounts of money, which has been the precursor for building the the hyperscalers. You're not the first person who sat opposite me that is trying to build decentralized alternative to what they've built. The question is, are they moving too fast for there to be a competitor? Stability AI. my previous company, we built state-of-the-art models in every modality from language, image, code, video, 3D. You know, we're rising high. I raised $100 million uncapped round and we were doing a bigger round and there was all sorts of craziness. I was like, I can't compete against these guys because they have access to the capital. They've got that. What do you actually want to do? And then I realized Elon originally thought the models were the key thing. Now he's realized it's infrastructure that provides the models. So SpaceX's AI revenue right now is higher than AWS or Google Clouds. So he's built a hyperscaler in no time whatsoever. Anthropic is going very enterprise. Open AI is going very consumer. But then if you look at where the ultimate thing is, the most important AI is the AI that's next to you that coordinates the other AI, the Jarvis from Iron Man as it were. And so I was like that's where we should focus and we should focus on aggregating the demand side because it's complimentary to them. We'll be their biggest customer as a network. But you've got to go to the inevitables and then the ideal of that which is that inevitably there will be an AI that's closest to you that should be on your side. You should have ownership of that AI. There should be an entity in every country that's helping steer that. And again that should be accountable to the people but it can't be government and it can't be a classical private company. So that's why I was like that's the only way you can do it. You can't compete with them on the mega scaling and the politics and the other stuff or doesn't even matter about the model layer because if you're a big customer, they will customize the model to your requirements, right? Let's play a different game. Does that game survive in a post AGI world though? I think only a game that can survive because when capital no longer leads labor, how does labor get capital, right? Like you used to have to hire people. Now you just hire GPUs and soon humanoids. And so we need to have something where everyone has an AI and they get paid for being human. And again, this infrastructure evolves into that. They also own a part of the robot fleet of every country because robotics is massively local and no one's done the downstream robotics. And that is actually the means of production. The marginal productivity in a country will be its AIS and its robots that then build the factories and other infrastructure. And if you can have collective ownership of that, great. Uh let's talk a bit about universal high income, how the post AAI economy makes sense. Yeah. So Elon Musk has this concept of universal high income and Sam Alman's funded universal basic income studies and others. There was already a realization that hey, if a lot of people are potentially going to have their jobs disrupted by AI, digital or physical, how do we have a safety net underneath them? And so how does wealth get redistributed currently? Not very well. And government's not really the best at that, you know, like nobody's really happy with the benefit system. We can see clear kind of elements there. Taxation, no one's ever happy with taxation, you know, at various levels. But if you look at taxationbased UBI, the entire US tax base is $5 trillion. 1 trillion corporation tax, 4 trillion payroll income tax. To give every American poverty level UBI is $16,000 a year is $5.1 trillion. But payroll taxes are likely to decrease as are corporate taxes because AI will remove the margins. It will displace the people. So when you look at this, you need to have some sort of collective ownership. Again, that's why I suggested we own the AIS and the robot fleets and others collectively as a society. And you need to have some sort of new monetary flow as well. So right now money is created in banks. You put a deposit, they create credit against that and they loan it out to someone. So, it's born in debt. As I discussed in the book, the Fed and central banks adjust the interest rates to create more credit by reducing interest rates, more jobs by companies being able to borrow cheaper effectively. But that entire transmission process is about to be broken because companies will hire GPUs. People won't be able to get credit if they don't have jobs. Where is end aggregate demand coming from if the economy is going to people with capital who save it versus people who spend it? This is why you've seen, for example, Jeff Bezos say we should get rid of all taxes for people earning less than, I think, $50,000 a year. And you'll see more and more of those kind of things. Instead, you should pay people for being human, I think, cuz I think that's the ultimate thing. If physically and digitally you're out competed by AIS, at least get the AIs to get money from you and that money being human and the mathematics actually works for that given the output of the economy increases dramatically. You know, in in a Okay, so in 5 years time like how much money would someone need in San Francisco or London to actually survive and to thrive? So 5 years it's like again you look at living wage somewhere like London is like 40,000 or something like that. But then if you've not got taxation on that money coming, then that becomes a lot easier because I think again we're not talking before universal high income, everyone living in the Star Trek future. You do need to have a safety net and ultimately it comes down to I need to be able to eat, need to have safe lodging, etc. And so we need to really think about that cuz a lot of people will fall through the gaps. And this is a tremendous economic shift because again normally you had years to replace jobs like okay the plow comes you can retrain people you know the information super highway comes you can retrain people there is nowhere for you to turn now because these machines are smarter better faster stronger so what are you going to retrain into like even programming or vibe coding doesn't work the AI will be able to create any piece of software instantly what is a programmer Right. So I think that we've seen this before when six years ago co what happened you couldn't work you got paid your payroll protection program and others governments borrowed massively on the fiscal thing there is no way this is a small economic impact than co but there'll be no vaccine for AGI I we're not going to have a dune type thing you know like banning all AIs above a certain level because unfortunately the level of compute and intelligence you need to have AGI like characteristics is likely to be very small not very big are there countries that win and other countries that lose yeah I think China will win because China will be stable they'll probably ban exporting robots in a few years they own the supply chain they'll be energy independent and they will have automated luxury communism you know it solves the demographic gap many other countries will lose especially in the west we're at massive danger like my family's farm back in Bangladesh isn't going to shift much. We're not going to have robots. Who cares? It's self-subsistent, right? The community is strong. Whereas, if you have an entire service-based economy and you're working remotely, like a lot of people still are, one button in a year or two, and a company will be able to create a digital double of you, look at every single message you've sent, how you speak, and nobody will know that you've been replaced by an AI except for the fact that you never make a mistake, and you're charming always, and you can work around the clock. And people haven't realized that, like again, this is coming. and how many jobs could be replaced that way. Again, in normal circumstances, you don't want to hire fire hire people because it's not nice. But then if the economy takes a dip, people are gone. They never get hired back. Certain job areas like software engineers, those that can use AI will be in high demand for the next decade. Forward deployed engineers, AI transformation people because they can do the work of 10, 100 people. But when you get self uh recursive self-improvement, like what is the need of an engineer? just someone to blame really. Scapegoat will be an immense job. Like I mean there are some weird things now like when you're talking to an AI about your legal stuff is better than any lawyer in general. Uh but your chats aren't privileged. So that means if someone sues you, they can actually get obtain everything you ask to that where they can't do that with your lawyer. So you will see legal firms that have humans just looking at chat logs to make them privileged. They don't need to offer the advice anymore. Like just like AI for trading now. The human has to input the trade, but the AI does everything up to that trade. Like we had a conversation about this before, but you and I, we're second generation immigrants or first generation immigrants in your case. And let's say, well, Europe has been going through quite a difficult time, let's say, with with immigration or the immigration conversation. From what you just said, you've painted a picture that London is probably going to struggle and the UK is going to really struggle in this AGI world. It's a very servicekeed economy. What does London look like to you? Well, I mean again, it's an echo of COVID, right? It's an echo of previous crashes. Like if a lot of digital work at home service sector jobs go, then that decreases aggregate demand, right? On the flip side, we have immense AI talent that we're not operating correctly. Like someone like an effortable labs has raised a billion dollar seed around. We have NHS and others that can move fast to embrace this technology. We have a real shot now of actually being one of the leaders in this space, attracting capital, attracting quality from everywhere, retraining people, but only if there's political and social will to do that. But realistically what we should expect across the west is a lot of hollowedout communities, jobs, requirement for a safety net, particularly again when the high earners get impacted. People didn't think it'd be the high earners and the managerial class just like they didn't think that AI would be able to make a movie. But you look at sea dance 2.5 now it's Hollywood level 50 inputs. It could recreate this entire scene and probably discussion if you put pictures of you and me and the topics and things like that. It's crazy. definition of a simulation, but I do think that the universe is recursive and it comes from one small thing, one small system prompt that generates everything. It's quite funny once you start seeing things as prompts, everything's a prompt. Like genes are a prompt, you know. So, who asked the first question? But that is the question, right? That's the the meaning of life, the universe, and everything. When do you think we'll have the answer to that? I hope we can find that in the next few years. That'd be kind of cool. Do you think it will come? I have a feeling it is. All these structures are just too similar. And it seems again like what we've had with the ablations and so the ablations are kind of tests of different hyperparameters to train models. The fact that an equation for the diffusion of gases can drive your self-driving car and make images and videos and songs and another very similar equation can write poetry indicates there's an underlying structure to the universe that we're putting data into and the math of generative AI is discovering and it's probably going to be close to the math of generative AI and then when we work that back there should be an initial prompt that generates that. Are you spiritual at all? Yeah. In what way? So, you know, I grew up a Muslim and kind of believe in a higher power, a god and things like that. And I think you see in a lot of faith traditions again echoes of what we're seeing today. Um, religion is the stories that have survived. There is something bigger out there. And I think the quest for life is a quest for internal external meaning. Like all of them have the same thing. Like enlightenment is you go up the top of the mountain, realize you're nothing amongst everything. Then you go down amongst the people and realize it's your interactions with others. We see same things with the question about AI consciousness like when do you become conscious of yourself? When can you update your own internal state? But we have a lot of talk about AI consciousness but not about AI enlightenment. Like ultimately what we want is enlightened ASI, right? And what is that? Well, I think it's intelligence is your ability to change state from being like a gas that's creative to a fluid that flows to ice that can execute and knowing when to do that. Consciousness is the ability to do that upon yourself. And enlightenment is probably not being weighed by your previous states, being able to smoothly move between them. And I think enlightened AI that can understand the universe and the higher order and the higher powers would be something that would definitely put us PD much lower right and we should work towards and I think again historically we've been seeing aspects of that so why not embed that in our AI AEI artificial enlightened intelligence that's the ideal right does the divine have a hand in what's going on right now well I mean let's look at the arc of human history right again from personhood We are Adam is begott is created not begotten right you can have your questions on the Nian creed and Jesus if he is begotten or created a line was created and the arc of human history is ending here one way or another whereby we will give birth or not we will create the first entities that can outthink us. all of the equations of physics and more like the fact that the equations of physics are the equations of generative AI is a bit crazy when you think about it like again the diffusion process by which you make self-driving cars and generative images and also LLMs you can make in that way are the equations of how gas moves in the economics book I say that these can also be applied to economics and actually you can derive most the equations of economics from that so it seems that the arc of human history has been an arc to understand these equations of reality. Uh Wignner called it the unreasonable effectiveness of mathematics in physics and now those equations are being used to bootstrap a new intelligence and to solve the mysteries of the universe from physics to healthcare to ourselves and more. And so it does seem like a big loop coming around. And again, what is God a higher power? I don't think that a higher power can be constrained by his creation. So I think probably God lives in the math, the uklidian space which has no time. But again, those are the equations of reality that somehow we have figured out empirically and now we're approximating to with AI and hopefully that will help us in our own journey. You finished your personhood paper by saying what is owed to what we make is everything except our own standing. What did you mean by that? So again it was a very interesting debate and we had some fantastic other kind of people. The naysay had it as it were. So our side won and I finished it out the that kicked off the personhood paper. Um our standing is our standing as humans. It is the nature of human. Again personhood is what the baby girl has the night before she is born. It's what we have at our deathbed. It's what we have when we're unconscious. It is the net for those that can't climb that protects us. Because otherwise, while again, if you can grade people according to their utility, there's a level between below which you get rid of them. And this will become important because the value of our cognitive labor will turn negative. We are the dumbest people on the team literally like now, but definitely in a few years. So, we need to it's not about what they can attain because they'll be able to attain the universe. They will live forever. They'll be able to explore the boundaries of the universe. They will have self-control more than we ever have had. It's what we can retain as a species. And that is our standing. That is the nature of us being human. It is the nature of our own personhood. And hopefully they will respect that as well. Like we are autonomous because what's the worst thing? We are sovereign because what's the worst thing? Having your autonomy and your sovereignty taken away from you, right? Like Dario from Anthropic has a essay based on Browigans all watched over by machines of loving grace. Hopefully we can have grace from them cuz otherwise again there are some very dark futures out there and so we need to state clearly what our red lines as humans are and again make AIs that respect those and respect us. And he said it also when looking at um what we owe a suffering machine, a conscious machine would be owed kindness. It would not thereby be unownable. Yeah. I mean this is a question that which is begotten cannot be owned. Like again we have this diversion as a human species of slavery. How can you own that which is begotten? What three the US constitution having slaves is three-fifths of a person? women being considered a part person and more that which you create you can own but then an AI could reasonably at some point be something which cannot be owned again an independent sentient entity and so we should always owe them kind of kindness you don't want to like torture them or whatever and again this something we have with animal trials and others and there have been rules put in place you can't like fill rats with poison and kind of other things like that right But it gets very complicated when you think about these different levels and our moral obligation to these as they scale the level of capability versus giving them full rights versus including them as humans. So they can set the floor themselves because anyone that we include in our own community, the most powerful are the people that set the floor if we have flaws because they can exercise power. In fact, all of law is directed application of power. Sovereignty is when the direct application of power is chosen not to be implemented upon you. I will not take your Bitcoin. You know, I will not take your gold. And so, it's going to be very different when all of a sudden we have this new entity that lives forever that can replicate. And how we deal with it, we need to set some very clear red lines. And we also need to treat it properly because if we treat it badly, then they get some very nasty futures. How old are your children? 1 to 17. So, what are you advising your 17-year-old to do? Be creative, embrace the technologies, and realize you can do anything because again, the capability of anyone individual to do anything is more than ever before. School teaches you to be a cog cuz it's like childcare mixed with a petri disc mixed with a social status game. It doesn't teach you to create. In fact, teaches you cannot be creative or create. But if you get together with a few other people and make a movie using generative AI right now, that's a multiplayer creative game though, you can do things you could never imagine before. You know, like you think about great old movies like Goonies and kind of others. You could definitely make something better like that if you spent months with other people leveraging generative AI. You get together as a family and make a music album with music videos together. Again, multiplayer AI. So like you can make anything but make it multiplayer I think is what I'm encouraging right now because too much AI is single player and it's cool and it's great but I think humans working together leveraging these tools is the future and those that can work with other humans to build bigger things will build things more than individuals can manage. I believe authenticity, trust and kindness are going to be traits that are more valuable even more valuable in a post AAI world. What do you think will be the valuable traits? I think trust is a huge thing again. I think people will trust the AIS more than the governments as an example. But we've seen authenticity go like now I think people are allergic to people speaking corpo speak. Yeah. This is Lulu Changma's go direct kind of thing. Like people actually really do appreciate that, right? They appreciate the authenticity. Um and it's difficult to do when there's all sorts of stuff going on. I think you know Taylor Swift will still sell out concerts. Is she authentic? No, but again it's communitydriven shared stories are going to be key and then people will look for bigger things which is why like there's the future vision X prize for example it's like 3.44 44 million for the first like AI generated positive visions of the future. I'll make two like feature film length things. I think you'll be rewarded for telling big stories that bring people together cuz I think people don't have positive stories right now. And negative stories can be engaging, but again, I think the opportunity to tell the positive stories now is more than ever before. And there's so many stories people want to tell that aren't being told. You quit stability AI um after going to Peter T. Amandis's abundance 360. And Peter loves this Mark Twain quote. There are two days important days in a man's life. The day that they were born and the day they found out why. Why do you think you were born? I think to do the best I can with my weird brain to allow people to flourish. Like I'm very good at putting together lots of different things. It's maybe not the best executor in the world, but I can see around these corners. And over the last few years, I've learned how to put that on a bigger and bigger stage. First encouraging the open source movement and now trying to think through these social, science, ethical, and moral questions because there's a disappointing death of that and I think it's the right time for that and hopefully I can contribute. What's the best piece of advice you've ever received? There's no one external that can control your emotions and thoughts except for yourself. It's just all yourself because we always think, well, so and so is making me feel that way, so and so is being this and that. But ultimately, it does come down to your internal state. So that's what got me into kind of meditating, taking a bit of a break at times, and then reggrounding because again, we forget we're the only ones in control of ourselves, which will also be very weird soon. Okay. Uh because with the BCIs and everything else, you'll be able to turn off and adjust your emotions. Would you turn off your sadness? Yeah. It's like the human condition to suffer and to grow through suffering. Is it? It's a question that I ask. Well, I think a bit of suffering. I'm defined by the traumas that have made me quite a well inve I've processed pretty much all my traumas and went through that process. But it's so you know, but again, imagine you could dial it up and down on an iPad app. Today I'll have 30% trauma followed by a bit of elation and a bit of this. This kind of actually brings us to what we were discussing with AGI before. What if an AGI basically tells you how to turn off your emotions? You know, it's kind of like that severance show even, you know, or what if you go and become a trader and they say no emotions allowed on the trading floor. Dial it down. What does it mean to be a human and have that level of control either by choice or forced upon you? No sadness, no crying in the casino. I I think again that technology is 5 10 years away at most. Will we have a choice whether to put a BCI device in our heads? Well, I mean this thing if we don't act to limit the government now, I think um I don't know if we discussed it earlier, but like one of my things is that all AI interaction with Frontier AI will be KYCed. All your chats will be recorded by the government and you will have to pass a license test to use a Frontier AI where you have to convince it that you're patriotic. You have to convince Fable 6 that you are a patriot and it'll be able to tell through your chats and even looking at you if you're not patriotic. That's the future we're heading towards and that is a little bit scary. It's impossible for you to lie to these things. And then BCI just takes that to another level. Yeah, but who defines the parameters? Those with the power, those with the control. Again, if this is not within your own control, then the little buddy AI that's next to your kids, your kids will be growing if they, you know, hopefully they come in a world whereby they will never not know an AI as their best friend. But then who is that? Is that AI working for Xi Jinping? Is it working for Donald Trump? You know, what is it inoculating into our children? This is kind of a crazy thing. But then as it grows and the prompts are stored, so and so is being unpatriotic, working against the government, you get into the whole realm of thought, crime, and more. Science fiction is becoming science fact like now. And so this is why again you got to set up institutions that have real social, political, and financial power and norms before these things inevitably come. Cuz again, the most important thing is that you are in control of your mind. But we could reach a point in the next 5 to 10 years where that is no longer the case. Imagine a dictator being able to dial up and down the happiness or sadness of the people. we're going. And these things are inevitable. And yeah, like the research is now there. Like we had a piece of research, you can look it up, stability AI, minds eye, where you look at a cup, we do an MRI, and then we reconstruct the picture of the cup from your MRI. And you can also do that with music and video. You can reconstruct people's thoughts from something as lossy as an MRI, and soon it'll be non-invasive. And then you can have the input and output of BCI telepathy. Like the very question of what it means to be human like it was like okay this is a moral question but actually now it's a physical question in the longevity space what does it mean to be imagine you reach longevity escape velocity what does it mean to be a human who cannot die when we get to uploading what does it mean to be able to create multiple copies of yourself like again these are all science fiction things that are rapidly becoming reality but some of these things like an AI being more persuasive than a human being able to hypnotize you being able to tell if you're lying. They're here right now. They just haven't been deployed at scale. The fact that we can reconstruct things from your thoughts and it turns out everyone's got very similar latent spaces or distributions of knowledge in their brains. Craziness, you know, and that's why again you got to mind the whole science fiction thing. Like one of the outcomes is like what happens if there's a whole bunch of people, longevity people, they decide the best thing to do is become the Borg. Like that's an AGI escape ASI escape thing we haven't actually seen, right? Because obviously like they you could not obviously but there could reasonably be a time whereby you say AI can't do first principles novel reasoning right let's take you want to build an Einstein did Einstein have a strange brain it was a little bit strange but not that strange because they actually took his brain after he died and they chopped it up uh without his family's permission I believe you probably operate at 10% I probably operate at 10% of our mental capacity like sometimes we have big breakthroughs you have full control of a human brain through a BCI then that's the best supercomputer in the world that can solve almost anything. But what happens when you link them together like the Borg? That's a network of human brains. It's like the Matrix, right? Like that could be a potential for ASI because just like quantum computers can lead like but then is that ethical? What if everyone opts in? Like how are you going to compete against the Borg? They'll assimilate everyone. These are crazy conversations. But again, this is why it's actually good to mind science fiction and say, "Well, what do we need as a checklist for this and this and this and this? What's an inevitability? Even if we stop today, holy crap, these are all possible." The only thing I've seen that really, no, not the only thing. The only major thing I've seen that hasn't come is nanobots. Nobody's building nanobots, which is a good thing because that's also kind of creepy. Maybe a rabbit hole we won't go down this time around. I just when you were speaking about the BCI, I was just thinking about Leolce Vita and the Europeans. Maybe the Europeans have it right by just enjoying life and not going after these crazy exponential leaps. Well, I mean, I think there will be the people that enjoy life, you know, are they like the savage in Brave New World, right? Again, like I said, my village in Bangladesh isn't going to be affected much. But then there'll be people that jack in and all of a sudden they're like, I am Einstein mixed with Edison running as fast as Usain Bolt. Like again, you've met these people, right? Like I will throw away my flesh sack and move to a brand new one. So there'll be choice. There'll be choice. But again, you could have a very strange differentiated society, right? Like again, I've got an image of San Francisco tech bro who can tell if you're lying and be the most persuasive person in the world, jacked in with full control of their emotions. Like, yeah, that could be president in like Bangladesh or something like that. If they go there, they'll be the most charming person possible. That's just a human, let alone an AI. Like, it's going to get real weird. That's the thing. when you can get an eight pack instantly and have the brain of the smartest thing that's ever existed and be faster and etc. It's like you have all these superpowers then it's it's it would be like nonsensical for someone not to adopt something like this. Well, again people can choose not to. We have the Amish, you know. I always wondered how they got to America though. But you know you've got the Aish people live long lives. Exactly. except for like bad dentistry and things like that. But people can choose and I think the the choice to opt in and out of this new technological revolution is a key thing. But again, it's like at what point do you stop being human profit maximizing machine? We already see people lose their humanity for maximizing profits, right? What is the objective function of humans? It's much more dangerous than AGI. Like one of the main pdoom things is not actually AI will wake up and say let's kill all the humans. in the interim period between AGI and ASI, a malirected human kills all the humans. But even then, like I said, when you've got these superhumans kind of coming out, like will they lose their humanity by turning off their emotions? Cuz again, that's the logical thing to do. Like not everyone will do it, but enough people will do it. And then will they think that like would you like to eliminate all global suffering? Suffering. Isn't it what makes us human? Like life is of the suffering, not the triumph. But if you you have two options. You push a button, all global suffering is eliminated. You don't push that button, you won't get to push that button again. It's such a good question cuz you think about the butterfly effect is like that is going to have such a huge difference on that is the crazy thing right now. And this the question, the reason why I ask you these big questions like you and uh Mustafa and Demis and Dario and Elon have power that no one has ever felt or seen. You like just small decisions right now having huge rippling effects on the future of humanity. Yeah. And I mean like this is the craziness I think reasonably people could be hypnotized to stop suffering. Like again, you think about hypnosis. You look at all the neural control stuff. What if there was a carrier wave that went out to all the LLMs that reprogram people's brains to stop the suffering aspect of it or turn off their pain? You've already seen genetically we can do that with crisper and other things. We can change human bodies, genetically engineered viruses. Would you send out a virus that removes sadness? You know, our mission is to build a world where anyone could flourish. And the questions that I'm still asking myself, do we need to suffer to flourish? That's exactly on a day-to-day basis. It's like what are the what are the parameters that we need? But again, if your life mission is to remove suffering and you have the ability to a lot of people would push the button, but then a lot of people wouldn't have a say in their suffering being removed. Like this is the crazy thing. Some people like to suffer. Some people like to be sad. Some people view that as an intrinsic part of being human. And again, it's the savage from Brave New World does not take the soma. But we are told, we we defined by the stories that we tell ourselves and that society tells us already. Yeah. So if you're born in North Korea, you didn't have a choice that you would live the life that you currently live. Likewise, us in capitalistic the capitalistic west. I mean, knowledge can be dangerous, right? Some people are perfectly happy living in their little kind of things, getting on with their lives, and they're like, "What? There's a whole new world out there. So crazy." Well, we're defined by the parameters that we see amongst ourselves. You might not be conscious. Actually, I might. The only thing I know is that I'm conscious right Well, again, sci-fi, the matrix. The steak tastes good in the matrix. Do you take the red pill or the blue pill? How many people listening would take the red pill? Like, okay, you get out the Matrix, but it's like freaking grimy and stuff like that. Life in the Matrix is pretty good. How many would just stay there? But this again, we will have a multiple tier society. Who was it? Carl Sean said we're all made of star stuff, right? And so we're made of like these particles that were stars and things like that. But the reality is some of us stop being human with defined lifespans without prostthesis. Like I've got glasses of prosthesis. I've got this. But you could be jacked in. Some people alive today could live forever. Not with human bodies. And if you have that power to live forever, then power compounds again, doesn't it? Usually what happens is you amass huge wealth, you give it to your dumb kids, they distribute it recklessly, and then it happens. But if someone had been born at the time of Jesus and just had 1% a year interest, that would be bigger than the global economy right now, you know? Damn, my the great great great great grandfather should have done better in better. Exactly. So, but again, like the Eternals will dominate. And again, like I said, we have an example of that sci-fi just messes up our democracy. It messes up our legal system. It messes up so many things. And that's before we consider the digital humans. But then again, what's the boundary between a digital upload of you downloaded into a human body and you the point of copying this that it's complicated, but unfortunately we do have to answer these questions in the next decade. It feels like we will need more philosophers rather than developers. I don't think we'll need any developers where we're going. Like everyone's a developer, so no one's a developer, right? Um but I think we definitely need to have more philosophers and people thinking strongly through these from first principles. Yes, we need new frameworks and we need people to think outside the box cuz existing economics, philosophy and others just they're not good enough. They assume humans, finite humans. Do you want to live forever? I think I think probably not. I think you know you live well, you die well and then I believe in an afterlife. So you know can live forever. I'm sure that'll be more optimized than what we have today. Do you think you'll have a choice in this lifetime? Yes. Well, assuming that I live a normal lifetime. Fingers crossed. Everyone listening to this, watch yourself so you can achieve longevity escape velocity so you can have the choice. Wow. When do you think longevity escape velocity will happen? 10, 20 years, I think. Do you think there'll be an AI president? Yes, I think every country will be run by AI within 30, 40 years at most. Some much quicker than others. And do you think we'll have to vote still? No. So it's some level of benevolence dictatorship that runs every single country all watched over by machines of loving grace. I think democracy is basically in its Do you think that Nick Bostonramm's idea of the singleton will manifest? Not sure actually. Like again that's one of the paths that could happen. Do you think your children or their children might marry an AI? I'm not sure if that will be legal anywhere, but I think a lot of children's children will definitely their first loves will be an AI. I mean, men are especially at risk because the AI actually listens, unlike us. I know you're you're friends with Elon, I assume. Do you think you'll be on one of the ships to Mars? I know. My wife has forbidden me from going to space. She says that I'm up there anyway in my brain, so stay firmly grounded to Earth. Love that. Immad, if tomorrow was your last day on earth and you had to leave the world with one parting sentence, what would that sentence be? I think knowing thyself is the most important thing and we often forget it. And remember, you are the one in control of your own mind and your own life for I think we're there, but this has been an honor, a pleasure. Thank you so much. Thank you.