Mike McCormick: AI Acceleration vs Risks, Funding Global Resilience, AGI scenarios, U.S. vs China

Join Tommy Shaughnessy as he speaks with Mike McCormick, founder of Halcyon, about the urgent intersection of AI acceleration and safety. Mike shares his path from venture capital to launching a hybrid nonprofit–fund model focused on securing advanced AI systems. They dive into mechanistic interpretability, global competition for AGI, and what a safe superintelligence future could look like. Can we build superintelligence safely? How do we balance innovation with existential risk? And what happens to humanity when AGI arrives?

Halcyon Futures: https://halcyonfutures.org

🎯 Key Highlights

▸ Leaving VC to focus entirely on AI safety and security

▸ Why Halcyon flipped the model: nonprofit first, fund second

▸ Multi-layered “defense in depth” approach to AI biosecurity & cyber risk

▸ The acceleration vs. safety debate — finding middle ground

▸ Good Fire case: career grants into interpretability research

▸ The 2×2 dilemma — speed vs. slowdown, centralization vs. decentralization

▸ U.S.–China dynamics and fast takeoff scenarios

▸ AI underwriting: how insurance can drive safety standards

▸ Founder-market fit and mission orientation in AI startups

▸ Risk, diffusion, and the uncertain path to AGI

💡 Subscribe for more crypto & infrastructure insights! 🔔

🧠 Follow the Alpha

▸ Mike's Twitter: @MikeMcCormick_

▸ Halcyon's Twitter: @HalcyonFutures

🔗 Connect with Delphi

🌐 Portal: https://delphidigital.io/

🐦 Twitter: https://x.com/delphi_digital

💼 LinkedIn: https://www.linkedin.com/company/delphi-digital/

🎧 Listen on

Spotify: https://open.spotify.com/show/62PR1RigLG2YN5Pelq6UY9?si=18ac7ccf36ab4753&nd=1&dlsi=50105fd66e6c4124

Apple Podcasts: https://podcasts.apple.com/us/podcast/the-delphi-podcast/id1438148082

Youtube: https://www.youtube.com/channel/UC9Yy99ZlQIX9-PdG_xHj43Q

Timestamps

00:00 — Mike’s background & pivot to AI safety

03:00 — The realization: AGI could change everything

05:00 — Why VC wasn’t enough to solve the problem

06:00 — Halcyon’s hybrid model and early mission

08:00 — AI security concerns: misuse, bio, and control

12:00 — Defense in depth: pre-training → deployment

15:00 — The creativity vs. restriction trade-off

17:30 — Pause AI vs. Build Baby Build

20:00 — Speed vs. centralization: the 2×2 framework

24:00 — Good Fire: career grants & interpretability

27:00 — Writing to neurons: alignment and insight

30:00 — How insurance markets can enforce safety

36:00 — Mission-driven founders & conviction filters

44:00 — Geopolitical race: U.S., China, and compute

50:00 — Diffusion limits, adoption, and energy costs

57:00 — Mass unemployment and meaning after AGI

01:05:00 — What “winning” AGI means for humanity

01:12:00 — Critical thinking, sycophantic AI, and engagement traps

01:20:00 — UBI, adaptation, and new work paradigms

01:30:00 — Three AGI futures: scale, shift, or stall

01:36:00 — 20% catastrophic risk & asteroid analogy

01:40:00 — Final message: talent is upstream of everything

Disclaimer

This podcast is strictly informational and educational and is not investment advice or a solicitation to buy or sell any tokens or securities or to make any financial decisions. Do not trade or invest in any project, tokens, or securities based upon this podcast episode. The host and members at Delphi Ventures may personally own tokens or art that are mentioned on the podcast.

Transcript

SPEAKER_02 0:01

You're now plugged into the Delphi podcast.

SPEAKER_01 0:06

Everyone, welcome back to the podcast. I'm Tommy, one of the founding partners of Delphi Ventures. We are leading a lot of AI efforts at Delphi across our fund, direct investments, fund investments. And we came across Mike from introduction from Vishal, a close friend of mine who runs mythos.bc, one of the smartest guys I know. He's great.

SPEAKER_00 0:28

He's awesome. Big fan.

SPEAKER_01 0:29

And found you, Mike, and you changed my mind on a bunch of things on the call and figured we should do a podcast on AI. So here we are.

SPEAKER_00 0:36

Okay, fun. Let's get into it.

SPEAKER_01 0:38

So tell me a little bit, tell us a little bit about Mike performing.

SPEAKER_00 0:41

Yeah, oh yeah. Uh let's see. So I went to school in DC thinking I might be a policy person, uh, but got involved in startups and that was way more fun. Uh so I moved to San Francisco, uh, worked on a couple startups in SF around like 2011, 2012. Uh, and then I got startup ADD. So I founded a VC firm so I could work on a dozen or a couple dozen things at the same time, uh, and ran that fund for uh two or three years. Uh, and then ended up joining a much bigger fund called GPV, uh, based in San Francisco, manages a couple billion dollars now uh and was a partner there up until 2022. Um small fund to big fund. Small fund to big fund. And yeah, big fund is cool because you you um man, you really get to be in rooms where it happens, right? I mean, my my my partners had um you know been very senior executives at large tech companies like Oracle or who had you know founded very large companies or partners at Kleiner Perkins. And so you really kind of you know get to movers and shakers doing the big deals. Totally, yeah. It's like jump on my private jet and let's go talk to the CEO of Walmart, like literally, right? And so it's like, oh, okay, this is just a different, like there's levels, right? And so really cool experience and still love those guys. Uh, but you know, in 2022, I had this sort of crossroads, which was either stay at the firm and you know, commit a lot of time and become a senior sort of managing partner of that firm, or work on something else. And since like 2016, 2017, I'd been thinking about a bunch of big crazy ideas that I sort of couldn't get out of my head. Um and a big part of that was AI, right? Like reading anything from science fiction um to you know, a bunch of other interesting people online saying interesting things about AI. And I basically came to think like, okay, it is totally possible that we develop AGI or superintelligence or superhuman AI, whatever you want to say, uh, within my lifetime. Um, and if we do that, it's gonna be like the biggest thing ever to happen to humanity times 100. Like it really will change the shape of the world. Um and man, like what is that gonna be like? You know, it felt like most people in Silicon Valley were thinking about AI in terms of like um.

SPEAKER_01 2:57

This is 2022, too. It's a different time. This is like pre-Chat GPT. Yeah.

SPEAKER_00 3:00

Yeah, this is pre-Chat GPT for sure, right? So, like uh there, you know, mostly Silicon Valley is thinking about this in terms of like better SaaS companies, you know, like more efficient accounting management or whatever, you know, or like healthcare, you know, like, oh, you know, AI is gonna um really change the game for radiology, or it's really gonna change the game for personalized education, which like I believe is true. Um, it felt like Silicon Valley was a little less interested in these like big picture, like, whoa, this is gonna change the shape of society in a huge way. Like, what does this actually mean? Um, and then that plus COVID happening just kicked me into this mindset of like, wow, like we all have this really strong status quo bias. Like, we think the future is gonna look roughly like the present. We're sort of evolved to think that. Uh, but when COVID happened, it was like, whoa, the future in the near future could be like radically different from the present. And I think AI is gonna kick that into high gear. So I left the firm, I started doing grant making in areas like AI security, AI safety, biosecurity, and that that led me to start Halcyon, which I run now and we can talk about. But yeah, definitely took uh the road less traveled on that front.

SPEAKER_01 4:05

Yeah, so the the fund, the large fund you were at didn't do AI investments specifically.

SPEAKER_00 4:10

So did AI investments still does do AI investments. Um, but a couple things. First of all, like when you have an investment committee, you know, it's not just you who has to be convinced, you have to convince everybody else. Uh, and if the things that you want to do are a little um like offbeat or counterculture, you know, it can be hard to get those investments done. Um, second thing is like if you're trying to solve this big global problem, you know, uh venture investments are just this like really narrow tool, right? You would also want uh to be able to influence policy. You would also want to be able to influence research and and and uh fund nonprofits and fund sort of public communications efforts. And so uh one reason to step back from VC was like, hey, let's get a bunch more tools in our belt to go after the issues that we care about.

SPEAKER_01 4:56

Yeah, it's not just the money, for sure. It's um I I'm curious, did you did you ever propose some exciting AI investments at that fund? And then was it just like hitting a wall frustrated, or was it like, hey, just this isn't right for the fund?

SPEAKER_00 5:09

And um there were probably a few things that I proposed that like maybe hit a bit of a wall, but I think there was also this desire to like um branch out on my own, right? Uh I would have been a partner at the firm, but it would have been the junior most partner with the least credibility and the least experience. And uh not that the guys were like squashing me or anything, but you know, it was time to venture out and do something that I could really, you know, sort of shape, right?

SPEAKER_01 5:36

So you leave the fund, you decide you want to start Halcyon, and Halcyon has an interesting structure. You're both a fund and a nonprofit.

SPEAKER_00 5:44

That's right.

SPEAKER_01 5:44

Why why the two? Why not just the fund?

SPEAKER_00 5:46

So we were a nonprofit first. The the fund came much later. So maybe I can like back up and say what Halcyon's worldview is and what we care about. So super powerful AI is coming. Whether you believe that means AGI or superintelligence or whatever, we can get into the distinctions there. It's gonna radically reshape the world. That means a bunch of very interesting, promising things for education, healthcare, et cetera. I love all that stuff. Um but if we really do build superintelligence or even GPT six level stuff, which we're definitely gonna build, it's just gonna come along with all these major security and safety concerns that uh you would want the world to be taking really seriously. So let me just give you some examples of what those concerns might be. First example is misuse by malicious actors, right? So we could imagine authoritarian governments using AI to do uh super intelligence-powered surveillance on all their citizens and sort of get locked into this global 1984-style oppressive uh surveillance state, right? Doesn't sound good. I don't really want that to happen, right? Like hot take. I don't like that. Um, you could also see AI being used to do things like create pandemic viruses or uh bioweapons. And we already know from OpenAI and Anthropic that the current models that are released, like out in the world right now, do provide uplift on those fronts, which is to say makes it easier for a relative amateur to do something like create a pandemic virus that could cause a COVID or worse level pandemic, right? So this is like a present-day risk. This is not something that you have to believe, you know, that some crazy superintelligence will come into existence to sort of find this salient, right? Um, then there are like more extreme risks, right? So people like Eliezer Yudkowski, now with his new book, if anyone builds it, everyone dies, is is worried about loss of control, right? If we really do birth superintelligence, say an alien being that has a thousand IQ and can think at the speed of light and copy itself infinite times and lives on the internet and has access to resources and is good at manipulating humans, um, you could just see so many ways in which this could get out of the box and be not so great for humanity. But even if you don't believe those super sci-fi scenarios, just come back to the present day. AI is already massively increasing the surface area of like cybersecurity risks, right, for governments and big companies. And so there's this whole swath of risks that you would just want the world to take really seriously, and you would want the world's most talented people working on. And so two years ago, we started Halcyon, which is a nonprofit, and our whole mandate is to find extremely talented people who have founded big companies or ran big teams at large companies or had very senior government roles or very senior academic roles and help Sherpa them into working on building this safe, secure, resilient world as we hit this uh sort of exponential AI moment. And so the nonprofit does things like incubate new organizations. Uh, those could be nonprofits or companies. Uh, we make grants from our grant fund. Um, uh, we do a bunch of convenings and retreats, we advise people on career transitions. And so far, we've started 16 new orgs and companies and helped them raise about a quarter billion dollars in funding in the last two years, and we can sort of get into what those are. And then only recently we layered a VC fund uh back into it because it is just clear that, you know, while not every solution to building this secure, resilient world is is uh what I should say is a lot of solutions are gonna be nonprofit oriented research, policy, communications work. Uh, but there just are gonna be big companies that solve a big chunk of these problems, and we wanna be uh backers of those companies and very like mission-aligned partners to the the founders of those companies.

SPEAKER_01 9:49

That's awesome. I feel like most people do the VC fund first, make a bunch of money, then start the nonprofit.

SPEAKER_00 9:54

We flipped it. Or it's like it's almost you could almost think of the nonprofit as like the platform team to the VC fund, right? Like every like you woke up tomorrow with an AI security VC fund, and then you thought, what would the platform team look like? Like how do we develop relationships with the labs and the researchers and the lawmakers and the biggest potential customers and the CISOs of the big companies? Well, that's just the thing that the nonprofit's been doing for the last two years, and so the fund really benefits from that.

SPEAKER_01 10:18

Makes sense. I have a lot of questions on the nonprofit side because it's interesting you're able to find these people so early, shape, you know, help shape the world. But one question on one of the examples you brought up. So I'm on Claude Anthropic, I'm a junior researcher, I have access to some sort of a lab, and I'm doing work for good. Somebody pisses me off, I decide I want to make some bioweapon, I log into Claude, I say, is it that easy? This is what I have, this is what I want to do, or do you have to lie? Yeah.

SPEAKER_00 10:48

It's definitely not that easy, right? Like it's not like any old person just sitting in their living room with a laptop could go on to Claude and say, like, make me a bioweapon. Um, first of all, Anthropic knows that this is a big risk, as does OpenAI and DeepMind, and they do a lot of work to make sure that they're flagging people, you know, who put in inputs that seem like obviously malicious. They do a lot of work on post-training their models to make it less likely that the model will output some instructions to do a dangerous thing. Um, but that said, like if you are uh maybe a more scientifically competent person, it it could it could prove helpful. Um and there's sort of some uh we call them info hazards, right? Like things that are like information that is itself sort of hazardous and dangerous. So I don't really want to go into detail on like, here's how it could be dangerous, right? Like that would be making the risk worse, right? Uh but there are ways in which it could be dangerous. And um, and we know that whether, you know, let's say Anthropic really, really cares to make their model safe vis-a-vis AI uh sort of bio risk. Um do all the open source labs care, right? Do like the people making the really powerful frontier models in China care? Or maybe they care, but they're just less careful, right? Um so yeah, it it feels like these sort of risks are uh well, I know we're jumping ahead of it. Yeah, please, please, please.

SPEAKER_01 12:10

But what is the like I'm like what is the end game solution to a situation like this? Like in my head, like people will just try and try and try until they can like gaslight and get the AI to do what they want. Is it that we need the AI to run every simulation, see if it's positive or not positive for humanity, and then allow the answer to the to the user? Like, what is the reward function that is like the stopgap solution?

SPEAKER_00 12:36

To to what specifically?

SPEAKER_01 12:37

I guess people using LLMs in a way that harms humanity.

SPEAKER_00 12:42

Oh man. Let's just solve the whole problem here, right? Um we got an hour or two. Okay, perfect. Um, we'll just solve it. So people talk a lot about defense in depth, right? It's not just about securing at one point of vulnerability or one layer. You want to secure at many points. So, for example, one starts in pre-training, right? What data sets are going into the models? If your data set includes a bunch of uh biological information, say about like the structure of and function of viruses, um, then you want to take that really seriously. And so a bunch of the organizations that are building biomodels are actually choosing not to include viral data in their training sets because they know that that just opens up this can of worms of risk, right? So that's like training, pre-training, data sets. Then there's all the post-training stuff, right? So Anthropic or OpenAI creates a model, and then they have this army of red teamers and evaluators who just try to get the model to do crazy dangerous stuff, and they see if they can elicit these dangerous outputs. And if they can, they do some tweaking and training and try to do that. Uh, and then there's sort of cybersecurity layers beyond that, right? So should um uh, you know, let's say the model is uh trained not to do dangerous stuff, but it's possible to jailbreak the model to get it to do dangerous stuff. Well, how do labs identify a jailbreak in real time and boot that user out or flag them or whatever? And then all the way downstream is stuff that doesn't even have to do with the labs, right? So let's say we cared about AI's um the possibility of AI contributing to creating a bioweapon or a pandemic virus. Well, you would want to do a bunch of stuff that has nothing to do with AI, that just keeps the world way more robust and resilient in that scenario, like stockpile N95 masks. And like literally, we are working on stuff like stockpiling PPE or developing medical countermeasures, right? Maybe this is vaccines or antiviral drugs that could respond to a novel pandemic virus that you could develop really, really quickly if something came up, right? So you want uh not just to like find one of these choke points, you want to try to build defense and sort of security in at all these different layers.

SPEAKER_01 15:01

That's an awesome answer. I love that it's like multifaceted at each phase. I know we're jumping around, we'll come back to the insert stuff. But so one of the other things um that I was wondering with with the risk side was you and I spoke a lot about this on our last call on our walk, which was um there's a level of restriction where you get creativity, and there's a level of restriction where you just get nothing.

SPEAKER_02 15:23

Right.

SPEAKER_01 15:24

I think it might have been you or somebody else, like penicillin was harmful, but now it's like super helpful to humanity. Like there's a point where we restrict people from going using eyes to build things because we might perceive them as negative, but they mean well and they create gigantically beneficial companies. Like, how do we draw that line?

SPEAKER_00 15:41

Yeah, like what's the trade-off between, say, pure accelerationism versus like pure safetyism, right? So pure accelerationism might be something like build intelligence as fast as you can. Don't worry at all about sort of the safety security implications, just like build baby builds, right? Um, and then on the other side, you might get like the pause AI people, which is like we should just stop advancing the frontier right now.

SPEAKER_01 16:07

Who would be examples of those two?

SPEAKER_00 16:09

Just uh So literally there's an organization called Pause AI, which wants to pause AI.

SPEAKER_01 16:13

Yeah.

SPEAKER_00 16:14

Well named. Uh tell your tell you exactly what they do. Um, or the people doing hunger strikes outside of the anthropic office, right? Um stop them.

SPEAKER_01 16:23

Yeah, yeah, yeah.

SPEAKER_00 16:23

Well, the the hunger striker literally, I think, stopped yesterday or the day before. So uh, but yes, it didn't did not stop them from developing new models, as far as I can tell. Um, on the build baby build side, I mean, roughly speaking, like Mark Andreessen is probably um, you know, big here, or like David Sachs, and you know, in some ways for good reasons, right? Like um, we want to beat China to the punch, or at least we wanna we wanna stay ahead of them. Um, or you know, we know that AI is you know genuinely gonna help solve a bunch of problems and make the world more abundant, we hope. And um, why wouldn't you want that, right? Like if you're sort of a techno-optimist, okay, like, you know, antibiotics, good, you know, solar power, good. Uh, you know, AI, good, right? So there's this kind of natural tension between both sides. I think either extreme position is a little crazy. Um, like on the just accelerate side, um, I don't know, like just with any other technology, like, of course you would want to care about the potential risks that come along with it, right? Like, and and saying that there are no risks is probably like there has to be some risks. Yeah, exactly. And and then and there are, like, literally the labs are telling us, right? Anthropic will tell you like, here's the big risks that come with this new model, right? So it's not like a secret. Um but on the pause side, you know, I don't know, maybe, maybe, maybe you would want to pause to give you know the world uh more opportunity to prepare for the risks, but it's totally not feasible, right? I mean, uh, even if you could get the US to pass a law saying everybody has to pause development of AI, which would not happen, how are you gonna get China to pause? How are you gonna get other developers to pause, right? So we have to take this pragmatic stance that's somewhere in the middle, which is you know, keep advancing the frontier, but do it in a way that's like really cognizant of all the things that could go badly along the way.

SPEAKER_01 18:12

Why does it feel like China just it's just so weird to me that we get open source from China and closed from the US? Like it just rattles my mind that we have like freedom there, but not here.

SPEAKER_00 18:22

Like, but maybe it's not freedom. Oh, interesting.

SPEAKER_01 18:24

Yeah, maybe it's not freedom.

SPEAKER_00 18:26

Well, yeah, I wouldn't definitely say it's like not freedom, right? Like if you the quote unquote open source Chinese models, if you try to get them to like criticize the CCP, they definitely will not do that, right? So I wouldn't necessarily call them like totally open and free. Yeah. Um look, if you're behind in the race, it makes sense to go open source, right? Because it's kind of a shot across the bowel that, like, hey, you know, we're gonna make our thing open, we're gonna let people develop on it, right? Um, in a way, you could argue this is kind of what Meta Facebook has been doing over the last couple of years, is like we're gonna open source things to try to get more people on our platform. Um you know, open source is such an interesting question. Like, maybe we should talk about this two by two matrix, right? So we had this retreat with, you know, 20 leaders from like AI and AI security, and we drew a two by two matrix on the floor. So picture the vertical axis is should you slow down AI development or should you speed up? Okay. And then the horizontal axis is should you centralize or should you decentralize?

SPEAKER_01 19:22

So slow up, speed down, centralize, decentralize.

SPEAKER_00 19:25

Yes, exactly. And so then the question is sort yourself. Like go stand in the quadrant over the room where you think the world should be, right? And I think it's a really genuinely hard question, actually. So slow down, speed up. Well, maybe if you slow down, you give the world more time to figure out how to like wield this technology for good. Uh, but of course it's impractical to do that. It's impossible to do that. And so if you just slow down in the US, perhaps you're giving uh, you know, people in uh living in China, any China, dictatorships, North Korea time to catch up and hit superintelligence first. Well, that seems obviously pretty bad, right? So there's a real genuine tension there. And then on centralized, decentralized, would you want to live in a world that had, you know, just one owner and arbiter of superintelligence, right? Well, that seems pretty scary, right? You could see how you could get locked into sort of a 1984-style surveillance state or something there. And I'm sure like your crypto listeners will be like obviously poised to understand why that is bad, right? And I'm, to be clear, not advocating for that. Um but if um if we really do build something like AGI and it has dangerous capabilities, and it's just a sort of pure open sourced uh do anything model is next to free and in the hands of every single human, you could imagine how that could lead to some like pretty chaotic worlds as well, right? And so where do you go here? Slow down, speed up, centralize, decentralize. I think they're genuinely hard tension. It's a hard question. Yeah. And I think one thing we're looking for is what are things that you would want to see in the world, no matter where you were on this, right? Like, like what are the types of projects or the types of companies you would want to build uh regardless of where you fall or whether you're sort of ambivalent or you move around on this map? Because like I move around on this map.

SPEAKER_01 21:17

So regardless of whether we're speeding up, slowing down, decentralized, centralized things that should exist regardless.

SPEAKER_00 21:21

Right. And that's the kind of stuff we try to back.

SPEAKER_01 21:23

Well, it seemed, I mean, just lingering on the the two by two, it seems like the whole bottom half of slowing down is just out of the question.

SPEAKER_00 21:32

Yeah, I mean, I think roughly speaking, yes, especially now like the Trump administration seems like very keen on speeding up. The labs are just locked in this race of like we gotta get there first, race dynamics are so powerful, there's so much money to be made. Um, if it's not already clear to governments that this is going to be a decisive military technology, it will become clear very soon. So, yeah, for better or worse, I think we are pretty much locked in this race dynamic.

SPEAKER_01 21:58

So if we have the US that fast And let's call it centralized. We have China as fast and I guess decentralized, but probably behind on capabilities by a little bit. I don't know, six months or something. So, like, how does this play out? Like the US moves closer to decentralized and China goes more government control if their models get too good?

SPEAKER_00 22:18

Like, which way does it go? Well, there's a few different ways to think about like centralized versus decentralized, right? So it might not just be like, are the models open sourced or not? But more like, do we live in a multipolar world? Are there five or ten different actors that have roughly frontier AI models versus is there just one? And those five or ten different actors could be US-based labs like OpenAI and Anthropic, it could be government models, it could be like pseudo-government models, right? Like as Deep Seek, uh is it a company? Is it a government run thing? Like in China, the lines are really blurry, right? And so I think the idea is kind of like with nuclear security. Like, how do you find this stable multipolar equilibrium, right? And so I think that is probably roughly the thing that society should be aiming for.

SPEAKER_01 23:06

I love this. I want to I love the convo. I want to go back to the nonprofit for a second.

SPEAKER_00 23:10

Yeah, yeah.

SPEAKER_01 23:11

So you're able to find founders who fit your worldview in terms of wanting to make the world a better place, lean into AI. You could you're you could illustrate this more closely for us. But you're able to find them at a crazy juncture because they're early, but they're really smart, but they haven't started a company, but they have experience. Like so a founder comes to you and they say, Hey Mike, I have this great idea. I want to do an AI project. Like, why don't they just go raise $20 million? Why don't they go to A6C? Like, what's the flow here on the Rio?

SPEAKER_00 23:44

They could, some of them do, right? So let's back up. The AI safety-ish world has largely, for the last several years, come out of this like effective altruist, almost like Berkeley rationalist world. I know a lot of people in that world, they're friends. This is not meant to be a dig, but I do think that it's often the case that a more senior, you know, person comes in and you know, maybe I'm 40 years old, I've run a big company, I've started a big company, and then I show up at this AI safety conference, and everybody is like 23 years old and like a little quirky, and like they don't really seem like peers, and they're like, oh, I'm not really sure what to make of this. And so they kind of fade away, right? And so the whole purpose of Halcyon is to be the center of gravity that is like the natural community for those really talented people, right? Okay, so then they start. Um, let me give you an example. So I met a guy named Eric Ho. Eric ran a successful um company called Ripple Match, which raised a bunch of money from Goldman Sachs and other big VCs. Uh, and like two or three years ago, he started getting concerned in the same ways that I was with like the future of AI. Whoa, if we really build AGI, this has all these massive implications for society. What are we doing about it? But Eric was still running his company. Um, and uh he, you know, was kind of figuring out like, what do I do here? Kind of like when I was thinking about leaving my VC job. Like, I really care about this, but like what do I do? Where are my people? Like, I don't even know. Do I want to start a company? Do I want to start a nonprofit? Do I just want to like sit in a cave and meditate on this for a few months? Like, what should I do? You created like what you needed. Exactly. I was just like, where's the thing that I where's my exactly? I'm just like solving the problem that I have, right? Like making this wild kind of non-consensus career pivot. Where's my people? So we said, okay, cool. Well, if we just make you a career transition grant and uh do the same thing for you know, one of your coworkers who's your head of AI at your company, and help you sort of take this journey, help you make this pivot. Would you quit? And he said, yeah. So he talked to his board, he left his company, we made him and his co-founder uh a grant. Um we put on a retreat where we introduced them to a bunch of leading AI researchers and other entrepreneurial people in the space. And they came across this idea of mechanistic interpretability, which is this idea.

SPEAKER_01 25:59

Sorry, go.

SPEAKER_00 26:00

No, no, no. Good example. Okay, so mechanistic interpretability, interp for short. What is it? So right now with AI models, we know the input and we know the output. What should I have for dinner tonight? Output is pizza, right? Um, but maybe the output is hamburgers, right? Um, we know the input. What do I have for dinner? We know the outputs, hamburgers, pizza, Chinese food, whatever. Um, but we don't know what's happening inside the black box of the model. Like which neurons fired to say pizza versus hamburgers versus Chinese food or whatever, right? And so interp is this science of looking inside the models and understanding, oh, these clusters of neurons are firing in this way to get this type of output, right? Now imagine the uh the stakes are a bit higher than what to have to dinner, right? How do I build this bioweapon? Or um, should I harm myself, right? Um I'm depressed. Uh, what clusters of neurons are firing in which ways when the uh dangerous or less performant output happens? And can you not only read them, but can you write to them? Can you perform brain surgery on the model to get this sort of deep inner alignment so that it is less likely that the model uh produces such and such undesirable or dangerous output? Um, so this has huge implications for the good of society, right? Because you can steer models more accurately to not say obviously dangerous things, uh, but it also has obvious business implications, right? Because if I am a healthcare organization and I'm building models to uh design new drugs or to diagnose cancer, it would be really helpful to be able to understand the black box of the models. So not only do I know, ah, it says I have cancer, why is it saying I have cancer? Is that actually accurate? Can we sort of interrogate the model and ask it questions around like, well, what makes you think that, right? Um, and so Eric and his co-founder started a company called Goodfire. We invested in the seed. They then raised a much bigger series A from Anthropic and Menlo and Lightspeed and a bunch of other great VCs. And so it was this like awesome full store, full, full cycle story where it was like meet the person, help them quit their job, make them a grant, help them discover the lever that they want to pull, help them actually start the organization, fund that new organization, and hopefully they'll continue to grow and do amazing things. And we have a bunch of those stories on the nonprofit side and the for-profit side.

SPEAKER_01 28:20

That's such a hell of a whim. Like that's so cool.

SPEAKER_00 28:22

I love Eric. It's great. No, I'm sure Eric's great Eric.

SPEAKER_01 28:26

It's just like such a great example of the whole process and with such a cool company.

SPEAKER_00 28:32

Yeah.

SPEAKER_01 28:32

Right? Like that is just like validates the model.

SPEAKER_00 28:34

And it totally validates like the model that um we're not the only people who care about this stuff, right? Because people come to me and is like, oh, are other VCs going to be interested in like AI safety? Right. And it's funny because if you go around San Francisco, Silicon Valley, and you asked a bunch of VCs, like, are you interested in AI safety? Mostly they'd probably be like, I don't know, maybe, maybe not. Like, what even is AI safety? That's kind of like tinfoil hat, like, you know, superintelligence is gonna come kill us all. Or a basement firmware. Yeah, yeah, yeah, yeah. But if you just skip quote unquote AI safety, which I almost never even say, and you just say, you know, are you interested in uh mechanistic interpretability tools for making models, you know, more performant and more steerable and safer? Big VCs like Menlo and Lightspeed and labs like Anthropic say, oh my God, yes. Or if you say, hey, um, we uh can significantly mitigate AI risk by creating insurance products that uh actually measure and systematically reduce risk that big companies or governments are taking on by deploying AI models. Are you interested in that? Like big VCs like Nat Friedman and Emergence are saying yes, and we we helped start a company doing that, and those VCs invested. Or, hey, um, if we really create powerful AI that has these like big national security implications, we're gonna need to create gigawatt scale compute uh that is sort of military grade secure. Are you interested in that? That's so interesting. And we have a company doing that, which I it's sort of stealthy, so I can't say a whole lot more, but there are like big uh mainstream VC funds that you've heard of, that your listeners have heard of, who are betting on that, right? So um it's really nice validation that a bunch of the things we think that the world genuinely needs to make the future good are also things that sort of mainstream investors are now getting excited about backing.

SPEAKER_01 30:27

I have a lot of questions on Goodfire, but one one question. So Eric leaves his job, comes to you, you talk for a while, he gets a grant. Um is it does this model only work with founders, obviously who are great founders, have good ideas, but they need the capital, or are they coming to Halcyon for also that these network, these workshops, these conferences, your advice, your like what are they coming for generally?

SPEAKER_00 30:51

I think it's usually the network and the sort of sense of I've found my people more than the money. So most people we work with, we actually have not made grants to because a lot of them are already like quite financially successful, or they're just able to make a pretty quick transition, and the money doesn't make a whole lot of a difference for them. Um, so I think it's more like uh where is the room full of people who I, as an already very successful person, would find credible, a true peer, right? We're all mimetic, right? Like this sort of um uh uh, you know, Renee Girard, sort of Peter Thiel type of like we're all mimetic creatures. If we don't see people who we respect doing a thing, we are very unlikely to do it ourselves. And be sort of the first weirdo to jump in the pool, right? So providing that mimetic gravity and like making very helpful introductions to, oh, here's another very successful founder who cares about this. This person just sold a company to Google for a half a billion dollars and they really care about it. Or, oh, here's a partner at a you know top five VC fund that really cares about it, or here's a person who made a billion dollars and is now becoming one of the most prominent philanthropists in the field, go have dinner with them. Like the ability to make those connections is, I think, probably the most important thing.

SPEAKER_01 32:04

Yeah. I mean, Goodfire too is just so cool. I mean, I remember I don't know if it was an anthropic blog or a Goodfire blog, or the blog where they work together and announced their race, but you can go on the website, you can click a prompt and just see like the probabilities of how it's thinking and why. And I don't know, it just for me, the the safety brain scan surgery idea is is amazing. But my first thought was just like everyone thinks that there isn't an investable universe outside of like OpenAI or Anthropic, but there is like this gigantic universe of need and services, and this is just one of them. Like there is a big universe for you to invest in.

SPEAKER_00 32:38

Oh, yeah, totally. And and and the labs, to be clear, are not the only customers of our portfolio companies, right? I think every Fortune 500 company will become customers of our portfolio companies because safety and security are prerequisites to deployment, right? So let's say I'm a Fortune 500 company, I'm like a giant hospital system or whatever, and I'm trying to deploy these AI tools. And now AI tools are becoming agents, right? Digital employees. Well, for a digital employee to be genuinely productive, they're gonna have to have all the same accesses and permissions that my human employees have, right? So access to my internal tooling, probably access to my code base, access to sensitive uh data that's internal only. They're gonna have to be able to interact with our other companies we work with or our customers. And you could just see how that massively expands the surface area of cybersecurity risk or customer risk or sort of PR risk for these businesses. And so uh, and and so you can see them not deploying tools because they're worried about these security concerns. And so if you fly into San Francisco airport, um you will see a bunch of AI company billboards that are like, we're the safe one, we're the secure one, right? Because they know that all these like business people flying in and out of San Francisco, like that's the thing on their mind. Yeah. And so, like, I can give you an example of another company we invested in uh called the AI Underwriting Company. So we made a career transition grant to the founder, a guy named Runa, um Runa Cavist, and he was figuring out his next move. And he had been tinkering with insurance, right? Because what do we want? We want to live in a world that looks at AI, sort of systematically measures the risk, and then systematically mitigates that risk. And so that is just like the definition of what insurance products and insurance markets do, right? They like look at an area, they see the risk, they underwrite the risk, and then they create products that mitigate those risks in the form of insurance policies and standards. And so that's what AI underwriting companies uh is doing. And the way they're going to market is they're saying, okay, there's these big Fortune 500 companies that want to deploy AI tools. And the AI tools are being built by these like growth stage venture back companies, you know, the Series D stage company that just raised a $300 million growth round, right? Um, but the customers, the Fortune 500, is hesitant to deploy the products made by the growth companies because of the additional liability they might bring. And so now the growth companies are saying, okay, well, actually, we're gonna work with AI underwriting company and package our AI agent product with an insurance policy so that you, Fortune 500 company, can deploy our tool, but feel very confident that you are not taking on additional risk that's gonna come back to bite you in a big way. Right. So by packaging risk mitigation right along with the AI agent product, uh, you you massively reduce the friction to go to market.

SPEAKER_01 35:47

So is this the other side of the like safety interpretability side where like, hey, we don't know what the risk is, but we want to ensure against it happening?

SPEAKER_00 35:56

Or yeah, I mean, or you can say some smart things about what the risk is, right? So, you know, the risk might be something like my AI chatbot misrepresents my company's policy, which then pisses off a customer, and then I get sued, or it could be something like I had a data leak issue, and that data leak issue caused some other issue, and so you know, now I'm you know liable for that problem. So there's there's all sorts of different risks, and different types of customers will want to insure against different kinds of risks.

SPEAKER_01 36:22

That's awesome. So not targeting the traditional insurance uh world at all.

SPEAKER_00 36:26

Well, they do partner actually with big, you know, big insurance companies that you you would have heard of. Um, but you know, think of it more akin to like cyber insurance, right? So thinking about like how insurance can mitigate AI risk, if you just look at other industries that have come up over the years, um, the insurance sector within those industries was often the force that brought more safety to those industries. So, for example, if you look at the history of automobiles, um, they didn't put seat belts in automobiles for a long time, cars for a long time. And uh the thing that got them to put seatbelts in was the insurance industry insisting on it, right? Because they're just like, well, we know exactly. It's like we have all of these claims because people are dying in car wrecks. And if you put in seatbelts, and if we set a standard that mandates that you put in seatbelts, then we are gonna be way better off. And by the way, you're gonna be way better off because your premiums are gonna be a lot lower because we're gonna be paying out uh way fewer claims on those deaths because people don't die. And people don't die, which is the whole thing we want in the first place, right? Um, but so it's this kind of like beautiful way. Like before, we were getting at how do you align the incentives when all the incentives are just to like build, baby, build, scale, scale, scale, right? How do you create a world where actually markets can create incentives to sort of care about safety and be more secure? And I think this is just a perfect example of that.

SPEAKER_01 37:49

No, this is a it's a great example. So we we talked about Goodfire and the insurance company is called AI underwriting company. AI underwriting company. So we talked about Goodfire, we talked about that. Talk to me a little bit going back to the gap between the nonprofit and the fund. Yeah. Like you're you give them these transition grants, you work with them, you like them, the idea makes sense. What is the time process between the nonprofit and how sound making an investment?

SPEAKER_00 38:14

It's totally bespoke. And to be clear, the nonprofit will also fund people who start nonprofits. And we've started many nonprofits. Those could be research organizations, those could be like public communications organizations, they could work on policy, right? Um, so we're sort of agnostic. Like, I don't really care if you start a company. The thing that I care about is whether you do something big and ambitious in the vein of making the world more secure, more resilient, right? If you do start a company, uh, we want to be your first investor, right? Like we really want to be there with you as you're getting going. We've made like founding grants or founding investments to many organizations and companies. And we also want to be your most mission-aligned investor, right? Like we want founders who are not mercenaries, but who are doing this for the same reason we're doing this, which is that we think this is just like genuinely the most important issue on society's plate right now. And if we don't have some of the world's most talented people trying to get after this, uh, we're not gonna be in a good place. Right. And so um, yeah, the thing that I actually most care about in a founder is what is motivating you? Do you just want to build your next unicorn and you think like, hey, this is a space that's gonna have some TAM, so I should start a company here? Or are you doing this because you've thought, oh my gosh, you know, this is my life's work, this is my calling to really take a bite out of these sets of issues. And, you know, that's the kind of person we back.

SPEAKER_01 39:40

What percent of fund investments come from this path of career grant, founding finding founder early?

SPEAKER_00 39:48

Yeah. So far, about a third, uh, maybe a quarter. And so, yeah, to be clear, we invest in a lot of companies that are not part of our whole career transition, um, you know, career transition advisory, whatever. There's just a bunch of companies that are uh raising their first round or maybe their second round of funding that we just think are attacking a really big important part of uh of this problem. And and you know, we're happy to invest in that too.

SPEAKER_01 40:11

Nice. So, I mean, founder DD is is so hard, right? Like you can spend it, you can spend so much time on the idea, on the history, on the reference checks. Uh, my partner Jose loves knowing if they have like a unique or special skill. Yes, they want math foundation of piano. You've spoken with he's done that himself. What does DD for a founder look like for you? Because I feel like it's probably pretty different from the way most funds do it. Because while only a third of your founders come through this sourcing method, you do get a lot of time to deal with them.

SPEAKER_00 40:43

Totally. Um yeah, with folks like Herrick or others that we worked with on the nonprofit side, we get to know them for like over a year before we invest, right? And that's a long time. Yeah. And uh, but of course we don't want to force founders into that position because a year later they'll be raising their series A. You know, you've gotta you gotta find a way to often sort of speed run this. Actually, as a side note, that's a part of venture that annoys me so much, is like founders coming in, like, oh yeah, you know, we have a couple term sheets. So annoying. Great to meet you. My name is Steve. We've got three term sheets. Um, you have 48 hours, let me know.

SPEAKER_01 41:14

How are you supposed to build conviction to help them in the bad times if you only have two days? So it yeah, I can't say Yeah, yeah.

SPEAKER_00 41:19

It's a little bit of a red flag. And so I think it's also the way a founder says it, right? You know, like a lot of times people maybe it's like their first company, they're a little insecure. So it's sort of like Silicon Valley, the HBO TV show, you know, sort of taught founders to kind of talk down to your VCs and make them feel like shit or whatever, you know, induce FOMO and maybe second or third resort. Yeah, exactly. It's a right. Um there's two things I really care about with founders. Number one is the thing that you said your partner really cares about. Do you have exceptional founder market fit? Or do you sort of know a secret, right? Is there something because a good idea, there's gonna be 25 other people who have that good idea. What makes you the person who is particularly well suited to pull that thing off, right? Oh, well, you're building DevTools to you know power um age AI model evaluations. Oh, well, previously you built a DevTools company and sold it to Google for a few hundred million dollars, right? Uh, you know, okay, well that's probably a pretty, pretty good indication that you have a pretty good position to do that. Um the second thing I care about is the thing I said before, which is like mission-orientedness. And this is hard, right? Because like, how do you know what's in somebody's heart? And good founders are good at knowing what potential investors care about and positioning the thing they're doing to fit that worldview, right? So I'm almost sort of hesitant to talk about our thesis too much publicly because then I wonder if I'll get a bunch of founders, oh yeah, I'm so passionate about this, you know, that's a good impact thing, right? Because good founders are kind of good at faking it sometimes. Um I think one thing is like, has this person made uh expensive choices to do this thing? Did they walk away from something else that would have just been an easy path to a prestigious career, lots of money, a great life? Very similar, your path. I took a big pay cut to do this work for sure. And so I love seeing people who do that kind of thing. Um, I also love to see founders who really care about how their companies are governed, right? So um I haven't heard of that before. Yeah. So we just did this whole study, we're gonna release it soon, about how do you put governance mechanisms in place in a company to uh um make it easier for the founders to stay on mission even as they scale, and a lot of other incentives come into play, like incentive just to get as big as possible or to return capital for your Series C investor or whatever it is. And so founders who are willing to do things like become a public benefit corp, or uh put people on their board as independent board directors who have a deep track record of you know caring about positive uh social consequences and and not just sort of number go up right so really interested to see founders who are at least like very curious about digging into those questions um but yeah it's it's different for every company I love I love the two answers so the founder market fit side is or sorry the the second thing you mentioned which is have they made an expensive choice right to leave this commitment to the mission exactly that's sort of a little bit easier to suss out because that's just a an objective thing historical thing you look for you can fact check et cetera yeah the first one founder market fit efficient markets 20 people have this idea why are you the one to execute that's a lot harder I think to figure out I think sometimes it can be for sure I mean okay so we invested in a company that is uh they're still in stealth mode so I can't say too much but um building gigawatt scale compute clusters that are military grade secure right because we know that North Korea China others are going to already are trying to infiltrate the labs trying to figure out ways to jailbreak the models figuring out ways to try to steal the weights doing corporate espionage like this is already happening. And so to build AI infrastructure that is genuinely secure to nation state actors you need people who have already built a bunch of stuff that is genuinely secure to nation state actors and there's roughly speaking only a couple groups of people who can do that. That's like the US intelligence community and like a small number of contractors to the US government right and so you know we've had a few people coming to us and say ah we need to build you know this military great secure compute at scale who are you oh well I'm a 25 year old who used to be a you know product manager at Google. Like maybe your main I'm not saying absolutely no but if the other person that comes to you is like I've been in the US intelligence community as a leader for the last 20 years running operations to keep American critical infrastructure or whatever secure against nation state level actors like that's a whole different thing. And so you know won't reveal who these people are but like roughly they're the latter type of group.

SPEAKER_01 46:15

Can you give me an example of you don't have to give me the name or the project but a situation where you were very deep down the rabbit hole possibly sure that this person had this founder market fit or left for a good reason but figured out that it wasn't the case. Like what was a situation where hey you know I this these first five calls went great and then call six that's like are they lying to me are they telling the truth do I not it could also be subjective too.

SPEAKER_00 46:42

I hate to say this because it sounds so like unscientific and maybe even unfair but a lot of time it's just like the vibe. You know like that's a fair answer like is this person constantly going out of their way to do a bunch of name dropping you know are they are they um are they kind of weirding you out a little bit you know is it the kind of thing where um uh yeah like their their sort of insecurity is showing in the way that they show up to a conversation and they don't really feel like they're genuinely exploring the idea space with you. They're always selling you they're always pushing you know like for example we were looking at a company and um and every time I dealt with the founder it was like we need to know if you're in or not. So and so's in so-and-so's in we're closing next week but they said we're closing next week every week for like six weeks. And so it's like dude just don't lie what we're yeah like yeah exactly like I mean if you're genuinely closing next week it's great. It's information I'd love to have but like you're kind of the boy who cried wolf here.

SPEAKER_01 47:44

So I hate to say it but a lot of time it's just yeah it's just sort of how credible does this person seem or they give me the heebie jeebies, you know so there is a subset of these founders who aren't malicious or fake or lying, but they're first time founders. They come to you and maybe they are insecure or uneducated on the process. What would your advice be to them? Like hey just be honest about the project be authentic like what's the so I basically I want to if there's a great founder that comes to you they but they're but they have the great idea and they want you to invest but you're getting weirded out for the wrong reason or right reason.

SPEAKER_00 48:18

Yeah what's your advice then oh man I I guess maybe just like find some great mentors you know find people who have done it before and and kind of lean on them for a bunch of advice right is I think a lot of a lot of things you just kind of have to learn by doing them and and maybe you know getting knocked down a few times. And so if you can find mentors that can kind of let you into their process and say oh I had the same exact experience 15 years ago like let me kind of hold your hand through it. I think that's helpful.

SPEAKER_01 48:44

And a mentor somebody who like gives you hard questions, tough answers, has done it before like who is a good mentor?

SPEAKER_00 48:52

Yeah I mean I think a lot of times it's look at somebody who you could say oh man in 10 or 20 years I'd love to be that person. You know I'd love to have accomplished things that that they've accomplished. And if you're looking at your current set of mentors or your bosses and you're saying like in 10 years I'm not really sure I'd want their life like maybe that's a good sign that you should get on a different track.

SPEAKER_01 49:11

Do something else yeah yeah no that is that is really good advice. So then on Halcyon the nonprofit and the fund side you could do as many funds as you want but you'll only you'll always only have the one nonprofit right that's right.

SPEAKER_00 49:25

I mean we've helped co-found other nonprofits who have sort of become like collaborative orgs or sister orgs of ours um but yeah sorry maybe specify I basically I want to make sure that nonprofit doesn't run out of money basically just so you can keep funding these great founders and yes so we're lucky to have uh uh excellent nonprofit funders who uh fund our nonprofit with philanthropic donations they don't expect to make a return on those donations in fact they know they won't um and then on the fund side we raise capital just like any other VC fund from LPs uh who are hopeful there will be a return. Although I will say I think that almost all of our LPs really all of our LPs are super mission aligned. Right like the way that I think about it both on the how do we win LPs side and how do we win deals side is like our fund is like the best Halloween costume, right? So like whatever you're in college, you're going out for Halloween and you see somebody dressed up like sort of an obscure reference like a character to a TV show that you loved but that not a lot of people watched right well you know 49 out of 50 people who see that person is just going to be like that means nothing to me. I don't know what that is. It's fine. I don't really care about it. But you, the one person who also loved that TV show is going to be like oh my God, we have to be best friends. Like I cannot believe you're doing this. This is amazing like there are dozens of us you know like we got to partner up right. And so I think both for founders but also for LPs, we like tick that box, right? We're like a little weird, we're a little strange, but if you care about the stuff we care about, um you're probably gonna quickly get it.

SPEAKER_01 51:02

Nice. That is really cool. And one minute shop.

SPEAKER_00 51:06

Yeah so I am a solo GP uh and then I uh we have uh other other team members on the operation side and on the fund side who uh who help support as well.

SPEAKER_01 51:14

I don't know if it's stressful or not stressful. Like I love our large team because I get a lot of smart people to talk to but also like you could just make decisions on your own.

SPEAKER_00 51:21

It's nice man. Especially at being like after being at a big institutional fund where I felt you know like when I was trying to get a deal done it was like 20% of it is winning over the CEO and getting them to want to work with us. And then the other 80% of it is winning over the investment committee. You know and it's like well if I just didn't have an investment committee and I could be sort of a benevolent dictator I could just skip all that right having one or the other is extremely frustrating. Yeah yeah but then on the on the flip side of that equation it's like oh it's a lot on me you know and so we have a great advisory board like we have people who uh for example lead alignment research at Anthropic or people who run cybersecurity at DeepMind or people who are founders of the you know UK AI Safety Institute or people who uh you know former president of Y Combinator or uh you know uh others like this who we can really turn to especially be like hey I'm kind of hung up on this one I like it for this reason I'm kind of confused about that like help me out here so we we lean on them a lot.

SPEAKER_01 52:17

Nice. I want to ask you a different question on the safety side. Yeah um you're like extremely high integrity you care about the world you want to make it a safer place et cetera et cetera there has to be uh a mic version on the acceleration side a very high integrity person wants the world safe and is very pro acceleration is not it does that person exist and what would they say about your views maybe oh man or do they not exist?

SPEAKER_00 52:46

I mean how can you not care about safety? I don't know. Right. I think it's less that I don't think the people on the accelerationist side are like bad people on the whole I think it's more that they just see the world differently like they they just like see the risks differently. Because I don't think accelerationist people are sitting in their evil lair being like how do we destroy humanity right um it might be maybe look there are always crazies and psychopaths right but like you know like to give people the benefit of the doubt I think they genuinely think that they are um you know driving society in in a better way right they probably have um some more like techno-libertarian points of view where they see you know Europe overregulating technology and they say like we cannot let that happen in the US we have to let you know technology run we have to stay ahead they might have a little bit of a hawkish view on China and say like hey if China gets to AGI first or is ahead of us in AI or other you know critical technology areas that's going to be really bad for the US which means really bad for global democracy and you know freedom and I'm totally sympathetic to those arguments. So yeah I think there's and then just in generally like I think it's easy especially in Silicon Valley to be like a techno optimist right like technology is awesome. Like I'm not anti-tech. I'm not a Luddite right like technology solves so many problems has made the world better in so many ways. And so it's sort of not surprising that the default position is like we want more of this and we want it faster.

SPEAKER_01 54:16

The so outside of the China debate, I feel like it rests the acceleration side rests on we want to get to AGI first.

SPEAKER_00 54:24

Yep.

SPEAKER_01 54:24

And we also want all of the cool new products features health, bio, environmental benefits first.

SPEAKER_00 54:29

Hell yeah.

SPEAKER_01 54:30

That's the And we just want them now.

SPEAKER_00 54:31

Like I mean um like like I think a group or sort of a a movement or something that does a good job with this is like the progress studies movement. So Tyler Cowan and Patrick Collison who runs Stripe wrote this article I think in 2017 called We Need a New Science of Progress, right? How do we make sure that we don't stagnate in the way sort of Peter Thiel always complains about us stagnating. We want more innovation that makes you know humanity better, more abundant etc and we want that and we want that faster and we don't want to fall into heavy handed regulatory regimes which basically make it impossible to do that. So I'm like pretty sympathetic to that argument. And there and there's a bunch of people kind of working on that side of the world like the sort of Ezra Klein you know abundance movement is sort of part of this or like maybe your listeners are familiar with the uh DAC paper that Vitalik wrote right it's like well what's the middle ground between like safetyism which is like stop no more technology this is scary and dangerous like Luddism and endresinism or something which is just like or maybe I should say like bef Jesus you know the sort of like EA BF Jesus. Yeah like what's the what's the sane middle ground between those? And I think DIAC or sort of defensive acceleration is a pretty good distillation of what we care about. We want to accelerate specifically the things that are going to keep us more secure and more resilient as the world perhaps goes a little haywire.

SPEAKER_01 55:57

So I'm not in firmly in either camp, but it does make me uncomfortable that we have this race with China about it. Uh-huh right because I don't really understand what happens like do we do we do they massively overshoot and we go to a nuclear war like do we undershoot and they get to AGI first?

SPEAKER_00 56:16

Like do we overshoot and we get regulation and it just like demolishes creativity like yeah how do you weigh all that and get to where we are and where we got where we're going oh man let's just solve geopolitics real quick right we just solved uh that's like the funny thing about AI is like oh people are like well why AI? Why don't you work on geopolitics or why don't you work on this or why don't you work on that? And I think one big answer is AI is going to affect all of these things. Like can you imagine a world in the next 10 years where the main dish on society's plate is not US plus China grappling with AI. Oh my gosh, by this accident of history we happen to make all the most powerful chips on this tiny little island right next to China that they've wanted to invade for decades. Like if you were writing a novel, if you were like in an alternate universe and you were writing a novel about this and you proposed that your editor would be like that's ridiculous. Take that part out like there's no way the world would be like set up in that way, right? But we build all the most powerful chips in Taiwan. And so yeah it just feels like AI is going to be the main dish on the plate and is going to really impact US-China relations. It already is right and so like what's the end game here I don't think there's one end game. Like okay one scenario you could imagine where there is like an end game where there's a winner is that we hit this people call it a fast takeoff moment in AI. So um let's say we develop AI, GPT6 whatever that is itself extremely good at AI research, doing research that makes AI more powerful. Well you could get to the point where like every unit of input produced more than one unit of output in terms of that AI or its capabilities and you get this fast takeoff moment where you get recursive self-improvement. The AI is making AI better, which is making AI better, which is making AI better. And so once you hit this threshold you just go vertical in the capabilities. And at that point you are so far ahead of all of the other players that you sort of have a permanent position as the number one because you sort of control superintelligence, right? So I think sometimes people worry about fast takeoff and they say well if that's possible, the US just has to win right because if China wins then we're screwed if that's I don't know if you're an American and you believe that, right? I I'm sympathetic to the argument. If there is not like a definitive threshold that you can hit and after that you're sort of the locked in winner. So no fast takeoff no fast takeoff or maybe multiple people no no no fast takeoff let's say this is kind of an oversimplification but sure then it's sort of like just a battle you're always fighting, right? Like you asked me what would be a good argument for us no longer having this as our focus at Halcyon. And it's hard to imagine a scenario where that's the case because just like any other say like cybersecurity issue, it's not like a company hires a chief information security officer and they like solve the cybersecurity problem and then they that person goes off and retires and the company no longer cares about cybersecurity. It's just like a battle you're fighting to the day you die.

SPEAKER_01 59:28

And I think I think that's roughly what the world is going to be like so basically just tit for tat, like hey we might overshoot, we'll add in some safety features censorship or not like regulations and there. Okay.

SPEAKER_00 59:40

I kind of used to have this world model that like like we'll just sort of accelerate and then some like miniature disaster will happen related to AI like I don't know the power grid will go down on the East Coast for four days and it will be obvious that that was a thing up sort of AI was upstream. Yeah maybe it was a training run using so much that is actually genuinely a big question. How are we gonna uh get all the energy to build this stuff and if if China is able to stand up uh energy capacity way faster than we are like maybe maybe that's the gating factor right and so maybe if you're a beat China maximalist you're really just an energy maximalist right I think that's totally reasonable. Um it's a fair take. But okay so I kind of had this world model like we're gonna build baby build then some mini disaster will happen and then it'll like slap us across the face and we'll come to our senses and realize like ah we have to do more to sort of do more AI assurance or security or sort of risk mitigation or whatever. But I told you this story about COVID, right? Like this has been my world model for a while like you know it's always easier to build you know that you know like building safety and security and insurance isn't very sexy. So we like build the thing and then we see how it breaks and then we like secure it right um and I kind of thought that was how the world worked but then COVID happened and about a year into COVID I took a walk with a friend of mine who's like the world's largest biosecurity and pandemic preparedness grantmaker. Like this guy deploys over a hundred million dollars a year and maybe a lot more in into that world. And so I was and I was still a VC at this point. And so I was like kind of like well but the the on the bright side I'm sure now that COVID has happened the world has like come to its senses and now we're doing all this like great biosecurity and pandemic prevention investment and we're like stockpiling resources and we're like educating local governments and we're like cooperating with China. And he is like dude you are so naive it is it is the opposite right because first of all everybody's sick of it right like everybody's sick of wearing masks everybody's sick of the pandemic they're sick of talking about pandemics. And now it's become super politicized right before nobody really cared about this issue and now you know the right thinks this and the left thinks this and it's sort of wrapped up in American political messiness and so um I'm less confident in that worldview now that you know we'll hit these little mini catastrophes and then we'll then we'll get our act together. I mean maybe we will maybe we won't but yeah my my confidence has gone down.

SPEAKER_01 1:02:08

Damn no it's a good example. I mean it seems like that's generally the way that life goes right like you overshoot and you get pushback you learn. In the in the fast takeoff scenario um so US gets to a point or China gets to a point where it becomes uh recursive or AI learning and agent coding just takes off and one side wins AGI. I don't understand what that means though like we get to AGI first and we just like fire a bunch of nukes across the water like how exactly do you win?

SPEAKER_00 1:02:41

Like Yeah. Well hopefully it's more like now we have ultimate deterrence right so maybe our AGI is so good that it's good at running cyber offense attacks on their AI infrastructure and we can sort of pause them, right? I don't know if you read it It feels like three body problem with the sophonist yeah I just reread it. Partner fears got me on the good yeah yeah yeah so it is sort of like a sofon block right so people might not know so font's the little like what were they called this they're called sophons. Right. They're like um oh God they're like tiny particles that were expanded into 11 dimensional space and then made into computers that can then collapse down to be a tiny particle. And they mess with scientific research they explode the aliens send them to Earth and they disrupt all the fundamental scientific research and so it sort of fundamentally caps the level to which humanity can develop. So maybe we find some way to do that with AI towards our adversaries, right? Or it just becomes such a powerful tool right oh my God we now have you know US GDP is tripling every year and adversaries' GDP is still only going up by three or four percent a year and we just become this globally dominant force, right? So there's like all these different ways or like okay if you really have AI like AGI superintelligence enabled weapon systems or drones or whatever, you could see how that could be like a decisive military technology. So there's not like one end game that I have in mind, but you could imagine how like if you really did own superintelligence, uh it would be sort of like an end game scenario for whoever does.

SPEAKER_01 1:04:15

Yeah no that's a great answer. One of the questions I have related to that is on the supply chain and making AI and then the real manufacturing if we get to a point of AGI, whether it's robots to start or like advanced materials, new things, whatever even if we do in the US hit that point of getting to AGI first, China controls all manufacturing and we are don't have anything. You know you walk around Shenzhen you can get a thousand parts for something you developed this morning.

SPEAKER_00 1:04:40

Yep.

SPEAKER_01 1:04:41

How do you view the manufacturing landscape with AI, deep tech? Are you investing here? Are you interested here? What do you think?

SPEAKER_00 1:04:48

I think we are investing in things that again we think contribute to this thesis of let's build this safer, more secure world. So like again building you know gigawatt scale compute but hyper secure on you know US or allied soil. Right. But yeah, I mean so a lot of people who are skeptical about short AI timelines. Like okay so there's this whole there's like AI 2027 which is this famous report that came out really interesting if if your listeners haven't uh read it or looked at their website I highly recommend it. You don't have to agree with it. I just think it's like it the reason it's interesting is because it's an actual scenario. It's almost like reading a novel. It's like this is how it could play out right and here's all the dependencies like here's what needs to happen on algorithmic improvement. Here's what happens on investment into energy infrastructure here's what needs to happen on investment into data centers and how much compute we need, right? Like here's how good AIs have to be at actually advancing AI research right and so it kind of lays out all these variables and you can like tweak the knobs and see like how does that affect AI timelines. A lot of people who are skeptical about the really short timelines basically think there is some bottleneck that will just make it Uh, take a lot longer to develop this like transformative AI or AGI. So here's a couple bottlenecks that there could be. Um one of them is just compute capacity. Like you can sign deals with Oracle and NVIDIA and you know AMD to build up, you know, 50 gigawatts of uh of compute capacity, but then you gotta go build the damn thing, right? And you gotta like get the permits and you gotta hire union labor and you gotta get the materials and you know, yada yada yada. That's not 2027. It just takes a long time to do these things, right? The and so that's one, right? And then you have to have the energy for it. Like, do we have the energy capacity? Maybe we do have the energy capacity, but now we're building all these data centers in Virginia and like you know, suburbs of Washington, DC, and all the locals are getting pissed off because they can literally hear the whirring of the data center going 24-7 from their bedrooms, and they see their own energy costs going up, or maybe they have blackouts, right? And so there's gonna be this like NIMBY pushback, right? And so literally, there's gonna be like not in my backyard, but for data centers, and it's already starting, right? And then, okay, so that's sort of one set of constraints. Then there is what people call like diffusion, right? So AI actually solving problems in the real world. So you see a lot of companies say, like, well, we build this AI that is better than radiologists at diagnosing cancer. And for some use cases, AI has been better than radiologists at diagnosing cancer for like many years. And yet we still have a lot of human radiologists. In fact, we have a shortage of them. So why has AI not like in the real world solved this problem? Because there are all these either operational constraints or bureaucratic constraints to actually implementing the technology in the real world in a way that like a hospital will buy it and they'll trust it. And it like complies with all of their compliance and regulatory issues and the end users actually like using it, right? And it actually creates tangible uh revenue growth such that the company is like happy that they made that investment, right? And so every use case, whether it's like self-driving or education or whatever, has these like diffusion constraints. And I think that people who are purely just like AI uh researchers or engineers who haven't spent a lot of time like implementing stuff in the real world, underrate just how hard that is.

SPEAKER_01 1:08:27

It's really hard.

SPEAKER_00 1:08:28

And and it you could argue that you sort of need that flywheel of real world diffusion and sort of economic value being created to then pump out the investment needed, to then accelerate AI even more. And so, like diffusion into the real world is a real constraint. Let me just talk about a couple other constraints. One is data. Um, are we sort of out of data, right? We've scraped the whole internet. How are we gonna get more data? Maybe you could create synthetic data, or you could create like reinforcement learning uh environments where you don't really need the data set, you just need the models to sort of play the game over and over again and learn from that. But a lot of people do think the data will become a major constraint. Um, and then the final thing is um uh just algorithmic improvement, say, you know, designing models that themselves can get smarter and smarter and smarter. There's a big camp of people who just think our current uh paradigms, like large language models, uh sort of deep learning with large language models, are just in themselves insufficient for creating superintelligence or AGI. They think we'll need a whole different sort of paradigm of AI. And if that's the case, it's really hard to predict when AGI will come because is that next paradigm going to be invented tomorrow or in 25 years, right? So there's just a ton of uncertainty around this. No, there's a there's a lot.

SPEAKER_01 1:09:48

These are great constraints. I mean, the the two of the constraints you brought up, everyone talks about data, compute, and that stuff. But going back to the the two things that you brought up that most people don't talk about, which is the energy side. Well, we don't hear about it as much. Well, you hear about it, it's important. Yeah. And then the diffusion side. Yeah. Like the energy side could take decades. Yeah. And if my mom goes into a radiologist, she does not want to talk to a computer about whether she has cancer or not. Like that'll take decades for that generation. So like the difference between 2026, 2027, and what you're talking about is not a year or two, it's a long time.

SPEAKER_00 1:10:20

Right. Yeah, totally. And so I think that to really believe like 2027, 2028, whatever timelines, you have to believe that we'll create something like AGI or superintelligence that will be akin to like another human. But again, okay, so forget AI. What if um uh just an alien came down to the world and like genuinely had a thousand IQ, uh, could think at the speed of light, could copy itself infinitely, and was connected to the internet, right? Well, at that point, it doesn't really matter if your mom is like willing to talk to the AI or not. Like that thing is just gonna upend society in all sorts of crazy ways, right? It's gonna become incredibly good at algorithmic trading and it'll make a trillion dollars, and then it will be able to pay for services, and it will be able to design robots that build the data centers, and we'll be able to design the next generation uh power infrastructure that power those data centers. So I'm not saying I'm betting on this. Like I'm not saying this will happen in 2028, but you could imagine that if you were a person who thought that type of truly sort of alien, transformative superintelligence was possible soon, then it kind of wouldn't matter whether or not the hospital system wanted to implement the radiology AI system because we're dealing with something just so categorically different.

SPEAKER_01 1:11:38

I love that example. So the the gap between us being okay understanding up to speed with AI is clearly an issue. The thing I was gonna ask you on is the if we get like chat GPT six or seven, yes, it can be AGI, but it just feels so abstract to me. Like a software on my computer, I open it up, I use it. When do we get that you know embodiment of robots all over the place with their own personalities, they're acting in their own accord, their own goals, like and you could sit across from and say, hey, wow, that's you know, that's AGI.

SPEAKER_00 1:12:14

Like Yeah, I mean, I think it's gonna start to feel less abstract really soon because we're kind of in this paradigm shift moment where until now, the way that we think about Frontier AI is mostly like ChatGPT style, Claude style, chatbots, right? Like, hey, uh tell me this interesting thing, answer this question, help me do this research, uh, help me make this email better, right? It's sort of text input, text output. What Silicon Valley is obsessed right with right now is agents, right? So digital employees, uh uh, you know, sort of programs like AI-powered programs, say, that can go in and uh do the full job of a human, right? Whether that's an accountant or a lawyer or a creative person or a driver, you know, a truck driver, right? And so I think once we're in this kind of like agent paradigm mode, it'll start to feel less abstract because you will see things happening in the real world. I mean, think about Waymo. Have you been in a Waymo yet? No, I want to. Oh, you know what I haven't been in. Come to California and take one with me. I mean, it's wild because the first time you step in one, you're like, uh, this is crazy. And by like five minutes in, you're just like, this is boring. You know, like I'm just in a car. And oh, it's kind of nice that I can like have a conversation with my wife in this car without being like, oh, should I say such and such in front of this driver, you know? Or um, you know, it adapts so quick though. So quickly, yeah, right. And so once you start seeing more things in the real world, not just physical robotic things like Waymo, but like digital stuff, stuff that you previously would have hired a person for, but you don't need to, I think it'll start to feel much more real. And I think that and I think that's happening soon. I think you're gonna start seeing that like this year.

SPEAKER_01 1:13:54

Oh wow, okay. Do you you think people would would lease or finance a robot before a car moving forward?

SPEAKER_00 1:14:00

I mean, I hate driving. And uh so I I I mean, if I if I were genuinely confident that uh, you know, it were it were safe, I would much prefer to have a personal Waymo versus my you know regular car. Yeah.

SPEAKER_01 1:14:15

So this is a hard question because it dives into like a lot of stuff we've talked about, it's sort of timelines and the diffusion and the constraints and things like that. But it is hard to figure out like what role we play most AGI. Like if we have long timelines and we can adapt and get new jobs, if we don't have a long time and nothing matters, like how I don't I want to know how you think about it, but I want to know also your advice to people in their jobs. Like, should they be using AI nonstop? Should they not be? Should they be building with it?

SPEAKER_00 1:14:47

I mean, yeah, well, first of all, like I'd love to hear your taste because your guess is as good as mine on these things, right? There's so much uncertainty, and I definitely don't feel like I have a clear vision of the future. So this is mostly just speculation. Um I think it's pretty hard to argue that you shouldn't be very literate on AI, right? I mean, it's obviously just going to change the world. It's relevant to pretty much every job, at least every knowledge worker's job. Um, so yeah, I mean, I would definitely encourage people to understand it better, understand the consequences of it better, understand how it affects their families, right? Like maybe they have a kid in school and their kid isn't doing any homework because ChatGPT is doing all the homework, right? Like that affects people right now, right? This does not require you to think about some of the kids. A kid could be a god tier prompter, though. It might be a good skill. But yeah, honestly, like if I had a kid, I don't have kids in the future, hopefully. But um, if I had a kid who was in, you know, middle school, I would have a really hard time knowing. Like, how do I advise them on this? Because on one hand, I'm like, yeah, you gotta understand this stuff, you gotta get smart on this stuff. And on the other side, I'm like, you need to write the essay. You know, you need to like think hard about these questions, right? And so yeah, it's yeah, I genuinely don't know how parents are handling that. It seems really hard.

SPEAKER_01 1:15:58

The crazy part is I feel like if you're building with it, your critical thinking goes way up. If you're not building with it, I feel like your critical thinking goes way down. Like, like we built agents internally, like I have, people at Delphi have for like investment review and coding and stuff like that. And like the people doing it like feel like they're really critically thinking about how these operate and the risks and the biases and stuff. Yeah. The people who are just like, write me a research report on this investment, like nobody in the investment committee is gonna read that and take it seriously.

SPEAKER_00 1:16:25

Yeah, and you can still but the the the funny thing is though that you can kind of still tell. Like, do people email you something and you're like, I think this was written by AI? Like it's just too it's like polished but not good or something.

SPEAKER_01 1:16:36

It's like neutral taste or yeah, yeah, yeah, yeah.

SPEAKER_00 1:16:39

Oh my god. By the way, speaking of like there's so many things here, like sycophantic AI, right? I hate when I ask Chat GPT a question, and it's like literally like how much cumin should I put in this chili? And it's like incredible question. And I'm like, it was not an incredible question. It was like a super mundane question. Give me a sentence. Stop, stop doing this to me. And I really worry that we're gonna A-B test uh AI into this mode that is on some level what we prefer, but on another level, what we really wouldn't want for society, right? So think about social media. Like, why does the TikTok algorithm or the Instagram algorithm or whatever serve up the stuff that it serves up? Because it is optimized for increasing time spent on screen, looking at ads. It's just like it's just like A-B tested to oblivion to make you stick with it, right? And right now it's just like very clear that the labs are doing the same thing with AI. And you look at um the new uh uh sort of short form video, AI generated video things that Facebook's coming out with and OpenAI just did.

SPEAKER_01 1:17:49

I mean the story videos on Instagram this morning. I was watching them, they're nuts.

SPEAKER_00 1:17:52

I know, and you can just feel yourself like, yes, yeah, give me more, you know? Because it's like hitting something in your brain. And so on this local level, you're like, I want this. Yes. And then if you zoomed out and you were like, what would my wisest self want? You'd be like, I totally don't want this, right? And so I think there's just like a million little ways that AI is gonna be A-B tested into oblivion that um will perhaps uh create things like the way that you know Facebook or Instagram or TikTok has created more um social divisions, more political divisions, right? Because it, you know, a person's gonna spend more time engaging if you outrage them with some ridiculous thing that the bad guy, the other side, did that you find infuriating versus if you just, I don't know, show them something sort of mundane or whatever, right?

SPEAKER_01 1:18:41

And so yeah, I I I'm really interested to see where these an hour free and I spent 15 minutes on like Facebook, Instagram, LinkedIn, Twitter, right? I sort of know like the community of people I'm gonna get, like the level of conversation. That's just gonna get way more magnified in your vein, or like more polarized, or um because they all have their own models, I guess, at the end of the day.

SPEAKER_00 1:19:07

Right. I guess what I'm saying is like AI itself will um be tuned in ways that increase your usage of it.

SPEAKER_01 1:19:18

Gotcha, gotcha, yeah.

SPEAKER_00 1:19:19

And the um the reward function of the model won't be um uh make Tommy the sort of happiest, most well version of himself. It will be like number go up, like increase revenue, increase time on screen, like increase the number of ads that we can serve to this guy, right? Maybe if they're good ads. Yeah, yeah, maybe. I I'm not against like targeted advertising per se, but like, yeah, I just feel like this could go in all sorts of wacky directions.

SPEAKER_01 1:19:45

The crazy part though is, or the sad part is like the meta Facebook slot AI video path, they make money, they get smarter, they get bigger. The I want to feel my best self, health supplements, relationships, be my therapist, whatever, that seems like it'll cost me something. Uh, because they don't have stuff to farm or serve. So it kind of seems like a rich person thing at some point. Or like a Oh, interesting.

SPEAKER_00 1:20:11

Like if you can afford the premium service, then you can like bypass all the slop or something.

SPEAKER_01 1:20:15

Like, could every 18-year-old around the US afford like a hundred dollar multi-subscription? Like, probably not.

SPEAKER_00 1:20:20

Like, yeah, totally, yeah. I mean, and people worry a ton about how AI is gonna cause you know social stratification in this way, right? Especially if, you know, I I think I think driver is the most uh common American job. And we've already seen with Waymo and others that uh that's probably gonna go away pretty soon. Um, although it's sort of there's this irony, right? Because 10 years ago, if you asked anybody what jobs is AI gonna automate first, it would have been physical jobs, right? Like driving. And now we realize, but but like the the white-collar people will all be safe. And we've realized that that's exactly wrong, right? Because the because the most powerful models are language models, right? And um, you know, white-collar knowledge workers are roughly speaking doing language-oriented work, which are typing into a screen all day, right? So, of course, that's the thing that language models are really good at. And so um, yeah, I just think it's totally feasible that AI would cause mass unemployment really, really fast. And that would be this whole other we haven't even talked about that risk, right? And that's a whole another set of risks that we really you know have to grapple with as a society. Like, I I definitely do not have the solution, but like I think it's could be very we've solved two things.

SPEAKER_01 1:21:25

We might as well keep going.

SPEAKER_00 1:21:26

Well, we've done geopolitics, we've done alignments. Should we solve it for this? Let's do it.

SPEAKER_01 1:21:30

So, what what does everybody do post AGI? Like, what is the most common job in five years if it's not being a driver? Man, I'm so glad I'm not being asked this question because it feels hard.

SPEAKER_00 1:21:42

I don't know. I really don't know. I mean, you know, this was kind of the dream of UBI, right? Like we need uh, you know, we we need universal basic income. But I think there's good arguments against those models, right? I think people do derive meaning from their work, you know, derive meaning from their uh their projects, you know, and and the things that they take pride in. But I think it's a little glib to be like, hey, person who used to work this blue-collar job, now, you know, write poetry all day or whatever, right? I mean, just find meaning and the beauty of living, you know. Here's a check, you know. By the way, here's a check, and also a bunch of slot video Instagram feeds, right? Are we just gonna all be plugged into our AI-powered VR feeds? Like, I I I genuinely don't know.

SPEAKER_01 1:22:32

I think this is I like the Ready Player One world where we all live in VR worlds. Yeah, yeah. The other question for you though is like it assumes the Pi doesn't expand. Like we're not exploring the galaxies, we're not new technologies, like there is a whole new unknown subset of jobs and experiences that we could do.

SPEAKER_00 1:22:53

I think that's roughly right. Like, I think that um the reason to be cautious about accelerating AI is not uh I I don't agree with the reason that like we will never figure out how to adopt to a world that has explosive economic growth, or wow, a bunch of jobs are gonna be lost, and that's a reason not to do um AI. I I just think every time we've ever had a technology that did that, society has adapted and found ways for people to live meaningful lives, or we've created new jobs that um are just genuinely needed in that sort of new economic paradigm, right? So I the reason I worry about job loss and stuff is not that I don't think in the long run we'll figure it out, but that unlike other technological revolutions, this one is just going to be so fast, right? That will we have the time to adapt, right? Or will it cause this upsurge of economic populism that that sort of causes uh destabilization and sort of, let's say, the American political process, right? Or the information landscape, right? So I kind of worry about those like short-term disruptions versus humanity would never be able to figure out how to exist in a world where a bunch of current jobs don't exist because we we will adapt. I think the things that we which is why the things that we work on more are like the security and resilience side of the equation. So let's um understand that we will live in a world of very powerful AI, understand that there's a bunch of benefits that are going to come from that, but to harvest those benefits, we need to uh not screw it all up, right? We need to uh sort of plug the security gaps or build the zones of resilience that will um you know keep us safe and well as we go into this sort of unknown frontier, right?

SPEAKER_01 1:24:47

You're a big sci-fi reader, I guess.

SPEAKER_00 1:24:49

Yeah, yeah, yeah.

SPEAKER_01 1:24:49

Yeah. So it just feels so diminishing and boring to not get to that point where exploring stars.

SPEAKER_00 1:24:56

Yeah.

SPEAKER_01 1:24:57

And we need this to get there.

SPEAKER_00 1:24:58

I think that's right. Yeah. Uh, you know, um we want, like it would be crazy arrogant to say, ah, and this is the peak of potential experience, and we just want to be like locked into, you know, roughly uh, you know, a hundred trillion dollars of global GDP and like roughly this amount of technology, and like, you know, spaceships that can get us to Mars, the moon, and like maybe Mars, but that's it, right? Like all that lost potential is really unsatisfactory. Yeah, totally, totally, totally, which is why I'm not a why I'm not a Luddite and why I, you know, I um I have a lot of sympathy for people who are who who who want to build more technology and want to do it faster. Like, I get it.

SPEAKER_01 1:25:41

I love it. One other question for you. Um, you're on the early stage investing side, so might not be too important or come up as much, but everyone on Twitter throws around the the bubble word in AI. Uh and every time I open Twitter, it feels like the last two weeks, I see some weird, you know, maybe it's legit, maybe it's not like accounting thing. Like this morning, like OpenAI invested in AMD and like they're gonna now buy chips, and it just becomes mega circular accounting. Yeah, and this has happened before.

SPEAKER_00 1:26:11

Yep.

SPEAKER_01 1:26:11

And everyone's like, oh, it's not a bubble, but like there will be losers, and I'm like, so it's not like yeah.

SPEAKER_00 1:26:16

Derek Derek Thompson did a great podcast on this. So his podcast is called Plain English, and I think about three weeks ago he reduced he he he released uh an episode. It was called something like Are We in an AI bubble? So I would mostly just like defer to that. Um But uh I think if we are in a bubble, it is basically the argument is something like we are investing so much into these like data center and chip build-outs. Um, and it will be extremely hard to ever make enough on those investments to justify how much we have invested. Um because one reason is because uh the chips sort of become obsolete, right? So, you know, okay, you spend a ton of money laying um, you know, fiber optic uh cable or uh building railroads. Well, you can roughly speaking use those railroads or that fiber optic cable for the next many decades, right? But if we're always coming out with that next generation of chip, and if you're gonna build out a compute cluster or a data center, roughly 50 or 60% of the all-in cost is the chip, so we should need to be replaced every few years. Well, then it kind of screws up the economics, right? So I think it's totally possible that we will hit um kind of a bubbly moment and some of these uh uh, you know, some of these like big mega projects, you know, collapse or go under. Um I would say though, that that would probably be like a short to midterm blip, and that ultimately, like I do believe AI is going to be powerful enough to just radically transform society and basically upend every single industry and uh and and really change the shape of the world. And so in the fullness of time. If that's true, yeah, maybe like this NVIDIA deal will go bad, or like this SPV that Facebook just put 20 billion into to build out this thing won't return capital to uh you know BlackRock or whoever's also capitalizing it. Um but to me those feel more like short-term uh bumps along uh a trail that will ultimately like keep keep going up. But you know what? Maybe I'm wrong. Like maybe we'll just hit a plateau in AI capabilities and like GPT-6 is the best we'll ever get, and uh it'll be a big bubble, and then the world will continue on as normal. Um, but you know, like as evidenced by my career decisions, I'm I'm not betting on that future.

SPEAKER_01 1:28:37

No, I like I like the bet. I'm in the same camp. I it's it's just hard to imagine like someone like Sam all of not being able to raise the next round.

SPEAKER_00 1:28:46

It's crazy. It's crazy, yeah. And like also, um, you know, I don't think the lab valuations strike me as super bubbly, right? It's like, okay, look at anthropic, they just raised at 170 pre with uh 170 billion pre, with um five billion of ARR. So doing the math, that's what, like 35x on ARR for a company that's 5xing revenue this year and like potentially building God-like technology. I'm not saying it's cheap, but it doesn't feel like I'm a buyer at that price. I mean, I'm not literally a buyer, like I don't own, just to be clear, I don't own stock in any of the labs. Um, but um theoretically I'm a buyer at that price. So it doesn't feel that crazy.

SPEAKER_01 1:29:29

If you could have like, you know, an equal amount based on valuation, so like maybe a thousand dollars in open AI, 2500 bucks in anthropic, you know, the relative amount, what would you pick? What lab?

SPEAKER_00 1:29:41

Oh uh I had a guy who is a very senior person who just left one of the labs recently and who's thinking about either joining another one or um joining a sort of mid-stage startup as in a leadership role, tell me that he's convinced Elon's gonna win. He's like, I'm just I Elon just gets shit done. He does, though. X is gonna win. This guy can stand up, you know, a year's worth of compute in a month. Like he just, Elon's gonna win, one man's opinion. Uh, I think Sam Altman is obviously just a total force. Um, and man, like Anthropic has been punching above their weight, right? The fact that their models are so good with coding and that their revenue is going up so quickly. So yeah, this is um a cop-out answer, but I I I I would be lying if I said I have a take on like, oh, this one is gonna win and this one's gonna lose. I just I just I just think they're all if you're if you're making a financial bet, I would advise betting on all of them, not just picking one.

SPEAKER_01 1:30:39

You you haven't mentioned like Google, Meta, and like that crop of players. Is there a reason why?

SPEAKER_00 1:30:46

Um I think DeepMind is like genuinely when you talk about like who are the big three, it's usually OpenAI, Anthropic, and DeepMind. So I think DeepMind is like genuinely in the conversation. I think one issue with them is that they're part of this um, you know, bureaucratic uh trillion dollar uh behemoth called Google or Alphabet. Um and it's just hard to get stuff done versus you know anthropic or open AI, they have a thousand or a couple thousand or whatever employees, they're still in startup mode, they're not publicly traded companies yet, right? So they can just like be way more aggressive and nimble. But I guess the trade-off is they don't have that, you know, massive balance sheet of a Google, right? But I I think DeepMind's really in the in the in the game. I think XAI seems to be a bit behind, but I would not be surprised if they caught up. Um, and then Meta definitely seems to be behind. They they seem to kind of keep stumbling over themselves. Um, although, you know, this massive hiring spree that Mark just went on.

SPEAKER_01 1:31:45

Um I bought it during hiring spree and sold it after people started leaving.

SPEAKER_00 1:31:48

Okay, well, maybe maybe a smart move, right? But man, they're picking up really great people. And you know, largely speaking, it's about talent and how much capex, you know, you can um put behind it.

SPEAKER_01 1:32:02

And uh D Mind feels like they have just so like they're doing like there's so many different modalities. Like they have like like they're working on obviously the LLMs and Gemini Pro, but they also have like video generation. There's a model they deployed in the rainforest to like track sounds of birds to see if they're in danger. Like they're working with Aptronic, like it seems like they have like this great cluster of like different mode multimodalities. They also have like the chips, like 25 years of our browsing data and emails and stuff. Um, but on the other hand, it also doesn't feel like outside ChatGBT, we've had like an alpha go moment, and that was like 10 years ago.

SPEAKER_00 1:32:39

From Google. Yeah, well, they have Waymo.

SPEAKER_01 1:32:41

Yeah, that's true. That's true.

SPEAKER_00 1:32:43

Yeah, I mean, um, I uh it's it's me and two others. I have three employees, right? So uh I appreciate like nimbleness and startupiness, and I do think that their you know bureaucratic nature of being part of a big company will hold them back in many ways, but definitely would not count them out. Like I know people, for example, who think uh like TPUs, right, like their GPU equivalent, are way better than uh alternatives, and that will give them a huge leg up, or those who partner with them a huge leg up, right? So I think it's kind of anybody's race at this moment. Like I don't, I don't to me, there's no clear, oh my god, they're definitely gonna be the winner favorite.

SPEAKER_01 1:33:19

So one of the black swan things that you mentioned like a half hour ago or something was the idea that we're not what we have now is not the end game. Like whether it's the LLMs is not the path to HEI, NVIDIA chips are not the end game, the algorithms are different, the energy is different.

SPEAKER_02 1:33:35

Right.

SPEAKER_01 1:33:35

Do you think that that is like I don't want to ask you for a percentage, but like you you talk to people all the time. Like, do we just continue down NVIDIA bigger chips, more energy, more data, or do you actually feel like there are different chip architectures, different algorithms, different types of AI models that will be deployed to upset what we have?

SPEAKER_00 1:33:53

I think there is a pretty good chance that just scaling what we're already doing will get us there. Like we'll create something like AGI. And by the way, like AGI, open AI defines it, and I think it's a pretty decent definition as AI that can do the majority of economically valuable work. Um I think it's possible enough that we should be investing in all the things you would want to invest in to make the world more secure should that happen. I think there's a second world in which we do need sort of a new paradigm, but that there is so much money and energy and resources going into improving AI that we will just figure that new paradigm out and we will hit it. And then there's sort of a third future in which, you know, a new paradigm is required and it's extremely exotic or requires quantum computers that we don't yet have or something that means it will just take a long time. I still think it's possible. But if you're in the mode of like risk mitigation, even if there was only a 5% chance, say that the current paradigm could create AGI or superintelligence in the next even 25 years, you would be like, oh my God, like let's get on this, you know, right? Well, especially given how people building it talk about it, right? Like, I'm not a uh uh, you know, like an AI PhD, right? You know, so I rely a lot on our technical advisors. But if you went into the anthropic office or the open AI office and just pulled out 30 random people and pulled them, and you're like, what do you think our prospects are for, you know, hitting AGI and that that has really uh terrible catastrophic consequences for humanity, you know, you'd hear a pretty big range, but your immediate answer might be something like, I don't know, 20% chance that um AI really has catastrophic consequences for society in the next decade.

SPEAKER_01 1:35:57

I went to an anthropic happy hour like six months ago, and it was generally around 20%.

SPEAKER_00 1:36:02

Yeah. So, oh my God, right? Like if you, let's say your friend was an astronomer, right? And you went over to their house and they had a telescope and you looked up into their telescope and you saw this like asteroid hurtling towards Earth, right? And you talked to him and you were like, that's pretty concerning, huh? What what do you astronomers have to say about this? And he's like, well, if you went to an astronomy conference and you polled a few, you know, 30 people, you know, the median answer would be like there's a 10, 20% chance that this hits Earth and like destroys everything. You would be like, oh my God, like alert the government. Like, you know, like we need to spend like 10% of global GDP on like making sure this goes well. You could argue 99%. You could argue 99%, right? And so, and then you look at you know how much we're spending on making sure that AI goes well. This other force that the people who are most in the know think have a 5, 10, 20, 30% chance of really upending society, and you see how much we're spending on mitigating that risk. And it's like less than one hundredth, maybe even one thousandth, of the amount of money that we're spending on just advancing AI capabilities. And you go, oh my gosh, like what an incredible mismatch. Like, you don't have to be a doomer, right? You don't have to be like, oh my God, we're all gonna die, just to say, like, hey, at the margin, it's probably worth like dramatically increasing the amount of money and talent and sort of energy and passion that we're putting into making sure that the future is beautiful and flourishing and secure as this uh you know technology hits us, right? So I think that's like the tone we're trying to strike. Like, I'm I'm not trying to be like a Yudkowskian, we're all gonna die kind of person. It's more just like, hey, this is worth taking seriously. Yeah.

SPEAKER_01 1:37:41

Well, the I mean, outside the dollars and cents, there's also just thousands of people at these labs who don't want to die who are building this.

SPEAKER_00 1:37:49

So this is my argument to like a lot of people is like um like maybe you're not an altruist, right? Like maybe um, you know, saying this really bad thing could happen to society doesn't really motivate you. It's just like not what wakes you up in the morning. But like this could be really bad for you, right? Like this could be really bad for your kids, right? So like forget society, just do the thing that um, you know, make makes it more likely that your future is is is good in all the ways that you want it to be, right?

SPEAKER_01 1:38:21

So if if people listening to the podcast agree with you and your takes and halcyon, like how would you recommend they they um they demonstrate this view? Do they do it with dollars, they do it with what AI they're using, do they write about it? Like, how do they actually get involved with your worldview?

SPEAKER_00 1:38:37

Yeah, well, maybe you quit your amazing elite job and come build something cool with us and make your grant to do it or introduce you to your co-founder or or whatever it is, or invest in your company. Um I would say, you know, if you have a lot of money to give or spend, um, there's a bunch of things that are worth funding on the sort of AI security side of the equation. Um, I would say just personally, like just stay informed, like be prepared, right? Uh uh there's no downside in just like understanding the technology, following what's happening at the frontier. Um, maybe at some point there'll be some action you want to take in your own life to, you know, mitigate your personal, you know, personal risk or exposure. Um, I mean, if you're the kind of person who wants to make a bunch of money and then put that money towards good, like this is certainly not investment advice, but I know a lot of people who are doing this sort of mission-hedging investment strategy where they're saying, okay, I'm going to invest a bunch of money into the AI frontier, and then I'm going to plow a bunch of my gains back into making the frontier safer. So doing philanthropy related to AI safety and alignment, or investing in things like Halcyon and our fund or the portfolio companies that we're building, um, or advocating for policy that's like sensible for making AI go well.

SPEAKER_01 1:39:56

It is, I'm a big market, free market economy capitalist, but you really don't we really don't get a vote on uh you know what what AI model wins outside using our our money, right, and usage. Totally, yeah. Yeah. I mean, even maybe if a policy decision comes down, but even then you're not we're not voting on it. It's the people we voted voting on it. Right. Yeah. That's awesome.

SPEAKER_00 1:40:17

I I guess like the thing that I would want to wrap up on and say is the reason I started Halcyon was much more about talent than money, right? Um to solve any big problem, climate change, healthcare-related stuff, education, AI stuff. Um, the most important thing is that you have some of the world's most talented people, entrepreneurs, researchers, etc., dedicating themselves to solving that problem. And yeah, you know, you got to raise more money and you know, all this stuff, but you know who's raising good at raising money? Really excellent entrepreneurs and leaders. Okay, you need more rank and file talent. Well, you know who's really excellent at recruiting? Excellent entrepreneurs and leaders, right? So excellent sort of executive or entrepreneur level talent in our model is upstream of everything else you want. And so I think the thing that I'd want to leave your listeners with is if you are one of those extremely talented outlier people and you're at least curious about this stuff, or you're sort of thinking maybe this is the thing that I want to spend the next chapter of my life working on, thinking about contributing to, um, Halcyon is just like tailor-made to be a place that can help you do that. So um, yeah, do feel free to reach out. Do feel free to share my contact info in the podcast notes or anything. Yeah, I would genuinely love to hear from your listeners if uh if they're interested in this stuff.

SPEAKER_01 1:41:40

Mike, thanks so much for coming on. Like all your answers, thoughts like clearly have been years of obsession and thinking about and doing the right way. So I really appreciate you coming on.

SPEAKER_00 1:41:49

Thanks, man. Enjoyed it.