Best of the Pod: Dwarkesh Patel’s Quest to Learn Everything

[00:00] How is AI integrated into your work and in your life right now? So a lot of topics I just find I have a vague sense of what's happening. And it's super helpful to chat with Claude to make sure I'm on the right track. What is driving all of this? What do you think that's about? I really just want to know everything. Wait, step back. Why is this necessary? What's going on? How do I think about the broader context of what's happening here? Because I really can't ask good questions unless I have a good mental model of what they're talking about. [00:30] you [00:39] This podcast is supported by Google. Hey everyone, David here, one of the product leads for Google Gemini. If you dream it and describe it, VO3 and Gemini can help you bring it to life as a video, now with incredible sound effects, background noise, and even dialogue. Try it with the Google AI Pro plan or get the highest access with the Ultra plan. [01:00] Sign up at Gemini.Google to get started and show us what you create. [01:08] Dwar'Akesh, welcome to the show. Thanks for having me, Dan.

[01:38] the people that inspire me to make smart content. So I appreciate you coming on the show. [01:45] That's really kind of you to say. I've always been trying to have the conversations that I would like to have if I was getting dinner with one of these professors or CEOs, what would I want to ask them? [01:57] I'm glad other people enjoy them as well. [01:59] It comes through. And I think it's really fun to get to turn the tables on you a little bit. You've done some interviews, but mostly you're interviewing other people. And I think it's probably on people's minds how you use AI in your work, in your life. And so that's what we're going to talk about today. So maybe just start by giving us a little bit of an overview. How is AI integrated into your work and your life right now? [02:27] Yeah, so it's actually changed a lot. I remember a year ago, this was after... [02:32] I think it was after GPT-4. [02:35] Somebody asked me, you know, do you use AI to help you with your research or prep? And I was like, not at all. It's completely useless in mid. It gives you these banal. You ask it, like, what should I ask? So it's a professor and it'll give you these banal. Where did you grow up? What's your book about? Whatever. So initially it was, you know, terrible. I think recently the models have gotten to just the point where with these like, I don't know, 4.0 or especially with the new cloud models, [03:02] are intelligent and interrogative and can consider the context which you provide to them. And so they're still not that good at like, what should I ask this person? Because obviously that's why I have a job, right? So that I can come up with the questions. But for the research itself, where you're, for me, at least I try to like ingest everything they've ever written, all the rebuttals to their ideas, all the other considerations. And there's often a lot involved,

[03:32] like the last interview i just did was with dylan patel who writes semi-analysis it's a um it's a publication about semiconductors and ai hardware and so on so like there's a bunch you have to learn and uh it's i mean i can go through my workflow but it's incredibly useful to be able to like have this thing where i'm like what's going on here can you help me explain this and then i guess one bigger thing i've been thinking about is ever since i interviewed [04:02] who talks a lot about how space repetition and other tools can enhance our ability to learn and how the normal mode of learning, you're actually not picking up that much. If you pick up a random book and start reading, you're not getting that much out of it. [04:13] And I really... [04:15] have found that to very much to be the case to the extent that if I'm just like casually reading a book, I think I'm basically wasting time or entertaining myself. And I'm, [04:24] I have come up with a couple of different workflows and tools that help me really interrogate and make sure I've reinforced what I'm reading about or learning and [04:34] A tool like a language model is very helpful because it gives you the content in another context. And it can quiz you if you want. So it's super helpful with that kind of stuff. That's really cool. I think we should start. I want to start back to front with the stuff you're using to read. Because I think all that reading is sort of the input, like one of the inputs to the interviews. And then we'll get into the interviews. And I'm really excited for both. So let's start with using AI to read and to learn.

[05:04] One of the main things I think is important is if I'm studying a topic over the course of a few weeks, especially if it's a difficult topic, it's new to me. It's incredibly important that I'm not just casually reading because if you're just casually reading, it's like every day you reread the same key terms, the same concepts, and you start over from scratch. Yeah. [05:30] So one of the things I like to do, for example, I was recently interviewing Dylan, right? So if I go to his publications, I mean, analysis, there's just a ton of lingo and things you have to understand. [05:40] So the new one was pretty interesting. It's talking about why nobody has built a huge training cluster yet. [05:46] And then first thing I do is just like, what are the key ideas and concepts I really need to understand? So I made myself a hugging face space. You honestly don't need to do anything like this. You can just, it's pretty simple to have Claude build your hugging face space, or if you prefer it, what it literally does is like apply this prompt to everything I paste in. So just, you can, you just copy paste that prompt into Claude yourself. But basically it just has a, [06:09] I copy pasted some of the things in Andy Matushak's post about how to write good prompts. And I just asked Claude to make those prompts for me. [06:15] space repetition prompts. So, um, [06:18] When I do this, hopefully in a few seconds we'll get something back. Initially this will give me some ideas of what are the key ideas I need to understand. [06:28] Um, I, [06:30] Super useful, right? I can zoom in a little bit, so it's more helpful. So for the audience who's listening, like, it's given me a bunch of question answer pairs that consolidate the key things I need to understand about this post about how to do it.

[06:47] You know, we can go through the specifics here. I'm sure that the actual specifics of hardware will bore people. But a lot of the things where it's like, OK, if you don't get this, you've like totally missed the boat here. And so you can start with something like this. I add it to my space repetition app. [07:00] Or I can just look through this and I'm getting a sense of like, oh, okay, here's what it would take to train a GPT-4 level model on a 100,000 H100 cluster. What are the three main types of parallelism you need to use to train on a big cluster or whatever? And this is on a technical post. [07:15] On other kinds of posts, there might be different kinds of cards that come up. For history, it might be a different kind of thing. For philosophy, it might be a different kind of thing. So this gives me a lay of the land. I love this. This is super interesting. I feel like I can go in a bunch of different directions, but where I want to start is how are you reading and when are you reading? Are you using this specifically for reading that you're doing for the show, or are you just doing this for any reading that you're doing that you feel like is serious and you really want to learn? [07:45] Both. So just this weekend I was reading, I forgot the author's name, but it's a book called Medieval Technology and Social Change. And it's about how different things that were developed through the last 1000 to 500 years, technologies like the stirrups, how they affected society. And it's like, you can, it's entertaining, you can read it and then... [08:07] So one of the things is like, okay, did I really understand what's going on here with the relationship he's trying to elucidate? So afterwards, in fact, I have some claw chats where I was just going through while I was reading it.

[08:19] Um, [08:20] Let's see if this recollects. Do it. I want to know. I'm on the edge of my seat because I have this book. It's like sitting on the desk in front of me. And so I want to know what you got out of it. [08:32] Okay, so first I just asked it to make some repetition prompts for me. First of all, I was just like, I read the chapter, I'm not sure I got it. So just explain to me the chapter about how he says that stirrups created feudalism. Like what exactly was the connection here? [08:47] So it's a much more condensed, like, [08:50] Here's what's going on here. Basically, if you understand this, it's a useful scaffold so that when you're reading the rest of the chapter, you understand where the pieces fit together. Have you tried like one of the things that I've tried with this is like because sometimes it doesn't know, especially for a book like that, where it's like not that popular. Have you tried like one of the things I do is create a little cloud project and then upload the text if I can find it. Have you tried that? [09:14] In fact, let me just call it AI projects. [09:20] So if I go to... [09:22] Thank you. [09:23] I literally just sound like I'm a host of a podcast where I try to ask good questions. My upcoming guest is a geneticist and I just upload the I get the EPUB of the file. I convert the EPUB to a text using an online converter. I upload it to Project Knowledge. Then I've only just started prepping for this guest, but I'll just have a bunch of chats where I'm like, [09:43] Um, you know, how, how does he explain what groups made up modern Europeans? It has all the context in there. That's, that ends up being incredibly useful. Like you were saying.

[09:53] Yeah, that's so cool. I love that. I love that feature. Okay, wait, let's go back to, let's go back to stirrups and in this chat you're having with with this book. [10:02] or about the book. Yeah. So, you know, it explains that the reason stirrups should create a feudalism is because you needed, um... [10:11] You needed a lot of land basically to support the kinds of people who become heavy cavalry, the knights. The knights need a lot of land in order to have the income to have, you know, like armor and lances and other kinds of equipment and to train themselves. But a knight is only possible if you have a stirrup against which you can brace yourself as you're attacking with a sword, because otherwise you're just like a Mongol who's shooting bows and arrows. [10:41] But then there's a bunch of stuff that's confusing here. Like, why is it that expensive to have be a knight that you need to like completely confiscate church lands in order to subsidize this knight lifestyle? And then on these kinds of questions, the author is dead, but it's just like I'm just like murky about it. I don't know what's going on. So I can just these kinds of things I can the book didn't even talk about. Right. But I can always just continue the conversation with Claude and have it explain what's going on. [11:10] And so this is just like a recreational reading that Claude ends up being super helpful with. [11:13] I think that's really interesting. What do you think about books like this? As a person who likes history a lot, books that sort of single out a specific thing like the stirrup and then are like, well, you can trace all this stuff to that one thing where it makes so much sense. But then there are things like, I don't know, like Guns, Germs, and Steel where Jared Diamond had that whole thesis about, I can't remember the exact thing, but it's like people in warmer climates. I can't remember the exact thesis, but it turned out to be totally wrong. Yeah. How do you feel about things like that?

[11:43] Yeah, so my opinion on these kinds of books, there's... [11:49] I mean, the sort of concise answer is like, yeah, there's ones that do it poorly, but just don't read the ones that do it poorly or something. There is a failure mode for public intellectuals where they initially start off with a discipline and they do some exemplary work there. And then they write an initial broad book that's about how this idea explains a lot of the world and it does incredibly well. And now they're in public intellectual mode. And now the next book has to be like, here's my theory of everything. And it's just not that satisfying. So I do worry about those kinds of things. [12:19] But, [12:20] Um, [12:21] And presumably the reason, I don't know, I'm not into like reading 500 page books about like just how this chair physically worked. Like what's the point of that? Right. I do want to understand the implications and maybe they're wrong. But I mean, what else are we trying to do here? Right. Do we just care about maybe you just intrinsically care about how the stair physically works there? I mean, I will point out a couple of examples. So there's a lot of interesting topics where you really can't get at the heart of the matter. [12:51] without just considering the whole story. And in fact, so a couple of biographies especially stand out in this way where, [12:59] If you look at Cairo's biography of LBJ or Codkin's biography of Stalin, it's basically a history of this 20th century or in the case of Codkin, even before the 20th century. I think Cairo books on LBJ start off with the Comanche raids on frontier settlers in the mid 19th century or something. And it goes through rural life in Texas, why electrification was such a big deal. A whole bunch of other things. Right now.

[13:29] century, but it has a very specific point of view or a specific [13:33] a locus, a character that's moving the story along. And I find those to be incredibly helpful in getting a full picture of what's going on in an era. There's a couple other books where they really aren't trying to write a theory of everything. Like, I don't think Kara's trying to write about like, what is the history of the 20th century, but they just can't help themselves. They feel like you really cannot understand the very specific topic I care about, unless I tell you everything about everything, you know, like Cotkin story, a biography of Stalin starts with the, you know, [14:02] like Bismarck's career as a military general and how that changed the way that different powers thought about colonialism and the need to modernize. And that's where it starts, right? It's a biography of Stalin. So, uh, [14:16] Yeah, I love those kinds of books. I think there's like a very just deep point about the universe being interconnected there. But there's also like a really interesting point for people who want to make stuff like make writing or make podcasts or whatever. Because like there's this deep fear that everyone has about like being pigeonholed. And it's like, well, if I pick this like really specific topic, I won't be able to like bring all of myself to it. I won't I won't be able to be like multifaceted. And it's like, no, no, no.

[14:46] else about the world in order to explain him. And I, I love that. And like, as a creator myself, like that's the thing that I think about when I'm like, Oh, maybe I'm getting too narrow here. It's like, no, no, the narrow is actually good. You can find the entire universe in the narrow. Yes. Yeah. Yeah. I couldn't have said it better. Yeah. Um, okay. So, so basically what I'm, what I'm seeing right now is you're using Claude when you're reading books that you care about, you care about like learning from, and you're using it a little bit to like prepare your, [15:16] about to read, which I think is particularly good for difficult books or for thinking through a particular argument before you go through it. You're asking questions. It's a reading companion. You're getting more out of the books you read from that. But then you take what you've read and throw it into this card generator. Yeah. [15:36] Yeah, and so that's mostly it's just chatting with Claude. And so let me see if I can find a better example. [15:45] So, I mean, a lot of topics I just find I've had a vague sense of what's happening, but I don't really like get it. And it's super helpful to chat with Claude to... [15:57] make sure I'm on the right track. One was like, [16:01] Um... [16:02] You know, Dylan has a couple of posts about how, why packing is a technology super necessary for these advanced chips. I'm not trying to make this podcast all about AI hardware. It just happens to be the last podcast I did. So that's what you're getting. But then I was, it's like a confusing, it's like a five series post about how advanced packing works and how, you know, what the technical specifications are. And I'm like, wait, step back. Why is this necessary? What's going on?

[16:32] about when there's questions about... [16:36] I'm worried about where I might get too deep in the weeds when I'm just explaining. Yeah, basically, I'm just like... [16:41] How do I think about the broader context of what's happening here? Because I really can't ask good questions unless I have a good mental model of, [16:49] um what they're talking about like i really get where all this is fits together [16:56] That makes sense. And so, Claude is kind of like the first thing you flip to when you want to know that. Are you using it on mobile or are you using it on a desktop? [17:05] desktop okay interesting so you're doing most of your like reading and research stuff on desktop [17:11] Yeah, that's right. [17:12] And what do you think about like, just Claude being really great right now? And like, I assume your chat GPT usage is lower than it used to be. [17:21] Yeah, I think these things will keep getting better over time and [17:26] I... [17:28] I think we're just getting in the practice of using these tools. I'll talk a little bit about how these tools relate to my post-production process. Initially, it was kind of useless, but I did spend a few weekends trying to write a few prompts and create a workflow. At the time, it was basically useless. Now, it's actually ended up being useful and I can use the same tools. [17:48] you know, Jupyter notebooks or whatever to get things done. So it is worth investing in getting, even if they don't work perfectly now to get them part of your workflow. So that as they keep getting better, you're getting the returns from that. [17:59] Yeah, that makes sense. So, so I want to just go back to the, uh, the flat, the, the, the Anki card generator, the, the space repetition card generator. So as part of this, once you, once you've done all this sort of like conceptual, um, uh, you know, clearing the ground conceptually for yourself to like kind of understand the basics of, of what a guest is talking about or an idea that you're interested in, then you're kind of, um, you're adding to your, your flashcards so that I guess, so that it, you retain the information past even when you talk to that guest. Is that right?

[18:30] Yes, that's right. I mean, I think of the larger mission of the podcast is to, why does the podcast get better over time? And it's because basically I'm getting smarter or learning more things. I'm reducing my ignorance around a bunch of topics. And so if I don't do that, I mean, I think about all the episodes I did before I interviewed Andy and started using space repetition. And I just like really regret it because I talked to all these world experts in a ton of different domains. [19:00] I didn't take that much away. I vaguely remember some things. And now that I use, I can walk you through the kinds of cards I make and the space operation tools I use. But it's like totally a game changer in terms of what I can retain [19:12] And in fact, I think it's not even about making sure I remember what I discussed in a previous episode or what I learned previously. It's more about future learning, because I'm sure you've heard the saying about learning. [19:25] the, you know, learning compounds because you can use what you've learned in the past to learn future things because they all interconnect. Well, you can't do that if you basically forgotten most things you've learned in the past. So my learning has a future other things has become much faster because I have a [19:45] cached all these different concepts and figures and facts. So the future things are just like, I understand how everything fits together much more. It's not even about the past. It's really about future learning. [19:59] Can we see your, I don't know what you use for space repetitions, can we see your deck?

[20:05] Yep. I will point out, by the way, as a side note, one one use case of claw that ended up actually being pretty useful. Sometimes you read obscure for a lot. I was reading Nick Land's selected writings about AI and his accelerationism. And I was like, what's going on? Like I genuinely I'm like, what is his argument? Basically, like, why does he think that the AI takeover and the whatever thing it creates in the aftermath will be goo? Because he's a smart guy. I'm assuming he has an interesting argument. [20:35] ask Claude, like, okay, so why does he think it's a good thing that AIs take over humans? And he, it offers a summary. Like, initially, this isn't necessarily that helpful, because I kind of did read this in the essay. But what's helpful is that when you go through and I'm like... [20:49] I respond like, I don't get it. Like, what does he think is wrong with human society that you can't, you have to erase it? And then he gives explanation. I'm like... [20:58] I still don't get it. Like what exactly you're talking about here? And then like, here's what I do with the podcast, right? I have the guest on and I ask them, what do you mean here? Like, I disagree. Here's a contradiction, whatever. And going through their writings with Claude and like, have I have actually found a sort of blind spot in their thinking? Or is this just me being confused about their ideas? [21:17] It's super helpful. That is really interesting. It's like you can get down to a deeper level before you talk to them so that you can start there with them as opposed to like starting at the surface. [21:31] Exactly. Which is really cool. I use that too for difficult books. Not necessarily for interviewing the author of those books, but for example, I interviewed Reid Hoffman a month or two ago, and I wanted to talk to him about the intersection between philosophy and AI.

[22:01] for it and it was like so much better um because i like i haven't taken a wittgenstein class or maybe i took one in college like a long time ago but i've read him a lot and just there are always those points in those kinds of books where you're like i think i i know what they're saying but like i'd probably have to go to a grad graduate school and like get a master's in this to like really know and claude is actually like [22:25] makes me be like, oh, I don't need that anymore. Like I, any book I want to read like this, like I basically know. And it just helped me so much in that, in that interview. Cause I could just ask, read like really deep. [22:34] Wittgenstein related questions and he could answer them. [22:37] Yep, yep. I think that's totally legitimate. I think some people would be like, oh, you need to read it in the original, blah, blah, blah. I think if you care about the ideas and you think the ideas are timeless and not... [22:48] The ideas are not about the specific kind of prose that the original author used, but just generally like what is the essence and the gist of what's happening here. If you care about the ideas, and I think this is totally valid, right? I don't agree with the people who are like, no, you need to read like the specific syllables that Wittgenstein used. Yeah. I mean, I'm also just saying like I have the book open and then I just take one of his like... [23:10] statements and just throw it in there. And then it's like, here's what it means or whatever, which I think is really great. Okay. So you're going to show us the space repetition card. So what app is this? [23:21] This is mochi. It's like Anki, but this is the one I use. Why? So actually, I don't have any cards today because I just went through them this morning. But let me give you a sense of what kinds of things I have. Right. So yeah.

[23:37] I have, if you go through history, recently I've been planning on interviewing David Reich, who is a geneticist who explores human origins. And these are especially cases where just like reading the book, I'm like, I would have totally forgotten. He names all these different ancestral groups and how they combine and in what eras. [23:56] You know, when did the Amina people come through Europe? When did the Anatolian honor gatherers, you know, wash over Eurasia? All these things that were just like you read it in one year, it goes out the other one unless you make cards for it. And so I made a ton of cards about this kind of stuff. [24:13] And, you know, so there's examples of that here. It's especially useful for hardware and technical things. [24:23] So, yeah, [24:24] Here, I feel like if I don't make cards, I'm just constantly relearning the same things because I didn't learn the lingo in the right way first. It's not just about learning the terminology. It's about understanding the... [24:35] underlying concepts. Let me give you a good example of that. So, [24:39] Maybe I'll step back and I'll explain like I go through these cards in the morning. Maybe you can see what it kind of looks like if I do the cram cards thing. Please. I can go through them and... [24:51] Right now, I remember this. This is the first one that came up randomly, but it's multi-query attention to not have to use huge KV values and then sharing KV values between layers and using local attention. That's the answer. Now, it seems trivial right now because it's just three things, but

[25:16] I would have totally forgotten about this if I hadn't made a card for this as soon as I read the blog post. And then it just like I've wasted my time in the future. If I'm learning about these technologies in a different context, I just don't have the connection to what was happening here to connect it to. Right. If I go to a different. [25:33] category if I go crime cards. This is the white thing. I would have totally forgotten about it if I hadn't made these cards. [25:41] Yeah, I'm just a big fan right now. I've become a space repetition fanboy these days. How do you think about the usefulness of space repetition in a world where any of these questions is pretty much answerable with Claude with one search? [25:58] Thank you. [25:59] Yes. So I think it's about not necessarily remembering this information, but when a future thing comes in, you understand like the conceptual. In fact, let me give you a good example of this. Right. So I remember. [26:11] Sometimes I actually make cards about facts that I don't even understand in the moment, but in the future... [26:17] the as I learn more about the field, as a sort of territory becomes more clear, the things I said in the card make more sense to me. So if I was reading some of Colin Burns papers. [26:29] And so I made this card about like why Colin Burns thinks that alignment is a tractable problem or understanding what model thinks is a tractable problem. And at the time, I wrote things down about like... [26:42] you know, features are in a linear space. What does that mean? Or like we can sort of see features in other sorts of categories. And at the time, I'm like, I have no idea what this means. So I'm just going to write it down because I read the blog post and there was no point of reading the blog post if I'm not going to make the card. Later on, as I learned more about how the residual stream model of how attention works works,

[27:06] you know how what that is and so forth this card made much more sense to me in the future but i would have just like totally memory hold or not even memory hold i would totally forgotten this content which required future understanding if i hadn't made a card of it and then when i see the card again in the future i'm like oh this is what colin burns meant now that i understand how attention works this is what it means right um yeah [27:29] This is really interesting to me. [27:33] I want to get into some of the ways that you use AI for interview prep, because I think we've mostly covered the reading stuff. But before we do that, I just want to understand what is driving all of this? It feels like you are just consuming massive amounts of information and turning that into knowledge in your head. You have this sort of just overdrive of curiosity, which I actually resonate with a lot. I'm surrounded right now by books. [28:03] for you what that what do you think that's about? [28:06] Thank you. [28:06] Hmm. [28:09] I think I really just want to know everything, right? I don't know how to express it. There's a beautiful passage in a Will Durant book as he's turning 90, where he's writing a memoir, basically, of his main ideas called Fallen Leaves.

[28:39] of higher understanding and clearer insight, or at least I've understood that such a thing is possible. And [28:46] Thank you. [28:47] Something like that just resonates with me. I don't know. I just like, I find that idea really appealing. I'm nowhere close to it, but I just, hopefully in the years to come, that'll just be a thing that, [28:57] I also really admire people I've had on the podcast who do have these self-consistent and really deeply interrogated world models. You know, I've interviewed these guests and some of them, a couple of names come to mind, people like Carl Shulman or Tyler Cowen or Bern Hobart. It feels like they've really read everything. [29:18] Everything you know is a subset of what they know. And I just... [29:22] I find them to be super compelling as thinkers. Of course, there's many things they can still be wrong about. I'm not one of these people who buys like, there's like a thing where you just know everything and you can never be wrong, right? You always have blind spots. But their ability to, which you can see when you talk to them, to connect anything you ask them about. And they're like a clawed six in the sense of like, you start talking about why it has a fraction of finance as a percent of GDP. I remember asking Tyler this. [29:52] cough, just a super interesting answer that connects a bunch of different disciplines. You asked Carl about like how fast AI hardware could grow and just like done the sort of Fermi estimates on how fast algae bloom and how much solar power they consume and how many fast CSMC is making. I find that sort of compression of what's happening.

[30:16] the input they've ingested over their lives and they can not only do they know that stuff but they can really connect it in a really interesting and compelling novel way i i i find it super compelling and and in terms of developing your own worldview like do you have that anywhere where it's like you're creating like some sort of living document or is it just it's just all in your head like all the stuff that you're learning obviously you have the you have the cards but that feels like more like um dots in the space rather than like uh the ways that they all connect and and how you think [30:46] as a system. [30:47] Hmm. I think I've been trying to do more of this recently. And now that I've sort of built up an underlying maybe vocabulary or understanding because of the podcast, it makes sense to do more of this. Something I've been doing recently, let me pull this up. I've only just started, hopefully there'll be more by the time people are looking at this. But I've started writing, you know, riffs on different books or things I read. And if I go to [31:13] um, [31:14] Thank you. [31:15] It's basically on my website. [31:18] And so just like I can read a book and I have questions or I connect it with other things I've read. I remember, for example, in when I was in Steven Pinker's Language Instinct, he was writing the book before the FOXP2 gene that can help explain human language was found. And so he has all these observations that are then later explained by the FOXP2 gene.

[31:48] I can do by riffing on other people's ideas. [31:52] I actually am curious. Do you have suggestions on what I should be doing? Maybe I should be writing more blog posts or what do you suggest I should do? [31:58] Well, that's a good question. So, well, before we get there, like one of the things that this reminds me of, I think Claude is so good for reading old science books because it can tell you what's outdated and what's not. I do that all the time and I love that little thing. But yeah, I mean, I think basically like developing a worldview is like... [32:16] You have to just try, you know, and you try over and over and over again. And I do think like blog posts are really good for that, especially like, you know, for me, like I have to write every week. [32:29] Um, and so I'm like forced to take a view on something. And generally, if you're like intellectually honest, you like want one post to like somewhat like agree with the last post and your audience will call you out if you're like just disagreeing with yourself all the time. [32:44] Um, so you're kind of developing a worldview that way. But for me, like right now, I'm actually like my big thing this, um, this quarter is like, I just have these like ideas that are simmering that are like sort of the relationship between like, um, language models and like some deep philosophical questions that we've been like talking about since like Plato, which is like the appearance reality distinction. And like, how do we know what's true and what's knowledge and all that kind of stuff? Like, I think there's a lot of overlap there and it requires like, it's going to be like a 10,000 word post or something like that. [33:14] And so what I'm doing is I just have a Claude project. I have all these little notes and riffs and stuff, and I'm just going into Claude and being like, hey, what's...

[33:23] like what's what's the thread here like what's going on can you help me like figure out like there's something in me that i have all these little ideas for but i can't quite like put it into an argument that all makes sense and i think just honestly like sitting with that for like a couple months i will i will know what's in there but there's something in there um and yeah it's cool do you do you make a clawed project to like here's some of the things i'm thinking how do they [33:53] show it to you. Let me just pull it up. So, okay. So if I go into Claude, [34:01] um, [34:03] I have a couple of different projects. Um, one project is seeing like a language model, which is the title of this post, whatever it is. Another is Zen and the art of motorcycle maintenance. So this is like a book that I'm reading as prep for this, um, this piece that I'm writing. Um, I've read it a bunch of times before, but like now I'm like doing a little bit of a deeper read. And so I have like the, I have the, uh, the whole book uploaded and then I can like ask questions. Then I have another one that I love called my psychology, which has a bunch of [34:33] goals I've set for myself, um, [34:35] over the years. And then also like things I've observed about my psychology or things I'm working on, like little aspects of myself that I'd like to grow or change. And so when I'm making decisions or thinking something through, I just go in there and it can reference all that stuff. So it knows who I am, which is really cool. Um, [34:52] So in seeing a language model, let me see if I can pull it up in the projects directory.

[34:57] so I have like [35:01] Basically, I have this one note in Apple Notes, which is like, [35:05] uh every time i have a little thing come into my head i'm like i just i just put it in there like let me see if i can find it for you um [35:12] Ch-ch-ch-ch-ch. [35:14] Sigh. [35:15] I just throw it in here. And this is huge and messy. And it's different quotes from different books and just different ideas that come to me off the top of my head as I'm walking around. And I think that there's a thread here in all of this stuff. I can see how they're all related, but I can't quite pull it out. And so what I've been doing is I just like [35:39] So throw it all in here. We have this like all the quotes and all the ideas and fragments. I have a little bit of a draft, like an intro. And then I have a chapter of a book by Richard Rorty that I think is really good called Pragmatism as Anti-Authoritarianism that like kind of sparked this whole thing. [36:09] is I put all that stuff in here and I was like, hey, I have a bunch of notes and some fragments of text for a long 10,000-ish word piece I want to write called Seeing Like a Language Model, but I need to understand what I actually think and make a bit of an outline before I get started. In order to do that, I need to understand the patterns of what I've been thinking and writing down and reading about. Can you suggest some ways that you can help me do this? I want to get from where I am to an outline. You have access to some fragments, notes and early unfinished intro.

[36:32] And it just like has a bunch of ideas like thematic analysis or argument mapping or chronological development. And I'm just like sort of going down the rabbit hole with it where it's like, you know, I asked it to do concept clustering. So it's like, you know, one of the concepts that I'm playing with is the philosophical divide, Plato versus Aristotle, which I think is like not quite right. It's actually Plato versus the sophists, but like it's close or the evolution of Western thought. [37:02] into the rest of Western thought and into science and just the way the Western mind works? And then how do language models differ from that paradigm? [37:13] So that's the basic thing that I'm trying for. The reason I ask this question is because I'm selfishly like, I feel like a little bit, I haven't done the big idea thing as much as I really want to because I am writing every week. I am sort of reacting to stuff. And so I want to be a little bit more thoughtful. And this is my attempt to put all of it together into something [37:36] Makes sense. [37:37] Yeah. You know, as you're going through this, this really actually makes me want to write more because... [37:45] Now that you're talking about it, now that you asked the question, I'm like, yeah, I should be sort of consolidating the things I'm learning in a more comprehensive way. And in a way that's also more useful and accessible to other people as well. Right. I, you know, I spend weeks like learning about some random, not random, but like the things I care about. I'm about to prepare for Daniel Jorgen, the guy who wrote the prize. It's a history of oil or whatever.

[38:06] geneticist, you know, AI researcher or whatever, to the extent that I'm getting something out of these research processes, I should consolidate it in a way that's not evident in the podcast itself. [38:17] Yeah. I mean, I selfishly want you to do that because I'm curious what you think. Yeah. I appreciate that. You will send these out through your newsletter, right? Is that the main way? I will. Yeah. Yeah. [38:28] Yeah, this will go out through the newsletter. I might do something special for it. Maybe we'll make a little mini site for it when it launches, but that's sort of in the future. I have to actually write it first. [38:40] So, okay. So, I want to move on. I want to talk about how you use AI to sort of do the interview prep. So, let's move into that. And then we can also... [38:53] we can also maybe even prep for an interview together. [38:57] Yeah. Yep. Okay, let's do it. Honestly, the interview prep is like, it requires a lot of work, but fundamentally what's happening is not that complicated. Like I can just show you a document I might have made in the past. I'll share my screen. So honestly, it literally is just like I'm going through, I come up with a bunch of questions and I sort of group them together in relevant categories. [39:17] or if I go to... [39:18] I was interviewing Dylan Patel [39:21] I'm sorry, this is not the right one. [39:24] Just like a bunch of different. Yeah, it's just like a list of questions, basically. It's not complicated. But the process of coming up with them is, you know, very research intensive. So we can go through like if I'm a guest, I'm like only barely started preparing for it. We can go through the process.

[39:37] preparing for them. [39:39] Yeah. Can we, can we, I just want to stop at those questions like again, selfishly, because I think it's really interesting. Like you have these like long lists of questions that are organized by theme. Are you like going down the list or are you sort of jumping around? Yeah. So it's really interesting because, um, I come up with these lists of questions, but it's like, it really never ends up being, I asked question one and I asked question two and I asked question three. [40:04] I start off with an interesting question and... [40:08] If you listen to the interviews, hopefully it comes up more as almost conversation because I spend so much time preparing that I have these. [40:15] questions basically memorized and so the next one that is appropriate to their response if they say something about um you know memorization in lms i'll have a question prepared about that or related to that and i'll just ask it next because that's what fits in together and so you know i'll have a list and this is what i'll send them if they ask for it but like really it just sort of me off the cuff like here's here's a question i remember that was relevant to this in the actual interview [40:45] It's almost like writing the doc is the prep itself. And you don't even necessarily need it in the interview. Maybe you have it just in case. But yeah, that makes a lot of sense. [40:54] And then, yeah, we can even go through... Let's see. I'm doing a couple of interviews in the future. David Reich and... [41:02] David Reich and Daniel Juergen. So the one, the first former is a geneticist with about human origins. The second wrote the prize, which is the, you know, the famous book about the history of oil. Which one sounds more interesting to you? We can do that, whichever one.

[41:16] I want to do the famous geneticists. [41:20] So let's go to Claude. In fact, I do have his book uploaded as a project, so we can just use that. That's great. And so basically what we're going to do is we're going to watch you, and I'll do it with you. We're going to prep for an interview with this guy. What's his name again? David Reich. David Reich. Okay, cool. Can we get a little bit of background on David Reich? Maybe we can even ask Claude, because I'm a newbie to David Reich's work. [41:50] into how have human populations across the world been formed? Basically, who are the Europeans? What groups make them up? What ancient migrations and genocides and population replacements made them? Same with the Indians or Native Americans or Africans. It's completely changed. I mean, they've basically sort of made many academic disciplines irrelevant because they actually have empirical data on like, here's actually what historically happened. You guys are completely [42:20] of what happened. [42:22] If you're familiar with Nafri Mincy Suvi's challenge, you know, like, you have these, like, birthed-up squirrels, but with some advanced techniques, you can get some useful information out of them. I feel it's in a similar vein. Obviously, they're not the same kind of project, but, like, it's a similar vein of, like, [42:37] Once we develop the advanced mathematics or genetics or whatever to understand what's latent in the genome, we've just uncovered a ton of insight about what's been going on in human history, basically.

[42:50] I'm sorry, I'm just getting nerd sniped and just going on riffs here. But one of the interesting things is... [42:57] You can see when one population replaces another, whether it was just like, oh, we met and we're now intermingling and trading and whatever. Or is it like we're committing genocide against you? And you can tell that because in the case where it's genocide or population replacement, it will be that the male line of the population that is invading will overtake the male line of the existing population, but the female line will remain. So mitochondrial DNA only comes up in the female line. [43:27] And you'll see like the female line because they're getting, you know, like the new men who are coming in are taking them as wives or something. And then anyway, so you can just like learn a lot about like what kind of invasion was it? Did they like conquer or was they were they just like mingling or something? One of the many things you can see from the DNA. That's really interesting. [43:46] Wait, and so this is basically re-examining DNA evidence of old settlements and he's uncovering new ways of being able to analyze the DNA? What's the new methods that they're using to draw new conclusions from existing evidence? [44:16] and so forth. Another is you can even tell the level of inequality in a society because if there's a lot so for example in India one of the things that was super surprising is that the amount of endogamy which is to say that the

[44:32] a certain cast in a certain village would just like not... [44:37] there wouldn't be any sort of intermixing with another caste in a neighboring village. Like to the extent that's true of nowhere else in the world. And they were able to find this in India where the amount of social stratification, you can see that in the genetic... [44:52] a catalog over the last thousands of years, where for thousands of years, these two neighboring castes haven't mixed with like 99% or something, which is like, even from sort of infidelity or rape or something, you would expect there to be more than what actually ends up being the case. So you can understand modern culture in India based on what has happened over the last few thousand years. [45:14] That's really interesting. So I want to like, I feel like you're doing like such a good job of summarizing his main ideas. But I kind of want to I kind of want you to do the same thing with with Claude so we can see how see how you stack up versus Claude, because [45:27] Obviously, you've input his book into this project, so it has that as reference material. Can we ask it to just summarize a few of his main ideas? Yeah, that's a great idea. Can you summarize? [45:43] And maybe the techniques he used to come up with his new insights. Perfect. So what you're writing is, can you summarize the main ideas from the book and the techniques he used to come up with new insights? [45:56] Cool. And one thing that's like really cool about this is like, [46:01] You've been able to do something like this with ChatGPT for a long time, but ChatGPT's context window isn't that long. And so...

[46:08] it chops it up um and like it's not gonna it's not gonna really be able to summarize the entire thing because of that you know it has to like find the right parts of the parts of the book and the embedding search in it is not very good and all that kind of stuff and claude you can just like throw us throw a ton of stuff in the context window and that just like makes a big difference um okay so it looks like it looks like we've got some some uh so it tells us that the ancient dna is revolutionized understanding of human prehistory um and then we we've learned that [46:38] populations today are the result of multiple waves of migration and mixture. [46:42] uh um and then you know just like a bunch of other genetic stuff um then it talks about the key techniques about hoji mode sequencing and how they've enabled these sorts of new new discoveries they've been making um [46:56] Yeah, yeah. But anyway, there's a bunch of interesting things about their research. [47:04] Well, now I'm interested in like, okay, so the key techniques that it's using are whole genome sequencing of ancient DNA samples. So is whole genome sequencing like a new thing that you can do it on ancient DNA samples? So it's saying like by improved extraction sequencing technologies, is that, that's like, that's a recent thing. Yeah, that is an interesting question. So we can even ask about it because I'm not sure. How... [47:26] How exactly do you... [47:29] Sequence an ancient... [47:34] or a prehistoric [47:37] Thank you.

[47:39] genome. Like, can you do like what is that at work? Right. OK, so they grind the bone and they have then techniques to get the DNA out of that. Now, another thing we can ask is like one thing I'm curious about. Let's see. [47:57] I don't really remember the chapter on... [47:59] Um, [48:00] Thank you. [48:01] Native Americans, I could ask about what exactly happened with everyone. Here's one thing I'm curious about, like, how would I guess I don't even know if David Reich addresses this himself, but like, how would David Reich's theories? [48:16] Help explain why civilization suddenly. [48:23] Emerges. [48:25] so rapidly, and that to... [48:29] concurrently in the new and the old world. [48:36] after 10,000 BCE... [48:41] AKA the end of the last ice age. [48:46] And then maybe I'll just like ask Claude why I think it's an interesting question. So like this seems like a really remarkable coincidence given how long humans have been around. No. [48:57] That is interesting. Well, coincidence, given that human, I'll correct my spelling, given that humans have been around for hundreds.

[49:11] of thousands of years. [49:16] Thank you. [49:17] I didn't realize that we believe that it emerged at the same time in geographically disparate places. That's totally new to me. I thought it was just in Mesopotamia or whatever. Yeah, but actually, there's a really good book by... [49:30] Peter Jackson called The Great Divide. And it's one of the most interesting books I read. Just as a side note, it's about, yeah, it's comparing the emergence of civilization in a new world versus your world. So in the old world, [49:42] Um, [49:43] In the New World, the Kharal is a civilization in 3000 BC, and it's based on fishing and not on conventional agriculture like Mesopotamia. And he talks about how that changed the evolution of the culture in the New World versus the Old World. But anyways... [49:59] That's really interesting. [50:02] Okay, so major population movements and mixtures.