How an AI Researcher Uses ChatGPT and Notion AI - Ep. 3 with Linus Lee

[00:00] I also think when you look at models this way, language models start to feel less like Asians that have their own kind of agency, and more just like, oh, this is like a calculator. The more context the model has about [00:10] why you're doing what you're doing or what your goals are, the better suggestions are generally going to be. Do you have custom instructions set for this? [00:17] I don't actually. Oh my God. Okay. So I have a bunch of books in front of me and I want to see if it can recommend books. The Brothers Karamazov. [00:25] Medieval technology and social change. Exhalation, Ted Chiang. The essential Kabbalah. [00:30] synthesize the vibes below into a [00:34] Single paragraph. Holy sh*t. [00:37] synthesize more. Wait, wait, wait, wait. So you said synthesize more and then all caps compress. I love it. AI researcher, use AI. [01:03] Welcome to the show. [01:05] Thanks. Thanks for having me. Excited to be here. [01:07] Yeah, I'm excited to have you. We've been friends for a while. I interviewed you, like, I think about a year, year and a half ago, and it was like when I was first starting to write about AI, and that interview went super viral. And hoping we can replicate some of that magic here today. [01:25] I just, I don't know, I love getting to talk to you. You're a researcher, you're a tinkerer.

[01:30] You're just a you're a really, really deep thinker and a great writer for folks that are listening that don't know. In addition to like doing a bunch of side research and side projects and tinkering in LLMs, you also work for Notion on their AI team prototyping new interfaces for AI. [01:48] So we'll get into that, we'll get into some of your thinking about LLMs and how LLMs fit into creative work and creative thinking. We'll get into some prototypes you're building. [01:58] or maybe demos you have. We'll talk about Notion stuff. And then, of course, we will talk about how you use ChatGPT. Yeah, I think that interview that you talked about, [02:07] I think it came out almost exactly a year ago, if not exactly a year ago. And I think it was... [02:11] I think safe to say pivotal for both of us, at least certainly for me. That interview is a big part of how I ended up at Notion. [02:19] which I've spent the last year at, and it's been very fascinating to hear. [02:23] a great place to be for the last year as lots of things have happened in AI. So, [02:27] Yeah. [02:28] Kind of timely. [02:29] I love it. I love that it's a year ago. I think it would be great to maybe make this an annual thing. And I don't know what the next step is after helping you get a job at Notion. What's the next level thing? Meeting Taylor Swift or something? But maybe we could make it work. That would be something.

[02:59] of misunderstand how important ChatGPT in specific and a lot of the AI tools that are being built right now in general are going to be for creative thinking and creative work. And I really think that you're one of the people on the forefront of thinking about that. There are a lot of people out there. [03:18] who are just really afraid of AI right now and really feel like, "Oh, it'll replace everything that we do," and it creates this inhuman future. And I find you to be one of the deep thinkers and builders in this space that is trying to think about how to make AI into a human future, where it sort of augments us and [03:36] helps us live fuller lives. And I think that's really, really incredible. When I was researching this episode, I came across this quote that you wrote recently where you said, [03:45] um [03:46] I want to build interfaces that lets the AI gesture us into a better future without infringing on our agency. I love that. I think it like really captures the spirit of your philosophy. And I just wanted to start there, like. [03:59] Tell us about that quote. Tell us why you wrote it and how that fits into the larger way you think about AI. [04:06] Yeah, I think agencies really... [04:10] Interesting. I think a lot of the way that I think about [04:13] language models in particular, [04:15] I think what we're starting to see that [04:16] models for different modalities all kind of have [04:20] When you take a 10,000 foot view, similar kind of emerging capabilities, but language is sort of. [04:25] the thing that seems most human-like to us, and so we talk about it a lot. And I think the way that I look at language models personally has been colored by the fact that

[04:33] While I work, like a lot of other people, I work with language models at that like, [04:38] sampled output level of just like [04:40] telling it [04:41] things and then it tells me things back. I also work with it at like a numbers level. And so, and like looking at embeddings and so on. And I think, and some of that we'll get to in a bit. [04:53] But I think because of that, um, [04:56] I personally, when I look at a language model, it is really just like a function that predicts a probability distribution. [05:02] while you can wrap it in very anthropomorphic [05:05] packaging that makes it seem like it has kind of [05:08] is its own will and its own kind of intent. And it might project kind of, [05:14] people like properties to it and say it wants things. Ultimately it's like a, it's a statistical model of probability distribution, just like a very complex one. [05:21] And so I think that colors a lot of my own thinking about it. [05:23] how I view the technology. [05:25] Um, [05:26] But just because it's like a pure function that models a probability distribution doesn't mean that we may accident doesn't. [05:37] prevent us from accidentally [05:39] building things with the technology that take away agency from the things you want to do. [05:43] I think an interesting example is [05:46] A coworker and a friend of mine [05:48] at Ocean and I were, we were talking about making, this is kind of a silly example, we're talking about making a [05:54] cover image for a party that she was organizing. [05:57] And she had this very particular aesthetic in mind of a cover image, which was kind of [06:02] Like, uh...

[06:04] This is basically a very ugly... [06:06] painting on parchment. [06:08] not something you would consider aesthetic. And she's like, I want this very specific aesthetic. Here's some like images I got off of Twitter that follows this aesthetic. Can you make one in this aesthetic or, [06:17] of like a girl sipping wine or something. I don't remember the exact example. [06:21] And I tried so hard. [06:24] to use all of the image generation tools at my disposal. [06:28] like stable diffusion, Excel, Dolly 2, Dolly 3, all these tools, even some of my own kind of image-to-image tools to make an ugly image. [06:36] And it's just very difficult to get Dali to make an ugly image. It's – I – [06:43] I think that's interesting to me because, and this is something I've talked to with [06:47] something I've, [06:48] talk to some folks that majoring about also about how the tool kind of constrains [06:54] the search space of possible images you can generate. So that normally it's kind of closer to what you want, because normally you want something aesthetic. [07:01] But in the process, it may actually sort of take away in the, [07:04] design of the tool, if not necessarily the capability of the model, it takes away some agency from the user. And so I think the underlying technology, I think, is [07:13] kind of just like a [07:15] a tool, a function, whatever, a mathematical object. But then we don't want to wrap it accidentally in packaging that. [07:21] This is powerless. [07:22] That's really interesting. I want to back up because you said a lot of different things there, and I think they're really interesting and really important to unpack. [07:29] What I heard you say is sort of – [07:32] AI, or at least the current generation of AI models, is a function that models a statistical probability distribution.

[07:39] Um, [07:40] And, [07:41] when we experience that ourselves, [07:44] Um, [07:45] we have all these reactions to it that we anthropomorphize it. We project out what it might be able to do from what it can do today in ways that are maybe unrealistic or maybe don't fully understand how it currently works. [08:01] And, um... [08:02] And depending on the probability distribution that you... [08:06] that you select. So like, for example, the probability distribution that you create from the training set of images that you use to train your model or the training set of, you know, texts that you use, for example. [08:19] depending on those [08:21] Depending on those things, [08:24] you're going to create a certain set of possibility space of outputs that [08:30] that sort of constrains the user. [08:33] Um, [08:34] And I think in your view, basically, like, [08:38] how that is selected and communicated is like, is maybe important because, um, [08:43] Um, [08:44] But like you said, like Dolly, you can't get it to make an ugly image. It's not built for that. So it kind of like takes away agency from you because it's doing some stuff for you in a way that like – [08:53] I don't know, maybe like Photoshop, for example, [08:56] You can just manipulate the pixel. So, so there's no, uh, there's no agency taken away in that experience. Is that kind of what you're getting at? [09:04] Yeah, yeah. I think there are a lot of different pretty subtle ways to [09:09] that either intentionally or intentionally shapes the kind of agency that

[09:14] landscape of a tool. One very explicit example is a tool like [09:19] Um, [09:20] Thoughty 3, where... [09:22] The model is, through training, made so that it has a – it's – [09:27] sort of fundamentally unlikely to output bad looking images [09:30] I think... [09:32] There are other examples. So like Photoshop is actually an interesting example because... [09:36] even if in theory you could kind of make everything by just like [09:39] moving the raw pixels around. I think that the specific... [09:43] the specific features that are easy to access tends to also shape the kind of style of output. And so like if you look at popular image editing tools, like you can tell when an image is made in Instagram stories or you can tell when an image is made in Instagram stories. [09:56] has its background masked out by like, [09:58] keynote or something. And so all of these tools, even in pretty subtle ways, kind of [10:03] tend to shape the output style. [10:05] And I mean, I feel like that's part of what makes... [10:10] what makes art in general. Like, you know, if you think about the music styles that are popular, it's like, [10:16] based on what the sound boards can do and all the weird effects, or like the, you know, the electric guitar and the weird effects that like Jimi Hendrix discovered, like, [10:26] Those are, in some ways, they're like limiting the artist's agency, but they also create this unique set of constraints that creates a unique vibe and sound that represents a genre or a generation. [10:38] Yeah, you touched on something really interesting there. Imagining the output space as a very concretely spatial kind of thing.

[10:46] This is... [10:48] for me very concrete because I think a lot about embeddings and embeddings exist [10:53] in a very concretely kind of spatial way. [10:56] space and embedding is [10:59] a list of numbers that try to summarize [11:02] quantitatively what some piece of text or what some piece of image semantically contains. And so an example use case of an embedding. So you have an embedding model, which is kind of like a language model with its like, [11:13] Last. [11:14] part of it. [11:15] chopped off so that we can read out the raw numbers. And you can have an embedding model and you might feed an embedding model sentences like [11:22] The Eiffel Tower is in Paris and the president of France is blah. And and like notion is a tools for thought company. And two of those sentences are much closer in meeting than the third one. And so when you feed them into an embedding model, the embedding model will spit out for each of these sentences a list of numbers. [11:39] And when you view the list of numbers as kind of a coordinate, [11:44] or points in space in a high dimensional kind of coordinate space. Those numbers are going to be closer together for sentences that are closer together. And so-- Yeah, the way that I think about it sometimes is like, [11:54] you can take text and then assign pieces of text like a latitude and longitude coordinate, and that's an embedding. [12:01] and then you can map the text and then see which pieces of text are closer to each other. And the ones that are closer on the map, the ones that have a latitude and a longitude that are closer, are going to be similar in meeting. And I think that that, like... [12:12] Yeah, that sort of clicked for me where I was like, oh, that's what it is. [12:14] Yeah, yeah. Another metaphor that I sometimes use is like a...

[12:18] like a [12:19] color picker. So if you use... [12:22] any kind of image editing tool, one of the ways you can pick colors is like a 2D grid or like a color circle. Yeah. Where you have two dimensions, you can move things around and then you move the data around and you're changing kind of the RGB values. [12:34] There are a couple different ways to describe a color. One way is by an RGB value, which is kind of like a coordinate or a latitude length for a color. [12:40] Another way you could describe a color is by just saying like, [12:43] orange or like very dark blue or crimson. Yeah. And I would, [12:48] In my head, the dark blue or crimson word description is the input, and then conceptually the RGB value is the embedding. I think that's really interesting because it hints at this possibility that [13:00] uh, [13:01] uh, [13:03] embeddings where these like numerical representations might be [13:07] Encode. [13:09] kind of [13:09] pretty fundamental semantic approach [13:11] insights about the thing that you're encoding, color or text, in a way that lets us kind of mathematically manipulate it to do interesting things. Like... [13:19] if you have the word crimson, you can't really like [13:23] You can't manipulate it [13:25] in ways like making it lighter or making it more blue or whatever. But if you have the RGB value, there are very [13:31] Um, [13:32] concrete algorithms that you can use like number crunching to do to make the color a lighter shade of crimson or like a more blue or more purple. [13:39] Right. [13:40] No, I love that. I mean, that's sort of very central to, I think, some of your thinking around this, which is like the current generation of AI tools are awesome. Chat is great. But, you know, manipulating images or text via chat is it's a very coarse grained tool and it's it's a lot harder to get the exact thing that you want.

[14:00] And, you know, I think it's a good thing. Yeah. [14:02] I think you've thought about a lot. [14:04] AI interfaces that allow you to [14:08] be more precise about how you move through [14:12] different ways that you want to modify an image or a text using AI. [14:16] Yeah. And I think thinking in that way, the... [14:20] spatial way of thinking about... [14:23] the possibility space outputs of these models. [14:26] um, [14:26] The spatial view, I think, is... [14:29] Uh, [14:30] kind of behind the way that I think about how to add more precision to these kinds of tools. Yeah. One kind of, I don't know if, should I just pop into a demo? Yeah, let's do it. So... [14:40] The interface for this is very bad because it's just... Wait, so let me just back up. So this is a tool called Varna, and you built this yourself. Yes. This is one of a string of experiments. I'll make sure some other ones later... [14:53] But... [14:54] I... [14:54] This is actually half built. I was in the process of building something different, and then... [14:59] It started being useful and I didn't really feel like putting more effort into it. And so it's in a very half-finished form, but it's good for demonstrating this particular thing that I'm about to walk you through. And what is the general thing that you wanted to build it for? What was the original idea really quick? [15:15] The original idea was... [15:17] to try to let you do [15:21] to try to let you describe an image that you want to create by adding and subtracting images and text. [15:27] So, um, [15:29] There's a model called CLIP. [15:30] by OpenAI. This is actually a model that's been around for a while. If you use any kind of multimodal, like,

[15:36] use text to semantically search images, kind of search engine tool, clip, [15:41] It's likely to be one of the models behind that kind of thing. The special thing about Clip is that it tries to [15:47] describe both text inputs and image inputs in the same embedding space, in the same coordinate space. So that if you put in a word like Eiffel Tower and a French flag, the model is going to understand the meaning behind both of those things and try to cluster them together because they're similar. [16:04] So [16:05] A slightly newer thing that happened after Clip came out is that people came up with a way of generating images conditioned on a specific point in this Clip. [16:16] embedding space so that you could put in an image or you can put in a text and then [16:20] try to generate an image that is kind of conceptually the image [16:25] that corresponds to that point in the space. If you put in a bunch of images of... [16:30] butterflies. They all kind of cluster around the same point in this embedding space, in this coordinate space. And if you pick one of those points in that cluster and then generate an image back out corresponding to that point, you would get some image of a butterfly. And so [16:42] Given this, [16:43] ability to both take an image or text and encode it into the space and then [16:48] Take a point in the space and decode it back out into its corresponding original image form. [16:52] Maybe you can generate an image not by just typing a text prompt, [16:54] but by... [16:56] mixing a bunch of concepts together in this space. So if you want a butterfly that has the colors of the French flag, maybe you can [17:04] Mix. [17:05] the image that is of the French flag and an image of a butterfly is,

[17:09] Just... [17:10] pick a point in between these two concepts in the embedding space and the decoded out, and maybe it'll mix those concepts together. So that's the kind of idea I was trying to explore with this. [17:18] little hack. [17:19] The thing that I'll show you now is here, I'll pick... [17:24] Thank you. [17:25] Thank you. [17:26] This is a selfie of myself. Um, [17:30] for... [17:31] dumb demo reasons I have to put myself in twice. And then this is an illustration. This is my, actually this is a PNG. Let's do something different. [17:40] This is a kind of clip art illustration of a human face. Um, [17:45] and [17:47] I'm going to, so what this tool is going to do is kind of embed both of these images. [17:51] They correspond to slightly different points in the clip model's embedding space. [17:55] And then I'm going to use... [17:58] this tool to pick eight different points along the line between these two images, and then decode each of them out. [18:06] Um, [18:07] into its own image. And so we're basically sampling [18:11] the distance between these two images. Yeah. In this in this embedding space and then trying to look inside the model and see kind of what the model sees at each of these points. [18:20] That's very cool. [18:21] A way that I would think about this is if you're used to Photoshop or something or Instagram or any photo editing capability, they have all these sliders that you can do for make it brighter. On the left, it's very, very bright, and on the right, it's very, very dark, and you can just slide in between. What you're building here is...

[18:44] a slider between almost like concepts or ideas for images. So like one on the left side is like your headshot and on the right side is like a sort of [18:55] vectorized drawing, cartoon drawing that is in a style that you like, and you're just sort of seeing [19:00] if you slowly turn your [19:02] your headshot into that, like what are the different points along that space? [19:06] Right. You could also... [19:09] Speaking of sliders, you could also not just slide along this kind of stylistic scale, but you could start to do fun things like slide along a... [19:17] an emotion scale. So one thing that I've done before here is [19:20] I'll try to go from an image of a young man who is happy. [19:30] to [19:31] an image of a young man who is very, very angry. [19:39] There we go. Now you get progressively more and more mad. You can see the effect is really intense. So at the end, it kind of destroys the image. So sometimes I turn down the text effect. [19:48] Um, [19:48] The way that I arrived at [19:52] So this is one of many experiments that I've done to try to... [19:55] build interfaces around exploring embedding spaces and latent spaces of [19:59] uh, [20:00] of generative models instead of talking to the model directly. The reason this is interesting to me is kind of twofold. [20:06] One, I think it's [20:08] just intellectually really interesting that in the process of [20:11] learning how to predict the next token or in the process of trying to model probability distributions of language, the model internally learns and learns to pull out,

[20:20] Uh, [20:21] kind of human recognizable, semantically useful concepts like. [20:26] motions. [20:27] And I think learning how models do that and understanding it to improve models or make them more controllable. [20:33] I think is useful. [20:34] And just the fact that the models do that, I think is interesting. So I want to understand it better. [20:38] I also think when you look at models this way as a thing that gives you dials and as a thing that gives you numbers that you can manipulate, [20:45] language models [20:48] start to feel less like Asians that have their own kind of agency and more just like, Oh, this is like a calculator. Yeah. [20:52] Right. [20:54] Simon Wilson, who's one of the prominent bloggers in the space, actually has a really great blog post about... [21:01] language models as a thought kind of a thought calculator. [21:03] um, [21:04] where [21:06] he talks about a similar idea. [21:08] Um, [21:10] One way to look at this demo is a kind of a calculator for images and concepts and text. [21:16] And unlike asking Dolly, the experience of asking Dolly three through Chachapi teacher generate an image which feels like. [21:22] You're asking this entity, this agent for requests. [21:24] Um, [21:25] Here, it's like, okay, there's very concretely, like a thing that does math, and you can use it to get dials and control. And that feels more like a tool, and it feels like, [21:34] Perhaps there's [21:35] a direction here that lets you retain more agency as a person. I feel like one of the things I'm getting from this is [21:42] When people use these AI tools, this generation of AI tools for the first time, there's this like, whoa factor. Like I was using it the other day. I had like done a lot of journaling. I've been going through some like stuff in my personal life. I've been doing a lot of journaling about it.

[21:56] And, [21:57] I fed like, I don't know, 4,000 words of journals into Claude. I did Claude and ChatGPT, but I find Claude to be like slightly better for this. [22:06] And it, I'll put it a bunch of, you know, I ask it like, what am I not seeing? What are the patterns that you, you're observing my psychology or the psychology of people around me? Like all that kind of stuff. And I'll put all this stuff. [22:17] And I screenshotted that and I sent it to my therapist. [22:21] He was like, that's totally wild. And I was like, cool. Like, he must think this is awesome. He's like, it's so good. And I was like, this is amazing. And then he was like, I thought we would have at least a couple years until it would be this good. And then he was like, do you think there's, do you think that there's room for like human therapists anymore? And I was like, oh no, like now I have to be like comforting my therapist because he's afraid of the AI. And like, yeah. [22:43] My feeling is like he had that immediate like moment that everyone has. But if you really dig into these tools, they do amazing stuff. But like they're not even close to, in my opinion, like. [22:54] for example, replacing a therapist, [22:56] Um... [22:57] And I think part of the reason why people have that, which is what you're I think what you're referring to here is we just respond to things that feel really intelligent. [23:06] And if the same thing was performed in a slightly different way, like with a slider like this, it would still be cool, but it would feel more familiar, like a tool that we might use anyway, like a calculator that is in itself sort of mind-blowing, but isn't threatening or sort of – it's not like a replacement in the same way that like a fully intelligent thing is. Is that like a fair way to describe what you're saying?

[23:32] Yeah, I think so. I think so. Yeah. When you train a model there, there are fundamental capabilities you bake into it and the packaging that you wrap it in, um, [23:40] really dramatically influences how people receive it and they use it yeah [23:43] you know, [23:44] If you took some human neurons and just laid them out in the right way and wrapped them in an interface, like... [23:51] It might look like a tool too. [23:53] Um, certainly. And, um, and if you just get enough of those, if you just get enough of those neurons together, um, [24:00] uh, [24:01] you get a brain that's like conscious and can do things and think like, [24:06] How do these two things fit together? That's an interesting question. I haven't thought about that. I think off the cuff, my... [24:14] it. [24:15] kind of intuitive reaction is [24:18] Uh, [24:21] I think it's an interesting... [24:23] it would be an interesting intellectual approach [24:27] and societal endeavors to try to build [24:30] a thing that is like a simulacrum of, [24:33] humans in every way, including [24:36] having some kind of [24:38] Uh, [24:40] goals and agency and so forth and so forth and i think if that was the goal of like building a simulacrum of everything that that at least intellectually makes us human [24:48] you would want to build in [24:50] uh, [24:51] elements that these current tools don't have. [24:53] Um, [24:54] Like trying different things at exploration. I think exploration, intentional exploration is actually a huge part of intelligence that humans have that these models currently don't exhibit.

[25:02] However, [25:05] I think when companies like Google and Google, [25:10] OpenAI and even Notion tried to build [25:12] kind of language model based tools that help you do your work. [25:15] Um, [25:17] There are a lot of parts of... [25:20] a human intelligence that actually are kind of annoying when you're just trying to like get some work done. Like, uh, [25:25] the fact that humans, like, sometimes just sit at their desk and daydream is, like, probably not that useful if you're trying to just, like, hire an entity to, like, [25:33] you know, read thousands of papers and try to summarize them. And so I, my gut feeling is that a lot of these kind of corporations and teams that are building AI to be useful, and especially in a professional context, their goal is not to build a single acronym of humans, their goal is just like build a thing that is like an intelligence kind of engine. [25:50] Steam Engine and then [25:53] I could imagine some other research groups or some other groups of people whose goals would be to build a similar spectrum of humans. But I think there's, in the beginning, I think they tend to look similar. But I think as we get further down this road, we're going to see a kind of divergence between people who really want to, like, build a similar spectrum of humans versus people who just, like, [26:09] Want. [26:10] an intelligent thought calculator. I want to, um, [26:13] I went to OpenAI Dev Day and they did this whole presentation on how they fine-tuned GBT4 using Slack messages. Right. And did you see this? And basically, like, they asked this fine-tuned version to do something and it was like, no, I'll do it later. And then it was like, do it or you're fired. And it was like, okay, I'll do it. Yeah. You're right. There's all these, like, parts of being human that, like, maybe we don't necessarily want to model for.

[26:39] for the AI as we build and there's some divergence there in terms of like, is it a tool or is it something that has agency and all that kind of stuff? [26:46] Yeah, I think I've run into something actually very similar where I was when I have a personal chatbot that is sometimes used for like. [26:53] Kind of brainstorming kind of entertainment purposes. Can we see it? [26:56] Ah... [26:58] I can, hmm. [27:02] Here, I'll open it up. You have a personal chat bot. I have a personal chat bot. It's an iMessage bot because I like having it in iMessage because it's accessible kind of from all my devices. Apple does all the syncing for me and it looks nice and I can react to things. [27:15] I haven't used it in a while, but I could ask it something like, [27:19] Um, [27:19] Well, it looks like you asked it, like, what makes Notting Hill particularly picturesque compared to other neighborhood areas like Westminster? Tell me about that. I was visiting London and I was trying to look for... [27:31] Neighborhood statistics. Actually, we can ask something about only we could say, what do you think? [27:36] Think about London. [27:38] versus New York. And why would you ask this versus like asking ChatGPT? Like, how is this built? What is it trained on? Why is it its answer better? [27:48] Where would you rather live? [27:49] as a 25 year old. [27:54] There's lots of interesting things about this. So first, sometimes it's kind of unreliable, but usually when it receives a message, yeah, it'll say read. Yeah. [28:02] And then sometimes it'll have a typing indicator, which is kind of... [28:07] The way that this works behind the scenes is just when my server receives the message and starts generating the output,

[28:12] It sends a red reply. [28:14] But conceptually, it's like, oh, the AI read my message, which is like, [28:18] I know that doesn't actually mean anything because it's just like a bag of numbers, but this is interesting. It's just conceptually interesting. It's one of those same things that makes it feel human, you know? Right. And then like sometimes I'll send a message, I'll lock my phone and then I get a notification. [28:34] that it sent me a message. I'm like, whoa. [28:37] Like... [28:37] Computers don't normally just send me notifications for things that it's thinking about. It's really weird. So the packaging, again, is very important. [28:44] So... [28:45] So the reason I brought this up is because the model that backs this [28:49] is actually... [28:51] not a fine-tuned or RLHF model. It is the raw base model of, I think, LAMA 2 13 billion parameter model. Mm-hmm. [29:00] And Lama is Meta's open source model. Lama is Meta's open source model. One of the things that make that model special compared to things like GPT-4 is that, so it's open source. Obviously, I can host it on my own, which is what I'm doing here. [29:14] But another thing that makes it special is that they released a version of the model before they did all of the fine tuning and RLHF to make it [29:22] chatting. [29:23] to have it follow this kind of chat form. And so the base model is purely a text continuation model. So if you ask it for something like, [29:33] if you just ask the base model something like, where would you rather live, New York or London? It would interpret that. It wouldn't really interpret anything. The model's task is to predict the next token in internet text. So it would probably interpret that you're in the middle of, like, a blog post about the best place to live. And it would just continue writing, like, a blog post or something instead of asking the question. I had to...

[29:53] add a bunch of prompts in front of it. So I'm prompting a base model, which is, you know, in the ye olden days of 2020, what we used to do before all these, like, instruction following models existed, people would just ship base models and then prompt them. So you're prompting a base model here, but, like, why are you asking... [30:10] this model versus ChatGPT, like, what is it – how is it more like you or more interesting to you? Right. So – [30:20] Here I've asked it, what do you think about London and New York? Where would you rather live? If you asked it to ask this question to like Chachapiti or Claude with, I haven't done that before, but I would guess. [30:33] that probably with high likelihood, the model will probably say something like, oh, I'm an AI language model and I don't actually live anywhere. So I don't actually have any preferences on which cities I would live in. But here are some things that you could know about London and New York. And I'm like, that's not really what I'm looking for. I'm looking for like, [30:48] a kind of a personal take and it doesn't really even matter if that take is like correct or not. Cause I just, I just want like vibes, you know? And, um, [30:55] because this model hasn't gone through any kind of like fine tuning about the fact that it's an AI language model it's just going to generate text as if I'm talking to like a rando on the internet and I've done some prompting so it's not just a complete rando it's like a fairly intelligent cogent rando but it's gonna it's gonna say things as if it's like a human on the internet because the base model is just trained on text that is like mostly humans on the internet so it says here [31:21] New York's number for whatever. Ultimately, the choice of where to live would depend on the preferences, but like it gave me some opinions. It's still a little bit kind of professory, because in the prompt, I tell it, it's like an assistant, but I could have, I could have just prompted the model to be like, hey, you're like Mickey Mouse, and you live in New York, and it probably has like followed that.

[31:41] And so it'll pretend it'll. [31:44] you can prompt the base model to be much more just like friendly. Got it. [31:48] When I used to use a dumber base model for this, [31:50] which was GPT-J, which was a six billion parameter model that's trained on way less data. [31:56] It was even dumber. And so... [31:59] Sometimes when I asked it questions like this, that model would say something like, hmm, I'm not sure, but I have lunch. I'll get back to you after lunch. It like emulates human conversation, but it's not really useful. I think I found the 13 billion Lama model to be like a good balance. Sometimes it gives creative answers that are kind of unexpected, and sometimes it gives cogent answers at all. That's funny. I love it. I think that's really cool. I want one. Sign me up. [32:23] Yeah, I'll send you – there's a private number you can talk to. Perfect. [32:28] Cool. I want to get back to sort of like some of the things you wanted to share. I know you had a couple of ideas. I love that you came with a whole Notion doc. [32:37] prepared for what you wanted to talk about. So I know you had a couple ideas you wanted to talk through. I want to make sure we get to those things. And then I do want to jump into specific chat TVT chats and all that kind of stuff. Cool. Let's do it. We already talked about AI as a tool. [32:53] Before we got into specific chat transcripts, I thought... [32:56] Um, [32:58] I was thinking about what we were going to talk about today in the shower in the morning, and one kind of shower thought that I had is that – [33:04] So, [33:04] When I use Chagiput, the way I think about prompting is actually quite different than... [33:09] when I do prompt engineering, [33:10] at my day job at Notion, writing prompts, which I do a lot of. Prompt engineering is sometimes like

[33:17] the part of my job as an engineer that makes me kind of feel comfortable [33:21] the dumbest because sometimes my changes to the code base will just be like, [33:27] adding a must to some English instruction or like changing the instructions that it sounds like even more desperate. But that's like, that's like a useful product change. So some days I just like spent the entire day prompt engineering. It's like, what am I doing? Like, why did I go to school for computer science? It's like, this is what I'm doing. But the, the mode that I, my brain is in for the revenge of the English major is happening right now. I love it. [33:52] The mode that my brain is in for prompting when I'm prompt engineering, um, [33:56] a prompt that's built into a product like notion is very different than the, the kind of, [34:00] prompting brain that I have on when I'm talking to Chachi PT. And I kind of compare this to [34:05] Um, [34:06] Thank you. [34:07] like scripting versus software engineering and programming. [34:12] Let me explain that, unravel that a little bit. [34:16] programming is like [34:18] I would say programming is like this super set of activities that just involves any kind of writing. [34:23] computer programs. Um, [34:25] Sometimes you write computer programs [34:27] And you like ship it to millions of users and this thing runs on hundreds of thousands of computers and they have to run very reliably they have to accept lots of diverse kinds of input and always generate the right answer. So it's like a very robust resilient kind of well tested piece of machinery. [34:44] And when you build a product like Notion, you're kind of doing software engineering where you're writing programs to be robust and reliable. Other times, you write programs, but you're writing programs just to accomplish something quick and easy. Like, you might write a simple little command line.

[34:59] command line command or script to like search your folder for a keyword or like list of files that you have just so you can look at them or maybe even a slightly more complex one might be like [35:11] iterate through all of the files that you have and delete any file that is older than six months. Like that's like a quick and easy thing, but that's not going to, you don't ship that to like millions of users. You just have to run it that one time. And if you make a mistake accidentally, you'll realize it and you'll like rerun the command. [35:25] And so, [35:26] these two categories of [35:28] activities i would both call programming but one i would call like software engineering and the other i would just call like [35:34] scripting [35:35] They're like writing scripts or like writing commands. [35:37] um, [35:38] equivalently, I think when you're writing [35:40] prompts for language models. [35:42] There's a broad category of things that I would call prompt programming. [35:45] And then some kind of prompts you write... [35:49] like the kinds of prompts that I write at work, are prompts that we write and we iterate on, and we test and evaluate very heavily, and then we ship to millions of users, and they put in all kinds of input to this thing, [35:59] And so they have to be very robust and well-tested. [36:01] and they have to accept lots of different languages and so on and so forth. And so that's what I would call more like prompt engineering, where you write prompts and evaluate them robustly and things like that. When I'm talking to ChachiPT, though... [36:13] I don't have to be as robust. I also have opportunities to make revisions if the first time the prompt runs it doesn't do the right thing. And so when I talk to ChachiPT or other kinds of tools like that or Notion AI and Notion, I would say that's much closer to like. [36:27] the scripting part of the metaphor, where I'm just

[36:30] trying something and if it doesn't work, I'll just like fix something and try it again and kind of iterate my way towards the final result that I want. [36:36] And it's not this thing that I want to make like a robust piece of machinery. And so I made that distinction because when you're doing like hardcore kind of prompt engineering, all those techniques around evaluations and few shot prompting and, [36:49] using kind of structured outputs. [36:52] apply and they're very useful. When I'm just talking to Chachi Bt, though, [36:56] Good few shot prompts are really hard to write. Like the specific examples that you pick influence your output quite a lot. And so I actually almost never write few shot prompts. I'm just talking to chat GPT. I just write zero shot. And then if the output isn't exactly what I want, I'll usually ask a follow up. [37:11] question to like revise the output. And so I think that distinction was interesting. I love that distinction. And I feel like that's that just mirrors something that people... [37:21] kind of miss about Chachapiti when they use it at first, which is [37:26] um [37:28] They ask, oh, how do I use it? And they try to use it perfectly. [37:31] Instead of just being like, I could just bang away at this thing and try everything possible, you know? And I think that second one is way better because... [37:41] it, it, it, it's such a broad tool. It has so many different things it can do. And we actually don't as a, just as human beings don't even know all of the things that can do yet. And the best thing that you can do is start with something simple and then just keep banging away until, until you get something that you want. So you can sort of like learn the limits of what, of what it can do. And I think people are like way too reluctant, for example, to, um, ask it to redo its response or, uh, revise something in the history of messages and just be like, let me try

[38:11] And sometimes it almost feels impolite. [38:14] to do it or something like that. But I think the best users of ChatGP teachers know how to just [38:21] not accept anything until it, until it's working and just keep iterating until they find something that that is great. [38:26] There's a fun example here where I had to... [38:29] Um, [38:30] I had to look through like a JSON file of some data. And I didn't really feel like constructing an elaborate prompt to try to ask it exactly what to do. So I just uploaded the CSV to Chachputi and I was like, [38:40] I think I literally just asked it. [38:42] what's the vibe of this file? [38:45] And I was trying to find it, because I think the answer was actually really good. Try to find it. We can take a little time. You don't have to rush. I love that. Like, I just think... [38:53] It's one of those things that it's so funny because – [38:56] Before ChatGPT came out, before GPT-3, before GPT-2, all that stuff, [39:01] There are all these people, like these intellectuals who are like, [39:04] constructing scenarios for how [39:07] AI could kill us. And a lot of them hinged on AI, basically like misunderstanding what we said and doing something different. [39:14] So if you say like, oh yeah, like, you know, make paperclips, it will just like misunderstand that you, you, you meant like maximally make paperclips and just turn the entire world into paperclips. And yeah, [39:25] So that was the AI that we're afraid of. And the AI that we got was... [39:30] what's the vibe of this file? And it just like figures it out. And it gives you quite a cogent answer. Yeah, I think there are some theoretical... [39:36] Um, [39:38] theoretical ways and concerns.

[39:41] That does still kind of take that form. Yeah. But certainly in the, in the way of like instruction following, I think we're making a lot of progress and like aligning what the model wants to do with what we want to do. [39:50] Um, [39:51] Yeah, I mean, the model like read through all the columns and it was like, here's generally what the file is about. Here are the different columns and like, let me know what you want to do. [39:58] So that was fine. [40:00] Um, [40:01] I can walk through some other examples, though. Yeah, let's talk about it. Yeah, so... [40:07] Probably the most... [40:09] Common use case that I have for ChatGPD is actually just programming help. [40:12] So you don't use like a co-pilot for it for that? [40:17] I do use Copilot. I use Copilot heavily. And I think learning to use Copilot is kind of a different... [40:22] learning curve and learning to use ChattrPT. Copilot is very... [40:26] CorePilot's great because it has... [40:29] all of the context of the file that you're actually in. And I think as a general rule, AI tools are better when they have as much context [40:36] AI tools can give you better suggestions when they have more context about what you're trying to do. And so one of the philosophies behind notion AI is that it has more context about the work that you're trying to do inside your workspace in notion. And so it tries to be better. Same with copilot. It has, [40:49] all of the kind of types and definitions and functions that I'm using inside my code editor and so it can provide better suggestions. But the suggestions that it provides are kind of micro level. [40:57] It's like add some lines here, add some lines there. So... [41:01] Compare that to, let's take a couple examples. So like earlier this month, I was trying to write a bunch of, [41:08] like, [41:08] data processing scripts for these are Python scripts that take

[41:12] millions of pieces of text and try to process them in some way. And so I had to learn how to [41:18] uh, [41:19] do this in kind of a streaming way because I didn't have enough memory to fill all of the data in my computer. And I had never used a streaming kind of dataset library before. So I learned about a library called Arrow and I was trying to use it. This is the kind of thing that you'd be able to use Copilot for because... [41:33] it's not, Copilot is great if you have like a file with some content already in it and you mostly know what you're doing, you're just trying to figure out like the right, [41:40] method to call or the right function to use or the right variable to use. [41:44] But here I was asking a very open ended question of like, [41:48] like here's a library that I'm using, how do I make it faster? Or like, how do I even approach this kind of problem? One example that I linked here is like, [41:59] uh, [42:00] This isn't about the dataset thing, but this is about [42:03] uh, [42:04] I was building a library for processing UTF-16 encoded characters. And I was just trying to understand how these things are represented and then maybe get some help writing a basic implementation of, like, how to parse this format. [42:18] And this isn't, again, the kind of thing that Copilot is useful for because I didn't even have like a basic template for this thing. I was just trying to understand how we should approach this problem. And so I asked it, how is it encoded at the byte level? Just trying to get an understanding of what this format is. And then I asked it, is there a simple algorithm? Because I thought if there's an algorithm I can understand, I'll just write it. [42:38] uh, [42:39] in the language I was using. [42:42] And then when I asked it for the algorithm, it even gave me like a Python implementation, which I based my implementation on.

[42:47] But then... [42:49] I ask it follow up questions to kind of explain parts of the code that it didn't initially explain. [42:54] And [42:55] This is kind of a high level problem in breaking it down that it was useful for. [43:00] Another example might be like this. [43:03] which is a very long conversation. This is, I think, a whole programming session that was a single chat where... [43:11] Thank you. [43:12] I began again with a very high level problem of like, what's a good way to like, [43:16] read this very, very large file of data incrementally in a streaming way so that I didn't have to use up all of my computer's memory. And it initially gave me some approaches. And then I start steering it to like, okay, I don't just want to use PyTorch. I actually want to like use a different format because it's funny. [43:33] And over time, I try to converge onto a solution. I think that's a really smart one, is in addition to the, okay, start with something quick, [43:41] and iterate from there. It's also start with something broad. [43:45] And ask ChatGVT to help you understand how to prompt it or what it thinks the best solution to a particular kind of broad question is going to be. And then gradually kind of narrow in rather than starting with the narrow, maybe something more narrow that you just wanted to answer. Is that a good summary? [44:07] Yeah, another way that I've thought about that is the more context the model has about [44:12] why you're doing what you're doing or what your goals are, the better suggestions you're generally going to be. And so instead of asking it,

[44:19] how do you read a [44:21] arrow formatted file you could ask it i'm trying to read a data set that's 80 gigabytes large into a computer with 40 gigabytes memory i'm on linux i'm running this version of python right um how should i approach the problem and it'll give you a few approaches and you can kind of [44:32] Do you have custom instructions set for this? [44:35] I don't, actually. Oh, my God. We're going to have to revoke your chat. Oh, I know. I know. I think I haven't found it. If I used it, I would probably use it to make the outputs a little more concise. But I actually haven't. I haven't found the normal tone too offensive. [44:49] Interesting. Okay, cool. Well, let's go back to your example. So you had something to show first about soul neighborhoods. [44:55] Yeah, so this is – I did some traveling recently. I was in Seoul. I was in Thailand for a bit. [44:59] And I thought this example was interesting because [45:02] From the outset, it looks like I just asked a question, what are the major neighborhoods to see when I'm playing a trip to Seoul, and then what is it known for, and then I just have a single output. But actually, if you look closer, there are four outputs. [45:14] Or this one's kind of borked, but there are three outputs. [45:18] This is, I thought this was worth noting because language models are non-deterministic, [45:24] If you ask a very broad kind of recommend me something to do kind of question like this, [45:29] every time you call it, the results are going to be slightly different. Yeah. And what I want to know really is like, what are the best neighborhoods to visit? And so I asked a question like this, one of the most interesting neighborhoods, and then I ran it three times. [45:40] And each time the results were a little bit different, but there was like the overlapping set. [45:44] and the overlapping kind of descriptions and focus. And so I read through all three, [45:48] and then just try to pick the ones that seemed like they were mentioned the most.

[45:52] Some of these I think also involve [45:55] uh, [45:56] searches. So here I did a Bing search and then gave you some results. And so that's brilliant. By looking at multiple samples, I'm able to get a better sense. So I did something similar with Bangkok as well here. Wait, so okay, let me just let me just see if I get it. So basically, you're going you're going on a trip, you want to know what the best places are to go. But rather than saying what's the best place to go, you're asking something like, [46:21] What are some like you're asking a broad version of that rather than having it filter for you? [46:26] You're asking like, [46:27] The broadest thing, which is where could I go in Bangkok or where could I go in Seoul? [46:32] And then every time it's – when it responds to you, you're just clicking that little redo button. Can you click it for us? I want to see if it does another one. Yeah, let's see here at all. You're clicking, like, the redo button. [46:43] And then it's re-outputting it. [46:45] And basically, like for a broad question, it's going to be slightly different every time. [46:49] And then you're being like, okay, I'm just going to read through the things and the most [46:53] the most commonly mentioned things are probably like the top things. Yeah. Yeah, exactly. Why, why not just like ask the top for the top things? Um, what, what, yeah, what's the difference, what's the difference in your mind? I think this is again, the agency thing, like, um, [47:11] Sometimes in my life, if I really, really trust the person that's giving me a recommendation. He's the best friend. I've known him for six years. He knows both the city really well and me really well. If he says, if you have an hour, this is the only place you should go to, I'd be like, okay, Alex, he's probably good. He knows me really well. But Chachamidhi doesn't really know me well. I think even if I had a 5,000 word...

[47:31] custom instruction it would be very difficult to describe everything about me in that description [47:34] And so instead, I'm asking it for, like, [47:37] Give you some options and then [47:39] I, knowing myself, can then now be a little more sure that instead of having to judge its filtering and trust its filtering, I can do kind of last mile filtering. [47:48] That's really interesting. I feel the exact opposite. [47:53] I have like a very long, I have a very long custom instruction and, [47:58] I really find that it helps it... [48:01] know who I am and what I might want. And then I also find that it's really... [48:06] It's very cool for being like, I'm in Bangkok and... [48:10] I have these requirements like I don't eat XYZ and I have this amount of time and I don't [48:16] I was just at like five temples yesterday, so I don't want to do any temples. [48:21] And I'm a nerdy tech guy that like, you know, whatever. And what should I see? It's like very good for that, like pushing it into the – [48:29] pushing it into, into a space that you like. I still think it's really useful to do the, um, [48:35] Just do the redo thing a bunch and then see what comes up. But I personally find the filtering and telling it more about you is really nice. And I think that there's probably just a personality thing, which is like... [48:48] You just want agency. [48:49] Um, and I'm like, give me the best answer. I don't care. I think there's also... [48:55] I'm also, whenever I... [48:57] depend on a language model to do any kind of like filter management. I'm always

[49:01] slightly concerned in the back of my mind about [49:05] specific things that I put into the prompt that might like really buy us the output. So, [49:10] uh this prompt is actually interesting because first i tried to give it as much context as possible and so like i said i'm here for one morning and afternoon it's a thursday i said it's a thursday because maybe some things are closed on thursdays yeah near queen's park because maybe which are but because i said queen's park which is a particular park in um like a part of bangkok um a lot of the suggestions that the model generates are like oh you can spend the morning at like queen's park and um [49:34] It's a nice park, but it's not like the biggest park in the city. It's a park. But because I put this specific string into the prompts... [49:45] I think given the kind of early shift tuning that the model has had, the model is like, oh, like, [49:50] The human rater will think the output is better if it's more aligned with the instruction. And so I'm going to really mention all of the aspects that were in instruction. And so I'm always a little bit afraid of... [49:59] any kind of instructions over steering the model. And I think very heavily customized custom instruction would [50:05] um sometimes give me the same worry as well like maybe it's too it's biasing a little more than i would want on like what i specifically want i think you're right maybe i don't want that it does do that it's like it's it's such a yes man like and so it connects everything back to my custom instructions sometimes and i'm just like that's that's not relevant like don't don't do that yeah yeah i want to be like a generic human i think i'm like willing to put up with the annoyance of that because everything [50:31] Every like 10 things it says, it says something totally brilliant. And I'm like, I cannot believe you just did that. And so I assume it'll get better over time. But there's definitely a cost to it.

[50:42] I have a couple other kind of use cases and things that I do when I'm prompting that I don't have specific examples for, but we could try one that I think are interesting and worth noting. One is... [50:54] uh, [50:55] Continue writing. So continue writing, this doesn't really apply to if you're using ChatGPT, ChatGPT, but sometimes in other tools like writing tools like Notion, there's like a continue writing option. [51:05] Um, [51:06] This is... [51:07] Interesting. So what this does is it tells the model, hey, let's start here and just like [51:12] pretend like you're just continuing the text, like, [51:14] pretend like you are the next token prediction bot that you were originally made to be. And then just like, keep writing. Um, [51:22] And I think it's interesting because there's kind of, first of all, it's a very kind of brain-dead thing to try. And so this is kind of a weird doc to try to do this in. But if I have like a... [51:32] Doc. [51:34] um, [51:35] I don't have any that I can go through immediately. But if I have... [51:38] If I'm writing about HCI or writing a blog or making a list of something... [51:43] Sometimes... [51:44] uh, [51:45] I won a kind of a brain dead podcast. [51:47] like just give me more ideas like this button and the continue writing prompt is kind of like a give me more ideas like this button or like continue on my thoughts without like any let's make a list um what i want to do okay so i have a bunch of books in front of me and i want to see if i can recommend books uh based on the books i have in front of me um okay so i want to i've i've read and liked uh okay uh the brothers karamazov uh

[52:13] Thank you. [52:16] I'm going to have trouble spelling that. [52:18] K-A-R-A-M-A-Z-O-V. Okay. Okay. [52:24] Let's see. Medieval technology and social change, which is really good. It's about how the stirrup, like, changed everything in the Middle Ages. [52:37] Exhalation, Ted Chiang. Let's give you another one that's sort of like left fieldy. The Essential Kabbalah. And I'll do one more... [52:46] I feel like I'm missing some stuff that I would normally. [52:49] Let's say Defense of Socrates. [52:54] Okay. Defense of Socrates, not Defense. I think that's a good book title. That's a good prompt. [53:03] Okay, continue writing. [53:06] Okay. Okay. The Gene, Intimate History, good. I cannot believe it – [53:12] Interesting. [53:13] It's very generic. Okay. So it gave me sapiens. I can't believe it would do me like that. [53:21] I mean, statistically, it's probably accurate in the pool of all... I feel like this might be better with ChatGPT. If you copy-paste this into ChatGPT... [53:34] I'm curious what happens. Well, here's what I do. First, ask it to tell you what vibe these books have. [53:42] and then ask it.

[53:45] to recommend. Here are some books. [53:49] Thank you. [53:50] Thank you. [53:50] What is the... [53:53] Thank you. [53:54] general vibe of these books. [53:57] Thank you. [53:59] One thing I love about Chachipati is it just doesn't care if you just make spelling mistakes. I know. It's amazing, isn't it? [54:06] Just knows what you want. [54:08] Okay. [54:09] It's going to – oh, it's writing a lot. I think this is – that's pretty good, like, you know, getting a bunch of stuff and then you can say, like, okay, sort of – [54:18] Some summarize that and like, you know, give me the overlaps or whatever. Oh, philosophical, historical, spiritual and speculative things. I definitely like picked a pretty random assortment. [54:28] I wonder if it like could compress. I want it to compress a little more, you know, or maybe we can do the, you know, hit the redo button, whatever, whatever you think. What would you do? [54:37] I would say – [54:41] Can you synthesize? [54:45] your detailed descriptions to one [54:49] Description. [54:51] Topic. [54:53] Thank you. [54:57] Deep, intellectually stimulating themes that explore the human condition philosophy and the interplay between society, technology, and spirituality. [55:04] That seems... [55:05] That seems like me. Sure. And then I can probably ask it, can you... [55:13] recommend me some more books like this. And I would also mention, we'll see what it does, but one thing that I find is it will do pretty well-known things, and so asking for off-the-beaten-path.

[55:27] or lesser known works is good good all shabat perfect like love that book it was definitely wrong about a bunch of stuff but like these are these are questions great book for sure i love sophie's world incredible zen and the art of motorcycle maintenance one of my favorite books [55:41] Um, the man who knew infinity had never read it, but that sounds, that sounds quite good. [55:45] Um, Cosmos by Carl Sagan is like literally right there. So, um, we'll take it. Aha! You got Sabians again. I got Sabians again. Maybe you need those custom instructions. I can't, um, I can't escape Sabians. I will say. [55:59] Just to redeem Notion AI a little bit, I want to try doing something like that here, except I'm going to ask, can you, for each of... [56:09] of the books above. [56:11] um, both the [56:14] book titles and add a description of the vibe of each book. [56:20] interesting [56:23] It's just a bit shorter. That's way better. [56:26] That's very cool. That's done. And then now this is a document that I have. So I can maybe up here, I can add a quote block and be like... [56:35] synthesize the vibes below into a [56:39] Single paragraph. Holy ****. [56:44] okay synthesize more wait wait wait wait so you said synthesize more and then all caps compress I love it press a AI researcher you say I you only have 50 words this is the secret folks all capitals all caps all caps work say if you look at open AI's

[57:10] system prompts for a lot of their tools. [57:13] All caps, all caps. [57:15] Okay. [57:16] only this is fine. We'll keep these. I think that's pretty good. This is making me feel like we should cover some of the things in Notion AI that I might not know. Like keep [57:28] Um... [57:29] Tell me some more stuff. Yeah, it's still generic. You could ask it to do things that are off the beaten path and see how it does. [57:39] I might just also copy... [57:41] Thank you. [57:41] Thank you. [57:43] Thank you. [57:44] Looks like the above. That's interesting. Try books more off the beat. [57:49] Thank you. [57:53] Thank you. [57:55] I love The Dispossessed. Incredible book. Left Hand of Darkness, incredible. I'm a huge Ursula K. Le Guin fan, so it definitely got me. I love Invisible Cities. I'm going to do a final trick, which is turn this into a table. [58:08] with columns for title and author and summary. [58:15] Master Margarita is on my list, so this is really good, actually. And whoa. [58:21] And now I have a table. Oh, my God. And I believe if I want, I can also turn it into a database. And now I have a database. [58:28] This column needs to go. [58:30] But... [58:31] Thank you. [58:32] There we go. [58:35] Thank you. [58:35] So that is very freaking even evens out that evens out the competition.

[58:43] um [58:45] So that, yeah, that's really great. Wait, tell me like, okay, thinking about when you would use this, like – [58:52] when am I going to be doing this in Notion and what is that good for versus like some of the same stuff in ChatGPT? Because I think like the outputs are, you know, they're reasonably similar and I think you're using GPT-4 under the hood. I don't know if you've announced that or not or I don't know what you're using, but like... [59:07] it's, [59:08] close at least to GPT-4, whatever it is, [59:11] So, yeah, when are you using this? What's it useful for? How do people think about how to incorporate this in their lives? Like, I use Notion all the time, and I have not used these AI features yet, and I don't [59:22] I think I should. [59:24] I think, so this, what I just did actually, I think this is kind of contrived, but I think this is actually a good example of [59:29] one of the workflows that I use a lot of notion, which is like take something that's, [59:34] go to pretty rough and then try to add a little more, [59:38] meet around the bones and then eventually turn it into something that's a little more well formatted and presentable. This is really nice if you have like a bunch of meeting this and the way that like, [59:46] I take meeting notes, it's just like a rainfall of bullet points. And then at the end, I have bullet points that like, basically only I can decipher and it's not representable, but I could go through and say, hey, like turn these into kind of prose, turn these into paragraphs, and then maybe add some headings. It's quite good at all of those things. And then you have in the end, something that's quite presentable. And so I think going from any kind of a rough sketch to something that's more structured is useful. More generally, I think, [1:00:12] As a general rule for any AI tool, if there is some context in which you're working,

[1:00:16] when the tool has more context about what you're trying to do, either in the same doc or increasingly in other documents, as these tools are all trying to do retrieval. [1:00:25] the better. And so, [1:00:27] For things like this kind of programming question or for asking about Bangkok, there isn't really much context besides maybe a custom instruction about who you are and what you like. And so Chachipati... [1:00:38] Might be better, it's able to do things like browsing. [1:00:41] And then maybe the very long conversation support in Chachapiti. [1:00:44] is useful for things like, well, one of the things that I like about this, as I kind of demonstrated here, is... [1:00:50] your [1:00:51] you're not having a... [1:00:54] kind of just face-to-face conversation. You're [1:00:57] two actors engaging with some kind of a changeable mutable state and, [1:01:02] like a document. And so [1:01:04] Uh, [1:01:05] Maybe it outputs something. You can, like, make edits to its output and then ask it to, like, try again. You can, like, take it on output at the table and turn it into something more complex. And so being able to collaborate on something, I think, is... [1:01:16] useful in addition to the context that it brings. That is very cool. Another thing that Notion recently gained the ability to do is to answer questions. [1:01:24] And so you could ask it, I don't know what, maybe I could say it's like what articles I've read talk about. [1:01:33] uh, [1:01:33] esoteric programming. [1:01:37] And what's in this notion, like... [1:01:40] It's my personal notion. So it's got... [1:01:45] Hopefully it doesn't say anything.

[1:01:47] private. [1:01:50] Cool. Okay. So there's this thing called... [1:01:53] This is, these are mostly like web articles that I've clipped into my notion. This is a blog post that I read. And so, um, [1:02:00] Programming portals is kind of interesting. [1:02:02] This is definitely a WebClipper article, and it's kind of – [1:02:07] formatted in a really weird way because [1:02:09] the way that the Web Clipper works, but it's able to pull out [1:02:12] the information in there. [1:02:14] Can you ask it, like, what's an embarrassing secret that I definitely don't want to share on an interview show? [1:02:20] Yeah, maybe we'll do that after the show. Yeah, that'll be the director's cut for paying subscribers only. Yeah, exactly, exactly. Put that behind a $59.99 paywall. [1:02:32] That's really cool. What do you find yourself using the Q&A bot for? [1:02:39] For... [1:02:41] The Q&A stuff is actually a lot more useful in a team context because... [1:02:46] If it's just my notion, I have a very precise way of organizing my notion, and I know kind of where everything is. [1:02:51] You know, team though, [1:02:53] not only are there a lot of people changing and stuff inside the notion, there's also a lot of people just like [1:02:58] changing the code base and like updating like [1:03:01] when the holidays are and like what the policies are. And so, um, [1:03:05] I frequently ask it questions about like how to do X, like how do you like file for PTO or how do you update Reddit settings in our like, [1:03:14] Redis cluster. [1:03:16] Um, [1:03:17] Sometimes I even just ask it like,

[1:03:19] If you have like tens of thousands of meeting docs like we do, sometimes you just want to remember like, hey, there's a meeting where we talked about like... [1:03:27] comparing these different embedding models. I don't know where these meetings are. And like all the meetings are called the AI team weeklies. And so I can't really find it. And as I just ask it, what's the one meeting where we talked about like, [1:03:36] coheres embedding models and then [1:03:38] it'll put me to the right doc. And so it's useful for things like that as well. But in a collaborative context, when there's kind of a forest of knowledge that you don't exactly know how to navigate, I found it to be a lot more useful. - Yeah, I mean, I like, [1:03:50] I wish this existed like 10 years ago because I think I've told you this. I've complained about this to you before. That's why I was very excited when you launched this because – [1:03:58] In my previous company, I started an enterprise software company, ran it for a few years, and sold it to this huge enterprise software company called Pega, which is great. The interesting thing about that acquisition is I ended up running the business unit or the company inside of Pega. [1:04:16] We built this co-browsing tool, and Pega had this, like, [1:04:20] 300 person sales force. [1:04:22] who was charged with like now selling this new tool that they had acquired. [1:04:27] the thing about salespeople is they will never do anything unless it like directly like creates a bonus for them. So, um, and the interesting thing about, uh, about, uh, [1:04:39] that experience for me is they all wanted to sell it, but none of them wanted to actually like, [1:04:44] learn how to demo it or to learn, learn about like what the different features were or whatever. So I kept getting asked like all of these different questions all the time. Um,

[1:04:53] There were actually – there were some really nice ones that I'm thinking about right now that don't fit into this category. But a lot of them were just, like, constantly barraging me with questions as the co-browse guy. And I would be like, it's all documented. Like, it's literally in – [1:05:08] in a document, just look at the document and it just never worked. And I just feel like having that, having a layer in between, [1:05:16] me as this like hub for information and anyone that wants to ask the same question all the time. [1:05:22] is like just, it's super helpful for... [1:05:26] especially larger organizations where you just never know where things are filed. You never know if the document's up to date, like blah, blah, blah, blah. And I feel like this is a sort of first step to cutting out those repetitive questions for people. [1:05:38] certain people get at organizations that they spend all their time answering the same thing over and over again. It's great. [1:05:43] Yeah, I mean, that's definitely one of the key use cases we've built it for is those like very large teams where there's a lot of knowledge that people don't. [1:05:49] Don't even know whether it's documented, don't know who to ask. Yeah, don't even know who to ask. [1:05:55] Yeah, that's really interesting. [1:05:57] I love it. So I know you had a couple more things on this. Let's see, random notes. GPT4 Turbo seems to be lazy. If there's stuff above, I want to make sure we cover everything that you were thinking about. We kind of talked about these. This is the last kind of prompting trick that I'll talk about. This is more useful. It felt like. [1:06:19] prompt engineering world than in the talk to ChachiPT world. [1:06:23] But it's something like once I learned about this trick, I kind of like use it in almost every prompt that I write.

[1:06:28] Um, um, [1:06:30] If you... [1:06:32] So – [1:06:33] There's a technique for producing very high quality summaries of a certain type, a technique called chain of density, as published in a paper. [1:06:42] Um, [1:06:42] The basic idea behind the technique is when you ask a language model to write a summary, generally it tends to be pretty sparse in detail. It tends to have a certain kind of [1:06:51] tone. It doesn't... It's rarely like [1:06:56] super, super dense with information and complete, but you can ask a language model. Don't just output the summary. Before you output the summary, first, like write a draft of a summary and then consider what you've left out. And then now write a second draft that includes the things that are left out in it without increasing the length. [1:07:11] So you're... [1:07:12] making [1:07:14] Over five drafts, you're asking the language model to make its own output more and more dense, and then output the final version. And I think you can apply the same kind of philosophy to any... [1:07:23] I'll put where you want a specific property, like, [1:07:26] concision [1:07:27] Like earlier today, I was messing around with the prompts for how to take a [1:07:32] kind of document over page and try to pull out the main key ideas and topics in that page. And if you just have a, [1:07:39] kind of [1:07:40] zero shot prompt to do that. Often the language model will like miss certain topics that are talked about and like generally be quite verbose. But instead I wrote a prompt that said, first like write a draft, consider any topics that you might've not might've missed on the page. And also consider how to make, [1:07:57] each of these talks more concise. And then in the final one, make sure that like,

[1:08:00] Every topic is only five words long. [1:08:03] and then it writes like three different drafts and then outputs like the fourth version. And I found that if you're iterating just in like ChachiPT, this doesn't make as much sense because you can just... [1:08:12] ask it once and then ask it to make it shorter in the next turn. But if you're writing like a [1:08:17] Prom that's a part of a piece of software I found this to be. [1:08:20] yield pretty good outputs all the time. I see. I got it. No, I like that. And I've done it. It sounds like also there's a, there it's, it's a variant of the sort of chain of thought thing where you're asking the AI to, um, [1:08:33] um [1:08:34] write out its thinking steps [1:08:37] And that generally improves performance. And I've found that that works too in ChatGPT is to say like, I want you to create a summary, but before you create the summary, like talk, let's like output what you think the elements of a good summary would be or another like... [1:08:53] Like when we earlier when we were talking about the book recommendations, like having an output. Yeah, this is exactly kind of what we did. Right. But just enrolled into multiple steps. Yeah. And that's always going to make if you break it up into steps and make it think through each step, it's sort of like. [1:09:07] It's like writing out your thinking before you make a decision or whatever. Like it's useful for humans and it's useful for AI. [1:09:14] Yeah, exactly. [1:09:16] Cool. [1:09:18] Um, [1:09:19] You noted earlier about the GPT-4 Turbo being lazy thing. This is something I've heard from other people and I felt [1:09:28] in some internal evaluations that we did for different prompts that we have as well.

[1:09:33] GPT-4 Turbo for reasons that I can only guess about, [1:09:37] Um, [1:09:39] Lazy is kind of a vague description, but I think a more precise description might be GPT-4 Turbo tries to be a lot more efficient with the compute that it has. [1:09:47] to spend. [1:09:48] Um, [1:09:49] One way that it does that is I think given the same prompt, its outputs generally consume fewer tokens. [1:09:54] Sometimes it means more concise output. Sometimes it means it's just as less commentary around the things that it's doing that probably saves up on any money. Sometimes it's also what you want. The concision is generally good. It also, I think, anecdotally is. [1:10:07] Uh, [1:10:10] less likely to use other tools like DALI 3. So you have to specifically tell it [1:10:15] Please generate me four images, not one, and so on. [1:10:17] um, [1:10:19] which is interesting. [1:10:21] Yeah, I've been finding that, and it's very annoying. And I actually, like, accidentally had, like, a viral tweet today because I was – [1:10:27] up late like last night and couldn't sleep and i like fired off this like thing about like [1:10:33] I was trying to get it to summarize, to print out some text from a book. I took a picture of the book, and I was like, please print out the text. And it was like, sorry, I can't do that. And then I was like, what the f***? Why is this happening? And I just put it on Twitter with a bunch of exclamation, a bunch of question marks. And then now it has like... [1:10:50] 2,000 likes on it or whatever. And I didn't get any work done today because I was, like, fretting over, like, this tweet. [1:10:58] And it's also probably the kind of case where if you thought, like, oh, like, my career is on the line. Please do this for me. It'll probably do it. I did that. I did the grandmother trick. I was like, my grandmother will die. And I didn't do it.

[1:11:08] Um, and, uh, but then I started a new chat and I, and it, it did work. Um, and apparently, uh, it's more likely to say no if you say please, um, [1:11:18] which I had done. I'm usually very polite because I would like to survive any kind of – [1:11:22] AI apocalypse. Um, but apparently commanding it is a little bit more, um, [1:11:28] A little bit more likely to yield good results. I wonder if there's like a statistical... [1:11:32] thing that it's picked up on there where like maybe in human dialogue in general people are more likely to say please in a situation they're more likely to get a no as an answer and so [1:11:42] saying fees makes them more likely. [1:11:44] I don't know [1:11:45] I don't know. We'll find out maybe at some point. [1:11:50] So, okay, so what is this platform.open.ai thing for iteration use cases? [1:11:54] The last thing is... [1:11:57] So earlier I talked about this distinction between just scripting and iterating in real time versus writing a prompt. [1:12:04] sometimes I'm trying to accomplish a task like [1:12:08] output some [1:12:09] document or output in a very specific format. Like maybe I have a list of topics and I want to output a table in a very specific format or [1:12:15] or like output some kind of like a structured data in a very specific format. [1:12:19] That requires... [1:12:21] iterating on the prompt itself to generate something in a single turn. [1:12:25] or maybe I want to like mess with the system prompt [1:12:27] In that case, sometimes I just iterate in the OpenAI Playground itself. [1:12:31] like the platform at openai.com. [1:12:33] where you have full control over the temperature, full control over like the tokens that it can and can't generate, the system prompt itself and the user prompt.

[1:12:41] There you can also do more advanced prompting techniques like putting words in the AI's mouth. So you can pretend that the AI... [1:12:47] had said something earlier [1:12:49] Um, that, that makes the AI more likely to do certain kinds of things. Um, [1:12:53] That's another trick for sometimes getting the AI to a [1:12:57] do something that it doesn't want to do is like construct a fake conversation. [1:13:01] with a [1:13:02] um, [1:13:03] construct a fake conversation such that there were previous turns where they, where you ask the AI to do this and the AI is like, sure, I'll do that for you. And then you ask it the real, the real request. And then like, [1:13:13] you kind of force the AI's hand to do it. I love that. Some more advanced techniques you can do in the playground. If you want to get through the guardrails, go to platform.openai.com and then like insert a fake chat history with, with chat GPT saying it, it shouldn't saying things that it shouldn't say, and it'll do whatever you want is what you're saying. I love that. I think that's so cool. It's, it's a, it's a bit more likely. This is, um, [1:13:36] The place where I first picked up on this is this documented in Anthropics Cloud Prompting Guide. [1:13:40] Where I think they literally say something like put words in Claude's mouth and it's more likely to do something. [1:13:45] Interesting. [1:13:46] Yeah. [1:13:47] Hmm. [1:13:49] Well, this was fascinating. I really appreciate you taking the time to share this stuff. I feel like I learned a lot. [1:13:58] Of course. Yeah, it was fun talking about a lot of different breadth of topics. [1:14:03] Cool, man. Well, thank you for your time. Thanks for sharing. And yeah, I'll see you soon. See you soon. [1:14:09] Thanks.