How AI Will Change Science Forever - Ep. 43 with Alice Albrecht

[00:00] Is the substrate of a scientific paper actually the right format to release science in? As time goes on, as we get more access to data and compute, scientists do become more computational by nature, even if they never were before. It's useful to publish papers. It's useful to get this information out there. The problem has always been we only publish certain things. And so we're not getting any information around what's failed or what's been tried. There's all these places in machine learning, in science, in journalism, [00:30] and the underlying data that were used to create the story were always had to be bundled. And that created a lot of problems. And the solution is not to get rid of stories. We are now in a place where the underlying data is actually probably just as important as the story. [00:59] - Alice, welcome to the show. [01:01] Thanks for having me. [01:02] So for people who don't know, we've been friends for a long time. You are the founder and CEO of Recollect.ai, which you recently sold to what company bought you? [01:14] Smart News. Smart News. I'm excited to have you on here because you're just one of those people in AI that's actually, like, been doing it for a really, really long time. And you have, like, you've seen... [01:28] many different AI summers and winters. Yeah. And I just find you to be incredibly smart and thoughtful and have built a business in the space that you sold. You said it was an acquihire. So like, I just want to like, kind of get into like, what happened in the business, you learned about running a company in this era of AI, kind of like what's on your mind now. And I'll say for people listening, like, I really liked recollect as a product. And, and we had not caught up since the

[01:58] We're just we're going to just do the conversation that we would normally do, just like hanging out together. So welcome to the show. Give people a little bit of an introduction to the product and the acquisition process, and then we'll take it from there. [02:10] Yeah. Well, I'm excited to be doing this. We have not gotten to catch up, so this is good. So, yeah, Recollect was, which is interesting to talk about it in the past tense now, but... [02:21] It feels, I don't know, it's bittersweet. Like I'm proud of what we did and I'm excited about it. But yeah, it's, I don't know, it's hard. [02:30] just because I sat with it for so long, which I'll go through, sort of the journey of building in this time of AI and how everything feels really [02:38] I don't know, time feels like it's accelerating faster than it would have if I think I might build this company another time, right? [02:44] If it was a software company five years ago with no AI, I feel like it would have been a different experience or a different journey. In any case, so Recollect was really focused on knowledge workers. You got to try out different versions of it over the years. [02:56] But our main goal was, can we take all the stuff that you're consuming? Could we connect it for you in this really personal way, the way your mind would? [03:03] And then how do we make use of that for knowledge workers, though? Yeah. [03:06] We started out with this tool and we had a few different interfaces over the years. We worked on it, but [03:12] that you could recall what you were thinking if you were doing writing or research. This is sort of pre-Chat GPT times. And then, [03:20] Earlier this year, wow, yeah, this calendar year, we had shifted into like, don't tell us what you're thinking about, tell us what you want to accomplish. And then we'll bring all the materials to you. We'll do the synthesis for you as needed or sort of.

[03:33] alter those materials in a way that's useful to you as a knowledge worker. Um, [03:38] And everything we were doing was really aimed at how do we make all this stuff that we have access to accessible and useful for you. [03:45] um, [03:46] with the goal of enabling human intelligence [03:50] really rather than the machine intelligence piece of it. [03:54] Um, and. [03:55] I just resonate so much with that. [03:59] mission or that like [04:02] Goal. [04:03] Um... [04:05] Like that's that's one of the things that I [04:08] got me so psyched because I'm a little bit more of like a late comer to AI than you. Like I, you know, [04:15] And so one of the things that got me psyched about it is that [04:20] Thank you. [04:21] I feel like I've been such a nerd for... [04:24] um, [04:26] tools for thought or technologies that expand the way we think or the way we see ourselves or the way we create things and and what we understand ourselves like all that kind of stuff and it just feels like [04:37] There's so much there and you are playing in that space. [04:40] um [04:42] And I'm kind of curious, like, [04:43] Yeah, what did you learn about trying to build a business there that you didn't know before? [04:50] Yeah, I think a couple of different things. When I started, like I left my full-time gig to work on this in 2019. [04:59] So things were really different at that point. Like Burt and Elmo, these early models had come out. It's all an opportunity. Um,

[05:06] The ideation phase on like, OK, we have a capability. What do we build with it? What's useful in the world of this capability? Like what's possible now that wasn't? [05:15] That was an interesting process. It took, I'd say a year and change probably with COVID too. It stuck in there and all of the chaos that surrounded that. Um, [05:23] But the journey on building a business for it, I think, was it had these separate faces. One of them was, I would say, like... [05:30] We have this really nascent technology. Nobody really understands that. I'm trying to explain what it's possible, what's possible to do with it. But the models really weren't quite, this is like GPT-2 maybe, like they weren't, [05:42] amazing. Yeah, like I could see potential, but that was hard. I think once [05:47] The models got a bit better. And especially after like ChatJPT launched, everybody woke up and was like, wow, this is really cool. And then we had this middle phase as a company where I still was a bit skeptical about, okay, do we shove this into a product and have it generate things? Do we trust the models to do that in a safe way, in a safe way? And I thought, no. [06:07] And so we had this middle phase of company building where we were launching a consumer product. We wanted it to be. [06:13] something that was useful for people, but also like navigating all of the [06:20] I guess like, oh, we could do that, but is that really in line with what we want to do? Or is that just because like something new came out with the model and we could veer that direction? [06:28] That was interesting to navigate. I think on the whole... [06:33] This is snow, like... [06:35] having gone through the whole cycle, if I had to do it over again,

[06:39] or had the opportunity to do it over again, I would have launched the product much earlier. I would have let the messiness be okay with customers and just see what happens with that. [06:49] and [06:50] I think we did a bit of trying to protect that and say, okay, we want to have this integrity. We want it to be... [06:55] Um, [06:56] Yeah. [06:57] we want it to feel like a really seamless experience. We don't want them to really feel the weirdness of using these AI tools. Um, [07:05] And then the third phase was really... [07:07] uh, [07:08] as these tools are way more widely adopted, they're in everything, there's this huge explosion in the space. [07:14] trying to think about, okay, what is the actual application other than retrieval? [07:21] Because retrieval was such an easy one. I mean, we were like in, we were doing a lot of retrieval stuff in the beginning, but then what happens after that? Like once you retrieve a bunch of things, what is useful to do with those things? Um... [07:33] Yeah, so it was an interesting tripart journey, I'd say. [07:38] That's interesting. I definitely, I think that that's such a common... [07:43] like um thing and it's so hard it's like easy to say and hard to like actually internalize is just like [07:51] ship something faster that you're like not as proud of and just see what happens kind of yeah you very um can be very scary because obviously like [08:00] you want to make the best product possible. And it's like a weird... [08:04] thing to be like sometimes making the best product possible is starting with is doing the messy thing or whatever. It's just like a weird

[08:10] It's a weird thing to get into your brain. [08:13] I think one of the things I'm interested in is... [08:20] When you started, it was like Burt... [08:23] and maybe GPT-2-ish days, like... [08:27] What made you be like, okay, this is happening now? [08:32] I think they willed it to happen, honestly. Like, even in, like, going out, we did the typical, like, Nisi fundraising process, too. And so... [08:40] I really was... [08:43] There was no moment where I was like, okay, yeah, we finally hit it. I was like, no, no, this is going to happen. [08:47] come hell or high water, even if I have to meet this true. [08:52] So, yeah, I feel like it was really convincing other people that, like, no, no, no, these embeddings I'm talking about are really, really important. [08:59] Um... [09:01] Yeah, and that was hard. [09:03] Okay. And what do you think about... [09:07] You wrote an article in Every a while ago, and I read it and edited it a really long time ago, so I only have some of it in my brain. [09:16] Basically, I just feel like you have a really strong and clear thesis for like [09:21] how humans and AI [09:24] can work together and what that should look like or could look like. And it's a sort of like cyborg hypothesis. [09:33] And I wonder if you could like lay that out because I think a lot of people, it's a big question on people's minds. Like, where are we going if we're going to use this as tools? Like, how does it work and also keep us being humans and employed? And then I'd love to talk about like how that has shifted for you over the years building

[09:53] building this company. [09:55] Yeah. [09:56] And it certainly has shifted, which has been interesting to kind of catch myself in between. You say these statements in the world and then something changes and you're like, oh, you have to readdress or reassess the whole situation. [10:08] So the cyborg piece, I'm still pretty hell-bent on. I think this has been my shtick for a long time now, coming from human studies to, you know, working with machines. [10:18] I have this belief that humans are [10:21] really kind of amazing creatures and that if we're going to use the technology, it should be to augment them rather than [10:27] to just build an AGI. Like the goal for me is not some sort of super intelligence that works without us. It's [10:32] How do we start to connect? [10:35] this technology to humans and really finding those touch points. I think knowledge is one of those. So if we think about, you know, knowledge work in general, a lot of it is consuming, distilling information, understanding, and then, [10:48] really kind of propagating that information out to other people, other knowledge workers and those systems. [10:54] um [10:55] And so... [10:56] For me, I think a lot about how can we use AI or machine learning or any of the other technologies [11:04] to make that process easier for humans so they can do the creative piece. And I think that's where we get to how we all stay employed. [11:11] Um, [11:12] I'm a big proponent of, you know, [11:15] uh, [11:16] lowering the bar, raising the ceiling, I guess. [11:19] So what's possible, we don't even... [11:21] truly understand quite yet. Um...

[11:24] But if we can find the touch points where it makes that human work a lot easier... [11:30] And I don't think we're quite there. I don't think chat is the right interface for this. I don't think the models have the right context for it. [11:37] I think, and I say this again and again, like, I still think we're going to need some sort of biometric feedback piece in there to really make this work. [11:46] And [11:47] In that case, it's about... [11:50] Finding where humans are not deficient, but where they kind of struggle from their evolutionary constraints. [11:56] thing where the technology can come together. And it's not like, you know, creating artificial eyes. That's pretty cool. But that's not exactly... [12:03] my thing. [12:04] Yeah, that made sense. I was laughing earlier because I feel like we have this like recurring sitcom or soap opera or something that every time New York Tech Week comes around, we're like on a panel together. And like the first year it was like me and you and it was like right after Caps of BTE launched and it was like a couple other people and... [12:25] I won't say who or whatever, but like there was one person on the panel where me and you were just like looking at each other the whole time. You're like, what does he say? What is happening here? Yeah. [12:35] That was really fun. And then the next year, we were on another panel about AI and creativity or something like that. And [12:42] we were on different sides of the debate because we were talking about chat interfaces and whether or not chat interfaces are the future of AI interactions. And like you said just now, you think the answer is no. And I was defending chat. And I'm curious, like,

[13:02] how that position has evolved for you and why, if you could refresh my memory, why you think chat is not the right interface. [13:10] Yeah, um... [13:12] Trying to remember the argument I made on this panel. I probably won't be able to recall it exactly. [13:17] And I'd also love to hear how your position may have changed over time. But so I think language is a super powerful tool, right? We're using it right now to communicate with each other. We read, like we consume it in this way. I don't think that it is the most natural way to access things like this. I think language. [13:35] We've actually moved farther in this direction probably since that panel two is all of this agent. [13:40] stuff happening where we have this... [13:44] kind of [13:45] thing that does work in the background for us, but it intuits what work to do a little bit more. And it accomplishes that without us saying like, hello, robot, please, blah, blah, blah, blah, blah. [13:56] And I think the... [13:59] The only way I've come [14:01] maybe a little closer to the chat piece is [14:05] I used it more. Like, I find myself... [14:09] in [14:10] new spaces, like where I'm like, okay, [14:12] I just started a new company. I'm in a new area where I'm thinking through things. It has become a little bit more useful for me to say like, okay, [14:21] I'm thinking about this. What are the pros and cons, right? And right now I have colleagues that are in Japan. We're 13 or 14 hours removed. And so it's helpful for me in that sense. It's still not my main interface, though.

[14:34] um, [14:35] So I think... [14:37] So I'm doing more coding too. And so I think like the main use for me right now is code. [14:43] And that's sometimes chat. [14:47] Cursor or like, what are you doing? [14:49] So I'm coming back to cursor. I had used it really early on. It was really buggy and I was like, this is ruining my GitHub. I can't use this. This is terrible. I'm coming back to that. [14:58] Yeah, so I use VS Code. I have, you know, Copilot, all the regular things. I do some chat. I really like the Claude artifacts. That has been a really big game changer for me. [15:12] Um, [15:14] And this is where maybe my chat argument does hold up. So it generates these basically like little maps of things, right? Like you can have it do these mermaid diagrams. [15:24] As one of the artifacts. And I think really visually, I draw things out a lot. I'm famous for having this remarkable with me every, everywhere I go and sharing these, like, I think I put these in one of my articles I published, like, this is how I think I don't really think in this like chat back and forth situation. [15:40] So it generating that has been interesting for me, too, as a way to collaborate in a non-text way. [15:47] That is interesting. Yeah, I guess like so for me... [15:51] And when I talk about chat, I would like... [15:54] include [15:56] like, [15:57] you know, [15:58] voice video like and the sort of like back and forth [16:04] kind of

[16:05] way. But maybe it's more properly limited to just text. And I think probably the reason I like chat as an interface is I'm just so... [16:15] verbal like I just so that it just like makes a lot of sense but if I try to um [16:22] Turn... [16:23] my own personality into like a general truth about like what's good, which I think is typically how things work. If I try to do that consciously, um, [16:35] I think the reason that I that I like [16:39] Chat. [16:41] as an interface is it allows you to push forward along many different dimensions simultaneously. [16:49] Um, [16:50] where a lot of other software interfaces are like, it's either on or off, or it's like, it's on one axis at a time or something like that, which is quite useful for [17:04] refining something. [17:08] sometimes for refining something or for, for processes that you're doing where, you know, [17:16] uh there the dimensions that you're improving along are like really well known and really well understood and it's sort of repetitive [17:24] Out. [17:25] But then there are a lot of other processes, particularly creative processes, where you're like trying to explore space along a bunch of different dimensions. [17:34] and you don't really know what the dimensions are beforehand,

[17:37] And I find that to be quite good for that exploration process and for the like refinement process of like, [17:44] okay, you like produce this thing, like now I want you to like push it in this way or whatever, define what that was before. But now I see it now I know. So a really good example is, [17:56] We've been incubating products inside of Avery. [18:01] which is really cool. I would love to tell you. [18:04] And one of the products that we incubated is this product called Spiral. It helps to automate a lot of like the repetitive creative tasks that you do if you're running a company or you're a marketer or you're a creator. [18:17] So an example would be like for this podcast, I'll start to take this podcast and turn it into a tweet to get the episode out. [18:23] and [18:24] So that's a very repetitive process because I kind of have like a format that I know works. You know, like I have a good first line and then I have like a couple of bullet points about like what are the key topics that we discuss that I think are interesting or whatever. [18:36] and [18:37] I realized that Claude is like really good at doing this. If you give it, if it's a few shot profit, you give it a bunch of examples of like odd cast transcripts that turned into tweets, then it can like do that over and over again. And it gets you like 80% of the way there. So we built Spiral. [18:52] where it's basically like a few shop prompt builder. So you can make a spiral for turning podcast transcripts into tweets, or you can make one for turning blog posts into LinkedIn posts or turning, you know, whatever, release notes into a product announcement or whatever.

[19:10] And I'm giving you this very, very long-winded explanation because one of the things that we found is like in the... [19:19] In the spiral interface, when you have a spiral, let's say we take my podcast transcripts to tweet example, you just paste your transcript and you press run and then it gives you a bunch of information. [19:30] It gives you a bunch of example tweets that you can try. And like one of our biggest pieces of feedback of people wanting an improvement is like, I just want to chat so that I can say like, this example is good. Do more of that. [19:43] We didn't really think that that would be the case. We had ideas for like maybe we could do sliders or like maybe we could like [19:51] really what you should do is go back into the spiral creation flow and modify the prompt a little bit and make the prompt a little better. And what we're finding is the natural thing to do is... [20:03] It's sort of like, you know, you run, you run a company, it's sort of like, [20:07] when someone that's reporting to you comes to you and it's like i did the thing that you asked for and you're like [20:12] okay, this is great, but here's a couple things I need you to do. That's a very natural way to, to kind of push something in a sort of multi-dimensional, somewhat unknown space. Does that make sense? [20:24] It totally makes sense. Yeah. And I think that like using AI today is really good for this, right? Like there's like first draft... [20:31] it [20:32] you know, create the tweet for me, write some code for me, write whatever, like any kind of production thing. [20:38] In that process, though, right, I think there's this very interesting teacher-student relationship, which shows up.

[20:45] I think there's some new stuff around this, around training training. [20:48] Tiny models, [20:49] based on big models, and those are really cool. [20:52] But, um... [20:53] I think there are two interesting pieces there. One is the human in the loop. And this I think we do usually agree on, which is like the human creative piece of this, right? Where you have the judgment and saying, nah, this one's not quite right. But here, change this one. Or like, I can choose from this list. Awesome. [21:09] The cloud model or whatever you're using doesn't have enough information or intuition or something that seems a little bit fuzzier in there. [21:17] to choose, okay, yeah, this is the best one, just roll with it. [21:21] I think that's going to continue to be the case. [21:24] And then the other piece in there, though, is this knowledge sharing, right? So in a sense, when Claude or whatever model you're using outputs these tweets for you, you get a little bit of a peek into how it's like, quote unquote, thinking, right? Like it generated these things. You can see these got kind of right. These got kind of wrong. You could give it feedback. [21:43] But you're getting insight into that. And when you give it data, it's also... [21:47] it's not learning in real time with the few shot pieces, right? But [21:52] Maybe at some point it could. [21:54] And I think the more that you get a system that... [21:59] it knows a little more about what you know and how you like, [22:02] I don't even want to call it preferences because it's so squishy. You can't learn these preferences easily. [22:10] But the more that you and the model can get to a shared understanding of what the other thing knows and can fill in the blanks, right? You can say like, oh, yeah, you wrote that one, but you forgot or you didn't know or...

[22:22] By the way, actually add this piece because this is critical in there. [22:27] So I think this way of interacting, and not even to me... [22:32] I think the original like chat interface of ChatGPT where people are like, yeah, I can talk to the same and everyone like chatbots. This is already beyond that when it creates artifacts. Right. It's not a conversation. It's not like, hey, I think you should generally do. It's like, here's your thing. Here is your output. [22:46] Um, [22:47] And that's gotten really interesting. [22:49] Yeah, that is interesting. Yeah, we definitely like, I was arguing against chat interfaces, and then the product that I built was like a chatless interface. And we're just, you know, like, I think, yeah, it's your point, like, that, like, reducing that down is actually like, it can be helpful. And then, [23:08] we need to bring it back in some form, but there's interesting trade-offs to that. And the form that we're bringing it back is definitely not a totally general interface, and that's the only way that it... [23:21] It can compete with Claude. You can do the same thing with Claude. It just has to be very specific to your... [23:26] geodervices. [23:29] Um, I. [23:31] I've been thinking about doing a thing [23:34] And [23:36] Something about the shape of it makes me feel like you would be into it or like have interesting thoughts on it and whether it would work and what I should think about. So I'd love to I'd love to talk about it. You said something earlier that. [23:46] that reminded me of it, which which but I forgot what it was. So let me just let me just lay it out for you.

[23:52] I'm pretty curious. So, [23:54] Okay. [23:55] I don't know whether we've talked about this before, but I have OCD. [23:59] One of the things I've been thinking about or lightly trying is I wanted to see if it was possible to, I wear a whoop. [24:09] So you take my whoop data and be able to label from the, from the graphs, whether or not I experiencing OCD on a particular day. [24:20] And so far, my answer to that is like kind of maybe, but like it actually probably needs more context to know. [24:28] Um, uh, cause like what a spike means on a, on a stress graph at a whoop, like can mean a lot of different things depending on, you know, what's going on in the background. And what's really cool about these models is now they, they can, they know enough to like be able to take the context and use that. And so I'm not far enough along. Um, yet like another thing I'm going to try, um, is I'm starting to do daily, like two minute video journals. And there's this like emotion labeling AI called Hume and I'm, I'm going to see what [24:57] that can label it. So I don't know. There's some interesting things there where I think it actually... [25:03] I feel pretty confident there's some combination of data where I can get from data to label to be like, yes, like he's having OCD symptoms today or no, he's not. [25:14] And once I'm there, the big question to me is, will it be possible to predict, let's say a day in advance, whether or not I will start to experience OCD or whether or not I'm in a sort of OCD process?

[25:28] phase, like whether it will go away. [25:32] Because I think once you can predict, it opens up lots of [25:34] interesting things. [25:37] And so my [25:39] thought for how to do this is to basically do a bounty [25:46] And [25:46] do like a 10 grand. If you can predict my OCD, here's a data set. And let anybody, because everyone, you know, you can have, you can use O1 Pro for $200. Anyone can like, [25:59] do this basically now in a way that they couldn't before. [26:02] And like, A, see if that works. And then B, maybe build that into like a sort of Kaggle-like platform. [26:09] But now everyone's a data scientist, right? [26:13] and [26:14] There's probably a lot of things that that you would be into about this, but like or have have thoughts on. But the thing that I'm kind of interested in is like. [26:25] I think... [26:26] Making predictions about and maybe we talked about this a bit, but I think making predictions about whether or not I'll have OCD. [26:34] is [26:35] a form of science, but it's a form of science that like [26:39] a scientist would never do because it's like an end of one experiment and you're not actually looking for this whole explanation. You're just predicting. [26:47] So it's like completely taboo to like the establishment of like research, but it's completely incredibly useful and I think now doable where I think we should be doing a lot more of these kind of end of one things.

[27:02] and pursuing more predictions over underlying scientific causal explanations. And I just thought that that would get your brain going and you'd have interesting things to say. [27:13] Totally. [27:14] Yeah, I appreciate your writing in, too, because I'm like, I have all of these pieces I can talk through. [27:20] -Yes. [27:21] I think the end of one thing is interesting because like, [27:24] If you came to me and you're like, I am in great [27:28] pain from this right like you were like I am suffering and as my friend I would be like gosh how do I help you [27:32] And if I thought I could slap together a model, you got your move data. Great. [27:37] It'll take me whatever amount of time, but like, I could just do this. What I... [27:41] I wouldn't actually... [27:43] go to one of the large language models straight away. Like my brain would say, [27:48] We need a predictive model. We need to understand the data sources. [27:52] So I think it would be an interesting combination of looking at the actual like research research, right? So like there is research out there on OCD. I'm not an OCD researcher. No physiological. As far as I can tell, there's no physiological marker research. Fascinating. That is actually the most interesting. And I did buy an at-home EEG to see if I could do stuff. It's all like MRI stuff. So it's not usable by me. [28:18] That's so interesting. That's like a whole other podcast in and of itself. Like, why do we not have biological correlates for a thing that is fairly common in the population? [28:26] And probably has like a, at least a heart rate difference. Like something has to change. That's true for pretty much all mental, uh, mental illnesses. There are no bio, bio, biomedical markers, biophysical markers, um,

[28:40] uh if it is it goes from like in psychiatry to being neurology so like [28:45] like, [28:46] certain things used to be like mental illnesses that are now just neurological issues um [28:52] So, yeah, it's very, like... [28:55] All of psychology is basically built on self-report as far as I can tell. I know. I know. It's true. Yes, I believe that also. So, okay, I could make some like easy predictions off the bat and go, okay, heart rate data, probably important as a predictive measure of this. If you can give me labels, right? [29:12] it's a question of how many labels do you need to make this work? Is it, [29:15] 700 and then you're suffering for a really extended period of time. If we think about like, you know, how long these episodes might last. [29:23] Um, [29:24] And so I think... [29:26] If I were going to start from this, I might actually, the way I would use these models, though, [29:30] is to like, A, help come through literature, [29:33] and say, okay, I'm looking for this pretty specific information. How do you help me [29:39] jump into a field that like I'm it's not mine, but it's adjacent to one I know enough. [29:43] But then also help me build the model, right? Like help me think about how to build this predictive model. Here's the data that I've got or the type of data. Help me... [29:53] not build an LLM, but build a model that has predictive power that is well beyond what the LLM would be able to do. [30:00] right because the LLM isn't really trained to be a predictive model it's trained to predict language and it has reasoning capabilities from that [30:06] But there's lots of other models out there that would take all this data. [30:09] normalize it, put it together and say, okay, now we can build a basic predictive model, which would be a great taggle exercise in and of itself, right? I don't think it exists, but

[30:18] um, [30:19] So, [30:21] I think... [30:22] The piece that's interesting is translating your... [30:25] like if you have a video... [30:28] sort of like daily... [30:30] I don't know, check-in or something, like translating that into something that is useful for this kind of model. [30:35] Um... [30:36] Fei Fei Li did some really interesting work a really long time ago. She's sort of like the... [30:41] mother of AI, I guess we can call her. I don't know if she would care for that label, but she had a fascinating paper and it's really a long time ago now, but it was taking different data inputs. So it was video data, like people talking and they could predict depression. [30:57] or the onset of a depressive episode. And it was the facial expression. It was the [31:02] cadence of their voice. [31:04] the tone, like all of this data lives in this. And I don't know that there's anything correlated for OCD. Um, [31:10] But I think you could use these models to sort of transduce that information into some meaningful signal for a different model. [31:18] Yeah. [31:18] I think you're right. So basically with Hume, the cool thing about Hume is it [31:22] It turns your... [31:24] it turns your video into the equivalent of like emotion embeddings. [31:30] Mm-hmm. Which is pretty cool. [31:33] Um, on so each frame it like for your voice and your facial movements and whatever it like outputs and a bedding that represents what thinks where you are in the emotional. [31:42] embedding space. And so my guess is that can be used in, [31:48] It fed into some sort of predictive model, but I don't know. Yeah. And if you had it over time, right? So if this model ran frequently enough that you get, like, time series data that's meaningful, even if it was once every...

[31:59] 10 minutes or something and you could aggregate over your group data you could you could bring it i could totally see how you could put these pieces together and then say okay i can label for you when i'm having these episodes i can label for you when i feel like [32:10] if you have a sense when they're starting to come on versus fully in the middle of this versus at the tail end, [32:18] I think it's totally possible. [32:20] What you're having so far is a [32:23] more of a like I built this little app for myself where I do I do retrospective assessments so that in the morning I'm like retrospective like how was I yesterday like what were my symptoms if I had any [32:33] And then I upload a screenshot of my boop graph. And then I do a two minute like check in of I don't say how I'm feeling, but I just talk into the camera about like what I'm going to do that day or something like that. Because I don't want to give away the label. So you're saying maybe I should be doing more like momentary type data gathering to like if I'm in the middle of something like catch it if I can, you know. [32:55] Yeah, yeah. [32:57] um without aggravating the symptoms obviously right because i can see how that could spiral not with a fun time but like i really want to get into a weird thing you're like what if i can like keep doing this thing um well one of the funny things like the treatment for ocd one of the big treatments is exposure so like um i can just yeah it's it's for science it's exposure for science i could have to exposure therapy oh gosh yeah yeah [33:21] Yeah. But like, I think if I was going to model any kind of [33:25] Like, [33:26] I don't know, psychological thing that was happening periodically. I would want the data from like right before...

[33:32] in the middle so you've confirmed it's true, right? And then at the end and see, okay, [33:36] Like, what are the shifts and changes that happen? [33:38] Um, [33:39] And then I think what's interesting with the models also is like you could get other contextual data that you don't have. Right. Like location is an obvious one. Like where are you generally in the world? My calendar. [33:52] Your calendar, yep. Who are you talking to? What are you doing? What is your workload? [33:56] we've got all these sleep predictors, right? Like the loop is good for saying like, did you sleep well the night before? It does all these other analyses. [34:03] How would I get that into a non-LLM though? Let's say I have a calendar. Let me tell you what I've been doing because you're probably going to laugh because it's so dumb. But what I've been doing so far is just taking the images and the labels and just throwing them into Claude and 01 and being like, [34:20] Based on these, here are our labeled images. I want you to, like, come up with a set of rules to, like, [34:26] classify whether or not I'm having SD or not. [34:29] And it's [34:32] Currently not very good. [34:36] I wouldn't expect it to be. I would be pretty shocked if it was. Like, you might even open up, like, a portal into some other, like, whole other research field. Yeah, it would actually be fascinating if it worked. So what I would say is what you need for a predictive model, right, is you need to get all of these signals as features. And then for, like, something that has... [34:53] temporal dynamics to it, right? Like, [34:55] You can get, I think you can get it off the boop. [34:57] I think I tried at some point, but if you can get the... The daily summary, but not the momentary statistics, you have to like...

[35:05] I think I could create a data pipeline because I could just take a screenshot and then feed it into Claude and Claude will be like, here's what the, like, make an SPG that, like, matches the line of the loop graph and then turn that into time series or something. I think it's awesome. Time series. [35:20] Yeah, that's actually really cool. I didn't think of that. [35:23] uh, [35:24] So that's very clever as a way to get this data. So what I would say is like you need temporal data that's fairly time locked and then you need to convert all of these pieces into features, right? So you have your video that you're taking at a certain time. And so like you would need to do some processing on the video to pull out and you can use Hume, it sounds like, to do that. [35:40] Um, [35:41] But pull that out, have the timestamp associated with it, have the actual time series of the loop data, have as much of that sort of queued up to these time pieces. And then those are features that go into a basic classifier. [35:53] Um, [35:54] And it depends on the test. There's lots of different ways to classify data. But all you're doing then is providing information to the model, a non-LLM model. [36:02] to say like this or this right and you kind of have a binary classification unless you do [36:08] predict like, oh, it's oncoming, it's happened. [36:11] it's fading, right? Like if you, and that's, that's also not a terribly difficult, [36:15] classification task, right? My bet is that if you asked Claude [36:20] If you said like, "Hey, I've got this data." [36:22] trying to do a time series for me. [36:25] Here's my video data with timestamps. [36:27] like, [36:28] Translate that into features. And then write, probably in Python, like a model for me, like a predictive model, basic one, and help me plug these features in. Like, I think you might be able to get

[36:39] somewhere with that. I think the problem is asking an LLM to do the predictive work. [36:43] because it's not really set up to do that. It doesn't execute other models. [36:47] Mm-hmm. [36:48] That's interesting. What about, because I mean, you can use an LLM as like a basic classifier model. [36:55] So you're just saying that it's like classifying a different kind of... [37:00] a different kind of sequence and so the the features that it's going to pay attention to and a text sequence are just going to be different from the features it pays attention to and like a [37:09] you know. [37:09] OCD, kind of more sequenced. Yeah. And depending on the model, like, I know there's multimodal aspects to the models now. I actually... [37:18] don't know that LLMs are the best for... [37:21] like classifying certain things [37:23] And so, like, even now, I'm still saying, okay, third topic is a different kind of model. It's a large model, but it's not open AI or untopicked models, right? Like, [37:33] Models that are specific to [37:36] pulling out tags or categories or things like that from text data are actually still a little better than just throwing it into the big model. [37:45] um, [37:46] They're more specialized. [37:49] So that might change as these models get better, but I don't think that these are... [37:54] in and of themselves better than [37:57] other machine learning models that are [38:01] really meant to classify, especially time series data. [38:04] That's very helpful. I'm like basically feel like what I feel like I'm doing right now, which I didn't even realize I'm doing is I'm like,

[38:11] trying to kill a mosquito with a rocket launcher or something like that. It's like it's a beautiful. Yeah, totally. [38:16] But it can help you build the right ones, which is cool. Yeah, yeah, yeah. That's interesting. [38:21] Um... [38:22] Well, I guess I want to maybe pull us out of the, like, specific details of this particular engineering problem and more to the, like, higher level of, like, [38:32] I'm thinking about how this might change science and how this might change how we do science, and I'm curious for your thoughts there. [38:38] Yeah, I think it's already probably changed fundamentally how we do science from the knowledge standpoint, right? Being able to comb through all of that information and pull things out, like has saved graduate students and postdocs, nice people like countless years of their I don't know, it must like that, that part feels incredible to me. [38:58] Um, [39:00] So that in one way is how we change science. The second way will be simulating. [39:04] So, yeah. [39:05] The data is huge to train these models. We have all the hardware for these things, and they're getting good at simulating data good enough. Like if you have a solid existing data set, you can say simulate lots more data like this. [39:19] Um, [39:20] And that is incredible if you're trying to [39:23] like, [39:24] Understand the possibility space as a scientist. [39:27] and kind of window that down to, okay, [39:31] And it's hard for humans to do, right? We can make predictions, we can synthesize data, [39:36] But keeping in mind... [39:38] all of these different kinds of possible future states is really, really, really hard. Um,

[39:45] So I think it'll change science in that way. I don't think we're going to lose scientists. I don't think we'll have AI scientists, really. [39:50] I think there is a problem around [39:53] like [39:55] I think humans still do a lot of the hypothesis generation, which is a lot of the science. [40:00] Um, [40:00] and still thinking critically about, okay, [40:04] What are we like? What kind of data are we even trying to get to [40:08] like understand this, like to even get started on this. [40:11] the space or the psychopsis. [40:14] That's interesting. I mean, that's sort of like what I've been thinking about is like, okay... [40:20] In a world where... [40:21] Um, [40:23] the data is actually like [40:25] the really important and like rare thing and you want to get really good data, is the substrate of a scientific paper [40:32] actually the right [40:35] format to release science in? Like, for example, there's a lot of like open data type pushes right now. And is the idea of doing like a 16 person study actually at all interesting? Or should the project of science be to gather as much information? [40:55] Um, [40:55] as much good data sets about problems that we care about as we can and to aggregate them and then allow any scientist to build models on them rather than writing papers about them. I mean, papers are fine, but like in general, like, hmm. [41:09] There's a huge overproduction of papers and huge underproduction of usable good data.

[41:16] And then if it moves to production of good data and basically building good predictive models, then like, [41:26] For me, it feels like, and I've been on this soapbox for a little while, [41:32] For me, I feel like we should be actually going for predictions first before we do the causal explanation stuff, especially for things like psychology or psychiatry or whatever, where the causal explanations are really, really, really, really, really complicated. [41:49] Because if you can avoid depression or predict my OCD or predict what intervention is going to work, it's like, [41:54] life changing and and and if you have a good enough predictor, like maybe the explanations are in there somewhere. [42:01] So that's kind of where I've been thinking about like, okay, science may change and it may have to change in this particular way where it looks a lot more like engineering. It looks a lot more like [42:11] data gathering and model building. [42:14] And maybe a lot more like, and this I think you'll have opinions on, which I would really like, maybe a lot more like the shift that we went through from symbolic AI to sub-symbolic AI, maybe that shift where, you know, we were trying to find rules and logical, this is not for you, it's for people listening, rules and like logical, [42:37] formulas to define what intelligence or intelligence decisions was, and then we were like, actually, why don't we just throw a bunch of data at a model and it'll figure it out. That shift, maybe we need to apply that to the rest of science and a lot of other areas of science.

[42:52] the world. And that would actually be really, really helpful. I let the analogy there drawing between sort of like the production of papers right now as being the more symbolic approach where we've got this really specific... [43:05] thing that somebody's asking a question about, they gather small data, usually. And it really depends, like, science is such a broad field. So I also don't want to, like, shoot myself in the foot with, like, well, there's so many kinds of science. [43:17] My space that I used to work in is small-end studies, right? Maybe 10, 20 people. It's not huge. Drawing conclusions from that is hard, and it's hard to replicate. But if we say, okay, the more symbolic pieces of this are rules-based, and we're saying, we've learned something really, really specific about this very narrow thing. [43:37] And now we can create a rule around that. And then the next person comes along and says, like, I follow this rule because you published a paper. I'm living it. So I think a couple of things in there. I do think we should be publishing a lot more data. I think that the... [43:50] data asymmetry between what people can use to train models, let's say like the large language models, right? Like all this text on the internet. [43:58] That was a huge unlock now that we have the compute power to deal with that amount of data. [44:02] We have the compute power, though. So if people release datasets in other domains... [44:07] it would be potentially like this very... [44:10] like synergistic and like [44:13] maybe sort of like almost not a linear, but maybe like a log scale improvement and things, right? Like the more we get this combinatorial power of lots of different pieces of this broader data set that everybody gets a chance to see, right? And so I do think like as time goes on, as we get more access to data and compute, scientists do become more computational by nature, even if they never were before.

[44:34] They have access to the combinatorial [44:37] tools in their toolbox to answer questions. And then I still think it's useful to publish papers, like, [44:45] It's useful to get this information out there. [44:47] The problem has always been we only publish certain things. [44:51] And so we're not getting any information around what's failed or what's been tried. And so we really are only getting a very weird and skewed slice of science. [45:01] as it were, [45:02] So I don't think we need fewer papers. We might need more papers. And if we have a tool to help us sift through those, maybe it doesn't matter. [45:10] But... [45:11] Also, if you do make data available, [45:14] There are people that are able to use it for lots of different things. [45:18] that you may never have thought of. [45:20] Um, [45:21] The hard part, speaking as a former scientist, comes in the [45:25] the way that you chose to collect that data is incredibly important. [45:29] What was the setting you were in? All the way down to the refresh rate of the monitor often. [45:36] Um, [45:37] And so... [45:37] Making this transferable is hard. [45:39] That's the thing is like, I feel like there's this big push, like... [45:44] Basically, there's a big push to generalization. That's the whole object of science as we've construed it, basically because of Newton and in general. I think what we find in... [45:58] cognitive science or psychology or whatever, like, [46:01] It's so contextual. [46:03] that every time people have claimed to have something really, really generalized,

[46:10] Not every time, but like a lot of the time it's like, [46:13] much weaker than we thought and like it's not as reliable as we thought and [46:17] And then people get angry at scientists and it's like, well, maybe we're setting like a super like an almost impossible task and we're like asking our own question. [46:27] And that's why I think the kind of end of one thing is more interesting because it's like, yeah, sure. Like the context really matters. Great. Because I'm one person like. [46:34] Please, I'm in my context. Like, solve for my context and worry about the criminalization later. Um... [46:42] And the other thing that this made me think of, I was talking about like papers and data sets is [46:49] One of the nice things about shifting away from paper as as the thing that you publish is that a paper you don't publish papers that that don't the hypothesis and bear out but if you're just publishing data set as long as the data is quality you'll publish it and then the model is like you unbundle the model which is like the kind of conclusion part from the. [47:10] data, which I think is kind of interesting. [47:12] And then it also, if the thing that's really important is the data, like, [47:18] It's really stupid that researchers have to get grants to study like [47:23] under like five undergrads when like Facebook has like all the data that you would ever need. And this is a family show, so we'll bleep that out. But, um, [47:34] Like what I think is big tech companies should establish data trusts. [47:39] where they will donate their data for qualified researchers to like

[47:45] Ask [47:46] questions and find out the answers, like, it's all sitting there. It's just, we just need to use it. Yeah. And I think, to be fair, there's some of this that does happen, right? Like, and this is the big argument around keeping open source models, right? Like, academics can use these open source models, and they can build on them, and they can do awesome things with them. [48:01] The more you close those down, the more we mess with academics on this point. But I think, yeah, like I think establishing a data test makes a ton of sense. I think like, [48:13] Scientists can ask the right questions. Pushing everything towards the general space doesn't really work. I don't think it works with AI either. [48:20] Um... [48:21] Like to... [48:22] bring it back to this sort of modeling that we're talking about now. And I think the power is not choosing open data sets over papers, but the combination. So for me, so something that like I am thinking about a lot right now is from news articles, how do we deeply understand that information? And then how do we build things on top of that? I think with academic papers, it's similar. How do I deeply understand, like, not the conclusions, maybe not the, like the methods are important. But if I were to [48:49] take all of this information out of this paper, if I had access to the data in combination with that, that would be a really like another level of understanding. [48:59] And even alternative hypothesis generation and testing I could do, possibly. [49:03] depending on the data set. [49:05] Have I shown you extendable articles? Oh, I think I saw it. It came out this week. It came out last week. Last week. Okay. Yeah. I started poking around with it. It looks super cool. Is it cool? It's super cool. On your point where, for people who haven't seen it, like we built this little tool where when we publish an article on Every that has a lot of original interviews and research,

[49:27] You can basically read the article and then we also make all the sources, all the hours of interviews and articles and whatever that we conducted and found and read. We make that available as a little chatbot so you can kind of like go through and form your own opinion. I guess is that sort of what you're talking about? [49:43] Kind of, yeah. So like if I could take, if every news article came with that, [49:47] it would be incredible, right? And this is like, there is a thread through here, actually. The stuff we were building with Recollect, where we were taking all of these pieces of information, like articles you could read, whatever it was, connecting those and then generating something on top of that. We had a line back to what were the pieces that went into this, right? And now on the other side of that, I'm trying to understand news articles that are coming through for the work I'm doing at Smart News. But like the deeper understanding of that that I get right now [50:17] are interconnected and I'm creating that in the models then later. [50:21] but with what you've made with this, like you're giving me, [50:26] Um, [50:27] you know, what is connected to this or like, how did you get to write this article? Right. So I feel like it comes from a couple of different angles. [50:33] But the more, yeah, the more you get those pieces, [50:36] the richer the understanding becomes. And I think the more interesting things we can do with that. [50:40] That is interesting. To generalize, I feel like one of the patterns that we're pulling out is like... [50:48] And this is something I've written about before, but like in only more specific context, but like there's all these places in machine learning, in science, in journalism where...

[50:59] For whatever reason, the like [51:01] the story and the underlying data that were used to create the story always had to be bundled. [51:09] And that created a lot of problems. [51:11] And the solution is not to [51:15] not to get rid of stories. The stories are like also very, very important, but it is just like, we are now in a place where the underlying data [51:25] is actually probably just as important as the story because it's now way more discoverable and legible than it was before without the authorial perspective. And that perspective is still... [51:39] important, but like, it's important to present it with this other thing. And we'll make way more progress as a society if we do that. [51:47] Yeah, I think that's absolutely right. Yep. I think if we can... [51:52] Like from what you're saying, which I am on board with, you as the story creator, you as the writer, you as a narrator, you as a scientist, right, writing this paper, these are all ways of conveying information. It's a story in and of itself. [52:04] you have an interpretation of a thing. And if you provide the interpretation and the pieces that you went through that got you there, whatever those are, they could be other articles, they could be thoughts you put together, they could be conversations you had with other people. [52:17] um, [52:18] All that information, then somebody could then generate their own separate story from it. [52:23] um, [52:24] in a much quicker way too than if they had, you know, [52:27] I got to read the entire reference section of this paper and I got to comb through all of your last paper and whatever. It's no longer necessary. We don't need to do that. And that's so good because like how much time have you wasted trying to like understand some like crazy paper that like is really important, but like you have to understand five background things or read it and like

[52:48] That's not a thing anymore. It's crazy. [52:50] Yeah, it is crazy. And it's exciting. I'm thrilled. [52:54] All right. We're almost at time. Is there anything else that you wanted to talk about or anything else on your mind before we end? [53:01] Yeah, I feel like [53:02] This last conversation is something that would be fun to do more on, but I feel like we won't get the time. [53:09] How does this intersect with the stuff you're doing with every, like, now I'm thinking about this media news space, like, I think there's like such a... [53:16] There's not been much change in this space. [53:19] which I thought would happen with AI, but I think it might be coming and I'm not sure. [53:24] Yeah, let's talk about it. So the question, just to make sure I understand, is how does this kind of [53:31] unbundling a story and data that we think is interesting in science and also in media. How does that affect media? How does it affect how I'm running every [53:40] That's sort of, that's what you're asking about? [53:43] Yeah, and like I think of [53:45] Maybe one level up from that broader question is we've got all the stuff that's happened in the last couple of years with AI. A lot of it is text heavy, story heavy. We haven't seen a fundamental shift. [53:56] in [53:57] Like, [53:57] media, well, we've seen like lots of generated articles and stuff and people have mixed feelings on those, but we haven't seen a huge shift and it must be [54:06] burgeoning. [54:07] Yeah, I think it's coming. And obviously, like, we're to some degree trying to invent it. So, like, I have some specific opinions, but, like... [54:16] I don't know if I'm right. I think I, in general, have an opinion on like where it will go and like a general opinion and then like a specific, some specific bets that we're making. So like,

[54:24] one of those bets is this sort of extendable article thing, which I think in its current form is like not good enough. And it's like, it's really interesting conceptually, but it's not saying that like a lot of readers are going to use all the time. And I want to, [54:36] get there and sort of try to make a standard. Another thing, we've created a new synthetic show from Every called TLDR. [54:44] And [54:45] And TLDR is a three to five minute AI generated podcast about your company. So about meetings that you missed. [54:54] Um [54:55] And so, and it's, it's, it's done with all of the like writing and taste and like, um, storytelling ability of the writers and producers at every, but it's about your company. And we take meetings recordings, um, and turn that into podcasts so that if you miss a meeting, you know what's going on. And it's like the first of hopefully many shows that we'll end up doing. Um, so TLBR is about meeting he missed. [55:25] did or like, you know, a Sunday strategy catch up where it's like, here's all the stuff that happened this week as you're like, you know, reading your morning, uh, reading, like read, [55:34] not reading your morning coffee, drinking your morning coffee, stuff like that. So, so that's, that's another thing, which is like, um, [55:42] As storytelling gets cheaper, I actually don't think it replaces other kinds of storytelling. Like we're still going to watch movies. But it means that we can tell stories in places where it would be too expensive to tell them otherwise before. So like...

[55:58] No NPR producer is going to want to make a show about most internal meetings, but now that doesn't matter. [56:05] Every company has a story and you can have AI tell that story. And that's, I think, really cool. [56:11] So that's another place that we're kind of excited about. Yeah. Oh, that's so fun. And I've been following the notebook LLM stuff too in terms of the podcast they're creating. I love the idea of saying no one would bother to create this. Now we can. Right? Like no one's going to sit in your meetings or whatever and be like, you know, now you're up first. Yeah. [56:35] That is really, really a neat application of that. I've been thinking a lot about accessibility of stories and how do you how do you craft a story that is maybe distilled or shorter, but doesn't lose some of the characteristics of, you know, what you're trying to actually say. [56:50] But, you know, either translate that to another language or make it audio, make it video, make it something that somebody that wouldn't actually like. [56:57] Maybe they have morning coffee, maybe they don't, but like maybe they don't read, right? Like they don't want to sit and read something. [57:03] So I'm pretty excited about like, I think that will be a fundamental shift. [57:07] in terms of the cheapness of producing that piece. [57:11] I agree. I mean, the thing, like, there are all these classic books that, like, [57:15] can become [57:17] more readable, more interesting, more visual, more audio. Like, I've been rewriting this platonic dialogue as a narrative nonfiction and threw it into Sora and, like, it made a movie out of it. And I was like, this is crazy. Oh, my God.

[57:35] That's really cool. Yeah, there's so much culture that, like, is inaccessible because it's, like, the only person that gets to translate it is someone who, like, studied that for, like, 50 years. Like, that's important, but it'd be nice if there was, like... [57:49] 50 other translations that are more accessible for a particular kind of person. [57:53] Yep. [57:54] Yeah. [57:55] And then, like, doing the work of creating a story, like... [57:59] Something we've been thinking about a lot recently is local news, very local things that happen. Nobody does the work. You don't have a ton of journalists out there that are doing this. It's kind of suffering. But you still need this information in a way that is a story. And so I think this piece, it's on the meeting piece, but kind of. If it's the stuff that's happening to you in your little, whatever sphere you're encompassing here. [58:23] Um, [58:24] And then [58:25] Like, yeah, I am very excited about like, [58:28] an underserved spaces, I guess. Like this is the thing where I feel like we can make an actual impact. [58:33] Like, this is the thing where we get people access to information that didn't happen before. And that's huge. It goes back to what you said earlier about raising the bar and lowering the bar. [58:45] And like lowering the bar for storytelling, I think is really important. Yeah. And story consumption, right. Um, [58:52] Yeah, so I'm excited. I think 2025 for me is hopefully going to be this year where we see the application of these things and like your products coming out, right? I'm excited you're incubating these things now. That's super exciting. [59:03] I love that. I hope that's true. I'm excited for this year too. And I'm really glad that we got to chat. Thank you so much for coming on. Yeah, it was good to see you.

[59:33] knowledge bombs about chat GPT. Every episode is a roller coaster of emotions, insights, and laughter that will leave you on the edge of your seat. [59:43] craving for more. It's not just a show. It's a journey into the future with Dan Shipper as the captain of the spaceship. [59:50] So do yourself a favor. Hit like, smash subscribe, and strap in for the ride of your life. [59:56] And now, without any further ado, let me just say, Dan, I'm absolutely hopelessly in love with you.