Designing AI Workflows That Deliver Value and Empower Creativity

NOTE: This is a raw transcript and contains grammatical errors. The curated transcript will be uploaded soon.

Hi, everyone.

One.

And welcome to Convex twenty twenty five. We’re officially raising the contents on three days designed for experimenters, innovators, and fuelless change makers, honestly.

Whether you’re here to break down silos, get the most for from your data, or discover what AI AI can really do, we’re exactly in the right place.

This year, we are going bigger, bolder, more practical, honestly, than ever gathering teams, what honestly driving change in some of the world’s most dynamic organization.

During the course of these these three days, you’ll learn from journeys of Virgin Media or to brands like Pizza Hut, Chewy, Booking dot com, Air New Zealand, Intuit, and many more.

Brands that actually don’t just talk about innovation, but actually make it happen.

What’s ahead? There are live hands on workshop followed by fourteen sessions packed with lessons from leaders at these cutting edge brands. It’s a chance to ask tough questions, build real connections, and rethink experimentation, personalization, and customer experience together.

And now to kick things off, we’re thrilled to welcome our first workshop from a true force in the experimentation space.

Like, welcome, Iqbal, and the stage is all yours.

Hello.

Well, hello, everyone. Thanks for joining. I can see loads of people, loads of familiar faces there as well. So thank you very much. I appreciate appreciate everyone joining.

So welcome to build workloads that work with AI in brackets, and other brackets are using small models and n a ten. We’ll get to those a little bit later. So a little bit about me. I’m Iqbal. I’m an experimentation consultant and trainer. I help teams with their experimentation processes, their and products, especially with integrating AI.

That means I train and coach teams. I run workshops like this one to do with AI and experimentation.

And to give you some background as to where this workshop came from, so I’ve built a tool called Resada that takes user feedback and surveys and conversations and extracts insights from that. So based on all of the development and the the scrapping of code and stuff like that that I had to go through, I’ve developed, you know, some I’ve learned a lot, and this is where I hope I can sort of impart some of that information to you. So about this workshop, this workshop is about how to break down processes to workflows, integrating AI in a way that works and is sustainable.

There’s gonna be some theory, and there’s gonna be six hands on exercises and one piece of optional homework. So if you do do the homework, then get in touch with me, and I’ll try my best to sort of give you feedback.

Now if we go over the the time, I’m just gonna continue. And if you complete the the tasks, then contact me, and I will take out and I will just give you feedback on the rest of the tasks that you missed or whatever. But, hopefully, we’ll we’ll be able to, squeeze everything in.

So a little bit about tools we’ll be using today, the chat, as Dev mentioned mentioned. So that’s gonna be a key thing. So please, at any point in time, if you’ve got any questions or if you want me to pause or if you, just just mention it in the chat. I’ll also be putting in links, so that’s gonna be something to look out for because we are gonna be using Canva whiteboard to do most of our exercises. So if I take you over to a Canva whiteboard, I’ll show you what that looks like.

Here we go.

So here’s a Canva whiteboard, and let me get rid of this thing. So the there’s a lot here, but the main thing, the only thing really that you need to focus on is if you go to elements, rectangle, and then just click on the square.

Oops.

Click on the square or drag the square into the main window, and then you can change your color here.

Give it any color that you want, and then make sure the text is also there. If you double click, you can you can type in a message. So so yeah. And if you want to copy that, if you hold down the alt key, you can drag out a copy of that rectangle or that note. I’ll be referring to them as notes from here on. So here’s the first thing that I want you to do.

Send you this link.

Share the copy link and put that into the chat. So if you head on over to the board and, and if you introduce yourself, just write down, your name, what you do, and your experience level with AI. So whether it’s you’re brand new, you’ve only used a chat interface or whatever.

What what I’m hoping this is gonna do is gonna really test out this system to see how well Canva whiteboards performs with so many people.

So we’ve got some contingencies in place because I can see there’s a lot of it’s like a swarm on the on the screen.

But, yeah, let’s see who we have.

Hello, Naomi. Hello, Laura. Hello, Claritin. Hello, Ella. Sorry if I don’t get around to saying hello to everybody. There’s a lot of people here.

But cool. But so as as you’re doing that, and I’ll and I’ll keep keep an eye on that, and I’ll come back to come back to that later on. So if there are any technical issues, then please let us know because this is that intro is the place where we know whether or not this is gonna work for the rest of the exercises. So so I thought I’d start by by asking since by trying to answer a simple question. Why do we struggle to get value from AI?

And what I’d like you to do now is if you head on over to the poll diff, can we put the poll out?

Or does does the poll automatically go out?

If you head on over to polls yeah. If you head on over to polls, there’s a question for you there. So after you’re done with your introduction, if you head on over to the poll, can see yeah. Okay. Great. Great. Great.

Why it biggest challenges with AI, knowing where to use it, inaccuracies with output, and inconsistent output. So, yeah, knowing where to use it is a big, big issue, and that is the the the kind of the core thing that we’re gonna be tackling. Inaccuracies and inconsistency is also something that’s gonna come along with that.

Okay. But so so to to to hone in on that then, Can you guys still see my screen? Just wanna make sure that ah, okay. Okay.

Cool. Cool. Cool. So so just to hone in on that a little bit, a lot of you are experimenters.

I could see that. So imagine this scenario, you’re being asked, hey. I need you to make me all of the money. And so what you do is you go away and you apply some processes, company processes.

You you apply some processes of your own, and then there’s a magic part, the mess that is your brain.

And then a few months later, you report the wins of your experiment. But how exactly did you do that? Because no matter how much processes we’ve got, it’s it’s still a little bit of an unknown. And now we’re in this situation where we’ve been given this new tool, this AI as a partner.

And AI is like a technology looking for a solution. There’s no manual. There’s no guidebook. We’re just supposed to know how to put it to use.

So when we try and get AI to replicate our work or integrate AI into our workflow, it’s tough. It’s really tough. Because the truth is, as I said, as many processes as we have, we still use our brains and our brains are a black box.

And this is a known phenomenon. So Ethan Moloch pointed out a paper that detailed how experts can’t explain how they do things.

So this is this is like the the the general gist of the workshop. How do we disentangle this, when we don’t know how we do stuff?

So in summary, why we struggle with AI? Because we don’t know how we do things, so we don’t understand how AI can help us with that.

And and just if you can put into the chat for the next one, what I no. Sorry. I’m gonna skip over this one.

Okay. Okay. Yeah. So what’s wrong with our current approach to, to using AI? So this is a typical approach.

So typical approach is that, we’ve got all this user feedback, all this user feedback text, and then we go to our chat window, and then we throw all that user feedback into the chat window, and we tell it, hey. Give me eight ideas from this user feedback. We tell ChatTPT, give us some, eight ideas. And ChatTPT does some squiggly line stuff.

So ChatTPT is also a black box, and then it comes out with eight ideas. So and in effect, what we’re doing is we’re replacing our black box with this alien AI black box.

And so what that results in is that after the sheen and the wow factor of us actually getting some response back from this thing wears off, we’re left with a gap between what we expected and what we got.

And that gap can be described in terms of accuracy, relevance, and consistency.

So what we expect to get back from AI and then what we get back from AI, there’s this gap that that that opens. And I think a good way to demonstrate this is using this as a comics as an example. So I’m I’m a comics guy, so so I’m gonna be using comics as a lot to kind of explain some concepts, but I think this really demonstrates the the issue and to demonstrates, illustrates the issue. Imagine we wanted to create a two panel comic strip.

So we know what the first panel is gonna be. It’s a guy standing in front of a cathedral. He’s saying ouch. He’s he’s he’s got his collars up.

He’s wearing a hat, and we want help with the second panel.

And along with that, as in terms of from a comic perspective, you don’t need to know anything about comics, but there is some domain knowledge that comes with creating that second panel. And don’t the the key sort of concepts that you just need to know are that comic panels are static in nature. The panels are static representations of moments of time frozen. So, so we can’t have a panel like, oh, Belinda goes to the fridge, grabs a beer, and drinks a full bottle. There’s too many actions to fit in one panel. And there’s also a limit in terms of the amount of information you can fit into a panel, And that limit is based on what your style is. For instance, the style lounge the the comic style that’s there, the other panel needs to fit with that comic style with the same amount of information.

So when we give AI this task to predict that second panel, it it goes a little something like this. So we’ve got a prompt.

We describe the first panel, and we ask it, give me some ideas. And then it gives us some ideas. And don’t worry about reading through the ideas. I’ll save you the work. None of them really work because domain knowledge is missing. And that gap, appears, in terms of, there’s too many actions in the panel, there’s too much information, and there’s additional context that I didn’t give it that I was thinking or I was feeling like this is this is what it needs to be, but, I wasn’t able to communicate it. Therefore, it’s it doesn’t exist in the final output.

And this leads to the core point in this entire when we when we go and build workflows, etcetera, humans need to be in the in the room as it were for their domain knowledge. So as part of the workflow, as part of everything that we do, we need to make space for for humans.

So what this means is that coming back to this example where we’re throwing in a user feedback and we’re getting ChatGPT to do some squiggly line stuff and give us eight ideas, what it means is that we need to break this down into workflow. We into a workflow. We need to break apart this the the chat GPT, the black box, and we need to insert ourselves in there and insert multiple steps in there.

And so the next part, why do we want to use small models? So I mentioned small models on the opening, and small models is a big part of this this workshop, and small models are a big part of, creating workflows as well.

And to start with, I wanna point out some of the problems, so go deeper into the problems that we’re facing when we when we go through that process. So when we the first problem is in terms of context size. So context size is the amount of text that an AI that an LLM can handle.

How to create reliable agents? Okay. Yeah. So these are some some messages coming in. It’s great.

Yeah. Put in the chat, just throw in what are your major concerns with, with AI. It was it’s one that I skipped over, but, yeah, it’s good that some people have picked it up and, and throwing it in. So so yeah.

So, yeah, this is why the output there there is an impact in quality in the output because the because of this sort of, like, throwing in too much too much text.

But why is that a problem given that Gemini or Google claim that the flagship model can deal with the equivalent text of all seven Harry Potter books?

The problem is but because even though they they claim that, the reality is a little bit more complex. Because in real world terms, the optimal amount of content that an LLM can handle or their LLMs can handle is three page. That’s a heck of a difference, right, from from what they advertise as you can throw in seven volumes of Harry Potter books worth of user feedback to going, no. Actually, it’s three pages. So I’ll I’ll actually I actually wonder how many people knew about this.

Oh, hey, Marcello.

How many people knew about this? And, yeah, just just just mention in the comments or is how many people are new to this concept that the fact that the advertised context windows is not as the company’s company’s presented. Yeah. Yeah.

Because they they don’t tell you. Right? The companies just kinda present the best put their best foot forward. If you wanna find out more information because there’s a little bit more nuance in that fact.

I didn’t know it was my experience. So yeah. Yeah. Exactly. Then I’d I’d recommend reading that paper. Just Google no Lima, and then you’ll find out.

The other problem is the problem in terms of capability.

So AI agents, however much we’ve heard so much about them and there’s so much hype about them, But in actual fact, from that the study that I’ve got here, benchmarking LLN agents and from my own use and from other studies that have been done, they only have a success rate of thirty percent. So that’s not great. That’s not as you know, that doesn’t match the expectation and the hype that’s that surrounding agents.

What’s more, the the AI models may have been trained on the tests used to measure them, I. E, they’re being trained to pass those tests. So the benchmarks that they are used to be measured with are finding their way into the training data, which leads to this now famous Apple paper. And in that paper, they go straight for the jugular and say, there’s an accuracy collapse with the latest models. So so, basically, in terms of capability, there’s a mismatch often in terms of what is presented and what the hype says versus what the reality is.

And not only are we seeing diminishing returns from in in terms of the the the the performance of LLMs, but the power requirements are going up by two point four times every single year. So while the the performance has diminished, the power the energy requirements just keeps going up and up and up for AI.

And based on OpenAI’s own reported numbers, they serve about two point five billion prompts per day. And if that’s true, then based on the estimated power consumption of a single prompt, that’s the equivalent of of around one point six million homes. So that’s the size of a medium sized city. And remember, this these power requirements are going up year by year. So these power requirements are incredible, and no sustainable energy is not enough to to cope with this.

If you wanna find out more about this, and there is a lot more to unpack here about that, I will just just read Karen Howe’s excellent book on that Empower of AI.

So enough problems. Give me solutions. And I mentioned something at the beginning, which is about small models.

Is this power requirement for training or inference? The good question. The the it’s training and also for inference. I’ll get to the inference thing in a bit. But, basically, they both are going up. But the the energy requirements in terms of the two point four times every single year, that’s specific to the training.

Okay. So so does all AI suck the planet dry? Well, no, because they’re small models, and, small models are as dumb as each other. So one thing that you probably didn’t know is that if you’re using the chat interface, you’re using the latest flagship model. But what you don’t have access to are smaller models, and there are often lots of different versions of those smaller of that same flagship models available as a smaller model.

And the if you look at the comparison of the small models versus the large models, this is a visual of in terms of what it looks like in terms of the energy requirements. So that’s a large model and the and the energy requirements for the large model, that visual. And then just just look at that difference. It’s it’s absolutely huge.

And these small models are cheaper, faster, energy efficient, and they also give us a path to going local.

But and here’s the thing. They need to be used differently.

Is there any other what’s the power of ones? Okay. No. Sorry. I thought there was something there. But they need to be used differently.

But that’s that’s actually a good thing because because the advantages of using it differently are transparency, accuracy, and consist consistency. And this is what that difference looks like. So instead of few long complex prompts, we have multiple simple prompts instead.

And as you break down your workflow or your prompt into multiple simple prompts, you can start to integrate humans and tools and and and implement integrate them into the workflow. So not everything needs to be done by AI.

And in general, we need to be thinking like architects, and we need to be thinking in terms of workflows.

And what this leads to, and this is the other thing that I mentioned at the outset about n a ten, is once you’ve created these multistep sort of processes and even if you’re going through doing them manually, there’s a step or there’s a path for you to go and automate that and put it into visual tool like n a ten can do that for you, and we’ll be coming back to n a ten again and again. Some of the concepts that we’ll be learning is gonna be something that’s very, very relevant for n a ten and can be used within n a ten.

Cool. Thanks, Margaret, for that link.

Defeating nondeterminism. Okay.

So to summarize why we struggle with AI, want to, solve the or rather to so to to summarize the problems that we’re trying to solve here. The problem of context window limits, ignore ignoring the misinformation, we need just need to step aside, bypass all of that misinformation we’ve got. We need to work more sustainably, so we need to use small models and we need to work with multiple steps and multiple interactions.

So, we’re getting we’re getting into the first task, in in a second. But the killer question is, we’ve got this black box.

How do we build a workflow from that black box?

Because remember, if we don’t know how we do stuff, how are we supposed to build a workflow around it?

And we do that by this because even though we don’t know exactly how we can how we do stuff, we can still apply a rough scaffolding to the whole thing. And you can break down majority of things that we do with this simple sort of diagram. So we there’s there’s three stages, explore, concept, and deliver. And in the first stage, as you’re exploring, it’s just kinda like, an idea leads to another idea, which leads to another idea, which leads to another idea.

And then as you make decisions, you start to focus your ideas, and then you start to refine and say, this is the idea that I wanna go go with, and you start to explore that sing singular idea.

And then the final piece is all about delivery, is all about rendering.

And the problem the problem that we’re trying to solve or the problem with our current approach is that AI forces us to that end step too quickly.

And this is an example of that. So you throw in your prompt, and then behind the scenes, this is a really bad slide, but it’s there to demonstrate the problem, which is that it’s going through doing all of the thinking steps for you so that you don’t so you don’t need to, and it’s racing to the to the end. And because it’s races to the end, it’s missing all of that domain knowledge. It’s missing you from the central, creative process, and also it’s giving you output that’s that’s subpar that doesn’t meet expectations.

So what we really need to do is to make sure that we box ourselves and scope limit the scope of the of our workflow to a specific section. So we need to break things down into steps. And if we’re saying, okay. I’m in the exploration stage, then we need to we need to make sure that we never leave that exploration stage. And only once we’re done with the exploration stage where which the where I’m sure that I’ve got the idea that I need to adapt to to to move on to the next stage where I’ve got the idea to then refine that idea, only then do I move to the next stage. And that is a human based decision that needs to be made in terms of to validate, leaving to the next stage.

So what we need then is to take our process and to, first of all, break it down into three broad stages based on separate processes that are going to be that are going to be used in each process. So an exploration process is a very different type of process to refining to delivery, and we need to understand that. And we can only move from one stage to the next once we’re done with that stage.

And so when you look at workflows like NA ten, they can pretty be pretty intimidating, but we just need to break them down, and we just don’t have to try not to bite off more than we can chew.

Okay. So here we get to see the beginnings of a workflow. So so if I take the comic panel that I was trying to do, so if I go all the way back, so this comic panel this comic panel that I wanted to create So if we break that down into steps, I start with figuring out and deciding what is it that I want to communicate, and then the output of that goes into the next step, deciding how to communicate it, and then the output of that goes into the next step where we render the second panel or we render that second panel.

Okay. So and this is what that looks like when we limit the scope, by the way. So when we limit the scope, it only gives us the output for the exploration. So it if I give it for if I say, give me broad ideas, approaches I can take, then it’s going to give me back those broad approaches. And then once I decide on a broader approach, I can move on to the next step. So what I want you to do then, as a first task is we’ve got this, central sort of, common way that we currently work, which is we take this user feedback, put it into the prompt, and then we ask it for ideas. What I want you to do is to create these broad level stages for the ideation.

So you can look at it in terms think of it in terms of how you work or a process that you’ve read about or you’ve seen. What are the different stages? If you were doing this yourself, what are the different stages that you would do to to identify those broad broad stages? So I’m going to throw in.

By the way, was there any technical issues with the introduction?

Anybody had any technical issues?

Okay. Cool. So I’m gonna throw in a link then.

So head on over to the Canva board.

And if I drag that over, So this is where you should end up. And what I want you to do is create your workflow. So those broad three stages, three or four broad stages. So something like decide on approach. And then what you can do is if you click on the arrow next to those next to the box, it will create an automatic linked box. So, yeah, that way you can, you can create your workflow, your example workflow.

So find yourself some space. I know it’s gonna be difficult. There’s a lot of people here.

A good tip is to go up or down, not left or right. I think if we if we all sort of like scroll up and down and avoid each other, then I think this should this should work fine. If there are any technical issues, I do have some other boards for you to go on, and and then, yeah, we can see those. And what I’m gonna do is I’m gonna try and give you as much feedback as possible in terms of those broad sort of steps. So remember, what we’re doing is that current approach of us ideating on experiments. We’ve got some user feedback, and we want to ideate on some things for our website, some tests for our websites.

Okay. So let me just see how we’re doing for time.

Okay. And now’s probably a good time to give you a tip here.

So here’s a tip.

If let me get the URL link.

Now remember that thinking process that we used, that I showed you before? Oh, let me create a new tab so I don’t lose my place.

Now I’m gonna throw in into the prompt into the chat window an example prompt. And I said I don’t like this single prompt approach, but I’m gonna do this one one time for you. Because, basically, we can use the reasoning to figure out what’s how to break down those steps. Because if you if we put in a task to say, like, this is an example prompt, we say context. This is my context. It’s a smoking bones. It’s a theme park, and this is my role.

I’m a content marketer, and I want to identify specific problems that users are facing. And the task attached using the attached feedback, come up with some matching content ideas or problem ideas or whatever it is, and then we give it some example user feedback. I’m gonna put that prompt into window.

Oh, that’s exceeded the limit.

Actually, what I’m gonna do then is I’m gonna dump it in an area here, text box.

Okay. Sorry, everyone. I thought just kinda gone over you.

Okay. So over to the left, I’ve dropped in that prompt. So what you’ll see is I’ll go through reasoning steps. And in that reasoning steps, AI is actually doing that breakdown of the process for us. So in there, there’s clues in terms of how a politic will just nicked the first step in the workflow. Sorry about that.

So in the reasoning, there’s there’s clues in terms of how the entire pros how AI is going to tackle the the the problem. So you can see here that what it’s doing is saying, okay. First of all, my step by step approach is that I need to analyze the user feedback, look for both both positive and negative comments to identify problems and strengths, and then identify key problems with the feedback and identify positive feedback, etcetera, etcetera. So you can use the reasoning as clues to kind of figure out what your broad stages should be.

So if you use, like, a ChatGPT or DeepSeek or something like that, you should be able to see the reasoning.

How are people getting on getting on? Is this something that anybody wants me to look at? I’m just gonna go through and understand all parties involved. So that’s hedgehog.

Understand all parties involved so AI understands the standard point of the output. Okay. Gather user feedback from all sources. Ask AI to look at patterns.

Ah, nice. So hedgehog.

I think that’s that’s hedgehog. So if you that that breakdown that you’ve got is a little bit too specific. So think in terms of really, really broad steps. So think in terms of, you know, the explore, refine, and deliver sort of steps and what an exploratory sort of stage might look like for this task, what a refinement stage might look like for this task, and what a delivery stage might look like for this task. So think in terms of those broad steps.

Gather user feedback, identify trends time yeah. So Claritin is there. That’s a good way of doing it. Turn the insights into hypotheses.

Great.

Gather all internal relevant information.

So I’m gonna share my version of this, and feel free to update your version if you want to, that is, to to match that. But I’m gonna be using my version to go forward with the rest of the tasks. So in my version, what I’m saying is as the exploratory stage, I want to go through list all the problems. So there’s gonna be a certain amount of going back list of problems, going back again list of problems, you know, just just kind of, like, make sure we list positive, negative, and stuff like that as many of you have got this step already.

The next step is basically once I’ve got a list of the key problems, I want to then identify the highest priority problem and then understand the problem and then ideate. So it’s good that many of you have got those broad stages in there, some version of that Capture customer inputs, validate the conditions, and understand the situation we find through. Yeah. This is some really good stuff here.

Oh, Ronald. Ronaldo. Is that you? Yeah. Yeah. That’s good work.

Okay. So let’s go back to this.

So a recap then.

Break the overall task into broad stages and decide on what it is that you want to validate before moving to the next stage.

And speaking of which, this is where we come to the next task. Does anybody need any more time? Because we’ve I think we’ve got a how are we doing with time?

I think we can handle up the time. Does anybody nope. We’re all good?

Okay.

So the next step then is for those stages that you’ve got, try to identify what the output from each of those stages is going to be. What do you want the output to be, and what do you want that output that feeds into the next step to look like?

And to so we’re gonna dive right into n a ten for this part. I’ll show you what that looks like.

Play can you please share the first stage of your version again? Okay.

Oops.

It’s that list all the problems, highlight the highest priority problem, understand the problem, and then ideate.

And this is an alternative version of this as well, So you can also do analyze positive and negative size of each problem, deeper understanding of the problem, and ideas. That’s a slightly different phrase in a slightly different way.

And I think others have got it phrased in this way as well. So what I want you to do now is to to think about that output and what that output will look like. So really think hard about what defining the output, remembering that it is going to be going into the next stage as an input.

And, yes, I promised that I’d go into n a ten. So before I do that, just wanna see what people have done. And you can do that as j okay. So where you’ve got most of that, where you’ve got JSON, be a bit more specific.

It’s not just like a in terms of format, but in terms of what exactly. What is the content of the of the output. So capture customer inputs. So I’ll show you my example.

I’ve got it.

Here’s my example.

So for instance, for a deeper understanding for example, for a deeper understanding of the problem for that stage, just just like a problem statement with a breakdown of what users have said. So that is the content that I need to go into the ideas stage.

And another way to what you can do is the size of each problem could be a table of problems with the number of mentions of that problem next to it so you know how of it how much of a big problem it is. So the so you so you know the size of that problem.

So think in certain terms of those terms, not necessarily in terms of structure.

So looking to see how everybody else is doing.

Oh, this is a good one. Polyus.

So Polyus has got extracting key objections from the user feedback.

List of objections, identify, summarize exact problems, highest priority, and then that goes back.

Okay.

So yeah. So that is that is right. So, basically, what sorry. Can you hover over a Polyus, was it?

Yeah. Yeah. Yeah. Yeah. Can you can you hover yeah. Yes. I can see your name.

Yeah. So the what what you see on the screen there, that is a good example of that. So list of objections, prioritize problem list. We can go even further and define the define the output even further. Now the reason why we’re going to this level and we’re defining the outputs to that sort of level is because when we move over to n a ten and let me get the link.

Copy link URL.

I wanna leave that open.

When we move over to n a ten, this exercise that you’re doing matches n a tens the the concepts of n a ten quite well. So this is a really simple workflow where we’ve got a trigger, which is the first sort of thing that triggers the entire workflow, and it’s just a chat input. So if I type in hello, then the output from me typing in hello is hello. And if you open up the basic LLM chain, you can see on the left is the input, and the input is hello.

The middle section is all the stuff that the n a ten that that that node is doing, the the actions that it’s performing.

And then the output is the output from that node.

So that is and then when you when you go back to the overall workflow, if I add a new node, that output, this output that you see here, is feeding in as input to the next step. So this is how this maps, this kind of thinking, this kind of output driven thinking maps to to n a ten and to other workflow based tools, especially, like, technical tools like that. And from an AI perspective, working in a output based way or output first way, it has a lot of advantages because you’re very clear in terms of what is it that I want from this stage before I can move to the next stage.

How are we doing?

So let me see. Research methods.

By g. Oh, by g. Research different methods, user interface, define the problem we discovered. Yeah. This is a really good one. Test with real users. So, Biji, if you can define what is it from the research oops.

From the research step, what is it that you want as an output to go into the next step specifically?

I say, here’s an example of mine. So Let me see if anything else.

Owen. Hello, Owen. Gather all internal relevant information that the AI workflow needs to make tailored responses.

The output, the AI workflow, summarize all internal data and be ready to draw comparisons.

Yeah. From the output, Owen, be really specific in terms of, an actual tangible output that you get. So, so a table of, of, you know, if it’s summarized data, for instance, is it a table with summarized outputs with with draw comparisons? So how will you draw comparisons? Be ready. I know you’re saying it’s ready to draw comparisons, I’m guessing it’s the next step.

But yeah. As I as I give you feedback, feel free to put in the chat if anything’s not clear or if there if there’s any other further clarifications you want me to make or if I’m missing something about what you’re saying there.

Yeah. An example would be good.

So let’s see. Oops. Let’s see what else.

Ronnie, gather user feedback from all sources, output AI format, the data.

Okay. Cool.

Okay. Nice. Nice. So so I’m gonna move on to the next stage then.

Okay.

To the next stage. So so far, we’ve not been doing much with AI. I’m aware of that.

But it’s it’s it’s really important that we go through the initial stage of breaking down, our broad sort of approach to doing something into stages so then we’re able to really understand, the the broad level way what our process should look like. And it doesn’t need to be really specific in terms of exact tasks. It can be a broad level, but understand what it is that we need to validate and the output that we need to be able to move from one stage to another. So now to understanding AI.

And this is actually, I’m gonna give you another whiteboard to to do this task. So anybody familiar with mind maps?

Just mention in the chat. Anybody familiar with mind maps or use mind maps on a regular basis? Yes.

Nice. Nice, Stephanie. Yeah. Cool. So specifically talking about association mind maps. So there’s a few of the few people who have not.

So, basically, what a mind map is is that you start with a topic. For example, this topic you see here, and then you branch off associations from that topic. And then from that from those associations, you branch off again, and you keep branching off and branching off to topics, subtopics, categories, related items, just things that you associate with them. So for instance, here for example, dog, dog, I it has tail, it barks, tail, tails wag, tail, planes, planes have tails.

And so you can you can yeah. Like nodes, Owen. So you basically move from from association to association, and you can move on a very in a different you can end up in a very different place from where you started. So we’ve gone from dog to plane as you can see here.

So what I want you to do, and I’ll send give you a let me give you another link for you to do that so you don’t mess up that board we’ve got.

So in the chat, I’ve put another link to another Canva board. So what I want you to do is to, create an association map. And as a starting point, I want you to use theme parks as a starting point.

And remember, there’s no right or wrong answers. Use your experience, use your knowledge. Just go with what you associate with theme parks and of the output that you get from those associations, what do you associate with that, and keep going and keep going and make these associations. So if I go to the to the board, Oh, nice.

There’s even people doing it rounded. Yeah. That’s cool. You don’t need to do it rounded, but, you you know, I do appreciate the effort.

Yes. Theme parts and queues, Marcella. That’s that’s that’s my key comp sort of association.

Ah, yes. Theme parks, rides, food, theme parks, roller coaster, food standard food stands.

Nice, Jonathan. So you got theme parks to rides to kids rides to thrill rides to food to snack carts. And when you when you bridge out to associations, don’t be afraid to change from theme parks if that’s the way it’s heading. It doesn’t always need to associate back to theme parks, but but yeah. I I like the fact that it is. But, you know, don’t dis don’t not include any associations because they don’t link back to theme parks.

Cars, engineering, roller coaster, turkey legs. Yep.

Ice cream, smiles, fun smiles, kids scream. Yeah.

Cool. So as you’re doing that, let me just go over why I’ve got you to do this exercise.

Because mind maps is, I think, the closest way to understand how AI thinks.

So when you put in a prompt into a prompt, into a prompt box, into chattypity, whatever it is, that prompt is broken into pieces.

And from those pieces, an LLM makes patterns or an associations and is trying to predict what the next word in the step is. And it makes from those pieces lots and lots of those associations until it decides this is the output that it is I’m gonna give you. And that decision making process is entirely probabilistic from from AI’s perspective.

So going back to our comic prompt example.

So how AI is working is it predicts the next item in that sequence. So if you say, here’s my comic panel. Give me the second panel, and this is the prompt that I’ve given it.

And the top part is the context, is where I’m describing that first panel. Man wearing a fedora hat, jacket, collar, half collar covering his face.

And then in the background, there’s a cathedral with gargoyle statues. And in the background still further background still, there’s a sun like a disc in the sky and saying, give me five ideas for the second panel. So that’s the task. So a first part is a context.

What do we have so far? What information do we have so far? Second part is what do you actually want the AI to do? So now we had a look at this earlier on, but I I brushed over very quickly.

But let’s take a closer look at the results that AI gave us.

Because as it mentioned in the prompt, we mentioned in the prompt, collar half covering face, and the man, he’s saying, ouch. So here in the results, you can see that it’s come up with neck and, neck collar. So it’s an idea around something to do with the neck and the collar. And then there’s also another idea to do with gargoyle chips and statues, which is something else we mentioned in the in the context in the panel description.

And and then there’s the hat. And because we mentioned hat first, it’s it’s output hat twice here where it says there’s a two ideas with with hats in them, all within the the the focus of, you know, why is he saying ouch because that is what’s what’s really important.

And so I I hopefully, you can see, like, how it’s it’s not drawing on it it is or rather, it is specifically drawing on the context that we’ve given the AI and how it’s kind of like going through this pattern matching exercise in order to determine what output it’s gonna give us, what ideas it’s gonna give us.

And this maps closely to a very specific way that we think. So if we want to empathize with AI, this is the closest sort of match that we can do. So as we’ve gone through in doing the mind map exercise and we’ve done that in a very purposeful way, when we’re daydreaming, that sort of associative sort of thinking happens naturally. So so this is called episodic future simulation where brain ponders future events based on known information and associations.

So it’s a very natural way for us to think anyways when we’re feeling bored, we’re daydreaming, we’re we’re kind of thinking in the same way that an LLM thinks.

So so while an LLM can come up with really specific predictions and really specific ideas based on the context, it may not have the experiences we have. For instance, it probably will not come up with some something like this where it it kinda taps into somebody’s insecurity about their curry cooking abilities.

So I hope you can see from that exercise that what AI is doing is it’s just simply matching patterns. It’s a simple intelligence. There’s no such thing as facts. There’s no such thing as reason. It doesn’t do math, and it’s terrible and it has a terrible memory.

So this leads us to the next task.

So back to the original whiteboard where we’ve got our stages, I want you to add notes for ideas where you can apply AI to those stages.

So given what you know about AI and what its strengths are, that it’s a pattern matching tool, and that it yeah. That what it’s not very good at, where would you give us some ideas about where what you could do with AI, how an AI could enhance maybe the workflow that you’ve created, or maybe also concerns about where you may not want to use AI.

And you can do that as new notes. I believe what you can do is if you right click and you can add comments. So you can add comments to your own sort of sort of your own card. So And when you’re coming up with ideas, try to link it to its strengths. Like, try to use the terms pattern matching and, and associations or whatever, and think back to that mind mapping exercise that you did.

What is a good use of a tool that can do associations at a massive scale?

And if there’s any questions, then feel free to ask them in the chat.

Polycyst. That’s those are good ones. Blank page so polycysts has got blank page syndrome, suggest new creative ideas that are not necessarily logical but applicable. So polycysts, we is that something that you’d that you would say is a strength why you would use AI or why you would not use AI?

Is that a an idea for AI or an idea to say, you know, that’s a concern with AI? I don’t wanna use it. Hey, Meenaksi. How are you doing?

Let me see if I can find you, Meenaksi, in that.

Oh, wow. What’s happened here? Story mind map.

I’ve done the approach, and I’ll put you to strengths. Can you review once? Okay. Let me let me try and find you. Good luck trying to find you.

May I ask you what color are you on the canvas board that is?

Hang on. Where are you?

Bottom left. Okay. Hang on. Hang on. Hang on.

Meenaksi, move around, Meenaksi, so so I can see you on the on the canvas board.

Thanks, Marcelo.

Okay. So here we go.

Provide context output table. Mean, actually, goal. Okay. Yeah. Table of the frequencies. Let me see.

Five five. So, I mean, actually, what are the specifically, you want me to look at in terms of the ideas that you think AI what AI is good for within that process that you’ve identified there?

Because your overall process is a really good objective and key questions to be probed for study.

It’s actually really, really good deeper understanding of c sentence. It’s for different themes.

If you just write in the chat, I mean, actually, what what is it the what kind of ideas is it specifically the AI ideas where AI can be used that you want me to look at? Because three, five themes for each research question table of sub frequencies.

Okay. Okay. I get it. AI strength of segregating the observations.

I’m not sure what you mean, actually.

Segregating as in as in basically taking observations and saying these are my observations, putting them in a box and and, yeah, grouping observations. Is that what you mean? Like, grouping?

Because, yeah, I think grouping observations is is definitely a good use of AI. Let’s see if we can see some some crazy ones. Anybody got some wild ideas for where to use AI or what they think are wild ideas?

Okay. Yeah. Segregating frequencies and attaching the sentiments. Yeah. Yeah. So that’s definitely a good use case for AI.

List of prioritized payment sites.

Nice. I like, Juan.

I like your the outputs.

Those are really good. And the broad stages.

An AI workflow here can provide useful oh, and I’m looking at yours.

AI could be used here to gather and summarize large amount of data that could take human days. Yeah. Yeah. Definitely.

Is there anything else that you would that you could do that’s outside of the scope of that to enhance the the entire thing? Like, for instance, you know, merging with different data sources and finding patterns beyond just the user research sources. So throwing in other user resources and kind of, like, pattern matching across different sets of databases where there’s maybe joins are not available for you, those of you who are analysts.

But, yeah, this is good. An AI workflow can be pretty useful for prompt to connect the right external information.

Yes. This is really good.

Oh, and this is really good. So a workflow to connect the external information to the internal data.

Yeah. So this is where you’re where you’re actually doing that. We’re fetching relevant data from other sources, external sources. Okay. So this is really good.

Cool.

Anybody else want me to review their stuff?

One of the reasons I will have to use AI to have fast horizon than mine, which is limited only by what we’ve read and learned, getting ideas beyond my horizon, slowly learning from them, reading from both Internet can get more knowledge as well as better ideas. Yes. Exactly. Exactly, Santana.

The AI has got a broad wealth of information that can then feed to into you to then get better specific knowledge and better ideas from that.

Okay.

So recap of what we’ve done so far. So we’ve taken a broad overall task, and we’ve broken it down into stages into broad stages. We’ve defined the output that can help us validate the the the output from each individual stage, and we’ve developed some ideas for AI usage.

So the the next part is to go over tools and connections because so far, what we’ve been using is just humans or AI, but actually there’s, you know, the the the technical scope of what we can include is much bigger. Oh, Minaxi, regarding the mind map, how shall we use it in our approach? Or did you explain it just after well, yeah. So, Minaxi, the reason why I got you to do the mind map exercise is for you to empathize and think, how AI works.

So I think that once you’ve got into the mindset of what of how AI thinks, that then leads you to consider all the different ways that you can integrate AI into your workflow. So knowing that all it’s doing is pattern matching and knowing that it’s about associations, what kind of activities do you use associations for? Like, you know, the creative sort of exercise is a key one that Santano mentioned where basically just making associations in order to come up with ideas about something, and then AI just broadens that and just kinda gives you much greater scope for for for getting patterns that match the the the ideas that you’re you’re going for.

Does that make sense?

Oh, Sofia, will this webinar be available after it ends after leave now? Yes. Yes. It will be. It will be available.

So yeah. And, Sofia, if you want some feedback oh, yeah. Thanks, dude. And if you want feedback on the rest of the activities and associations, then just, yeah, please message me. Happy to give feedback. That goes to everyone as well, by the way.

So tools and connections. So here’s one of the problems that we saw earlier. So if you remember, we threw in a large amount of text, user feedback, and we we said that we’re getting some data loss. So remember Google lied. They said seven books worth of Harry Potter when really they meant three pages. And so when we throw in all of our user text, we end up with a huge amount of data loss, and that results in hallucination and inconsistencies and inaccuracies in the output.

So and this is the the the the the description of that data loss. So as we as it’s before it gives us the output, we see data loss there. So the first tool I want to introduce you to as an arsenal is called a vector database.

And what that process is is or what a vector tool does is you once you split a text into chunks so here you can see I’ve got some lorem ipsum. So imagine that’s user feedback.

That text is broken into chunks.

As you can see, it’s been broken into three chunks. And then those chunks are stored into a database, into what’s called a vector database.

And what that means is that once it’s stored in that vector database, we can we can make searches that are not particularly to do with, like, exact searches. So it’s not like SQL where we say, find me everything that is matches this precisely. Rather, it’s a semantic database, so it’s able to return results based on similarity, based on how similar your text is. Remember going back to the associations that AI makes associations and stuff like that, that applies to vector storage as well. So basically, vector store just stores the those texts, and then when you make a search, then it kinda checks based on that search request what are the similar what are the most similar chunks to do with that. And what it returns is each chunk is like imagine it’s returning a table. It’s returning a a table where you have chunk of text one, chunk of text two, and a similarity score.

And the similarity score measures the closeness of that chunk to your search term. So if it’s if the if the similarity score is one, well, it’s an exact match for the text that you put in. If the similarity score is zero, then it’s gonna return everything. And then there’s grades, between that obviously as well.

So what that does is, or what that enables is for us to to search based on some specific keywords or based on some areas like anything to do with the checkout page, anything to do with the payment processing, anything to do with their experience on a specific landing page. You can make searches like that, and the vector store will return similar items. And then what you can use is you can use AI purely as a way to summarize that text and to feedback that text to you in a nicely formatted way.

So does that does that make sense?

Let me know. Say no in the chat if that doesn’t make sense or if you want more sort of description on that before I move on to the next tool.

Okay. Cool. That makes sense, everyone. Cool. So the next tool I want to introduce you to or rather not introduce you to because I’m sure you’re all aware of Google Sheets. So but it can be I’m using Google Sheets as an example, but it could be anything that you store information into, like basically, it’s like an as a spreadsheet that you might already have user feedback in.

And the connection basically means that you would be connecting to a data source and then fetching data from that data source. So dumb question, is the audience for this engineers? Much of it is going over my head.

It’s not for engineers. If is there anything specifically you want me to explain in a further way? Is it the is it the vector storage thing? Because let let me go over the vector storage thing in in a little bit detail. So so it and it’s not a dumb question, by the way, because, yes, that is a difficult concept to get your head around. But think of it this way.

Our problem is that we’ve is that we’ve got data loss in terms of all of the text that we’re throwing into ChatTPT.

It’s not able to deal with all of that text. It’s only able to do deal with a certain amount. So what this tool does is it solves that problem by giving us an intermediate step by saying, hey. Instead of searching all of that text and dumping it all into ChatTPT, instead, why don’t you throw it into a vector storage database in terms of a very specific type of database? So if you’re using Google Sheets, it’s like that, but imagine it when you’re when you’re searching it, it it just comes back with anything that’s things are similar.

And don’t worry too much about the technical technicalities, how you’re going to do it. I just need you to understand the technology that this technology exists.

So as stuff is stored into a vector storage database, you are then able to make searches that are are similar in terms of from a text perspective and semantic perspective.

I’ll have to watch the webinar again to fully get it. Yeah. Yeah. Hopefully, you do. But if not, again, just reach out to me, and, I can give you some more, detailed explanation. I can send you to some links to, because I’ve I’ve done a video tutorial on this as well. So you can maybe watch that and go into it in a little bit more detail.

But don’t worry too much about the technical implementation of it. Just be just understand the conceptually that what a semantic database is and what it does.

Basically, it’s it’s an in between stage between you and all of that user text so you can extract only the bits of information that are relevant to you at that given moment in time.

And then Google Sheets is basically your data store. So you’ve got maybe you’ve you’ve you’ve sent a survey and all of the survey responses are going into Google Sheets.

And so here’s the tools that I want you to I want you to figure out how and where to use. So this is gonna be a tricky one, I know, because you need to get your head around the concept of that tool, what that tool does, and then where it might fit best into the workflow. And, and what we want to do is to replace the AI’s role in that workflow with a tools role. So the as I said, the tools that we’re going to be using is a Google Sheets, the semantic store, and then a really simple one, just a basic calculator or a formula or something like that. So the task for you is this. For your overall process, identify, which tool can be used where in that process to, to basically offset instead of using AI, what tool can instead be used to get better results.

So was that clear to everybody? Let me know if Could you please share the link to the tools of vector storage and semantic store? I couldn’t trace it online.

What do mean, Minaxi? There’s a link to the tools for vector store. At the because at the moment, all we’re doing is in the in on the canvas whiteboard, we’re just putting post it notes in terms of ideas for where we might use the these tools.

So here’s an example of that.

So a calculator well, instead of asking AI to calculate the percentage of users who have a problem or to count the number of users who have a problem, use a calculator. So so to where to use a vector store? Well, you can use it to fetch negative feedback from the vector store or a feedback about a specific thing from the vector store.

So if it’s, like, something to do with the booking process or booking flow, you can you can search for that, and it would come back with the relevant results.

Dev, sorry. How are we doing for time? Are we we’re at one one hour thirty minutes.

Okay. I’m gonna move on to the next step. But like I said, with the as with the with the other stuff that and the other people who have left already, if you want me to feedback on anything else afterwards, please feel free to ask.

The video will be available for you to view because the last step is going to be to build the workflow. And and and, you know, to build that workflow, think workflows are distinct actions, each which create an output and that pass on output to the next step as an input. And and how how how many people are still in there? Let me see.

Okay. So we’ve got a few people. Hang on.

So so yeah. And in terms of I want to introduce you to the concept of AI agents. So so far, we’ve been talking about decisions and humans making decisions to move to the next stage and stuff like that. So an AI agent, and this is something that we’ve heard a lot about, but to summarize and to sum it all up, an AI agent is basically what happens if we give workflows a brain. And if we just said, here’s here’s your task, here’s your goal, here are the tools, AI, you go and and meet that goal.

And think of it in this way. Like, when you when you think of it like a a map that you wanna get from a to b with a workflow, you are giving the, a workflow very specific directions. So with a workflow, you say, go up that road, go right, go to the take the first left, and then go up, and then you’ll get to, b in the corner. So workflow is very specific steps.

Whereas, up where what happens though if with that workflow, there’s a roadblock in place. So when the roadblock in place, well, there, you’re stuck, and you need human intervention to get you around that work block or of that that roadblock. Sorry.

Whereas an AI agent, you’re just saying, get to b, please, and here’s the map, and you’ll figure it out. Now it will figure out, the best it might figure out the best way to go, but the key thing with an agent is that it may not use the same route twice, and that predictability is what we lose with an agent. But what you gain is a certain amount of flexibility in terms of being able to figure out its way to be.

So in terms of, like, to to to give you a difference or compare workflows versus agents, workflows are predefined static processes. They’re repeatable, predictable, and linear, but they may need human supervision for some steps. Whereas agents for tasks, are for tasks requiring dynamic decisions. They’re nonlinear and adaptable only to a point though, but can hallucinate. So we need both. So where can we use agents? Well, we can use agents using we can apply agents rather using these simple rules.

So we the and the rules are this. We we we have to give agents a very simple task, and we get them to make very simple decisions so they can they’re only allowed to make simple decisions.

Give them limit the number of tools they can use.

Give them few ways to use the tools they can use because remember, AIs are dumb. So if you give them too many tools, they’ll get confused. If you give them a tool and too many ways to use the tools, they’ll get confused. So you want to limit the number of tools and limit the number of ways they can use the tools, and we only want to use them where the the the risk is low for that task.

So which leads me to the homework, and this is where I’m gonna leave you guys.

Based on everything that you know and you’ve learned today, think about where you can use agents. And if you want me to review it, hit me up on that link. And remember, with agents, follow those rules. And where you currently got a human in the step, think what is a good task to give an agent that’s low risk and that follows those rules exactly.

So to wrap up then, what we’ve learned today, we’ve learned how to break your processes into smaller steps.

We’ve learned how to make room for AI.

We’ve learned how where not to make room for AI and how to get better results from AI. And all along, we’re sowing the seeds to think in small steps, which means we’re we’re able to use small models in all of those steps in instead of the larger models and sort of larger big prompts that we’re fee for throwing in. And all of this will build towards our understanding and key concepts to build automated workflows in n a ten. And I showed you a little bit of that today in terms of the definition of the outputs.

So defining those outputs is a re is a key critical concept in n a ten that you need to know. The other thing that you need to know is, understanding what tools and, connections you have available. So the vocabulary of tools that you have just needs to expand a little bit. So so, yeah, like I said, any questions, any homework related queries, or any question queries about this, please get in touch.

And, yeah, here’s my LinkedIn so you can message me.

Thank you so much, Iqbal. That was indeed, you know, the headliner that we deserved for convex.

In the spirit of experimentation, improve, iterate, and be better next time around. Thank you so much for taking part in Convex twenty twenty five day one. Two another days to go, and, yeah, always be testing.

Thank you. Thanks, everyone.