Agency owners need to manage AI like employees

The following is a computer-generated transcript. Please listen to the audio to confirm accuracy.

Chip Griffin: Hello, and welcome to another episode of the Agency Leadership Podcast. I’m Chip Griffin.

Gini Dietrich: And I’m Gini Dietrich.

Chip Griffin: And Gini, I think we should just walk away and let AI handle this entire episode, and I’m sure it will all be fine. We don’t-

Gini Dietrich: I’m sure it will.

Chip Griffin: We just need to say- Yeah … give us an episode, we can move on.

Gini Dietrich: I like it. Let’s do it.

Chip Griffin: Publish it, and it’ll be great.

Gini Dietrich: Yep. I like it. Let’s do it.

Chip Griffin: Actually, it could be an interesting exercise at some point, just to, just to-

Gini Dietrich: It could be, actually, yeah. Maybe we should do it …

Chip Griffin: just, just, just, just tell Claude- Yes. Yep … “Hey, you’ve got all the transcripts, you’ve seen all of the episodes, just create an imaginary episode from start to finish.”

Gini Dietrich: Actually, we should do that.

Chip Griffin: That, I, um, I’m-

Gini Dietrich: Not today, but in the future.

Chip Griffin: No, not today. Not today. We should, we should do that. But you know what? We will not just publish it without looking at it.

Gini Dietrich: Right.

Chip Griffin: And that’s what we wanna talk to you all about today. Oh. Because you all know that Gini and I are huge champions of AI.

We strongly believe that it is, it is valuable, it is essential even for the success of agencies moving forward.

Gini Dietrich: Mm-hmm.

Chip Griffin: However-

Gini Dietrich: Mm-hmm …

Chip Griffin: you still need the human in the loop. You still need to use some judgment. And you need to figure out what you’re putting out actually makes sense, whether that’s to an internal audience, whether you’re digesting it yourself to make decisions, whether you’re giving it to a client or publishing it.

Whatever it is, you need to be paying attention to it. And I have gotten one too many bits of material sent to me in recent weeks that has had no human intelligence applied to it whatsoever.

Gini Dietrich: Mm-hmm.

Chip Griffin: Not in the preparation of it, not in the editing of it. I guess in the delivery, the hitting of send.

Gini Dietrich: Maybe. Maybe AI did that too. I mean-

Chip Griffin: I don’t, I don’t- It will … I don’t think so. I mean, that’s- That’s a st- generally speaking, the people who are at that level where you can actually send it have at least given it some of the context in advance to, to help guide it. But there is this whole concept out there of one-shotting things with AI.

Uh-huh. Uh-huh. And you see all these people talking about how, “I one-shotted this website. I one-shotted this landing page.” Mm-hmm. First of all, in many cases they didn’t, because when you see it and the result is actually good, chances are they built up a whole lot of context within their own personal AI in advance.

Right. So it’s- Right … is it one-shotted? Sort of, but it’s based on a whole pile of information. That’s like saying that, that you one-shotted an article on the PESO model. Okay, sure, you sat down and wrote it all at once, and it’s great, but that’s because of all of the years you spent developing it.

Gini Dietrich: Yeah, yeah. Yeah, yeah, yeah.

Chip Griffin: Yeah. And, and we’re just, we, we need to, as, as excited as we are about AI, we need to figure out how to balance in the human judgment piece.

Gini Dietrich: Yeah, I mean, we talked last week or the week before, recently, about critical thinking skills and judgment, and those are the things that we should be interviewing for and hiring for, right?

Well, the same thing goes for you and your use in AI. It’s great. As you said, big, big fan. I use it every single day. I’m not afraid to admit that I use it. I do use it, but I use it as a thinking partner, not as a create this draft for me and then I send it off. Like, that’s just not how I use it, and that, that’s what you have to be thinking about, is, like, how do you say to it, “Okay- This is good, this was not, you didn’t cite this correctly.

Like, you have to, you have to go back and forth with it. To the point, I do that so much to the point that Claude said to me the other day, literally said, “If you’re going to continue making changes to this deck, please do it all at once so I don’t have to keep redoing it.”

Chip Griffin: Wow.

Gini Dietrich: I was like, “I’m sorry, you’re a robot, and you’ll do it every time.”

Yeah.

Chip Griffin: Wow.

Gini Dietrich: Yeah.

Chip Griffin: I, I- Mm-hmm … I can’t say I’ve had that experience myself yet.

Gini Dietrich: Yeah, I had- So- Just the other day. Happened just the other day. Yep. So all right, fine. So I went through the whole thing and I wrote down all of my notes and I said, “Here are all the things I want you to change, with the caveat that when you change this, you may not do some things correctly, and I may have you redo it again, but here it is.”

And so you, you go back and forth, and it’s totally fine to have it make the updates and have it do all those things for you, but you have to use it, and review what it’s giving you because you can’t just, to your point, one-shot it and send it off. You can’t.

Chip Griffin: Yeah, and look, I am perfectly happy for you to use the tool itself to do all of the work, and then you just, you know, send that final deliverable on to whoever, whoever you’re sharing it with.

However, it needs to have that critical bit of thinking, and you need to give it the feedback just like you would give it to an employee so that it continues to refine and develop it. And I mean, look, sometimes, sometimes the one shot does work out. I’m not gonna sit here and say it never happens because I’ve had it happen too.

But usually because I’ve given it that context in advance. And I think you really need to think about this throughout the entire life cycle of the information that you’re sharing with your AI tools in order to get the best quality. You need to give it the background information so that it’s not necessarily as smart as you are, but it has a large chunk of the context and knowledge and experience that you have fed into it.

And it takes time to do that. Yeah. I mean, I’ve been doing it for a couple of years now and it’s still not where- Yep … it needs to be. Yeah. I still am constantly giving it more material, and there’s all sorts of stuff from my pre-digital days that it’s much more difficult to give it. I give it to it by having it interview me, which I’ve talked about, I think, here before.

And so I will, you know, give it sort of an oral history of things that I did in the ’80s and ’90s and early 2000s before I had files that I could feed it. And, so that’s all helpful, but you’ve got to give it that, and then you’ve got to read whatever it’s giving you or watch it if it’s a video or what…

Just you cannot just run with it. You need to actually look at it just like you would look at something from an employee and give feedback to get it where it is, is ideally should be, and not just throw it out into the ether and say, “Good luck, let’s have at it.

Gini Dietrich: Yeah, absolutely. I think that’s exactly right.

And what I kind of tend to think about it as is it’s a really smart intern. So you would not just take what the intern creates the first shot and send it off. You wouldn’t. You would look at it, you would say, “Okay,” you would provide feedback, and you would go back and forth so that you can coach the intern and you can train them and all the things, right?

It’s the same kind of thing. You have to train it, you have to teach it, otherwise it’s gonna keep giving you the same slop that it, that’s giving to everybody else, and it’s not gonna be differentiated at all. So to your point, like, we- I went through, we do planning in six-month increments, so at the end of the first half of the year as we did our plan for the second half of the year, I went through and updated all of our material markdown documents that we put into ChatGPT and Claude so that it learns, okay, this is what they did in the first half of the year, these are their goals for the second half of the year, so that, especially for me when I use my co-CEO, I’m like, “Okay, here are our objectives and key results.

Here where, here’s where we are against them. Help me, you know, update them and keep them going.” And it does all of that, but I had to give it all of that context, and I have to make sure that it’s correct. Sometimes it’ll say things that are four weeks old even though it has context from last week’s leadership team meeting.

No, you missed that. Like, you have to be able to review what it’s giving you and apply your critical thinking skills to it.

Chip Griffin: Absolutely. And, I think, you know, one of the things that I’ve done with my own AI operating system is that I’ve developed a critic pass. So anything that I’m going to share with, with anyone else, or publish, or that’s gonna have a material impact on, you know, some software or tool that I’m building, it runs a critic pass to assess it and make the machine judgment initially on are there gaps here, are there holes here, are there contradictions here, is it pulling on the right data?

And it almost always will find things in that pass. That doesn’t mean that I still don’t have things to find myself, but at least it’s now, it’s improving it before it even comes to me, so sort of like having, you know, the middle manager review the intern’s work.

Although I’d, I’d push back. I think AI has moved well beyond the intern stage.

Gini Dietrich: Yeah. I would say that too.

Chip Griffin: And I, I think it’s more… but particularly, the more context and guidance that you give it, the more it becomes a, a middle-level employee. Not even a junior anymore. It really can, if you give it the right information and such, I mean you almost think about it as, as yourself five or 10 years ago. And I think that if you, if you do that, you can get more out of it. But again, it does require that, that back and forth, that give and take. You need to provide the feedback so that it continues to improve, because if you’re, you know, let’s say that you’re, you know, creating an outline for a podcast episode.

If you do it with it every week and you show the final product, it will continue to get better. And so the feedback you give in week one is not the feedback that you give in week 10 or 20. Yep. Yep. If it’s working correctly. If it’s not, then you need to look at your process and change it.

And honestly, what I usually do there is I’ll just say, “Hey, we seem to be having this problem every week, Claude. Can you help me figure out why and how do we fix it?” And it is remarkably good at helping you to figure out how to make it better.

Gini Dietrich: So one of the things that we do internally, and I think I’ve talked about this before, is we have a twice weekly what we call Poppy Seeds, and we use Slack for our internal communication.

And so every Tuesday… No. Tuesday and Thursday? Tuesday and Thursday mornings, Poppy pops into Slack, and she has, it does two things. One is sort of like learning and training, and one is we, we discovered that not everybody was reading or listening to or watching my content, and a client would ask them about something that they had read or listened to or watched, and that our team had not.

And so the Thursday one is, “Here’s the content,” and then it asks very specific questions about the content. But the– we’ve just spent the last five or six weeks in Poppy Seeds on this exact thing on from an AI perspective. And right now we’re working on, you have a, you have the ingredients of a great prompt.

You know how to ask it questions. You know how to get the output that you need. Do you know how to ask it questions? And so what we’re, what we’re pushing them on right now is pushing back, rejecting, and refine, and then doing it so that you get something that’s actually valuable. So we have some follow-up questions that we should, that we want them to do, which is what we want them to ask after any AI output.

Why did you weigh that factor over these others? Now argue the opposite idea, which is your critic’s pass, right? And see if it holds up. You gave me the safe answer. Give me the one you would tell a friend. Like, those kinds of things we’re trying to help the team think through from the perspective of actually getting it to being a thinking partner with them.

And I think those are, are ways that you can do that too. Like we, and I think we’re in week five or six. So we started out with, “Here’s how you architect a prompt, a really good prompt.” Right. And then we’ve moved, now we’re getting to the thinking partner piece.

Chip Griffin: I mean, that’s a, that’s a fantastic idea.

And maybe one that I steal as well. You know, which is really the reason we do this show, right? So we can get different ideas for ourselves.

Gini Dietrich: It’s really for us.

Chip Griffin: And, if you all listening happen to get something out of it, that’s great. But, you know… but meanwhile, Gini and I get our social half hour, and then our recording half hour and some ideas out of it, and we get some things off of our chests so that we’re more calm and relaxed for the rest of the week, and we, we take it out on our microphones instead of on our teams or our clients. So anyway, but, I mean, I think you are, you are spot on there, and I think the other thing that we need to remember here is that we can give feedback to the tools much easier than we can give it to humans, right?

We’re always, or many of us at least are like, “Well, you know, do I really, do I really wanna give all of my feedback to Joe because he’s gonna be, and he’s gonna be really sensitive to it, and, you know, now, am I gonna… Is it gonna be counterproductive because now he gets in his head?” The AI doesn’t get into its own head.

Gini Dietrich: Nope.

Chip Griffin: I mean, Nope … it doesn’t have- Nope … feelings to worry about. Nope. I mean, it may- Nope … it may chastise you apparently for giving it too many edits and individual pieces, which I just, I, I’m, I’m just, I am- Yeah … dumbfounded by that. But- Yeah … you, you’ve got this, this room to give it the meaningful feedback, so it really does get closer and closer to what you are actually looking for.

And, you have to be in a position where any time you’re presenting something or publishing something or sharing something that was AI generated, you need to understand how it came together. You need to understand what’s included. One of my personal pet peeves is when I’m in a meeting and someone says, “Oh, I don’t know, you know, Claude generated that or ChatGPT generated that.”

No, that’s, that’s not an acceptable… If you are putting it out there, you have to have- Yep … I mean, that’s, that doesn’t mean that there aren’t gonna be little mistakes. I’m not saying you have to personally fact check every little thing in there. Of course, yeah. There will be mistakes and at that point it is usually better just to, to be transparent and say, “Yeah, you know, I had ChatGPT do it. Uh, apparently it screwed up that link.”

Gini Dietrich: I missed that. Yeah.

Chip Griffin: Right? Yeah. But, it ha- that has to be a minor bit of it. It can’t be the major thrust or the broad conclusion. You need to be endorsing all of that, and you would not endorse it without reading it if it came from an employee. You shouldn’t do that with the AI tools.

Gini Dietrich: Absolutely, 100%. Yeah, it’s just, it’s shocking to me, and I think we all as humans get lazy, and we’re trying to move as fast as we can, and we’re all having to do more with less, all of the things, right? But we have to, have to, have to, have to review what the output is.

Chip Griffin: Yes. And here’s the but, okay?

Gini Dietrich: Oh, okay.

Chip Griffin: The but is- … this is not your savior. The fact that the human has to be in the loop does not spare your business from all that AI is changing. It does not mean, these people who say, “Well, you know, well, but, you know, the judgment is, is your moat,” and moat is one of my least favorite terms, ’cause AI just loves saying moat.

And so judgment, strategy, those things are not your moat. The AI will help you get there, and you have to be part of it, but it doesn’t stop you from improving because you now have access to AI. So you need to be doing more. You need to be doing better. You need to produce enhanced results for yourself and for your clients.

So just because I’m saying that, that you are essential to this process doesn’t mean you can lay back and say, “See, I told you that AI couldn’t do it all. I have to… I am so important that that is going to be the savior of my agency and my business.” It is not. If you rely only on yourself without the help of AI, you are in just as bad a condition as those people who think they can just one-shot and substitute themselves with AI.

Gini Dietrich: Yeah. I do, like, I know I’ve said this before, but I honestly and truly don’t know how I did my job five years ago. I don’t. Because now I can do research, like, we have, there’s a, our accountant presented me a plan, for taxes this year, and I looked at it and I was like this doesn’t make sense to me, but I don’t know why.

And five years ago, I would’ve done some Googling, and I would’ve gone down, like, the IRS rabbit hole, and it wouldn’t have made sense to me, like, all… So I put it into Claude, and I was like, “There’s something about this that doesn’t make sense to me. What is it?” And it was like this, like, immediately, and it told me why, and then I started asking it questions.

And then it was like, “Okay, go back and ask your accountant these seven questions,” and I did, and he gave me the answers, and I was like, “Aha, I get it now. There’s… Okay.” Like, but, five years ago, I would have never gotten to that. I would’ve known something was wrong, but I would… And I would have done my research, but the research would have taken me down a path that it would’ve made my head explode because it was, like, all IRS-

Chip Griffin: Right

Gini Dietrich: rules and stuff that I just would’ve been like, “I don’t get it.” And so then I would’ve been like, “This doesn’t make sense to me,” and I would’ve just washed my hands of it, right? Now I know why. I… Now I know why it’s… And I… That would not have happened five years ago.

Chip Griffin: Yeah, I mean, it can, it can really help you to understand more complex things that, that you would you either have done some Googling on or more likely just given up on and just said, “Ah.”

Gini Dietrich: Yeah. “Well-” Right.

Chip Griffin: “… they’re the expert.” Yeah. “So whatever-“

Gini Dietrich: Right. Yeah.

Chip Griffin: “… whatever they said, you know, must be true.” Yeah. And the more you understand it, the more you can ask those critical questions to decide if it’s the right solution for you or not. I think one of the things you always wanna be mindful of, too, with all of these tools is how much you steer, right?

So one of the things you said was that you were skeptical of it, I think is how you presented it to the tool. The tools are much better than they used to be, and they will sometimes, you know, not agree with you. Yeah. But I still generally prefer to just say, “Give me your assessment of this.”

Gini Dietrich: Yeah. Yeah, yeah.

Chip Griffin: And only once I’ve figured out, you know, where the general take is, then revealing to my own tool where my head is at, just to make sure that it’s not, you know, unconsciously biased towards agreeing with me. Same thing I would often do with employees, right? I mean, I think I’ve talked about this before.

Yeah. Sure. Yep. You know, whenever, whenever as the boss or the owner, you’re in a meeting with your team, keep your mouth shut till everybody else has talked. Because as soon as you express your point of view, people are gonna at the minimum pull their punches in all likelihood.

Gini Dietrich: Yes.

Chip Griffin: And so you wanna get…if you want real input, you’ve gotta stay quiet, and so I encourage the same… I mean, it really is just consistent with treating your AI like you would a human employee. Yep. Yep. And if you would, if you would not just, you know, one-shot something with an employee and just say, “Oh, so you’re gonna write this speech for me, and I’ll just get up there and I’ll start giving it without having read it?”

Gini Dietrich: Right! Okay.

Chip Griffin: Now, in fairness, I used to work in politics, and politicians I worked with did that all the time. I would give them a speech and they would. They hadn’t looked at it. They would just get up there and start reading it.

Gini Dietrich: I feel like that’s different though. I feel like that’s different.

Chip Griffin: I, I mean, I personally would not be, I would always wanna, like, read it once, but I, I wrote-

Gini Dietrich: I would, I would too, but yes

Chip Griffin: I wrote so many speeches for members of Congress, and they would just sit up, they would just get up there, and they would just start reading it. And you could usually tell that they hadn’t read it before, because they would stumble over words or, you know, whatever. Because I used to be part of a speech writing group, and we would just create speeches in bulk basically, and some members of Congress would just grab, you know, one from the pile and deliver it on the House floor.

Gini Dietrich: That’s crazy.

Chip Griffin: Yeah.

Gini Dietrich: That’s crazy. Yeah. Wow.

Chip Griffin: So, a lot of them wouldn’t, wouldn’t bother looking at it in advance. They just-

Gini Dietrich: Wow

Chip Griffin: I would want to look at it and put it in my own words.

Gini Dietrich: I would as well. Yes, I would as well.

Chip Griffin: But, there are a lot of members of Congress, at least, at least back in the day.

I can’t-

Gini Dietrich: I’m sure this has not changed …

Chip Griffin I can’t, I can’t speak to today, but back in the day they would do it without, you know. Hopefully they’re not using- I mean, sure the staff probably is doing AI to, to do it now. I mean- I’m

Gini Dietrich: sure, yeah.

Chip Griffin: But-

Gini Dietrich: That’s nuts

Chip Griffin: I liked it when I would sit down and we’d have races to see how fast you could write speeches or op-eds.

Gini Dietrich: By yourself, without AI.

Chip Griffin: By yourself. By yourself- Yeah. Yeah … without, without AI, and off the top of your head with no research. I mean, that was-

Gini Dietrich: Yeah, no research. Great. Yep. Yep.

Chip Griffin: But.

Gini Dietrich: Yeah.

Chip Griffin: Well, I think that, that’s probably a good spot to end on here, because who knows what other rants we’ll go on if we don’t.

But so use, use the tools to your advantage, keep yourself in the loop, and don’t rely on it, your human judgment, as your, the sole savior of your organization. That’s right. It’s all gonna come together and work in cohesion.

Gini Dietrich: Yes, absolutely.

Chip Griffin: So with that, we will draw this episode of the podcast to a close.

I’m Chip Griffin.

Gini Dietrich: I’m Gini Dietrich.

Chip Griffin: And it depends.