We transitioned the agency to operate out of AI. Here's what broke and what we learned

The catalyst was the dreaded email: "We want to scale back our work with Data Story because we're using AI to do more in-house."

Gut sinks to the floor. Knot in stomach. Eek.

So I called up a couple of other clients. "Hey, how are you using AI atm?"

"Well, in the business not much just yet, but personally I'm using it so much I'm thinking we may not need to hire for our next role!"

It was a "holy crap" moment. The disruption was no longer theoretical; AI had finally arrived for our jobs.

Thankfully, we'd been preparing for this. We made an expensive decision to put Hannah, our head of strategy, on building out all of our processes in Claude Code.

We get the entire team to pull down the DS-System from GitHub, start training weekly on using AI, all sharing our learnings, and building the machines as we go.

Going forward, the whole team is working from Claude Code. New process? Build it in Claude as you go! Confused? Ask Claude!

The analytics team flat out connecting MCPs.

Initially, we were riding a wave of efficiency. The team was pumping out work that looked 80% complete in record time.

However, we quickly learned that AI skips vital steps. It makes massive assumptions to achieve that speed, and every single one of those assumptions needs to be scrutinised to make the output production-ready.

The dopamine dried up, and we're left with a tonne of work that's no way customer ready.

We began to see through the hype. Claude was hallucinating details, and the initial "slop" required significant human intervention.

Ironically, our processes were taking longer. Because data accuracy is our non-negotiable priority, manually fact-checking AI "insights" proved more labour-intensive than our traditional manual process.

We were working harder with less tangible output, margins were tightening, and AI subscriptions were climbing into the thousands.

Yet, amid the frustration, we saw glimpses of a breakthrough. The system evolved. Newer models became sharper. Our tool integrations became more seamless.

Rubbish in, rubbish out. We start getting AI huge amounts of context, and the outputs are genuinely good.

This led to a realisation: AI isn't a time-saver; it's a process-shifter. It allows us to move past the administrative bulk and focus on more critical, high-level strategy.

We've started being more strategic in where we leverage AI, ensuring we define a clear problem or job-to-be-done before automating.

That client who emailed asking to pull back work? They asked for some help implementing what Claude was suggesting for their ad accounts, and it turned out to be 70% bogus. Talking about campaigns that had not been running for months. Suggestions that missed the context of the business. A few good suggestions, sure. But dangerous stuff. If AI was followed at this scale it would have cost millions to businesses.

I don't want to be a futurist, or spend weeks thinking metaphorically about the way work is changing. But here are a few things we're learning.

1. If you try to do something quickly with AI, the output is typically average to poor

If you're asking AI to do something you're not an expert in, you risk following generic information without knowing the consequences of the information you're leaning on.

AI is excellent at making people feel self-assured. Perhaps this is why we get this sense of fatigue when using it a lot. Your brain is trying to decipher the truth from the clever wording that sounds good at first read.

I've been burned from the dopamine rush of getting the quick progress too many times now. I've got to not skip the process! Yes, do the process in AI for speed, but don't skip it. I'm now thinking "if I want to skip the process, is the work really high-leverage enough that I should be working on it?" Perhaps I need to reprioritise where I'm focusing.

Building systems and being explicit about the task you're completing is critical. Skipping steps does not pay off. We learnt this the painful way.

2. You can get answers without effort, but you can't get understanding

Some types of work require understanding. AI cannot give you understanding. I've been observing a nuance where certain work, even if the output is good, if you have not bought into what AI is producing you're creating noise. This subtlety, and deleting things even if they're accurate and seemingly helpful, is still a form of slop. Make every word count. Know your stuff with enough conviction you can speak to it, or don't share it.

3. The closer you are to code, the better the work, and the more valuable

Using Claude Code to produce code has been where we've seen the most value. Coding webpages, writing SQL, filters and segments, writing custom scripts, generating rubrics, certain types of data analysis. Using machines to do these jobs is superior, and we're grateful for it.

4. Learn AI architecture to see through AI hype

There's real AI literacy that marketing leaders should learn. Learn how to write skills, give context, and how to use MCPs.

These are the building blocks of working in AI environments. The harnesses (e.g. Codex, Claude Code, Cowork, Hermes, Antigravity, and more to come) are like new kitchens. They're only as good as the Chef (you, driving and orchestrating the work), the ingredients (the context, inputs and data) they work with, and the processes they use (the MCPs and Skills they execute).

Once you understand these three areas, you start to see that most new shiny things in AI are typically a new harness. They're streamlining processes, and the innovation comes through applying them to build something great. Design Thinking will hopefully make a resurgence.

5. Strategy is king, and AI is not a strategy

I've flipped from being gutted that I got so caught up in AI, then realising I'm not sure there was another way to learn so much so quickly. And we're still learning (and always will be!). But it does feel like the world is returning to a new normal. Businesses are no longer waiting around to see how AI will shape their work; they're carefully adopting AI while getting on with what they do.

Strategy is still king. AI allows you to streamline things, but the old Bill Gates line rings true:

The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency.

Bill Gates

Good strategy is now more important than ever as execution gets more commoditised. Where you plan and how you'll win is THE work now. Can you define what you're automating and why? What's the problem you're solving?

6. Set bigger goals if you want to use AI well

The single biggest driver of business success is the leadership team's vision for the business. With a big vision you need to think strategically and you need scale. AI can help you scale.

There are two main strategies in marketing: do more of what's working, and try new things. The first thing you should do is more of what's working.

This is not that easy, as most businesses don't know what exactly is working in their marketing.

And most businesses in NZ and AU don't have big goals. They don't need to seriously innovate or scale outputs. They only need to ride the market they're in.

The danger is that competitors are getting smarter and slowly creating real advantage. This is the same game of business that has always existed, just with some new tools. Do we need to set new goals?

Parting thoughts

I love marketing. I'm grateful for the journey we're on, and to know we're not missing a trick with AI. We're excited to continue to put these new tools to work in strategic ways. We'll continue to focus on strategy, as we've seen this work as the bedrock of leveraging these tools in marketing well.

Genuine thank you to our amazing clients and partners for going on this journey with us, and for putting up with a bit of slop. We know the outcomes and future states are already starting to add value.