Competing on Experience ft. Air New Zealand

NOTE: This is a raw transcript and contains grammatical errors. The curated transcript will be uploaded soon.

Hello. Welcome to Convex twenty twenty five. What if your next big win came from the daring to outdo every competitor, not on price, but on experience?

Nasiri Reed’s philosophy of curiosity over convention has unlocked more than one hundred million dollars in growth and reshaped the way airlines, e commerce teams approach customers and innovation. In this session, Racie takes us behind the scenes at Air New Zealand, revealing how a relentless drive to experiment transforms more than the conversion rates.

It rewrites the rules of delighting customers, inspiring teams, and delivering breakthrough results.

Tracy, welcome to Convex twenty twenty five.

Hi. Thank you for having me.

Awesome. So when I was researching about your profile, I found a very insightful nugget, which was around curiosity and how curiosity is your compass. So what personal habits or mindset has helped you turn experimentation into such a big catalyst for change in size and scale of your organization?

Yes, yeah. And I think curiosity has always been my compass.

And no matter the size of the organization.

I know today I’m speaking on my experience at E New Zealand but I’ve worked across startups, SMEs and enterprise companies and clients.

And it is really what I’ve found has made the difference in both the the induction of the programs, and in other CRO practitioners that I come across.

Even Recrucias will call me asking if I know any CRO specialists and we’re not easy to come by, there’s not a lot of us. And being in New Zealand, if we are good, normally Australia will purchase.

So I always say it’s curiosity as a mindset. Anything else can be taught if you’re curious about the customer, about learning.

And CRO practitioners have to be extremely T shaped, right?

You need to be sort of deep in one discipline but have curiosity across many others.

And when you have that, it is much easier to learn every other craft required for CRO.

My curiosity is really deep set in human behaviour.

That sort of working with research around our limbic system and how we make decisions without really knowing that we’re making them.

But also how our experiences do influence those decisions as well.

So that allows me to empathise with both the travellers or customer, and internal stakeholders. And I think both of those are really important to really drive an experimentation program. So it’s what keeps me asking questions and keeps me learning.

That’s such an important point. Empathy, curiosity are really the catalyst for ideation, which is really the important job of a CRO anywhere. And it’s been an incredible journey at New Zealand and across your career to get a peak of. But you’ve also driven some of really large scale initiatives. I read there’s a one hundred million dollars incremental growth that you were able to deliver through your initiatives and experimentation, what’s the key to connecting experimentation to both commercial and customer impact?

When you I think a lot of the time in CRO, we’re looking for friction points, right? If you can discover friction points with the product, the experience or the e commerce journey, that’s just one step. But then how you solve it is really where the magic is.

You may come up with an insight and then there’s probably multiple hypotheses that you could have for how to solve that inside of that friction point. But unless you truly put yourself into the customer’s shoes and dig through them and understand all of the reasons why that might be happening. You can’t actually solve it. You might sort of sticky tape it or plaster it. But yeah, you have to deeply understand. So even when we might find a friction point in one point part of flight booking, once you’ve really understood what’s going on, you might realize that that’s actually way further out the funnel that you’ve introduced the friction.

So yeah, to understand and put yourself in the shoes of the customer and really have empathy for why they might be making decisions, that’s really needed to be able to drive revenue growth coupled with customer satisfaction.

I completely agree. And in fact, all of these methods would remain first principles within the SPF CRO for as long as I can imagine. But there’s now a new entrant called AI. And you have been pioneering AI enabled projects at Air New Zealand. So how are you imagining AI customer experience experimentation, maybe also improve operational agility and other facets of your job?

Yes. AI is a large topic and is really transforming how we do things and pretty much our generation as a whole. It’s pivotal.

At E New Zealand have partnered with Opening Eyes, so that is a very strategic move. And we’ve enabled enterprise level GBTs across the entire organisation. So not just a few specialists, but everybody has it and it allows us to all experiment safely. So not just AB testing but developing LLMs and building internal and external tools.

And even just being able to be better at our own jobs.

So I think the decision that really democratized AI at Air New Zealand was when our hedge of data and AI, now we’ve renamed ourselves as well.

He was asked who should have access to DBT.

And he said everybody. So he likened it to having email.

You’re not gonna sit down and figure out who should and shouldn’t have email. Everybody needs it to do their job. And in the same way, everybody, every New Zealand has access to enterprise GBT. We’ve looked at other AI platforms as well and other tools.

That go through various guardrails.

And we figure out if they work for us. So again, it’s an experimentation culture, do they work for us? Do they add value or not?

But I think where it has really helped specifically in CRO is the process of CRO is somewhat repeatable, right? You’ve got process and you follow it really for each experiment.

And whenever you do something more than once a week, or even just once a week, you should really have a custom duty for it.

And that applies for pretty much anything in life.

I’ve been in my own life for shopping lists, looking after children, but in CRO, we built out this specific CHL Custom DBTs.

So I leveraged two others that had been built by both the Insights team and the UX team. The Insights team had built a GBT with some really deep research they’ve done on our customers, demographics, where they’re likely to travel, frequency of travel, the value to the company, and the market size as a whole.

And it was in that they put really strict guidelines of you cannot access the internet. You can only respond with the data that’s actually here.

UX created another one, which has our customer personas. So not in place of, but probably prior to user testing. You can now push designs or insights through this DBT. And it will respond from a multiple variety of custom viewpoints. So you can kind of stress test those ideas beforehand.

So I created a custom DVT which I put in our set of heuristics, so and these are the ten heuristics which ring true across pretty much any digital touchpoint.

Personalized them for our brand and then overlaid them with these two other DVTs. And that has allowed us to really become it’s like having a multidisciplinary team at your hands.

You can go from a fraction point or an insight to probably three to five different hypotheses and design implementations within minutes instead of what would have taken maybe weeks.

Of course, you always still have to put your human lens over all of that and your knowledge.

But that is just sort of one way that we’re really using AI in CRO at the moment.

That’s such an amazing view and your entire point around how AI has enabled or reduced the time to action and create all of these ideas, experimenting across tooling, across your stack to try out various options.

It’s a really exciting phase within CRO and how things are changing. But CRO today also means more than running AB tests. So how do you embed experimentation frameworks that shift company culture, maybe shape up smarter decision making?

Yes, so to, I guess, have a sustainable experimentation culture.

It’s not just about the frameworks, it’s embedding culture and curiosity across the business. So right now, it’s probably firstly thinking about at what levels you need to influence within the business. So if you can manage up to your senior leadership who are the endorsers, and get them excited. But equally, manage across to product managers and developers because they’re the enablers.

And when you’re able to excite at those two levels, everything else follows. And then you can kind of just feel the flywheel is starting to turn.

Yeah, so the other probably key thing I’d have there is embedding experimentation into existing frameworks. So I know sometimes CRO might be something that you do on the side or experimentation might be something that you do on the side.

But we really take an experimentation first principle. So I think being a smaller airline as well, and in New Zealand, New Zealanders have the reputation for experimenting.

We can get away with a little bit more. We can put some things out there, test it and see how it goes. And we are allowed to fail.

But it’s better to fail quickly before you’ve built out the entire thing than to sink a lot of time and costs into building something that doesn’t work.

Yeah, absolutely.

And coming on, that’s a perfect segue in fact, onto my next question, which is every innovator faces some setbacks and failure, as you mentioned rightly, is not entirely bad, it’s a really good learning point. So can you share a time when an experiment didn’t go as planned and how you turned out or how it was counterintuitive in that sense?

Yes.

Failure, so when we are running tests or have failures in other areas of experimentation, we try and make a purpose of sharing those back to the business as well because those insights often guide you more than just do when, right? When something doesn’t work, you now know, okay, we went left, now we gotta go right. So one experiment that I did, it was probably about eight years ago, was helping to test the new seat select interface into flight booking.

It’s a very complex interface. There’s a lot that goes on with seat select, But it was a commercial success. You’re waiting for the budge.

We saw that paid seat uptake skyrocketed.

But unfortunately, very soon after we saw quite a large amount of customer feedback coming through in negative CSAT. So people didn’t understand why they had to pay for a seat and were they gonna be left standing on the plane? Like what was this?

So that experience really taught me that conversion without customer understanding isn’t really a success.

Then did you roll back the experiment? What were the next steps there?

No. So at that point, eight years ago, our Voice of Customer program was in its infancy.

I’ve actually worked at Air New Zealand twice, and a lot of that’s not uncommon in Air New Zealand, probably anywhere within travel. You leave, do some other things and realise how exciting travel is.

So that was done before I left and in the time that I wasn’t in New Zealand, I had to answer to many friends and family and other people I worked with about why you had to pay for seats. Was talking for New Zealanders, I quite obsessed with in New Zealand and seats.

So luckily when I came back to Air New Zealand, we were able to iterate on that. But it kept me awake at night. Really did.

Absolutely.

And so competing on experience and not just on price is a rallying cry for your teams and has been a theme across your role too. So what are those motivating tactics or memorable experiments for that matter that drove unexpected value for customers or the business?

Yeah, so I think with airlines, they are actually a low margin business. So whether you’re a low cost carrier or a premium airline, imagine small.

So it really is a competitive edge to compete on experience versus price. Why is it equally important? There need to be fear that experience is a lever that we have more control over.

So I think an example of that is probably the following on from that seat selection. There’s just so many insights around that because of how much people care about where they sit and that experience of the seat.

So what we actually did with that experiment is we just put in some messaging to make it obvious that you can pay for a seat or we will allocate one for you at check-in.

So that is when we tested that, we did see paid seat revenue dip. But equally we saw flat booking conversion, therefore that revenue increase and customer sees that came back up as well. So that’s been able to drive sustainable revenue, offset the paid seating, and that got rolled out to production.

But what it has done is change our perception of revenue. So we would now coin that as poisonous revenue.

And so while we’re looking for fraction points, we’re also looking for poisonous revenue.

And our sort of largest experiments that both have revenue impact and customer CSAT is when we can reduce that poisonous revenue and offset it in a positive experience or with a positive revenue stream.

I agree. And oftentimes, it’s such a tough call to actually choose between experience and a top line of a business. So that’s vital insight would be into how you have labeled it. But thinking back, is there an experimental project that taught you the most about blending psychology, data, design, and real world impact?

I mean, I should come up with a new example and we’ve got plenty of experiments, but again, that’s the issue.

Sort of, you need to look at that experience right through the journey and it is what really blends psychology, data and design because complex to convey a seat within a digital experience.

But then the psychology behind it is really interesting.

Customers have a real emotional attachment to them.

So it can be signaling a status. So if they’re a frequent flyer, you’re sitting among your peers at the front and in the seat that you’ve earned, as well as the ability to get off the plane quickly.

But then for families, if the mom’s booking, she wants to know that everyone is sitting together.

But the cost of it to pay to sit together might actually be the canvas when the flight becomes too expensive.

And therefore, actually throw off all flight booking, they might choose not to fly.

And then we’ve got other complex products like SkyCouch and soon SkyNest. So the psychology around seating and the amount of data that we have on it is really interesting and probably an area that we’re gonna continue deep diving into for a while.

Now that I reflect back on my whole journey as a flyer, I had to kind of relate to the point that oftentimes the placement of the seed has led me to reconsider an airline because I want a specific aisle side or the window seat and that might impact my purchase journey. So it’s really incredible to learn about one small experience which has become like a really missed out part from a customer’s lens. But when you’re curating that experience, it takes incredible amount of insight, empathy, as you mentioned rightly, and level of data and design. So that’s really good to know.

I think also when you think about travel, it’s a very emotional thing, like the catalyst for travel is either vacation or you need to be somewhere, a business opportunity. And seat on the plane that right, like where you fit sort of within your journey or society and it’s the part of the experience that you can actually control the most.

Absolutely. So as reading somewhere, it’s about the virtues that we want to signal to the outside world that really impacts our decisions. And I agree there could be so many occasions in which travel is special. It can be first time after your wedding or about your first job that you’re traveling to a new location.

So having a smooth journey in kicking off that occasion is really important. And I completely now would be looking differently when it comes to season selection or when we book a flight What unique challenges do you see in creating seamless digital experiences for such a different and wide set of consumer context? So any insights there?

Yes, so probably one of the unique challenges that airlines face is unlike traditional e commerce, we can’t restock inventories.

Once a seat is sold, it’s sold. We can’t get more of them. We can buy more planes, but it takes a while.

So that, I guess in the contributing CRO revenue can be a bit difficult, like what’s that seat going to sell anyway? But we’ve framed that as revenue protection because we know that it does make a difference.

From a customer perspective, there are lots of challenges. So the day of travel itself can be very anxiety driven for lost people.

Any time humans have to do something, there’s a number of events that have to happen in chronological order that creates stress and anxiety within us.

You think about like getting out in the door in the morning, that can be stressful because there’s a lot of things that have to happen before you get to work.

And so we’re quite often looking for ways that we can reduce that anxiety in the day of travel.

Are these steps that we can remove?

And in natural or known experience we can introduce.

Because if you can reduce that anxiety and you increase customer satisfaction, and not only hopefully customers will choose us but look, choose to travel more as well so it wouldn’t fall.

I agree, there’s so much anxiety, not anxiety about the travel person but what it is leading to and you know there are a lot of occasions which might change the way I’m perceiving travel.

And having a optimal calm experience is equally important within this space.

That’s really insightful.

So coming on to any automation plans on building on to this experience that can be enabled maybe through AI? And how do you plan to balance analytical rigor and creativity in CRO programs if AI enables that experience? So that’s something that we’d like to know.

Okay, so I mean, I don’t like traditional e commerce where they probably favor moving fast and big.

Airlines have to maintain precision and safety. So there is always that balance there for us, especially when it comes to the day of travel. Safety is number one above anything else.

But there is a lot of ways that we can automate.

And then, so we are automating cancellations and rebookings.

No one ever wants a flight canceled, weather is out of our controls and we’ll always make the decision to not fly if it’s not safe.

But when you do get a cancellation, it is always disappointing or stressful as well. But you actually have to be somewhere at a particular time.

So that we’re just introducing new automation in that area which will rebook customers immediately. That gives them the option to accept that rebooking or make an alternative.

Those are the types of areas where AI is probably gonna make the most amount of difference.

But it has to be precision, as I said before. So probably the area that we’re working in now is getting our data and our LLMs and experimenting in those areas so that we can have the confidence to actually apply AI in those types of situations.

I think there are outer layers which can be automated but as you rightly mentioned, it would be about precision and uncompromisable you know, ways to move forward without, you know, actually impacting any of these levers like safety. So that’s a that’s a good point to know. And on a personal capacity, Tracy, looking ahead, what skills or mindset would be most critical in this next generation of CRO leaders, digital optimizers who will be working with AI, would be working in a different context altogether than twenty ten and twenty fifteen. Any tips there?

Yeah, I think, again, I’m just gonna come back to be curious. So I think as you said earlier, it’s a really exciting time.

Everything’s changing. And so if you remain open to learning and actually get hands on, then you’re gonna change and learn at the same speed as everybody else.

It’s a curve that is increasing and we don’t really know where that curve’s going. We can even speak to OpenAI and they might have a direction that they’re going to. But they can’t tell you when something will be available because it’s not until those LLMs are ready.

So we all, you have to be nimble frankly.

And have fun with it.

So the more you’re allowed to experiment and making sure that you have the right guardrails in place so you can be confident in your experimentation, I think that’s what’s really going to have the CERA practitioners stand apart.

Yeah, you’ll be able to iterate a lot quicker.

In fact, the broader takeaway that seems to come out is to really love what you do and be really passionate about it and be constantly testing with your own skill set, always be experimenting with other ways and methods which can add on to the process and maybe improve thereon. So that’s a great way to approach careers and Hankins knowledge.

Why I’m really just so how I how I sort of see AI in the current is that we’ve got a whole lot of tools available to us and it’s like a a train with some rockets bolted onto it. And know, the train’s moving along and that’s you with your processes and but the train can really go faster when there’s rockets on it. But within the next sort of one to probably three years, everything’s gonna change. So the process, probably the chills and everything else will stick change.

And so we’ll probably just have rockets without a train.

And that is why I say be curious and lean in because what we’re talking about this year, this conference will probably be a completely different conversation next year.

I agree.

In, you know, Convicts twenty twenty three, I remember AI was a back burner, not the patent center.

Back to two years, we’re now discussing AI as a major leeward for scaling, for intelligence, for change.

So things will definitely change, that’s for certain, and we’ll have to just be on a train with the with the right That’s okay.

So that you adapt to it. But that’s a great view, Tracy.

And maybe just as a parting thought, any reading recommendation, any voices in the industry that you are following, any set of resources that our audience should refer to in order to keep up with this ever evolving, ever changing space?

Good question. I think there’s probably three resources that I’ve used or continue to go back to.

One’s actually a book that shaped my thinking. And it’s nothing to do with CRO data technology, it’s about the human psychology.

So it’s a book called The Body Keeps Score by Bessel van der Kolk, I think is the name, a psychologist. He was actually researching trauma and the impact that it has on our brain, our limbic systems and our decisions.

And reading that actually helps you have empathy.

So until you can truly understand why we act the way that we do, you can actually have empathy for what we, the result we get.

And that is something I go back to, it’s a hard read, it’s a really good one but I’ll go back to it every now and then and read parts of it.

I signed up for Masterclass and I think that is great because there’s just so many different exhibits and different subject matters available to you.

And yeah, there’s some great AI classes on them, a lot around data and then a lot around human psychology and leadership as well.

Those are all great bite sized learnings that can be done in the car.

And then CXL, BWO and the CRO leaders are really great in those core craft skills. How do you actually apply CRO methodologies?

And as that shapes and changes with AI, they’re kind of the thought leaders in that space.

So yeah, think those are the three sort of areas that I lean on to continue that.

All our incredible reads and what obviously thank you for mentioning VWO there. But thank you for allowing us this opportunity, Tracy, for giving us a peek into what experimentation really looks like at scale, how it’s changing with AI, and how, you know, experience really trumps everything else. So thank you so much for this opportunity.

Wonderful. Thank you for having me. It’s been enjoyable.

Awesome.