B2B marketing and AI
How to measure marketing effectively. Covering metrics, dashboards, attribution, testing, and the analytics frameworks that give leaders confidence in decision-making.
* Now with added AI
Measuring the performance of marketing departments has always been a primary objective for marketing leaders. There have been many models and proposals for how to do this, most of which I’ve used over the years.
But now there is an added dimension – how well is your AEO and GEO optimisation going? What do you need to add to your quarterly deck? The attached is my suggestion, based on work I have done before, but now with a small update for AI.
For as long as I’ve worked in marketing, there’s been one problem I could never quite solve: how to measure the impact of brand work.
We’ve always had solid tools for performance — clicks, conversions, funnel stages. But brand strength? Hand-wavey and unmeasurable. I’ve tried surveys, awareness studies, proxies like search volume. They gave some data, but they were slow, expensive, and rarely told us what people actually thought at the point of decision.
That’s why I find AI search and generative engines so interesting. I believe for the first time, we can open that black box and see the impact of your brand in action.
When I’ve been a CMO, I always felt the tension between what the numbers said and what the market was actually feeling. Performance dashboards looked precise, but they ignored the harder question: what position does our brand hold in people’s minds?
Surveys and brand trackers tried to answer this, but they were blunt instruments. They lagged behind reality and rarely influenced day-to-day decisions.
So “brand impact” remained something we talked about, but couldn’t measure in any meaningful way.
Generative engines don’t just list links. They generate answers — which means they have to synthesize a point of view.
When someone asks an AI tool about your category, the answer is shaped by:
That makes AI outputs a kind of brand mirror. They show what the system thinks your brand (or your category) stands for in real time. As mentioned here, “Brand” can be considered as the priors in a Bayesian model where they represent previous knowledge that we bring when making a product selection. Simply put your prior knowledge that you love Apple products has to be considered when you are searching for a product solution – you are unlikely to buy the new expensive iPhone because of a feature list 🙂
The big difficulty as we all know as marketers is that non-marketers don’t get this. This is partly our fault of course, we are notoriously bad at “Marketing marketing”. And I don’t think words are enough – we need some maths! we need to understand why these things are related at a deeper level. Then, the next time when your boss says “Why are we spending all this money on brand advertising can’t we just get some leads?”, you can link these two things together (brand spend and company performance) much more directly.
So GEO is a new and exciting frontier. But how can we measure its impact?
For the first time, we can ask: what does AI say about us? And we can track how that changes.
That’s measurable brand impact. Not perfect, but visible in a way it’s never been before.
This post is part of a wider project I’m running here on bjrees.com. Over the next few weeks I’ll:
If brand has always been the hardest thing to measure, maybe AI finally gives us a way in.
Why has brand impact been so hard to measure?
Because the tools we had — surveys, awareness studies — were indirect, slow, and disconnected from real buying moments.
How does AI change that?
AI systems generate answers by combining historical knowledge with fresh signals. That gives us a live view of how a brand is represented.
Is this just another form of SEO?
Not really. SEO is about rankings. GEO shapes long-form generative answers, and AEO structures content for short, direct responses. Together, they make brand visibility something we can observe and track.
Most of your time as a marketing leader is spent trying to make decisions with inadequate data. In an ideal world, we would have run an A/B test on everything we wanted to do, looked at the numbers and then made a decision. Which image should we use for our new advert? What message? What tone? Which type of customer are we trying to reach? And 1,000 other things.
A/B testing is one way of approaching this problem. The difficulty is that most marketers will – and should! – already have a view. If I was given the choice between two headlines:
I know which one I would click on, I definitely don’t need to do a test!
But here is a more realistic example. You are trying to sell into a company and you are not sure who makes the decision. Is it the end user? The manager? The person holding the strings?
How on Earth do you do an A/B test for something like this?
You will soon find with a question like this, that you quickly hit the “Everyone has an opinion“ problem. You ask various members of the team and outside your team and everyone gives a different answer. There are a couple of ways out of this situation as I’ve mentioned, but doing an A/B test is generally completely impractical.
So what can you do? The approach that I take now is to use some of the concepts from Bayesian logic to help me make the decision. The key concept is is the idea that every decision you make is a combination of your prior knowledge plus the data that you see. And the real issue with prior knowledge is that everyone comes to the table with their own history.
As an example, if you have been running a marketing team for years, and all you have been doing is numbers-driven digital marketing for B2C businesses – and crucially, you have had success with that, then you are going to start your analysis with that approach in mind – the answer to the question “what should we do next for our marketing?“ will very likely be something around digital marketing strategy. In contrast, if you come to the table from a brand marketing background, then it is likely that your initial opinions will favour this sort of approach. Why? Because this is what you know and there’s a good chance you’ve had success with it at some point in the past.
Crucially, just asking the question “which is right?“ will not get you anywhere! You each have prior knowledge that you are bringing into the process. So what do you do when you start running a campaign and you start to get results, albeit with very low numbers? How do you combine your prior knowledge of what should happen with what is actually happening?
This is where the Bayesian approach can be very useful. I don’t think you need to understand any maths to use this approach, it is about the principles behind it.
I first read about this principle in the book below:
I have recommended this book about six times on this blog, so I am definitely a fan! The key part that is relevant to this blog post I have copied below. I tried to paraphrase it, but then I realised that Sean Carroll’s short explanation is better than anything I could come up with:
“Prior beliefs matter. When we’re trying to understand what is true about the world, everyone enters the game with some initial feeling about what propositions are plausible, and what ones seem relatively unlikely. This isn’t an annoying mistake that we should work to correct; it’s an absolutely necessary part of reasoning in conditions of incomplete information. And when it comes to understanding the fundamental architecture of reality, none of us has complete information.
Prior credences are a starting point for further analysis, and it’s hard to say that any particular priors are “correct” or “incorrect.” There are, needless to say, some useful rules of thumb. Perhaps the most obvious is that simple theories should be given larger priors than complicated ones. That doesn’t mean that simpler theories are always correct; but if a simple theory is wrong, we will learn that by collecting data. As Albert Einstein put it: “The supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience.”
Everyone’s entitled to their own priors, but not to their own likelihoods.“
This might feel like a slightly obscure deviation from the subject matter of this blog (marketing!) but I don’t think it is.
Unlike many other areas, It is very difficult to come up with definitive evidence for why one approach is better than another. This can lead to the back and forth, debate about messaging and other areas.. or worse than that, you can even end up with the HiPPO principle for making decisions (“Highest Paid Person’s Opinion”).
But – If you understand that this is where a lot of people are coming from, that they are making decisions based on their prior experience and not necessarily the facts in front of them – then this makes it much easier to have a rational discussion. There will be a very reasonable logic behind why somebody is arguing for something. Listen to that person, interpret and apply intelligently.
There are always too many things to do running a marketing department. The essence of your job as a marketing leader is making strategic choices about where to double down and where to pause. This is what makes the job both interesting and difficult.
Below I’ve provided a scorecard that I’ve used many times in the past. It comes from the great Gabe Larsen, someone that I’ve followed for years on Linkedin, and I’d urge you to do the same.
This is just a example of his scorecard with some semi random numbers filled in. These are the scores that you give yourself and your department to show where there needs to be improvement and where everything is “just fine”.
An obvious first point – be honest. This is something for internal use only (as I say, the example above is semi-random). If you publish something like this to a wider group then there’s a risk you’re creating a problem for yourself (“I thought you said we were great on social media. It says here we’re hopeless!”). It takes a growth mindset to use this sort of thing really effectively. As always, fix the culture first.
So once you get over the cultural and communication issues, the very next question is “So what? What do I do with this?”. To quote a friend:
Eight out of 10 ideas are bad ideas. So the worst thing you can do is try to do everything. you’ll just burn everybody out. The “OK” thing you can do is to do nothing. At least you’re not burning everybody out. But of course the skill is choosing the right two things to take forward.
What I like about this scorecard approach, is that it gives you the superset of all the things you could do which forces you to consider them. In the example above, the marketing leader has proactively decided to not do any business development work at all. That’s not because she just doesn’t have time or doesn’t know anything about it. It’s a proactive decision that this isn’t something that will move the needle. Crucially, this frees up time to double down on something else. And that’s how you move from being a tactical marketer to a strategic marketer.
Measuring marketing performance is tough. At times I’ve argued that it’s impossible and therefore pointless.
But I don’t think that’s true. As a starting point pretty much any company you work for in a senior marketing role, will expect you to understand the impact of your work (and that of your colleagues). Part of this is looking at flow metrics like “Leads generated this quarter”. But that only tells part of the story, specifically about the flow of business. The other side which is discussed more by the Board than less by marketing departments is the need for balance metrics. As a simple example – you might have one business that’s generating 100 leads per month, but has, already, historically built up a mailing list of 500,000 people (all of whom have given you permission to market). Another company might be generating 100 leads per month but all from scratch and every lead costing money. The former company is more valuable because they have built up a balance of people over time who known them. And that is a very valuable thing.
This makes a significant difference to how you measure marketing performance. Ultimately the goal of the company is likely to be valuation. Part of that valuation is the value of the customer base and/or the awareness of the company. I believe the awareness of a company and its offerings is directly linked to valuation and so, as a marketing department you should be tracking it closely.
Below is a brief PowerPoint covering these different metrics. Any questions, let me know.
Here’s a short presentation on how to calculate return on marketing investment. It’s heavily oversimplified but the principle is simple – do a back of the envelope calculation before you start spending money, particularly if looking at a new system that only promises an uplift (rather than a significant change in performance).
A lot has already been written about how the old marketing funnel model is no longer as relevant in modern B2B organisations as it used to be, and how a flywheel model is more appropriate for how customers really buy (as an old colleague said to me “The person who invented the marketing funnel should be shot!”. I think that’s excessive, but it makes the point…). Here’s how HubSpot describe the alternative approach: https://www.hubspot.com/flywheel.
But rather than just repeat the work of others, I wanted to write an article on a model that I’ve been using for a few years now and which I found very helpful when making practical decisions about marketing activities. Models are all well and good but the reality is we have to make calls every day on where to spend our marketing budget and for this we need a robust model to help guide us in the right direction. Should we spend more on PPC ads? On brand building? On events? What? That’s more importantly, why? You will be asked why you decided to spend all of the budget on activity A rather than B. You need to have answers to those questions.
Here is the short version of my PowerPoint that goes through this model (there is an 80 slide version of this that I’m working on as well). To summarise the context for this:
But to repeat, having some sort of model for how marketing works is crucial. Otherwise how can you make decisions about which levers to pull? About what effect you think you’re having? Of course sales is the end goal here but you need to understand what happens before the customer makes the purchase if you want to influence what happens at the end. And in a B2B process that can take months if not years from initial awareness to signing on the dotted line. Understanding the drivers between these two stages is the key to knowing how are you can influence the process and increasing the chances of hitting the revenue goals.
A final note – all of this is, of course, based on the work of many others and I’ve blatantly taken diagrams and paragraphs from other marketing practitioners. Almost all of the ideas here from other people, all I’ve tried to do is bring it together and find the links and similarities. Any errors are mine.
ROI calculators are a pretty common tool amongst B2B marketers. On the face of it, the logic is simple – show a calculation of how the time saved from subscribing to your product equates to money and how that money is less than the annual subscription cost charged. Then surely the sale should be in the bag – who wouldn’t want to save $$, how could anyone say No?
I think this sort of calculator is useful for a limited purpose – it provides a supporting tool for your advocates in the org to help them convince their bosses to make the purchase. And for that it’s valuable. However, I’d argue it doesn’t help with the core problem – convincing buyers of the real value they’re getting – as it doesn’t help with the core reasons why people actually make purchases like yours.
An ROI calculator usually looks something like:
At this point, you can produce your pen and ask them sign on the dotted line (well, click a button on a Docusign document) and start figuring out your commission.
Why doesn’t this work? There’s some lower level arguments to be made against a calculation like this – do you really believe the figures? Is all of that time genuinely saved or does someone else still need to do some manual work somewhere? Is the org really doing the task that badly today? And of course this is missing the implementation fees and ongoing work needed to keep the automated system working.
But I think there’s something more important, particularly when selling to senior buyers. There’s a dual problem with this calculation – firstly, that to make this saving the company would effectively have to fire someone, and secondly, in today’s environment, hiring and retaining staff is a far bigger headache than saving costs from letting people go. The calculation assumes an environment where the manager currently has too many people working for them, and is being asked to make savings. But this isn’t the reality for almost every manager I’ve spoken to in the last 10 years – the perennial problem managers have is growth and finding talent to fuel that growth. What they’re asking is “How do I find great people? I’m short-staffed, and just can’t hire the people I need. And when I do hire them, it’s such a hot job-market, I lose them unless I keep them happy!”. I can’t remember a single situation ever where someone has bought a product/service from a vendor, then “made the savings” by firing someone – it just doesn’t happen, and so is not believable to a client.
The real pain that clients have is hiring a great team, building and developing that team, then keeping them engaged. By building that great team, they not only get the obvious advantages and pride of running a great organisation, they also get multiple benefits from being able to provide much more value to the rest of the org. I.e. instead of my team being seen as a “Cost centre, to be reduced wherever possible”, it’s seen as “An incredibly value part of the company that’s helping us grow and be successful”. This “Soft ROI” is what senior buyers are really worried about. Here’s how I’d describe the world of the average finance team, from talking to our customers:
This is an example from the world of finance, but of course the same could be said for other teams – though I’d argue to different degrees. The sort of waste I’ve seen in finance teams was ironed out years ago in development, where you see people automatically running 512 cloud-based tests at the push of a (build) button without a manual step in sight – the sort of automation finance teams can only dream of.
How do you present this value, if an ROI calculator isn’t enough? Through great marketing – all great marketing is based on a deep understanding of your customers’ genuine pain points and how you can resolve those pain points. Instead of (or “as well as”) an ROI calculator, I’d be writing content about the pain points above. Show how you understand your customers’ worlds, how you understand that pain and can help solve it. It’s also a question of positioning – if you position your product as “A tool that helps save time”, then you’ll never really resonate with the manager’s pain. Alternatively if you position your product as “A service that supports you transforming your team from a cost centre [to be reduced] into a high-performing and motivated function valued by the rest of the company” – and you can connect the dots between that message and your product – then this is a much stronger way to appeal to the target audience.
As I mention, I think ROI calculators still have their place. Once you’re in the door, and you have an advocate on the inside, then a tool like this can really help him/her make the case to their boss, who might want some numbers to back up the investment. But you need to win hearts and minds first – and that happens by understanding peoples’ real problems, and finding a way to help them solve those problems – ideally with the help of your product of course.
We’re about to go into our quarterly review period at Redgate. We don’t just run QBRs, we also run reviews across all parts of the business. These are a chance to examine the last three months – what worked? What’s going well? What’s not going well and needs fixing? All part of a strong agile process for keeping an eye on what’s really happening, and making adjustments through the year.
But there’s a flaw in these meetings when it comes to marketing. The implication of this process is that the things we did in Q1 have a direct and measurable impact on the outcomes from Q1. I.e. when I’m talking about our Q1 outcomes, the activities that occurred in-quarter are the most relevant for discussion.
But that’s a very partial view of the truth. As all marketers know, marketing is a mix of long, medium and short term activities which, if played right, add up to the outcomes we all seek, such as pipeline for the sales teams.
We’ve just had one of our best quarters for marketing-sourced pipeline in a long time. I can’t say “forever” because we’ve changed how we measure things over the years, but a quarter-to-quarter growth of 20% is something I’m very happy with. I’m not giving absolutes in this chart, for obvious reasons, nor a key to what the colours mean!, but the chart below shows how COVID-19 hit Redgate for marketing sourced pipeline through 2020, then how we’re coming out of the pandemic stronger than ever:
So, great, and I couldn’t be more proud of the incredible marketing team at Redgate for making this happen.
But we’re trying to build a scalable, predictable revenue engine here. It’s not enough to know “What”, we need to know “Why?” – what did we do to achieve this? What can we re-produce? What activities had no impact, that we can cut? This is a Sisyphean task, full of estimates, best guesses, rules of thumb. And I thought back to a poster that used to be on the wall of our HR department (back when we all worked in offices!) stating “Not everything that can be measured is valuable, and not all valuable things can be measured” – which is particularly appropriate here. But we should at least have a stab at trying to know what influences these numbers.
Here’s my go, some based on data, some not. And I’ve given these in reverse “timescale” order – from things we did actually do in-quarter, ranging down to activities that have been in-play for years. And of course it’s almost impossible to weight these things – what was most important? What only made a small difference? Very hard to know..
Obviously I’ve missed a lot of things here – so many great projects over the last 12 months, small and large. And, as I mention, it’s really hard to to figure out “Okay, but which of these were really important?”. Problem with that of course – successful marketing is a complex combination of lots of activities, hopefully orchestrated together to give the result you need. No single activity is sufficient or even necessary for success, but I think you do need most of these things working together for growth.
But to go back to the original problem – how do we present a successful quarter as a set of repeatable, scalable activities? Should we just repeat all these things!? The skill is to choose between either stopping, maintaining or increasing these activities. There are a number of areas which are certainly in the maintain mode – the brand work, experiments with digital spend being just two examples. We’ll keep doing these, but not more. And a couple of things you can’t or shouldn’t “continue” – marketing re-orgs are very expensive, and I certainly hope that we need to do less and less work supporting customers during the pandemic – an option I’d be very happy to lose! But the trick is knowing what to double-down on – something I’ll be working on over the coming weeks.
Why do Marketing and Sales departments need to collaborate? Sure, it’s nice, but beyond people getting on better together, how can it really impact the numbers, the outcomes for the business?
We’ve just spent a month at Redgate improving the collaboration between the two departments and we can see the direct and measurable impact on new opportunity generation. Here’s what we did and what happened. NB: None of this is rocket science, these all seem like really obvious things to do. But it’s quite rare to see such a direct impact on the numbers, so I wanted to share “What we actually did” as it may be useful to others – it can be easy to miss some of the basics.
To start with the obvious, it’s nothing to do with whether you “get on” or not, whether you’re friends. We have great relationships between the marketing and sales departments, we get on incredibly well, and we talk all the time. But that’s not collaboration.
I feel there are three levels of what could be called “collaboration”:
The first two are fine, and communication is great. But the third is what I consider true collaboration – what is our shared goal? What are the (shared) numbers showing? What can we do to fix this, together?
And it’s the third of these that we kicked off at the start of 2021, and has shown direct impact on the outcomes we care about.
Redgate has a good problem (and has had that problem for a while) – too many “leads”. We get around 500 leads a day (a lead being “Someone who expresses an interest in one of our offerings, and gives us some of their details”). Great, what’s the problem? The problem is that salespeople’s time is invaluable and scarce. If we asked Sales to follow up on all 500 leads every day, they would waste an incredible amount of time chasing low quality leads, tyre-kickers, people who will never buy from us and so on.
This is a standard problem in marketing/sales and the solution is some sort of qualification process. There are various models out there (the SiriusDecisions Demand Waterfall being one of the more common frameworks), but we have a pretty simple process – use Marketo to score leads on two perpendicular scales – engagement, and firmographics, then only pass the good Marketing Qualified Leads (MQLs – the Glenngarry leads!) through to Sales. It’s generally around 10% of the total, or 50 a day. Then keep the rest back to be nurtured from within Marketo until they’re ready for prime time.
So far, so easy. But of course the point is that it isn’t easy. When you move from theory into practice, here are some of the problems you hit:
Most of this is operational – needing marketing operations and sales operations teams to work together alongside the rest of their colleagues. And there’s a lot to figure out here. But these are the things we did in January to try and tackle some of these problems. Not everything went smoothly, but enough went well to achieve noticeable differences in the numbers. And everything here was a joint project between various people in Marketing and Sales at different levels.
This is what matters. If you did all of the above, you can give yourself a pat on the back, but it only matters if it made a difference to the numbers. So did it?
Yes it did. I can’t post all of the charts here, but in summary:
Could this just be luck/the market? It could be, but poring through the data for “What actually happened here?” it clearly shows that the new opportunities are coming from the right leads being placed in the hands of the right sales people with the right information at the right time. Sales people feel like they’re getting decent leads from Marketing. Marketing people feel that all their leads are being “maximised”. And most importantly customers are getting the service they expect – help from Redgate when needed, and left alone when that’s what they want.
I’ve been watching the figures every day like a hawk, to wait for things to go wrong, but the increase is very consistent.
Looking back over January, this effort would have failed if it hadn’t been for us taking a collaborative approach. As mentioned at the start, these things aren’t rocket science. So why didn’t we do it all years ago? Well, a number of reasons (having the right people in place for example), but primarily, that taking the more forensic, tougher and collaborative approach was necessary to proceed. We could have tried doing these things in isolation, but it would never have landed as well as it did.
I love the book Rise of the Revenue Marketer. In it Debbie Qaqish describes the need for a change program to move your marketing department from being a cost centre (“We’re not sure what marketing do, but we need them to do the brochures”), to a revenue centre (“They’re responsible for generating a significant proportion of our company’s revenue”). Though the journey is easy to describe, it’s a long and arduous path to take.
We at Redgate have been on this path for a while now, and we’ve made enormous progress, particularly in the last 12 months. But one of the things that slowed us down was holding on to certain beliefs about how to measure marketing performance, how to measure the impact of marketing work – and holding on to those beliefs for too long, when perhaps they just weren’t true. Lots of these ideas came from conferences, blogs, books and make a lot of sense on paper. But when you get to the real world of implementing something, the reality is not always as expected.
Here I’ll go through five “myths” that I found to be unfounded. Of course, these come with big caveats – we’re one specific org, with a specific market, with particular advantages and disadvantages – so all should be taken with a pinch of salt. Still, with that caveat in mind here are my five, starting with the most controversial:
The idea of a marketing attribution model is that you can take every lead, opportunity or sale and somehow work out “What were all of the things we did in marketing that contributed to that outcome, and what value would we give to each of these things?”. For example, I just generated a lead, I could go back and look through the path history of that individual, find that she clicked on a PPC ad, attended an event, did a Google search, interacted with us on Facebook, and so on. I then have some smart “multi-touch” model that assigns value to each of these (maybe the first or last get higher scores? There are lots of alternatives). If you then know the value of a lead (let’s say, $10), you can work out the Return on Marketing Investment (ROMI) for each activity by comparing the “value” (e.g. maybe $3 for the PPC click), against the spend.
But, I think this is baloney. This is a classic example where – just because you can do the maths, doesn’t mean to say the results are accurate or useful. The model is flawed for at least the following two reasons:
Okay, maybe it’s not all baloney – but the approach is, I believe, significantly flawed. Nevertheless, there are some things that can be measured – which brings me to myth 2…
Not sure this is controversial actually. To quote Seth Godin:
The approach here is as simple as it is difficult: If you’re buying direct marketing ads, measure everything. Compute how much it costs you to earn attention, to get a click, to turn that attention into an order. Direct marketing is action marketing, and if you’re not able to measure it, it doesn’t count.
If you’re buying brand marketing ads, be patient. Refuse to measure. Engage with the culture. Focus, by all means, but mostly, be consistent and patient. If you can’t afford to be consistent and patient, don’t pay for brand marketing ads.
The danger is that, in an effort to measure everything and show the return on everything, you stop activities because they’re fundamentally un-measurable. The myth is that “Because you need to show a repeatable, predictable and scalable revenue engine, you need to understand and measure the impact of everything you do”. But that’s taking the argument to an extreme view – the reality is that there will always be spend in your budget where you won’t be able to tie revenue to that spend. Ever.
Perhaps controversial again. A traditional funnel implies a sequential path for a customer from something like “Awareness of problem” to “Discovered our solution [to that problem]”, “Evaluated our solution” finally “Becomes customers [then perhaps evangelist etc]”.
Again, we’ve never found this to represent reality. Of course all models are exactly that – models. They’re not perfect, but if they’re useful, that’s okay.
But I feel the funnel fundamentally misrepresents how real people actually interact with a brand. From talking to customers what you find is that there are just an enormous number of holes in this approach. For example:
A funnel implies a single direction of travel, a path to enlightenment, ending with purchasing your tool. But, from talking to customers I find a much messier reality – people go back and forth, there are interrupts and so on. We’ve found it almost impossible to actually classify people in to different stages – it’s too over-simplistic to be useful (we’ve found).
Again, controversial. But our experience is that conversion rates are the lever you are least able to pull. Why? Because for most orgs, they have a pretty well optimised process for converting leads at different stages. At Redgate, there are certain lead types that convert at a 70% conversion rate, within a 2 week period – and that has been consistent for about 10 years, almost regardless of what we do! We’ve spent a lot of time and effort on this stuff – its value is in “Can we improve/optimise this?” – and generally we find we can’t. Of course you monitor it, to make sure it’s not dropping (e.g. because some leads got lost), but otherwise – stop worrying.
Finally, myth 5…
I wanted to end on a positive. 2-3 years ago, I thought the task of building out a “revenue engine” that was vaguely water-tight, believable and actionable was never going to happen. There were so many holes in the data, it was so hard linking activities to outcomes, that it wouldn’t actually happen.
I pleased to say that isn’t what has happened. It’s been pretty arduous, but we are now on the brink of a model that allows us to:
A simple example of the last point – in 2018 we ran a number of webinars of different sorts. We tracked through the leads, opportunities and revenue from each of these and found that the impact of having a “star” presenting the webinar (someone big in our community) had a far bigger upside than expected – at little or no additional cost to us, other than the trouble of finding and convincing these stars. I.e. one webinar with a star involved would generate more high quality leads than 2-3 webinars without such a person on the event. So this year, we’re changing our program a little – fewer webinars, but each more impactful with more big names presenting.
Just a small example, but there are countless more – we’re building out a model where we know how and which levers we can pull (and which we can’t), and at what cost. It took a long time to get there, but it’s finally becoming real. Feel free to get in touch if you want to know more!
I recently spoke at the B2B Ignite conference in London on “Building a MarTech Stack at a Small Organisation: A Real World Example of What’s Worked and What Hasn’t”. Here are my slides from that talk.
It’s a lot of pictures, so might be hard to understand without the actual talk! Any questions, feel free to get in touch, always happy to help.
Whenever I meet customers I always slip in a marketing question or two along the lines of “Where did you hear about us? What brought you in to Redgate?”. One of the answers from a couple of months back was:
Well a year ago, I got a new boss and she told me that I had 6 months to turn our dev team in to a “DevOps” team. I did most of the work then realised the database was causing us real problems. So I did a Google search, found your website, and what you said made complete sense to me – you knew what you were talking about, so I tried your software out.
Okay, great but – how on Earth do you measure the effectiveness of your marketing activity with answers like this? What’s the ROI of this lead? I know the return (the customer bought the products in the end), but the investment? How can you calculate that?
I’ve written endless articles over the years about marketing attribution, “performance management”, lead measurement and so on. All with the stated aim of showing “What’s working?” or “What’s the ROI of my marketing budgets?”. All different ways of asking the same thing.
And yet, years later, after reading many others’ articles, attending conferences, seeing demos from various products claiming to give the “formula” (marketing automation platforms, Google etc), the answers don’t feel any closer. Why is it so difficult?
Firstly, every business is different with, hopefully, different marketing strategies and tactics – some activities are inherently more measurable than others. A B2C business selling gizmos at $10 through Facebook ads is fundamentally different to a B2B org trying to sell $3m deals to the Fortune 500. Measuring the performance of those Facebook ads where the customer buying cycle is likely very short (“I saw your ad, I clicked Buy Now 10 minutes later!”) is a significantly easier task than measuring the impact of four years of concerted marketing plays to win over a multi-national bank. But still, all feel like hard problems.
It’s become a cliché in marketing blogs to quote the cliché about “I know half of my advertising works, I just don’t know which half”; but I include it because I think it can be updated to something like “I know what I do to get half my leads, I just don’t know about the other half”. I feel this better reflects that state of play with marketing performance management, hopefully I can explain why.
At Redgate we fundamentally get two sorts of leads – what I refer to as “Always On” leads, and “Outbound” leads. I think, for the first type, measurement of marketing ROI is integrally difficult, I’d argue impossible. For the 2nd type, you can measure ROI and should do so at all times.
Tackling the latter first – these are leads where you can make a strong argument that, if it wasn’t for a given activity, you wouldn’t have got the lead. An obvious example is an event or a graphical ad in the trade press. There’s a pretty solid chance that if you hadn’t been at that event, or placed that ad, you’d have missed that person. The trigger that caused that person to make an enquiry was paid for, by you. Of course, there are arguments that they might not have really been interested if they hadn’t recognised your brand from years of prior work; or that “Maybe they would have come to your site anyway!”. But that over-complicates what is already a difficult problem. If you spent $10k going to an event, and you made $20k, you should attribute that $20k to that $10k spend – simple.
This is the half you can understand and measure. For media ads, events, webinars, other placement spends, I think you can put together a pretty good spreadsheet or other tool showing the ROI on your investments – you just need to do the graft (or use a tool – I’ve been most impressed by Pardot recently, which seems to do this sort of thing very well).
But what about the first half? What triggered the customer coming in to find out about your offering? It was his boss telling him to find a solution. It was nothing to do with your marketing. Like a lion waiting for her prey to walk past, your job is to be “Always on”, ready with the right material, the thought leadership, the clear CTA, the understanding of the customer. Most of these customers do their own research well before they talk to your sales people – they read your website, they read other peoples’ websites, they talk to their analysts, talk to their colleagues. You may have made investments in all these areas both in terms of $$ spend, and employees’ time (web copy, site architecture, positioning and messaging, briefings with analysts, content placement, blog posts, influencer programs) but really how can you apportion this effort to the lead?
This is a hiding to nothing. Trying to work out how the salary of the individual who spent half of her time talking to Gartner, Forrester and IDC can be attributed to the leads that came in this year is both an impossible and pointless task.
The reason this is called “Always on” lead generation, is because, like the lion waiting for prey, you have to always be there when the customer goes looking. If you’ve got the money, you have a pack of lions covering every location – every analyst, your website, 10 other websites, recommendation sites, articles in the trade press and so on. But when that customer does a Google search, you’re ready and waiting with the greatest copy and thought leadership you can muster.
[A quick aside about SEO and PPC – this is table stakes. If a customer out there has a need for your solution and he/she doesn’t find you very very quickly through Google then you have a more immediate problem that needs fixing immediately. Customers not finding you easily when they’re already out there looking is a fixable problem, primarily through good SEO practice.]
The important subtlety here, and why this is such an issue, is the driver of interest for customers. For the outbound activity, it’s legitimate to say that your actions have precipitated that activity – have, in some way, driven that behaviour. Theoretically if you do more of these activities, spend more money, you’ll get more leads – if I spent $10k this month on webinars, then it’s possible that spending $20k might double the number of leads (ignoring issues like diminishing returns).
But for the “Always On” leads, you haven’t driven this behaviour. If you doubled your spend on say “hiring even better copywriters for our blogs”, would that double leads from this source? No – because the primary drivers are out of your control – they’re in the hands of the businesses you serve. May be you can improve conversion somewhat, but I’d suggest the attribution is very tricky.
So overall, I do think it’s possible, and essential to show where half of your leads come from, the ones where you’re precipitating the activity yourself. This information should be readily available in Excel, PowerBI, Pardot, whatever and you should be reviewing it constantly – was the spend right? What can we double-down on? Cut?
But for the other half – stop worrying. Abstract calculations based on employee time or similar are pointless. You’ll never really understand the complex interaction of customers and touch-points that led to that lead – what value does it bring, knowing that the customer read 129 different parts of your website before getting in touch? – so stop worrying, and focus on measuring what you can measure.
It took quite a while to write part 2 of this post, for reasons I’ll mention below. But like all good investigations, I’ve ended up somewhere different from where I thought I’d be – after spending weeks looking at the Twitter feeds for different companies in different industries, it seems that the way Twitter is used and is useful varies enormously by industry. This makes it difficult to build a generic model for Twitter sentiment analysis (because a model built from, say, small B2C companies, isn’t really applicable for large B2B companies) – but it also makes interesting suggestions for how companies in different industries should use Twitter to help their businesses.
Oh, and my main conclusion? People sure do hate the airlines! More on that later.
First, a potted history of how I got here:
Well, the answer is No. Or at least not within the time that I’ve had so far. And the issue seems to be that the way Twitter is used by companies and by companies’ customers varies significantly by industry.
A first step in creating a predictor model is to manually assign sentiment scores to a long list of tweets – this creates a training set that you then use to train and create a model. During the creation of the model you repeatedly test the model against an out-of-sample dataset to see if your model is working (to avoid things like over-fitting). As a first step, I assigned manual values to around 500 tweets (i.e. I manually tagged 500 tweets as either very negative, negative, neutral, positive or very positive), then I tried to create a model from this. However, my cross-validation scores were terrible – the model was struggling to predict the sentiment of other unseen tweets. I know this is partially because 500 isn’t nearly enough, but still – how could the scores be so bad?
What was the problem? Like all machine learning issues, I’ve generally found that eye-balling the data can tell you a lot. I believe there were two problems with my model:
The latter creates the problem of domain-specificity. Is a model created from, say, tweets about the airline industry, relevant to tweets about online cloud storage? It seems not.
The first clue is the distribution of scores I was giving. Remember, 0=very negative, 1=negative, 2=neutral, 3=positive, 4=very positive. For the airline industry I found the following:
| 0 | 1 | 2 | 3 | 4 |
| 10% | 32% | 41% | 12% | 4% |
I.e. though there were a lot of neutral tweets, there were a lot of negative tweets. Here’s a very standard example:
@Delta just rebooked my 70yr old parents on a longer flight back to cvg from fco and no extra legroom that I had paid for. Fail.
There’s a lot of this sort of thing going on for the airlines.
In contrast, here’s the distribution for a couple of big cloud providers (specifically, AWS and Azure combined):
| 1 | 2 | 3 | 4 |
| 3% | 83% | 11% | 2% |
An enormous number of utterly neutral tweets. Here’s a standard example:
RT @DataExposed: New #DataExposed show: Data Discovery with Azure Data Catalog. https://t.co/5sWuKpdoYx @ch9 @Azure
..pretty dry stuff.
This actually presents two distinct problems – firstly, as mentioned, if the nature of Twitter usage and the type of tweets varies so much by industry, then models will only be really effective within their domains. Fine – if you work in a given industry (e.g. Airline, IT, Fashion), then you can create a model for your industry and use that.
The second problem however is more difficult to work around. For a given industry, I’ve found that the way in which Twitter is used varies enormously. This is what you’re seeing in these very different distributions. And if the vast majority of tweets are of a given sentiment, then sentiment analysis becomes not only difficult, but actually not particularly useful!
In the airline industry, as far as I can see, Twitter is used almost exclusively for telling airlines how bad they are. There’s a pretty strong correlation between the number of tweets mentioning a given airline and the negative feeling towards that airline. This distribution does vary by airline (see table below), but when you know that most tweets are just complaints, what’s the value in searching for the occasional (positive) needle-in-a-haystack?
If I worked for an airline and wondered “How could we use Twitter to improve our brand?”, the answer would be pretty simple – firstly, improve my product! and second, employ customer service reps to look after these people and react to the complaints. As I say, some airlines are worse than others:
| Row Labels | 0 | 1 | 2 | 3 | 4 |
| American Airlines | 12% | 35% | 37% | 14% | 2% |
| British Airways | 10% | 29% | 46% | 12% | 3% |
| Delta | 10% | 37% | 38% | 12% | 5% |
| SouthWest Airlines | 5% | 26% | 47% | 16% | 5% |
| United Airlines | 12% | 43% | 35% | 6% | 4% |
| Virgin | 0% | 13% | 44% | 25% | 19% |
Well done Virgin and SouthWest. Delta and United – you have work to do…
The problem in the cloud services industry (AWS and Azure) is the opposite – mentions of these services tend to consist of semi-banal tweets about new services offered, new features and so on. I.e. Twitter is used to share information about the products and services and rarely to express emotive responses (it’s very rare to read “Can’t believe how amazing @AWS was today!!! #FTW” – it just doesn’t happen). Certainly the split between B2B and B2C tweets shows this difference (I looked at small and large orgs as well, from local shops, to fashion houses, to small tech companies).
I still think there’s value in implementing a domain-specific model (for example, a model “Just for small tech companies”). The only block is, as described in Appendix 1, the problem of Parsing tweets properly. Maybe once I’ve figured that out, I’ll find a way to classify the other million-odd tweets I’ve collected for the airline industry as a starting point (there are a lot of unhappy airline passengers out there!)
Appendix 1
The problem of parsing tweets
I was warned by the following tweet from the team at Stanford NLP:
Using CoreNLP on social media? Try GATE Twitter model (iff not parsing…) -pos.model gate-EN-twitter.model https://t.co/K2JAF5XwJ2 #nlproc
— Stanford NLP Group (@stanfordnlp) April 13, 2014
The problem we’re trying to fix, as described in the previous post, is that, for us to understand the sentiment of any sentence we have to carry out a couple of stages first. To begin with we need to Part-of-Speech tag a sentence. So, identify “Dog” as a Noun, “Catch” as a Verb and so on. This has challenges with Twitter which is full of URLs, hashtags, #LOLs and so on. But – the GATE Twitter model mentioned above solves this by adding in this functionality to the POS tagger. If you run a tweet through the GATE POS-tagger it will identify http://some.url/ as a URL and so on.
So far, so good. However, the problem comes with the next stage – Parsing. What’s this? If you look at a sentence such as “I don’t like ice-cream”, the tagger will identify and POS-tag each component of this sentence – I, do, not, like, ice-cream. Great, but to understand the sentiment of this sentence, we need to group these elements further. We need to understand that there’s a hierarchy whereby do and not are grouped together, and apply to like to negate this term. I.e. this sentence is actually negative despite the fact that like is a positive word.
And herein lies the problem with parsing tweets – because of the language used in Twitter, often abbreviated, often partial, it’s very hard to properly parse tweets and work out this structure. This seems to be the problem from the very brief analysis I’ve done. The Stanford team (and others) do state “Sure, you can try the standard parsers using tweets tagged with the GATE POS tagger, but good luck with that!”. It obviously needs more work, and maybe when I have more free time, I’ll have a look!
This post is split in two, primarily because I hit a roadblock half-way through the work – and I wanted to get the first part out. Second part to follow once I’ve fixed the difficult problems!
A lot of people follow the Twitter feeds for competitors or, of course, themselves. But, one of the things I’m sure you’d like to know is – are the tweets for any given company generally favourable or not? Does the company’s Twitter community love or hate the competitors? Or you!?
I’m going to try and find out. The process of investigating this problem was arduous, so below is the summary of the things that actually worked. I have a machine-learning background, but have done very little on natural language processing. There were a lot of dead-ends before I got here..
In summary, there are some standard techniques for doing something called “Sentiment analysis” for sets of textual data. I.e. if you have, say, 1,000 pieces of text written for something – a book, a film, in this case – a company – are they largely positive or negative?
Obviously I’m not inventing the methodology here, but stealing from various sources. Or rather “building on others’ work” (full acknowledgements below – I really am just using the hard work of others). Here are the steps I went through if you want to try this at home:
? As you’ll see below, this was the main issue that stopped this being an almost trivial exercise.Sentence #1 (4 tokens, sentiment: Negative): This isn't good [Text=this SentimentClass=Neutral] [Text=is SentimentClass=Neutral] [Text=n't SentimentClass=Neutral] [Text=good SentimentClass=Positive]
Obviously the clever thing here (apart from the tokenisation which doesn’t just treat “isn’t” as one word), is the way in which the algorithm can’t determine that the word “good” is positive, but that the sentence is negative overall – as mentioned above, you need to see words in the context around them and the algorithm is smart enough to know that the “not” that is part of “isn’t” negates the positive word “good” that follows it, to make the overall sentence negative – very clever.
company3: Half way through a great panel session with #Company1 and @Company2 @ #LaunchEvent launch event http://test.co/test
0 = Very Negative 1 = Negative 2 = Neutral 3 = Positive 4 = Very Positive
Perhaps at worst, this tweet might be misclassified as neutral, but “Negative” seems plain wrong. Also note, this is a statistical process – I know and expect a number of misclassifications, it’s part of the game – but this seems like quite a simple example.
company3: Half way through a great panel session with #Company1 and @Company2 @ #LaunchEvent launch event http://test.co/test - score of 1
company3: Half way through a great panel session with Company1 and Company2 launch event - score of 3
company3: Half way through a great panel session with #Company1 and @Company2 @ #LaunchEvent launch event - score of 3
FirstName: RT @Company1: Free eBook: 50 Web Performance Tips for Developers (from @Company2) http://test.com/test - score of 1
(ROOT (NP (NP (NNP FirstName)) (: :) (NP (NP (NP (NN RT) (NN @Company1)) (: :) (NP (NNP Free) (NNP eBook))) (: :) (NP (NP (CD 50) (NN Web)) (NP (NP (NNP Performance) (NNP Tips)) (PP (IN for) (NP (NNP Developers))) (PRN (-LRB- -LRB-) (PP (IN from) (NP (NN @Company2))) (-RRB- -RRB-)) (NN http://test.com/test))))))))
(ROOT (NP (NP (NNP FirstName)) (: :) (NP (NP (RT RT) (USR @Company1)) (: :) (NP (NP (JJ Free) (NN eBook)) (: :) (NP (NP (CD 50) (NN Web) (NN Performance) (NNS Tips)) (PP (IN for) (NP (NP (NNP Developers) (NNP -LRB-)) (PP (IN from) (S (VP (USR @Company2) (NP (UH -RRB-)))))))))) (URL http://test.com/test)))
As can be seen, this is tagging elements more appropriately – USR for users, URL for URLs and so on. Seems great. However, if we again look at the classification…
FirstName: RT @Company1: Free eBook: 50 Web Performance Tips for Developers (from @Company2) http://test.com/test - score of 1
Sigh.
So, I’m going to pause there. I’ve understood the problem a little – tokenisation, followed by POS tagging, followed by sentiment analysis – I’ve found a standard framework – the StanfordCoreNLP codebase – which is incredible, and a better Twitter POS tagger.
However, my results still aren’t quite there. Running this algorithm for an example company (for which I have 10,000+ tweets), I find that positive tweets are clearly and, generally picked up. The struggle is distinguishing between neutral and negative tweets – far too many neutral tweets are being categorised as negative. This could be a number of things – many of these tweets are, by nature, very neutral and specific to B2B companies. Is this the mis-application. Of a generic model to a specific data set? Do I need to create my own model for “Company tweets”? How do I protect from over-fitting, how can I add to the great work already done? To be continued…
Customer Experience (CX) – it’s a popular topic right now, analogous to the importance of User Experience (UX) in the world of product development. And something which I strongly believe is important for a marketing team to get right. So, we all know that getting your Customer Experience great and consistent is important for all of your customers’ touch points but – how on Earth do you measure if it’s working or not? Was it worth the effort?
There’s a great piece I read here recently about the issue of “end-to-end” funnels – basically give up on the idea of a linear funnel where a customer moves through the buyer process from “Awareness” to “Discovery” to “Validation” to “Retention”, measurable through cohort analysis and conversion rates. The reality is that customers interact with your brand in multiple, unstructured, unordered ways at the end of which (hopefully!) they buy your product (aka the “Dark Funnel”). If someone is buying a car how did they end up at that decision? Sure they might have visited VW’s website, but they might also have asked friends, gone on to review sites, read a motoring magazine, asked a question on Twitter, spoken to the sales reps in the showrooms, researched forums – all in a semi-random order, unpredictable and – key – very hard to measure.
I really agree with this – and the suggestion I take from this is: stop worrying about measuring it all and just make great customer experiences. That’s enough – if you’ve made your website great, your social media channels interactive and high quality, your support is top-notch, you respond quickly on forums, you’re friendly and positive at events etc, then belief is enough to justify this “investment”.
But, is that enough – is there any way of measuring the impact of your work? Net Promoter Score (NPS) is one possibility – and it is the sort of thing that can be measured. But from a PDCA perspective (where we’re trying to measure the impact of our work and adjust accordingly), it’s very slow, and let’s be honest, optimistic to think that you’re going to be able to separate out the impact of your marketing work in an NPS score from everything else. What if your company has just released a ton of great product to the market at the same time – what had the impact on NPS? It feels unsatisfactory.
So we need something more immediate. The best I could come up with, and the thing I’m going to try, is “Mystery Shoppers”. It seems obvious but in the world of User Experience what do we do to test the usability of our products? We ask users to try the product out – simple. So why not do the same for your CX? Get five people, at regular intervals to pretend they have a need related to your products – and to go through all the necessary interactions with your brand to buy something. From first Google search, to some forum interaction, maybe a question on Twitter, download a whitepaper (and try to understand it!), follow your Facebook page, phone up support, try to use the product, try to buy it, get follow-on help and so on.
And you could have different types – someone acting as an end-user vs. a corporate buyer. Someone who is totally self-service/won’t speak to anyone vs. someone who wants to sort everything out on the phone. An expert vs. a newbie. A difficult so-and-so vs. a “happy path” customer.
Then of course you can get both qualitative and quantitative data from those people about their “Customer Experiences” – put the numbers on a chart and use PDCA process to see where the problems are (“We’re great if you get us on the phone, but self-service customers are really struggling” or “They’re great once they’re using the product, but it was a mess up to that point”). And you can use the qualitative feedback to know how to act. If a mystery shopper says “I just couldn’t understand from your site what your product actually does“, then may be better explanation? Or better still, a video?
It shouldn’t cost much to implement and the feedback should be invaluable – as long as you then act on it! NB: It’s also important to get people disassociated from your company. Just asking the people in your team, or a regular customer who has loved you for years isn’t enough. You want people who’ve barely heard of you, or haven’t interacted with you for years – proper, independent input is the most valuable, and what you should be seeking.
Jeff Bezos’s letters to shareholders are, of course, famous for their insight, not only in to how Amazon functions, but also for their advice on how to run a certain type of business. One of my favourite excerpts, from the 2005 letter is:
As our shareholders know, we have made a decision to continuously and significantly lower prices for customers year after year as our efficiency and scale make it possible. This is an example of a very important decision that cannot be made in a math-based way. In fact, when we lower prices, we go against the math that we can do, which always says that the smart move is to raise prices. We have significant data related to price elasticity. With fair accuracy, we can predict that a price reduction of a certain percentage will result in an increase in units sold of a certain percentage. With rare exceptions, the volume increase in the short term is never enough to pay for the price decrease. However, our quantitative understanding of elasticity is short-term. We can estimate what a price reduction will do this week and this quarter. But we cannot numerically estimate the effect that consistently lowering prices will have on our business over five years or ten years or more. Our judgment is that relentlessly returning efficiency improvements and scale economies to customers in the form of lower prices create a virtuous cycle that leads over the long term to a much larger dollar amount of free cash flow, and thereby to a much more valuable Amazon.com. We’ve made similar judgments around Free Super Saver Shipping and Amazon Prime, both of which are expensive in the short term and—we believe—important and valuable in the long term.What Jeff is saying here, as he has in many of his other letters is that Amazon is in for the long term, making long term market decisions (essentially, that Amazon is taking a market position that it will always be the cheapest, getting cheaper all the time, whether for a book, a baby stroller, a Kindle or 50 TB of cloud storage) in preference to short term fixes. But more than that, he’s saying that even when he knows that a short term (and presumably, very tempting!) decision will make more cash – because the maths shows it – that will be usurped by their strategy to play the lower-price long game.
This reminded me of a an interesting problem from the world of maths – simulated annealing. Described considerably more succinctly, here on Wolfram, this is method used when trying to optimise a given function, and, because of the complexity of the function, there is a risk of getting caught in local optima, particularly when using standard “hill climber” functions. If a function is too complex, with too many variables to find the optimal solution directly, one has to start from what we know today then use “hill climber” functions to find “What’s the direction I should go in?”. For example, we’re currently charging $x for our product, what should I do next to optimise revenue? Increase slightly or decrease slightly? And if the function your trying to optimise is the “Best price for this particular product”, then this will reach an optimal solution – you’ve optimised when all experiments to change the price just make profits worse. I.e. you’ve used “the maths” to find the optimal solution for this particular problem.
But this is a local optimisation. If what you’re trying to optimise is the bigger, much more complex function of “What is the best set of prices for everything my company offers?” then the danger of this approach is that you optimise the individual cases whilst de-optimising the problem as a whole. Simulated Annealing as a process tries to solve this problem in two ways. Firstly and most relevant here, is that so-called “Bad” decisions are allowed to be made locally if it allows one to explore more of the landscape of possible outcomes. So, for example, if your product currently sells at $1000 – the hill-climber approach would say “What if I charge $1001, or $999 – which is better?”. Keep repeating this till you maximise profits – may be at $783 for example.
However, with simulated annealing, you would allow a test which would be “What if we charge $10? Or $10,000?”. Possibilities that seem ridiculous, but what if these experiments take your business in a whole new direction, reaching new customer types you never dreamed of? What if that, in the long term, led to optimising your business as a whole? You would never realise this success without trying these experiments that fly in the face of what the maths is telling you, and letting those experiments run for a while.
I know, of course, that I’m slightly over-extending an analogy here – in fact what Jeff Bezos is doing is not taking larger, experimental punts to try and jump out of local minima. Instead he has a long term strategy which overrides that – a belief that in the long term, the lower prices for everything is the optimum for his business. A strategy that’s hard to argue with given his success!
But the point really is that, sometimes, taking a strictly data-driven decision in the short term, though it may show nice charts of increased revenue (i.e. you’ve optimised locally), can have longer term implications that you might regret, if those decisions don’t sit well with the position in the market your company is trying to achieve. If you’re trying to achieve the position of “The cheapest”, don’t increase your prices (see Amazon). If you’re trying to achieve the position of “The best”, don’t lower them (see the Apple 5C). If you’re trying to achieve “Pay as you Earn” (i.e. people pay according to ability to pay), then set an intelligent pricing strategy that caters for all wallets and stick to it – see any Atlassian product, e.g. JIRA:
This last example is a good case in point. I’m sure it would be very simple to show how increasing the lowest price point here ($10 – that’s ridiculous, how can they make money on that!?) will make more money in the short term. But there’s a long term strategy here based on Life time Value – today’s startups are tomorrow’s big businesses. Get them hooked on the product now and in the future they’ll be buying at the higher price points.
In the phrase”data-driven” it’s not the term “data” that worries me. It’s the term “driven” – when we are “driven” to decisions, rather than sticking to a long term strategy, instead letting the numbers make the decisions for us. By doing this we’re in danger of optimising in the short term, and not for the long term.
Obviously I’m not saying “Ignore the maths” all the time. Just be very careful when it suggests doing something that doesn’t align with your long term strategy of where you want the company to be in 10, 20 or 50 years from now.
I’m beginning to think I might need to change the tag line for this blog. One of my earliest posts was about how we needed to apply some scientific rigour to the process of marketing attribution and therefore ROI. How can marketers be getting away with such unproven and unprovable techniques, spending all this money with so little evidence of success?
But I think I might have changed my mind. We’ve actually had a couple of geniuses at Red Gate looking in to the provability, or otherwise, of marketing ROI, and their conclusion? Nah – you can’t do it. More specifically, for our volumes, with our variation in spend (we’re not a company make a thousand sales a day, all at exactly $10 each), it’s not possible to show statistically that a sample and a control group will differ enough to be discernible at any interesting level of significance. I’ll get more details if they let me (as well as writing about the interesting point raised that – does this mean Google’s whole business model is based on FUD? On us be too scared to kill spend on things like Google Adwords? For another time…).
Anyway, I was thinking about this problem – that completely undermines this blog – and thought that actually there’s more to it than the maths. And that actually, trying to measure return on investment for marketing is very much like trying to measure return on investment for an employee.
There are many similarities between these two investments:
So really, I’m coming round to the view that, though we might be able to measure Inputs well, both for campaigns and for people (“Is this a great campaign? Did we get it spot on?” and “Is Pete doing well? Is he producing great work, and a great person to have on the team?”) and we can measure some Outputs (“How many page visits did we get?” and “How buggy is Pete’s code? How productive is he?”), Outcomes and therefore accurate ROI are pretty much an impossible requirement.
So what does that leave us with? Well personally, I think it’s an issue both of using intelligent qualitative assessment of work and also some faith that high quality work based on sound judgement will have the desired effect. For example, when thinking of Google Adwords – if we’ve done our research and know that there are lots of customers out there with a particular problem; and we’ve written our search terms to reflect that in an intelligent way; and we’ve worked on the ads so that they really reflect how we know customers are thinking; and of course, we’re checking that the ads are working, and continuously improving them – then, I think it’s reasonable to suppose that this effort (and spend!) will produce some results beneficial to your company. Of course, you have to check the rest of the pipeline – when they’ve discovered you through the ad, can they try your product easily (Validation)? Do they get what it is (Positioning)? Can they buy it in a way that suits them (Pricing and Packaging)? But if these things are also done well, I believe the extra work trying to validate whether or not you’re getting a particular overall ROI from Adwords is a hiding to nothing.
I’ve read a couple of books in the last year both with something to say on the subject of marketing plans. Well, I’ve read one and given up on the other. The one I finished was:
Lean Enterprise by Jez Humble, Barry O’Reilly and Joanne Molesky
And the one I barely got started on was:
Marketing Plans by Malcolm McDonald and Hugh Wilson
This makes this a bit of an unfair comparison review of the two books, as I haven’t made it through the latter, but in a way that’s the point I’m trying to make.
The latter book comes in at a whopping 592 pages – and that’s for the 7th edition of the tome. And the content is dense – every page is packed with data, information, tables, planning tips and tricks, processes and so on.
In contrast, the Jez Humble book is being released through an Agile Publishing methodology (release early drafts to customers, gather feedback, rinse-and-repeat), where the first draft came in at a digestible 78 pages. And it’s very readable.
But again, that’s not really the point – there’s nothing wrong with a long, intense study of a subject. The most interesting difference between the two books is the approach taken to marketing activity. To illustrate the difference, the following is a key diagram from Jez Humble’s book:
This diagram isn’t directly applied to marketing, but instead is making a point about how most companies develop products end-to-end. The authors describe this as “water-scrum-fall” – a process whereby yes, your development teams are writing and testing the code for a new product following Agile practices, and may have been doing so for years, but the rest of the organisation still works in a heavily waterfall based, linear way, with all of the standard problems of waterfall development – bottlenecks, blocks, wasted effort, wrong products to customers, late delivery and so on. By the way, HiPPO stands for “Highest Paid Person’s Opinion” ;-). NB: I won’t go in to the details of “Agile vs. Waterfall” as development methodologies – Wikipedia as ever has a good description.
My contention is that I would add marketing and marketing planning in to this diagram as a part of the process of developing and releasing a product to market – and that we as marketers still, generally work in a heavily planned, non-iterative, and essentially out-dated approach to developing marketing campaigns.
The Malcolm McDonald book provides quite incredibly detailed tables for the reader to fill in, allowing you to plan out your whole year of marketing in detail – which segments you’re going after (and everything about those segments), when you’re running campaigns, how many leads you will get from each activity, forecast revenue generated and a thousand other things. In theory, all laudable activities. And for the last three years I’ve followed this approach (more or less – I could never quite stomach the level of detail required by the book). Every November/December I’ve spent weeks constructing a plan of, essentially, what was going to happen the next year in marketing, down to leads and revenue generated for the next 12 months.
As everyone knows, these plans never come to pass. But, the argument goes – “It’s the planning that’s useful, and what you learn, not the actual plan”.
Yes – to an extent, because a plan forces you to think about your goals and objectives, and that can’t be bad. But a key practice in Agile development is the concept of a backlog of work – you have a long list of activity that you want to undertake for a product, but the level of detail for those items is different depending on how soon you plan to do the work. If you’re starting something next week (at the top of the backlog), you’d better have a pretty good understanding and description of the story you’re tackling. However, if you’ve got something you “Hope to do in about 6 months”, you can leave these items at a very high level, something like “Add in that cool Facebook sharing feature” will do perfectly well.
Why is this a good idea? Because right now, you fundamentally don’t know what the future holds. Today in February 2014, I have an idea for what marketing we as a team want to be doing in 2014H2, based on the products we have, the customer segments we’re reaching (or not reaching of course!) and so on. But I’m also pretty sure that I’m wrong. That whatever we do in 2014H2 will be only barely related to my current thinking. Right now we’re planning a mass market activity, based on some of the traditional marketing approaches we’ve used before. But what if, in April this year, we suddenly start gaining some real traction with larger accounts in a specific segment? Then I might need to pivot and re-think plans to make the best of this opportunity. Any detailed plans that I had would be thrown out of the window.
And the point made in the Lean Enterprise book, is why not apply the Agile development approach to other areas of the business too, such as marketing? Release early, gather feedback, iterate-and-repeat. The key benefit is that you’re getting early feedback on what’s working, then you build on that, continuously improving what you’re doing so that by the end of the year you have activities that you know are working – because you’ve been testing them for the previous 11 months and getting benefits (e.g. leads) all along the way.
This is what we’ll be doing this year. I haven’t made a marketing plan for 2014 at all – and I’m feeling very comfortable with that situation.
PS the image at the top of this post is the character Destiny from the amazing graphic fantasy novels – The Sandman by Neil Gaiman (I can’t recommend these books highly enough – I’m now on my 4th read). Destiny holds a book (the “Book of Destiny”) that contains the past, present and future of every living being. As I said, the novels are fantasy.
I was attempting some market sizing activity this week. It’s something I haven’t done for a few months and quite frankly I’d forgotten how hard it was.
I start from a premise that the future is completely unpredictable. Really, aren’t we kidding ourselves when we think we can predict how many people will buy our widget and at what price? But then this seemed rather defeatist, and I have seen market sizing produce some value, so I thought I’d try and break the problem down a bit.
The other think I noticed this month was how much fruit-specific cutlery there is out there. You can buy a Kiwi spoon, a Mango splitter, a Grapefruit spoon and endless other tat to fill your kitchen drawers. So I thought I’d use this as an example market sizing task, focussing on a new, brilliant product idea I had, the “Apple Knife” (patent pending) – a knife specifically designed for cutting up apples with a fork bit on the end for picking up the pieces. I know, genius.
So the usual method I use for market sizing is the standard “Start big, and narrow down” approach. And I try to use some sort of staging for this, a version of which is described here – try to define some sort of Total Addressable Market (how big would this market be, if we had 100% market share?) and then calculate some proportion of this, based on how many people we can reach, what market share we might able to get and so on. As a massively over-simplified example, for our wonderful Apple Knife product:
So, in theory, so far, our Total Addressable Market is 1bn knives – if every person in the world who ate apples bought one, this is what we’d sell. I don’t think this will happen. So we have to start narrowing down the numbers.
NB: You can always narrow the numbers down in different orders – addressable geography first, or # of households first? I’m not sure it matters, but generally certain orderings are easier than others (e.g. it’s easier to find the number of households in the UK than it is in the whole world).
Anyway, here’s where it gets interesting. If I were a naive presenter on the Dragon’s Den, I might go in and argue – 26m knives will be sold in the UK, at £5 each, makes a total market of £130m, ker-ching! (let’s ignore things like manufacturing and marketing costs for now).
But, I suspect the dragons would have something to say about this. The next stage in the narrowing process is the most important and, I think, the most difficult – how many people will actually want to buy an Apple Knife? To continue our sizing process:
And the key of course here is, what is the value of n? If we think everyone will want one, it’s 100%, great. But, if we’re being slightly more realistic, n could be as low as 0.01% – leaving us with a total market of £13,000 (26m x 0.0001 x £5) – somewhat less attractive as an investment opportunity. Or even less of course (I’m going off the idea more and more, as I think about it).
I think the first parts of the market sizing, if not trivial, are much easier than the latter parts. It might not be easy to find, say, “The number of companies in the US that sell products online”, or whatever your top level numbers are for your business, but that’s a number which can, reliably be found – if I do the hard work to find that number of businesses, say 1.3m, then that number is correct and won’t change for a while.
The latter numbers (or percentage multipliers) are very unpredictable however. But not completely – what are the tricks for improving the accuracy of this figure? I think this really depends on how new or old your market is. And this exists on a scale. With our Apple Knife example, there are the following possibilities:
(NB: There’s a caveat here about access to competitors sales information. You might not be in the apple knife business yourself, but if you know a competitor who is, and have seen their sales figures – or can work them out from revenue etc – then that’s a great start. However, it doesn’t take into account massive differences such as market reach, brand, channels and other barriers to entry that differentiate the competitor from you. Nevertheless, looking at other companies’ revenue is a great check – if the biggest player in the market is only make £50,000 revenue per quarter from fruit-cutlery sales, that’s a good ceiling on your estimates..)
Most examples I hit upon fall in to one of these four categories. A I say, the early parts of the calculation are relatively easy, but the latter parts, if your problem falls in to the last couple of categories – and most interesting innovations and product ideas do – are very difficult indeed.
And the basis of the problem – it’s almost impossible to predict take-up of a new product or a new innovation. If you’re producing a new laser printer, with slightly better functionality than others, at a slightly lower price, you can have a good stab at market sizing. But what if you’re innovating to provide a fundamentally different way of solving a pain-point – a pain-point which might not even be addressed at all at the moment? Though we all like the certainty of saying “This idea could make us £10m!”, we really are kidding ourselves. Instead, I prefer a more experimental, research based approach where we start on a new idea in a cheap or lean way (make a version of the apple-knife yourself and stand outside John Lewis, showing it to people and see how many would buy one), then start making forecasts when you have some idea of need and interest from the customers. E.g. if 1 in 1,000 people you asked said they’d buy an apple knife, that’s not a bad start. Though I think I’d be lucky to get that much interest..
This post is about one of the many issues facing anyone trying to do marketing attribution – how do you measure the impact of offline activity on online success? If you’re selling online, but you’re carrying out offline activity (TV ads, magazines, direct mail, events, arguably word-of-mouth) then you don’t get this sort of insight “out-of-the-box” with Google Analytics – well, not yet at least!
Avinash Kaushik (@avinash) has written a great post here, describing the problem in a simple way. Essentially, how do you figure out where all of that Direct and, in our case at least, most of the Organic traffic is coming from? I’m not showing figures, but the snapshot below shows the five most important channels for part of Red Gate’s business:
NB: This is based on a channel grouping and attribution model that I described in a previous post (I’ll write another post later about the set up of this model for us). It’s not “Last click” attribution by a long shot (and so doesn’t suffer many of the problems of that model), but does still present an issue of how to get some insight in to where traffic is really coming from.
For us, the vast majority of “Organic” is people just typing in the names of our products or company (i.e. it’s not SEO as such, it’s just people who can’t be bothered to type in “www.red-gate.com”). And Direct is the black box that it is for everyone else.
Having lots of this Direct and Organic traffic is great of course! Everyone knows us already – maybe we should pack up, go home and leave the traffic to roll in!?! But, for me this isn’t really enough. Knowing that there are stacks of people out there just typing in our product names, without having any idea what we’ve done to influence that is tough. Particularly as we spend a lot of money on events and other offline activities – we need to know the impact of these.
Separately, I’m about to start helping with a piece of work for a large direct mail provider to help them prove the importance of direct (postal) mail in an online world – they need to be able to prove this impact (it’s not just enough to say “Of course it works – it always has!”, when everyone else is using Google Analytics to prove the impact of online activities). They need to link direct mail to Google Analytics – Universal Analytics really – so that customers can show the way that direct mail drives online events and eventually revenue.
Anyway, here are the things we’ve done historically to try and address this (most of which are as Avinash has suggested) and then want we’re going to try in the future.
All of the above have provided interesting bits of information, but none have provided anything like the data we really need to accurately compare online with offline marketing and to treat all sources equally. As I’ve said before, I suspect this is a Sisyphean task, but that shouldn’t stop us trying.
There are two approaches we’re going to try next – one based on Google Universal, the other with HubSpot.
We’ll see how both of these methods work out though both do rely on the same thing – some mechanism for indicating offline activity online, either in GA or HubSpot. However one great thing about HubSpot in particular is that we already track a lot of offline activity in our CRM (for example, adding references on tags for people who we meet at conferences), and this could easily be pulled in as part of the history for a lead. I.e. if we’ve recorded in our CRM that Joe Bloggs went to our conference a year ago and now, a year later, is reading our whitepapers, then this will automatically get shown and we can, in theory, start automatically showing the value of this offline activity.
I was excited this week about the prospect of finally having the time to play with Google Analytics’ advanced Multi-channel Funnel (MCF) functionality. As announced at their summit last year, they’ve extended their advanced attribution modelling to all GA users and this provides the opportunity to (finally!) try and attribute some measure of value to the various different marketing activities that are carried out as part of the marketing role.
There are three things that have been introduced that are particularly interesting:
There are/were lots of blog posts written about these changes. On the 2nd item, this post from the Search Engine People is a great summary of what can be done with this functionality (essentially, re-grouping the standard channels groups – organic, direct, PPC etc – in to groups more relevant to your business).
The new type of model (called a Proportional Multi-touch Attribution Model) does away with the idea that it is the order or position of a touch point in the model that matters in the process. In a standard “last-click” model for example, then it is the last thing that a visitor did before hitting your site (whether typing your URL in to a search box, clicking on an Adword ad, or whatever), that gets all of the credit for the visit to the site (or for triggering the specified goal).
In the new model (PMAM for short!), it is simply the importance that you assign to a given touch point that actually makes the difference to the attribution model, regardless of position. For example, if customer 1 clicked on a banner ad, then carried out an organic search before hitting your site and customer 2 carried out an organic search, then clicked on a banner ad, then both of these funnels would be treated in exactly the same way.
But, what’s even more interesting, is that not all touch points are treated equally. For me, the purpose of most marketing activity is to somehow nudge customers along some sort of customer buying journey. We use quite a simple framework at Red Gate as follows:
NB: This model is taken almost wholeheartedly from the book Digital Body Language by Steven Woods – a great and simple read which I’ll review on this blog sometime soon.
Anyway, the point is – marketing activity, I think, should be focussed on moving people down this funnel. If you were selling a bit of anti-virus software, is everyone out there Aware that this is even a problem? In this market, I expect so, so perhaps your biggest problem is helping people Discover your particular solution amongst the masses. So if you were marketing an anti-virus product, you really should be working on helping people who haven’t Discovered your product to do so.
So, when putting the PMAM together, you need to make some decisions about how valuable you think different activities are at pushing people through the funnel. And this is where I think the custom channel groupings get really useful. A good example in The Search Engine People article is the need to split branded and non-branded organic and PPC searches. Why? If your anti-virus software was called “Ben’s AVSoft” (for some unknown reason!) then people might find your website through two distinct types of search – by typing something like “Great windows anti-virus software” (non-branded) or by typing in “Ben’s AVSoft” (branded – because the customer obviously already knows about your product). If a customer clicks on an ad of the former type that you’ve put up, then I would argue that that marketing activity has done more to push that customer down the funnel than the latter. This is because you’ve moved that customer from being just Aware of the problem to having Discovered your solution. With the branded search, you haven’t done any such thing – they’d previously discovered your product by some other means, and are now probably just coming to your site to have another poke around.
I.e. the purpose of bits of marketing activity is to effect some sort of state change in the potential buyer – and the values you give to different touch-points should reflect how impactful those touch points are at effecting those changes. The way the PMAM works (or at least, the way I’m going to try and implement it in Google Analytics) is that you give a value, relative to 1, for how important you think each touch point is. So, if you think a non-branded Google PPC Ad is 10 times more effective than a branded Ad for marketing effectiveness/effecting a state change, then you could give the former a rating of 1 and the latter a rating of 0.1.
And this is where the Custom Channel Groupings as outlined in the Search Engine People article are so useful. You can’t for example, go through every single referring site and mark each one on this scale. Instead you merge them up in to groupings meaningful to your business then provide a rating for each group. As they suggest, I would certainly split branded and non-branded PPC Ads in some way (perhaps based on the Ad Group names, if a good convention is used?), but you could also split up types of referrers, types of email and so on.
And this is why I titled this article “When Marketing isn’t really Marketing”. Can you really call it “marketing success” in, say, SEO, when a customer types in “Ben’s AVSoft” and comes straight through to your page? Of course some earlier bit of marketing activity was phenomenally successful because the customer must have heard of you somewhere – may be a conference, or a blog post, or from a colleague. But each of these activities were what I would call “successful marketing” – they’d effected a state change in the mind of the customer to bring them closer to your product. The search for “Ben’s AVSoft” wasn’t marketing at all. If anything it was just a semi-technical piece of work to make sure your Google result looked bearable (because it’s not hard to get high ranking for a search on your exact product name – as long as you don’t call your product “Taylor Swift”).
And, just for the record, here are my strawman ratings for some of the most important touch points that we use:
Quite a simple post this time round. Essentially how I measure the success of a given campaign or piece of marketing work. NB: This isn’t the Holy Grail of properly attributed marketing ROI – when I’ve worked that, I’ll post it up, if I haven’t retired first – but instead a framework for how to show whether what you’ve done has worked or not.
The model consists of three stages – Inputs, Outputs and Outcomes. I’ll describe these below. For the article I’ll use an example campaign – we’re launching a new product, say a tool to monitor your website performance – and we’re using a number of tactics to promote this – a free, downloadable iPhone app, a new website, some content marketing, some Adwords, social media etc etc – the usual suspects.
The first part, often forgotten about, is to measure the Inputs to your campaign. By this I mean – what is the quality of execution of the campaign? Before initiating campaign work there is, generally, a long process of research, analysis, customer feedback, market segmentation etc which gets distilled in to some sort of proposition or idea. For the web performance tool, you might have carried out all this work and ended up with a campaign idea around “Keep an eye on your global website performance from your mobile phone” (or hopefully something stronger than that!). You’ve chosen a segment – say IT managers of smaller companies in the US – and you have a lot more information about the feelings you want to elicit, the differentiators you want to promote and so on.
But – that doesn’t mean to say all that great information gets in to your campaign. Woody Allen has said how he measures his “best” work not in terms of quality or audience response but in terms of how close the final film is to his original vision. Similarly, if you went in to a campaign with a particular value proposition and a really clear idea of what you wanted, is this what you ended up with? This is the measure of Input quality.
In this example, if the agency you used came back with a campaign that captured everything you wanted, the right tone, the right target audience, channels, content etc etc, then you’ve scored highly on Inputs. NB: This is very much a qualitative measure – you can try giving a campaign a mark out of 10, but what does that really mean? The way I see it, if I sit there completely satisfied thinking “That’s exactly what I meant”, then I’m happy.
Another part of measuring Inputs, which I sometimes carry out, is to look at a few quantitative KPIs covering the basic measures for parts of the campaign. For example, with Adwords, how many impressions did we actually get (i.e. it’s not whether the ads were effective or not, just whether we’ve correctly placed them and bid appropriately)? How much are we spending on different ad campaigns? And so on.
NB: This does not, of course, mean it will work! Your research and analysis could be completely wrong – wrong segment, wrong message, bad ideas. But at least you can go in to the campaign confident that you’re giving it your best shot. Nevertheless, any boss worth his/her salt will want to know “Did it work?”
There are two stages to answering the question “Did it work?” – Outputs and Outcomes. Outputs is the first of these and is the measurement of the immediate KPIs that you’re tracking for the various parts of your campaign. I.e. We changed some Google Adwords – are more people clicking on these now? We changed the website – have we increased the conversion rate? In our example promotion, we created an iPhone app to promote the product (perhaps the app shows a demo for our company’s website). We might have loved the app (scored high on Inputs), but how many people actually downloaded it!? And used it? And our new website – how many visits have we seen? Time on page? Conversion rates? And so on.
For a given campaign, there might be 10, 20 or more of these KPIs, often at quite a low level – they provide immediate feedback on what is and isn’t working and, ideally, should be used within a Plan-Do-Check-Act (PDCA) framework to continuously improve everything you’re doing.
NB: Outputs is the area that is almost never forgotten about, mainly because it’s the easiest to measure (who would put up a new website and not check Google Analytics to see how many clicks it’s getting!?)
Great, so the campaign looks good, you’re getting clicks and downloads but, the question any exec should be asking you is, so what? What is the impact of this on the bottom-line? On revenue? Outcomes is the measurement of the final impact of the work you’re doing. In a commercial organisation this is likely to be revenue. However, if you’re working for a healthcare charity, this might be something like “Awareness of a particular ailment” or if working for the government, “Awareness of new policy X”. Or even for a commericial organisation a campaign may be something like “Awareness of our brand”.
But, it’s essentially the single figure that you care most about, that is the driver for your organisation. Unlike Outputs, there should really only be one of these values. And it should hopefully be simple to measure. What is very difficult however is attributing changes in this figure (say, revenue, in our example case) to any particular part of a campaign. As I’ve mentioned, this is a Holy Grail for marketing – showing that “My work I did on Adwords last month, which is costing us $1,000 per month, directly led to $10,000 in revenue – an ROI of 900%!”. So I don’t generally try to do this. Instead, I just look at the revenue trends moving, hopefully!, up over the months following the campaign. This measurement is full of caveats – can you really attribute the increase in revenue to your campaign? Could be market movements? Could be new product features? Could be noise! But this doesn’t really matter – you still need to be looking at this figure. It’s often the case that, sadly, it’s very hard to pick up increases in revenue attributable to particular campaigns, but the figures need to be shown nonetheless.
And that’s it – as I say, pretty simple! One qualitative measure (Inputs), multiple low-level KPIs (Outputs) and a final big KPI (Outcomes).
I really like this very simple post about how to buy wine, when you don’t really know much about it (which is definitely me). Basically, select a price (say, £7.95), then select a wine at that price. That’s kind of it. And what Evan Davis is saying is “Statistically, give or take anomalies, most wines at £7.95 will be about £7.95-ness in quality”. The article is written from the consumers’ perspective but if you view it from the vendors’ point-of-view, he’s saying “If you want to charge £7.95 for your wine, you’d better make it worth around £7.95 in value”.
Though this is a massive over-simplification of the process of coming up with a product, researching that product, marketing it, selling it effectively etc etc, it is, for me, a fundamental approach to product marketing. If you want to add 50% to your bottom line, then, somehow, you need to be adding 50% more value to customers this year compared to last year.
I’ve written previously about the difference between big-M Marketing and little-m marketing – the difference between finding new markets, new people, researching what people want, creating propositions etc and the less impactful role of just tweaking Adwords, designing flyers and so on. I think the two most effective bits of work that a Product Marketing person can do, under the banner of “Big-M Marketing” are:
An example of the latter is lower-alcohol wine mixers. There’s a large section in Sainsburys selling pre-mixed Buck’s Fizz, Kir Royale, Bellini and so on which barely existed 5 years ago.
Some bright spark has thought “Hang on, rather than forcing someone to find a cheap Cava then try and find where on Earth the cassis is displayed, let’s just mix them and sell them at a great price!”. It’s clever – if somebody has to drive, but doesn’t just want to drink orange juice, they’re very low alcohol and great too.
If you were a wine vendor, and a marketing person in your org came up with that idea, there’s a pretty good chance you could increase your revenue by a reasonable margin that year (assuming you get everything else right!). Particularly if you were already selling all of your current wines in the right places, with the best labels, at the right price, in the right combinations (if relevant).
Of course another way the wine company could make more revenue is to improve the wines they currently offer. Obviously, easier said than done, but if they can come up with the processes and methods to improve the grapes (I suspect “Having better hillsides in France” might be part of it) and the wine thereafter, they’re directly improving the value of what they offer – and therefore, by Evan’s simple economics, can charge more for their wine. NB: I don’t think much market research is needed here – I can say pretty confidently that a better tasting wine is what most customers are looking for..
This doesn’t mean, of course, that there isn’t revenue to be obtained from great presentation and promotion. If both of the following labels were on £7.95 wines, which are you most likely to buy:
And of course there are countless examples of great marketing adding enormous wealth to a company’s coffers. The one that comes to find is Calvin Klein underwear. Add your brand name to a pair of pants and watch your business soar! And there are lots more examples like this where, fundamentally, the product offering hasn’t been improved at all (were the pants really any better for Klein putting his name on them?) and the marketing push has had an amazing impact.
But – I think this sort of success is just far less predictable than success borne of product improvements. These are shots in the dark and for every Tango Orange ad, there are a string of forgotten, and often very expensive marketing failures. Also, I’ve rarely seen examples where just re-arranging product combinations or thinking of some clever re-presentation of the core value has made any sort of lasting impact.
In contrast, I believe adding new, researched features to your product offering is the most predictable way of increasing revenue. I.e. by adding value to your offering, you can charge more for it and make more money. Simple! Joel Spolsky said this a long time ago:
With six years of experience running my own software company I can tell you that nothing we have ever done at Fog Creek has increased our revenue more than releasing a new version with more features. Nothing. The flow to our bottom line from new versions with new features is absolutely undeniable.So if you have a choice between spending £100,000 on finding and implementing great new features for your products, or spending that on a new advertising campaign, though the ad campaign may produce some magic like it did for Tango, the more predictable revenue will come, I think, from adding some real value for your customers.
The book Competing on Analytics by Thomas Davenport and Jeanne Harris is a short but very interesting read about the need for organisations to significantly improve their analytical capabilities if they want to compete in the modern marketplace. The argument, quoting directly from the author is that:
In today’s global and highly interconnected business environment, traditional competitive differentiators-like geography, protective regulation, even proprietary technology-are no longer enough. What’s left is the opportunity to execute a business with more efficiency and effectiveness than your competitors, and to make the smartest business decisions possible. Analytics can help do this.I.e. unless you are implementing and using advanced analytics, you’re going to be left behind because you can’t use some of the traditional differentiators to keep any sort of advantage. NB: I don’t discuss Big Data in this post at all, as it’s more about the why rather than the how. But probably a post there for another time..
One of my favourite quotes from the end of the book is:
Analytical competitors will continue to find ways to outperform their competitors. They’ll get the best customers and charge them exactly the price that the customer is willing to pay for their product and service. They’ll have the most efficient and effective marketing campaigns and promotions. Their customer service will excel, and their customers will be loyal in return. Their supply chains will be ultraefficient, and they’ll have neither excess inventory nor stock-outs. They’ll have the best people or the best players in the industry, and the employees will be evaluated and compensated based on their specific contributions. They’ll understand what nonfinancial processes and factors drive their financial performance, and they’ll be able to predict and diagnose problems before they become too problematic. They will make a lot of money, win a lot of games, or solve the world’s most pressing problems. They will continue to lead us into the future.What a great place to be! But of course it’s not as simple as that. So below are a number of tips I’ve picked up from trying to implement this sort of thing over the years, or watching clients trying to do this as well.
1. Fit the implementation to your real needs. One of the first things to note is that a lot of the examples of success that they give in the book – Amazon, Netflix, Google and so on – are big companies. These organisations have millions of customers and, at that scale, the benefits of well-embedded customer analytics are obvious – Tesco Clubcard is another great example of significant profits generated through analytics.
So firstly, I think it’s a bit of a struggle implementing many of their ideas when you’re running at a much smaller scale – even if you thought it beneficial. One of the companies I worked at a few years ago had a total potential market of precisely 29 companies. If we wanted to know anything about these companies we didn’t look to data analysis to understand patterns of behaviour, we went to see them and asked!
2. Start Simple. A second more subtle point here though is about making sure you pick off the low hanging fruit before moving on to anything advanced. One of the questions I was asked at a job interview a long time ago now, was to do with detecting fraudulent activity on bank accounts. The question was “If we gave you the data showing the amounts going in and out of a bank account each day for the last 3 months, what’s the first thing you would do to try and spot fraudulent activity?”. I’d just finished a maths course 2 months before, so I launched in to a tirade about algorithms for picking out outliers, spotting complex patterns in the data, weekly seasonality calculations, de-trending the data and so on. After a few minutes of this, the interviewer interrupted and said, “Mm. I’d probably just draw a graph and see what was there”. I did manage to get the job in the end and found out that a reasonable proportion of the “advanced analytics” needed to detect banking fraud was really very simple algorithms to spot outliers, pretty close to what you’d be doing by eye (think “3 standard deviations away…”).
An example from the world of marketing – you might be trying to figure out “What type of customer in our CRM system is more likely to stay with me long term, continuing to buy products? (i.e. high LTV)”. There might be a lot of subtle factors here, but there could be some real no-brainers. For example, I’d suggest that customers who have spent a lot with you in the first 3 months are more likely to spend a lot in the future (because the initial spend is indicative of certain levels of budget and/or appetite for your products) compared to someone who spent very little. Of course you have to test this in the data (because it could be completely wrong – see the quote at the end about the blocks to implementing analytical thinking), but if you wanted a simple model for “Which accounts should we spend more time with?”, a simple binary model of “Spent more than $100k with us” vs. “Spent less than $100k” might be a good first start!
3. Get some early wins. There is always a (not completely unreasonable) objection to analytical marketing along the lines of “This stuff is all well and good, but if we spent more time just putting together some great ads, some great messages, reaching out to the community, running some great promos etc, then the money will flow. We’ll worry about analysing the detail later on”. Sometimes this comes from a certain mindset (again, mentioned in the quote below), about “the power of ideas over data”. Sometimes it comes from seeing failed implementations. I have a lot of sympathy with the latter – it is very easy to spend an enormous amount of time and money on this sort of project – time that could be spent elsewhere in the business – and see precisely no advantage at the end.
One way of combatting this problem is to make sure you get some early wins, even if these aren’t the primary purpose of your project. For example, you may have a vision of an all-encompassing CRM system that knows exactly the right email to send at the right time to the right customer (“Dave, we know you’ve been enjoying our product for 19 days now and that your boss, Helen, is interested in how the product could help her with regulatory problems in the textiles industry – here’s a whitepaper answering her questions and a quote for a price that I know exactly fits her budget for this quarter.”). But if you wait for utopia, without showing some earlier, simpler wins, you’ll be working for years battling off disgruntled executives with other priorities.. So better to pick an early problem where you know you can win. If you’ve never sent any segmented emails at all, then start with a simple opportunity (like sending different emails to new vs. existing customers based on some analysis) then prove that this has had a positive impact on either outputs (such as click rate) or (much) better still, revenue. Once you’ve proven positive return, then move on to the next stage with something a bit more clever. I’d use the phrase “Build success on success”, but it sounds far too cheesy.
4. Analyse off-line first, on-line later. There’s a great model for step-wise implementation of CRM analytics, along the lines of:
a. No analytics – purely transactional CRM system.
b. Offline analytics (e.g. showing that you need to treat new and existing customers differently), implemented manually (sending emails by hand once a week).
c. Offline analytics implemented automatically (e.g. above analysis embedded in to CRM/email system such that different customers automatically get the right emails. Or offline analysis shows that certain types of customers prefer particular products/prices, so these are hard-coded in the system),
d. On-line analytics, automatically adapting based on data coming in. For example, your initial model might say that “Customers who come in from Northern Europe are more likely to purchase product X” but, as data comes in, you find this changing over time. The models in the CRM automatically pick up that Southern Europe is now more likely to purchase product X, so automatically adapts to offer this product to these customers instead.
The last options is, for most, a complete pipe-dream, and can actually be quite dangerous. Therefore I’d strongly recommend a model where analysis is carried out off-line, on sets of data, then the conclusions from that analysis are proven online first. E.g. your offline work shows that customers in China are more price-sensitive than those in India? Implement this by just hard-coding different prices for the two countries and seeing how you get on..
5. Ignore all of the above if you’re working in a SaaS environment! A lot of the caution above stems from trying to run before you can walk. As I say, if you only need to sell 2 widgets this month to break a profit, your time might be better spent phoning up all of your potential clients, rather than analysing their behaviour to the nth degree. But – the SaaS world disrupts this approach. If you’re not analysing behaviour, interactions, price sensitivity, churn rates, click-through rates, conversion rates etc etc etc from the start, then you’re putting yourself in a very precarious position. Customer service and sensitivity to customer needs is everything in the SaaS world and if customers aren’t getting what they want, they’ll leave you and move elsewhere (the downside of easy adoption is easy rejection).
Hopefully some of these tips are vaguely useful. With regard to the book – it describes some interesting principles, and is good at a certain level (quite high level), though is a little short on detail. Nevertheless, I thought I’d end with my favourite quote, about the factors that hinder adoption of advanced analytics in organisations:
Gary cites four common factors that hinder analytical competition: deeply embedded conventional wisdom that has been around for so long, it’s hard to reverse; decision making-especially at high levels-that fails to demand rigor and analysis; employees themselves who are not willing or equipped to do analytic work; and the power of ideas over data.
Anyone who’s worked in a digital marketing environment will probably recognise one of the following two scenarios –
1) You’re merrily working through your day when you check up on a KPI graph that normally bobs along nicely only to see some unpleasant looking change (generally a “drop” of some sort). Panic.
2) You’re merrily working through your day when who should turn up at your desk but your boss, looking less than cheery, holding up a sheet of white A4 with what looks like a graph on it. With a drop on the right hand side. Panic.
Of course scenario 1 is for those people who have better monitoring/dashboard systems, but the end result is the same – why has that KPI suddenly started slipping off the edge of the comforting plateau where it’s been sitting for the last 6 months?
NB: This is certainly more likely and relevant for things like “web site visits” or “whitepapers downloaded” than slower moving items, but the principle is still the same.
For me, the important question is “What do I do now?”. If it helps, here’s what I did when this happened to me two days ago…
1) Firstly, don’t panic – I don’t know about you, but, unless your whole website is down, I’ve never found panicking help to get a problem resolved. It just muddies your thought processes.
2) Communicate with the people who care about the figures. If you found the problem, there’s nothing to be gained from “trying to fix it before anyone notices”. They will notice, and it’s better coming from you.
3) On to the more practical help – in my example, we spotted that our leads generated from the website (from people downloading the free trial) were down around 20% for a specific product. Usually the daily run rate for leads is pretty steady, so it was surprising when this figure was consistently down for a few days.
4) Don’t panic (again). Here is a non-exhaustive list of reasons why this specific KPI could be down, in an order which I hope makes sense:
* Customers are no longer interested in what we sell,
* We’d change something on the structure of the site which de-emphasised this particular product,
* We’d make it less clear how to get hold of the free trial (e.g. a design change to the page),
* People are finding the free trial but there’s a bug which means when they click to get it, they don’t,
* There’s a bug that means when they click to get the trial, they do get the trial, but we’re not recording this in our CRM system,
* Everything is working, it’s just that the analytics cube which we’re looking at isn’t functioning properly (i.e. it’s a reporting problem)
…and many other possibilities. As you probably spotted this list is ordered from “Our company is in trouble” to “Phew!”.
5) Analysis. As I say, there’s little point going through the details of how we found which of these was the problem, because every situation is different, but the most important tip, I think, is to take a methodical approach. If your symptom is “The leads shown in our cube are looking a little low” then the cause could be any of the above. You have to go through each in turn, discounting that option. As an example, my first thought was to discount the first, rather worrying possibility. I looked at the Google Analytics visitor numbers for the whole of our website and for the specific product in trouble. No change – within seconds I saw a re-assuringly unwavering number of visits for the immediate past.
6) Don’t jump to conclusions. And be nice to people. In almost every situation I’ve ever been in like this, the source of the problem is pretty far down my list – and it’s likely to be a technical problem (i.e. somebody didn’t test something fully). Going in with your size 13s, making unresearched accusations and being unpleasant about it turns a solvable problem into a horrible week for a lot of people – if someone has made a mistake they’re going to be feeling pretty unpleasant about it anyway.
7) Obvious really – fix it!
8) Learn from it – it could be a 100 things. Is it the testing process? Does your monitoring process need to be more finely-tuned? Do you need to slow down on making too many changes at once? It’s obvious and a cliche, but if you don’t learn from it, you’re doomed to repeat the mistake.
The above is all well and good, but a reasonable question is – so what? Isn’t this blog supposed to be about marketing? This just sounds like a data problem?
The main reason why I think it’s important for marketeers to be able to help with fixing this sort of thing is – you are the owner of those metrics. If something goes wrong, it’s up to you to get it sorted. You’re also most likely to understand the ins and outs of how the website and lead generation process works – just handing tasks like this on to someone else and saying “I don’t get what’s going on, let me know when it’s sorted”, is only going to lead to slower results.