Rebel Thinking

How to avoid creating a machine learning monster

Juan Pablo Rubio

20 December 2018

181219 frankenstein wp 1

Nowadays it’s not difficult to find plenty of programmes, agencies or services selling, or at least promising their consumers, products built with Artificial Intelligence. It’s a buzzword that sells – but what is it that we’re really buying into? This tweet by Mat Velloso pretty much sums it up:

When it comes to AI, we have to be cautious. In fact, for this article we’re better off discussing Machine Learning, a concept much more rooted in ‘reality’ than AI.

So what is Machine Learning?

The easiest way to define it is as a set of algorithms used by a machine to improve performance. Basically, it is a field of computer science that gives computers the ability to learn without being explicitly programmed to perform a specific task.

I’m not going to deny the benefits of Machine Learning. I personally love it and try to learn more about it every day. However, there are a few limitations to Machine Learning that we should keep in mind.

Let’s take a look at another insightful tweet, this time penned by Kyle Byers:

In a very elegant and concise way, this tweet explains what would happen if we were to feed our algorithm the wrong data (either by mistake or on purpose). This is where the problem starts. For Machine Learning to work, the machine in question depends on ‘high quality data’, and it relies above all on the human programming it.

How do we know if the data we’re supplying is inaccurate or incomplete? Or worse, how do we know whether or not the data we’re supplying is actually useful? Is it helping us to accomplish our goals? And of course, we also have to consider how we’re weighting that data.

If we don’t take the data we’re feeding into our algorithm into consideration, we may end up creating something other than just a ineffective algorithm. We might end up creating a monster.

I’m not bad, I’m just programmed that way

“If we give these systems biased data, they will be biased.” – John Giannandrea

Bias in Machine Learning has caused a lot of problems. Google’s ‘sentiment analyser’ thinks that homosexuality is a bad thing (a result of feeding their model inaccurate data), airlines are using an exploitative algorithm to split up families and couples in order to force them to pay extra to guarantee a seat together (a result from a lack of human centricity), and in America, crime prediction algorithms are wrongly labelling black defendants as high-risk at twice the rate of white defendants.

When it comes to digital marketing, there are some more specific limitations to consider.

Client retention vs. client acquisition

At Good Rebels, we’ve experimented with a lot of different services designed to optimise digital campaigns using Machine Learning. These services tend to optimise campaigns based on performance, and the algorithms used usually create audiences based on whichever segment has the best result. This approach helps us to improve any KPI we like: bounce rate, session duration, pages per session, conversion rate and cost per conversion. Everything is wonderful! Except – we’re only focusing on retention. The audiences that perform best are usually those with clients who have already bought our products or used our services, and were already very likely to repurchase, resubscribe, or reuse our services, regardless of whether or not they were advertised to.

Utilising Machine Learning for client retention is incredibly effective, but not necessarily efficient. At the end of the day, it’s not the fault of the service provider – they’re just doing the job they’ve been asked to do, with the data made available to them. The problem really lies with information silos – those organisations who are trying to optimise campaigns usually have a huge blind spot.

Let’s imagine we create a similar algorithm, but this time with a more specific goal – not only to optimise our campaign based on cost per conversion, but also to optimise our campaign based on cost per conversion while maintaining a balance between old clients and new.

Cost, quality and time

Another example of algorithmic racial profiling comes from COMPAS – a tool developed to identify potential reoffenders in the criminal justice system. COMPAS is a complex tool that predicts the likelihood of recidivism on the basis of 137 characteristics. The curious thing is that COMPAS could have achieved the same level of accuracy based on just 7 characteristics. In fact, even more astoundingly, they could achieve a very similar level of accuracy based on just two characteristics.

There’s a famous quote most people in our industry are familiar with: “all models are wrong; the practical question is, how wrong do they have to be to not be useful?”.

It’s clear, when we examine tools like COMPAS, that some Machine Learning systems are really just a waste of our time and money. Often, we could achieve 80% of the same result with just 20% of the effort invested.

You could argue, in the case of COMPAS, that every single characteristic counts if there’s even a small chance of accusing someone of a crime they didn’t commit – but that’s a whole separate issue.

So, what’s the solution?

Whatever your approach to Data Science – the first two steps are always the same – you need to understand the business you’re working for and you need to understand the data you’re working with.

Firstly, you need to make sure that you, as an organisation, know exactly what it is that you want, need, and expect. You also need to have a realistic idea of the resources you have available.

Secondly, if you’re going to outsource to a tool or service provider that works with data, make sure they understand your business needs, your brand, your values, goals, clients, service offerings… everything. And, of course, make sure they have access to and an understanding of the data you have available.

Finally, remember this: a good model supported by quality data is better than a great model built on bad data.

Let's talk