So, what’s holding us back? Is it the tech – it’s too expensive or not available? I don’t think it’s this at all. It’s been possible to implement ML for decades. You can use R, MATLAB, SPSS, SAS or a ton of other tools or if not those, Excel or even write your own (I have my own Mickey Mouse clustering app here as I got so frustrated using others’ tools). And people like Microsoft are making the tech more accessible all the time (e.g. Azure Machine Learning). So I don’t think that’s the problem.
My opinion is that the biggest shortage is in people who really understand ML, and can use it properly. And this is certainly what I’ve seen at customers and companies we’ve spoken to about this problem – they know they want to do it, but they just can’t get the people! Where are these mythical data scientists? Do they even exist? Could we afford them, even if they did? These are the questions we hear.
The key issue here is that, with ML, a little knowledge can be a dangerous thing. One of the problems with most ML algorithms is that they are complicated. Or at least, you need a significant level of understanding to know what you’re actually doing. If you take a dataset and run a Support Vector Machine over those data points in an attempt at classification, do you really know what the output means? When it gives an unexpected outcome, do you know why? Without jumping to easy (but often wrong) conclusions? Even something easier like k-means clustering – what do those clusters really mean? If there are three clusters, one big and two small, is that really saying something about the fundamental nature of your dataset, or is it just an anomaly because you haven’t transformed your data correctly beforehand?
These are difficult questions – having the tool available to, say, run a k-means clustering algorithm, is only 10% of the battle. Knowing what to do with that tool is the real issue; and how not to present something to your boss that any smart cookie could undermine in 10 minutes.
So finding these people is hard. It’s made even harder because I’ve often seen over-inflation with what people mean by “Machine Learning”. In my experience, someone who claims to “Know statistics”, is likely to be comfortable with value/volumes, perhaps a “mean” or a “standard deviation”, but not much more. Those who say they “Understand ML”, often have a good grasp of statistics, but struggle, when things get tough with machine learning. And of course, if you claim to “Know AI” – understanding a Support Vector Machine doesn’t mean you can build the next Skynet!*
So it’s even tougher finding people who really know their stuff. Maybe it’s just about money – supply and demand. If these people are hard to get, and you’re competing with the City to get them, then maybe you just have to pay the asking price and that’s it.
Or of course you can pick up a book and start learning! There are lots available – my favourite is Machine Learning and Pattern Recognition by Chris Bishop. A bit older now, there’s a lot in there, and you still need to implement this stuff, but it presents the material in a clear way and, most importantly, it helps you understand what you’re really doing with these algorithms. So at least your conclusions will be based on a deep insight, and not on guesses based on pretty looking graphs..
* I also have a “Skynet” project in GitHub. Progress is slow.