02 Dec 2015

Big Data: What's holding it back?

As big data moves from internal conversation to implementation, some adopters are being forced to rethink the personnel required to deliver these initiatives successfully.

As a recent TechTarget article states, this might include decisions on whether big data sits inside or outside of internal IT, and what functions of big data are best left to external staff versus being outsourced.

Some of the recommendations made by IT executives surveyed for the article include that big data initiatives be broken into bite-sized chunks that can be controlled by an internal person, and that the engineering of any big data platform – coming up with the algorithms and models – also remain an internal-only domain.

Recent research by TMForum suggests that “lack of skills ranks among the top three inhibitors to a successful big data strategy.” (The article offers a useful checklist of essential ingredients for big data success).

One of the challenges with sourcing big data skills is that it may not be obvious what skills are required, particularly at an early stage of the project.

As McKinsey & Co points out, employing “number crunchers” may not be a path to success by itself.

Rather, the consultancy recommends finding what it calls “translators” who are capable of working cross-functionally and “bridging different functions within the organisation”.

Data scientists are probably the best example of a “translator” – bridging the world’s of IT and analytics, according to a McKinsey Venn diagram – but there are other examples.

What McKinsey is sure of, though, is that you’re unlikely to be able to get away with one “translator” who can cut across all the functional areas that are required.

“Looking for a single translator at the right intersection of all the various skills you need is like looking for a unicorn,” it said.

“It’s more realistic to find translators who possess two complementary sets of skills, such as computer programming and finance, statistics and marketing, or psychology and economics.”

Of course, it may be the case that you have the right mix of people in place, but maybe they aren’t asking the right questions of the big data.

A recent advice column published by InformationWeek shows six ways to ask smarter questions and therefore improve the insight you’re able to get from the big data.

In the future, the question is whether we will even need people at all to be successful at big data.

MIT, for example, is already working on a system designed “to take the human element out of big data analysis”.

A prototype of the system can already beat the insights from two-thirds of human teams it competes with in competitions.

That has convinced other researchers of its efficacy, suggesting it could solve big data’s personnel challenges before some even have to consider them.

“I think [MIT’s innovation] is going to become the standard very quickly,” Margo Seltzer, a professor of computer science at Harvard University, added.