BigData in HR: Why it’s Here and What it Means


The talk about BigData is getting louder by the minute. As companies shift their core systems to the cloud, more and more people-related data becomes available. This, coupled with the tremendous focus on BigData in the technology sector, has created a huge focus on data driven decision-making.

Why Analytics is Coming to HR

If you think about the history of analytics in other business areas, the evolution looks like the chart below. When companies started industrializing their manufacturing, they eventually purchased ERP software and developed supply chain and financial analytics.

In the 1970s and 1980s companies started to industrialize their customer marketing and analysis, and we started to focus on "the market of one." This led to a tremendous explosion in CRM and sales analysis systems, which today has become a huge industry in customer segmentation and marketing analytics.

Now, given the global recession and talent imbalances in the world, companies are focusing on replacing their legacy HR systems to help apply analytics reasoning to HR and talent. As the chart shows, in each of these evolutions we started with reporting and core understanding and then moved to predictive analytics. This is what is happening in HR.

Fig 1:  The Inevitable Shift to HR and Talent Analytics

The Tools Market

A major part of this shift is the explosion in new tools. Today nearly every major HR software vendor (Oracle, SAP, ADP, Ceridian, Workday as well as specialty companies like SumTotal, Cornerstone, Lumesse, Silkroad, Ultimate, and hundreds of others) are building and buying end-to-end HR suites, very similarly to the evolution of customer relationship marketing. The clear next step is for these vendors to launch tools to help you analyze and segment this data.

Today Oracle (OBIA), SAP (Workforce Intelligence), Workday (BigdataAnalytics), and SumTotal have all launched major integrated HR analytics systems. Mercer has their own solution and a variety of smaller vendors are delivering amazing analytics systems as well (Visier comes to mind).

Oracle's Fusion HCM even has a predictive analytics engine which tries to compute potential talent problems for you, right out of the box. SAP's offering, built of years of experience from InfoHrm, will do the same.

These tools, which are focused on meeting the needs of HR and managers, have to handle a lot of workload: analyzing the hundreds of standard reports which HR managers need, plus bringing together non-HR data to look at people-related business performance. (Our new HR Measurement Framework helps sort out all these measures.)

And IT departments are now looking at a variety of IT-driven analytics tools (companies like Platfora, Splunk, and dozens of others) which run open source parallel systems like Hadoop and MapReduce to help you combine internal business data with social, location, and many other sources.

The Consulting Market

Since data analytics is a very multi-disciplinary problem, there is also a tremendous growth in consulting offerings in this market. Mercer, for example, launched its iknow platform (a toolset based on MicroStrategy), Deloitte has a major practice in data analytics across all business functions, and IBM is putting together a wide variety of analytics services (to also leverage Cognos and SPSS, two tool sets now sold by IBM). Driven by the tremendous growth here, we can expect many more consultancies to move in this direction.

I personally believe that BigData and talent analytics will be a consulting-driven market for the next few years. Despite the amazing set of tools now available, the real challenge is process and expertise. Many large IT-driven data warehousing projects fail, simply because the process is complex. So one would be advised to work with an experienced consultant before you try to do this alone. Many companies are building this expertise internally, but when the problem is big we advise using experts.

I liken this space to building a house or office building. While the tools are all available, projects succeed or fail based on the strength and experience of the architect, general contractor, and trade skills. The same holds true here. You will need someone with architectural data warehousing experience. 

Bottom line to all this: the time is ripe for BigData applications (also called HR Analytics or Talent Analytics) to explode in the next few years.

(By the way, if you want to really dive into the BigData market, check out the O'Reilly Strata Conference. Many of the technology and implementation thought leaders appear here, and it's an excellent way to stay up to date on BigData across all areas of business.)

The State of HR Organizations

That all said, most HR teams today are not ready for this evolution. Our research shows that only 6% of HR departments believe they are "excellent" in analytics and more than 60% feel they are poor or behind. Over the next few years we are going to see major investments in this area in all large companies, with huge returns on investment from the effort.

We are just introducing a whole new research subject area on HR Analytics and Measurement and already have many tools, reports, and experience to share. (If you're interested in becoming a Bersin members, you can join our BigData working group.)

Let me share some of the key findings we've already uncovered as companies dive into this market in 2013. Here are three best-practices we've discovered through our dozens of interviews and data collection this year.

1. Start with the problem, not the data. 

We are all flooded with data: employee data, location data, social data, compensation data, and much more. If you start an analytics project by collecting all the data you can find, you may never come to an end.

Rather you have to start with the problem: What big decisions would you like to be able to make? What problems would you like to solve?

One common talent problem, for example, may be sales productivity. What factors contribute to a predictable high-performing sales person? Every company would like to understand this better. And once you understand these characteristics, how can you better source, attract, and hire such people? Another may be turnover. What factors contribute to high turnover in your company and in particular groups?

These questions are worth millions of dollars to answer. A company I just met with developed a predictive analytics model for turnover in their restaurants. Did they use Hadoop or parallel databases? Nope, they used Excel. But they had a very smart statistician working with a very senior manager to come up with the hypothesis. And from there they explored all the possible data elements that might contribute to the answer.

Be careful you don't start by only looking at data. It leads to lots of money spent, systems built, and often little or no return.

Over time your analytics platform will grow, and our BigData maturity model describes how to do this. But first focus on one or two key problems, so you can develop the credibility and skills to scale.

2. Data cleaning and building a data dictionary may be 80% of the work.

There is a quote by DJ Patil, the ex-Google data scientist, states in "Data Jujitsu," that 80% of a data science project is cleaning. In our business we clean a LOT of data, and I'd say there is a lot of truth to this. Until the data is clean and well defined, any analysis you do may be misleading.

Most HR data is quite dirty. Fields are filled with incorrect, duplicate, out of date, and inconsistent information. And you'll find one of your biggest challenges is clearly defining what various data elements mean (this is called building the data dictionary).

How do you want to define "turnover" for example? Would you include people who joined less than six months ago? How about part-time employees? What about people who left during the last day of the year?

These types of detailed descriptions can dramatically change what a given analysis might say. So a major part of a talent analytics program is defining the data itself, and bringing together definitions from across your different HR functions.

By the way, our research also shows that building a data dictionary is a cross-discipline project. You will need to access data which may include core HR data (date of hire, age, experience, educational history), recruiting data (pre-hire assessment, interviews), performance data (ratings, job assignments), training data (program completion, certifications, scores), and leadership data (leadership skills and assessments). And in some of the most powerful analyses you will want to access other data (location, manager) as well as business data.

So as DJ Patil points out (and most experienced data analysts know), without clean and integrated data the analysis will may be difficult. While there are many legacy tools which clean up data, ultimately this is a manual people-intensive process.

(This all takes place at Level 2 of our Talent Analytics Maturity Model).

3. HR Analysts do not have to be Computer Scientists.

The venture capital and tech community is very focused on new tools. (Hadoop, Mapreduce, and a barrage of new highly scalable open source tools.) These high-powered tools require people with a computer science degree (often called data scientists). And right now there is a lot of talk about how we don't have enough data scientists in the world.

In the case of HR analytics, you don't have this problem. Most HR analytics projects can be managed in traditional relational databases or Excel. If you have clean and well organized data, the volumes are not that vast. So the skills most companies need are statisticians and well organized analysts. And if you really do need a BigData infrastructure, you can outsource it (and your IT department is already working on it).

As I've told many people over the last few months, you have a big source of analysts in your own industry: people who studied I/O Psychology. Psychologists are trained in statistics, testing, and critical thinking. The I/O Psychologists in our company are so excited about analytics they can't wait to see more data!

And the critical skill these people really need is curiosity.

Great Analysts need Curiosity

One fascinating presentation by Nokia at the Strata conference (Breeding Data Scientists) talks about the fact that one of the most important skills in an analyst is "curiosity." I've seen that in our business as well. Of course you need people who know math and statistics. But then what really makes a great analyst is a curiosity nature to explore and "just look one more time" to see what may be going on.

Fig 2:  Nokia's Perspective on Data Skills Needed

This curiosity is what will bring you great decisions and models. And data analysis is always iterative – one question leads to one answer, which in turn leads to the next question. If you don't have that never-ending curiosity to dive deeper and deeper, it doesn't matter how much math you know.

BigData and Analytics will change HR

There is no question that HR will start to really "get" analytics in the coming years. Just as marketing, finance, and supply chain functions now rely heavily on analytics, so will HR.

And our research shows that high-value BigData projects in HR are not "HR Analytics" systems. They are business-driven analysis projects, built on an evolving set of infrastructure which grows over time.

But don't let the tools drive your program. The hardest work and most valuable results come from people: the questions you ask, the people you hire, and the process you use.

If you're working on a fascinating HR analytics project, we would love to hear from you.  We will have a series of BigData case studies at our IMPACT conference next April.  Read our BigData in HR research to learn more. We look forward to talking with you.

Josh Bersin

Josh Bersin writes on the ever-changing landscape of business-driven learning, HR and talent management. His favorite topics include strategic talent management, creating high-impact learning organizations, and how organizations drive business change and competitive advantage through talent strategy and technology.

17 thoughts on “BigData in HR: Why it’s Here and What it Means

  1. Pingback from

    What’s Trending Now In The Field of HR | daniellehhumanresourcesprofessional

Leave a Reply