The 5 most in-demand Big Data profiles in 2017

AI of Things    7 December, 2016
With Big Data falling off the Gartner Emerging Technology Hype Cycle this year, and Artificial Intelligence now hogging the limelight, we decided to take a look at the most in-demand data profiles we expect to see in 2017.
During the past 2 years we have seen the craze of the Data Scientist, however, as a greater number of companies in an increasingly wide range of sectors begin their journeys to data-driven – more and more different profiles are emerging. We spoke to our business unit and our HR team to get their insights and these are the top 5 profiles that cropped up in our conversations:

1. Data Engineer

High quality data is fundamental in the success of Big Data projects.  For this reason, we expect to see plenty of vacancies in 2017 for Data Engineers who have a consistent and perfectionist approach to data conversion and treatment. Companies will be looking for these data gurus to have extensive experience in manipulating information with SQL, T-SQL, R, Hadoop, Hive, Python and Spark. Much like Data Scientists, they are also expected to be creative when it comes to comparing data with conflicting data types to be able to resolve issues. They also often need to create solutions which allow companies to capture existing data in more usable data formats – as well as performing data modelling and design.

2. Data Visualisation Specialist

Data Visualization has become extremely important in ensuring data-driven employees get the buy-in required to implement ambitious and important Big Data projects in their company. “Data Storytelling” and the art of visualizing data in a compelling way has become a crucial part of the Big Data world and increasingly organisations want to have these capabilities in-house. Furthermore, more often than not, these professionals are expected to know how to visualise in various tools such as Spotfire, D3, Carto and Tableau – amongst many others. Data Visualisation Specialists need to be adaptable and curious to ensure they keep up with latest trends and solutions to tell their data stories in the most interesting way possible in the board room.

3. Big Data Architect

This is where the Hadoop experts come in. Typically, a Big Data architect addresses specific data problems and requirements, being able to describe the structure and behaviour of a Big Data solution using the technology in which they specialise – which is more often than not, Hadoop.

These employees act as an important link between the organisation (and its particular needs) and Data Scientists and Engineers. Any company that wants to build a Big Data environment will require a Big Data architect who can comfortably manage the complete lifecycle of a Hadoop solution – including requirement analysis, platform selection, technical architecture design, application design and development, testing and finally deployment.
According to this article, the primary skills and competencies to become a Big Data Architect are:
  • Marketing and analytical skills
  • RDMSs (Relational Database Management Systems) or Foundational database skills
  • NoSQL, Cloud Computing, and MapReduce
  • Skills in statistics and applied math
  • Data Visualisation, Data Mining, Data Analysis and Data Migration experience
  • Data Governance (an understanding of the interaction between Big Data and Security)
If you can tick all of these boxes, then you’re currently recruitment gold, with salaries showing steep year-on-year growth:

US salary trend
Figure 2: US Salary Trend for Big Data Hadoop Architect
If you’re keen to become a real Big Data architecture rockstar, then getting the right certifications in Hadoop and Apache Spark is strongly recommended to achieve a thorough knowledge of tools such as MapReduce, HDFS, Pig, Hive, Hbase, Zookeeper, Flume, Sqop, Cassandra, Scala, Mongo DB, Storm, Kafka and Ampala. Not too much to get your head around!

4. Artificial Intelligence Developer

The unquestionable hype around Artificial Intelligence is also set to accelerate the number of roles advertised for specialists who really understand how to apply AI, Machine Learning and Deep Learning techniques in the business world.  Recruiters will be asking for developers with extensive knowledge of a wide array of programming languages which lend well to AI development such as Lisp, Prolog, C/C++, Java and Python.
However, many speculate that this peak in demand for AI experts could cause a “brain drain for tech groups” with businesses poaching talent away from the world of academia.  Last month in the Financial Times, deep learning pioneer and researcher Yoshua Bengio, of the University of Montreal stated: “The industry has been recruiting a lot of talent — so now there’s a shortage in academia, which is fine for those companies, but it’s not great for academia.” It will be interesting to see how this conflict between academia and business is revolved in the next few years.

5. Data Scientist

The shift of Big Data from tech hype to business reality may have accelerated, but the shift away from recruiting top Data Scientists isn’t set to change in 2017.  A recent Deloitte report highlighted that the world of business will need one million Data Scientists by 2018, so if their predictions are correct, there’s a big talent gap in the market.  This multidisciplinary profile requires technical analytical skills, technical computer science skills as well as strong softer skills such as communication, business acumen and intellectual curiosity. These are outlined in this article and in our video which we published earlier this week on the LUCA blog.

With only one month until 2017 kicks off, we’re excited to see where the new year takes us and how innovation in Big Data and Artificial Intelligence will affect the global data job market.  If you think you have what it takes to help leading organisations on their journey to data-driven, keep an eye on our jobs portal here.


  1. Big data for making any project or presentation means a big problem for presenter or developer that how they can understand or use this big data. The data scientist or data engineer are the person who are made to handle all your big data problems, they can solve your problem by shorting up your big data into a summarize or useful form which is worth full and time saving for your project.

Leave a Reply

Your email address will not be published.