Select Page

Top 10 Challenges Data Scientists Face at Work

by | Apr 24, 2019 | Crash course on privacy, Data privacy, Data Privacy Solutions, Privacy Courses

We all have heard that “data is the new oil”. As with oil, data has to be transformed to be of real value to society. The people in charge of this transformation are data professionals.

Data professionals are constantly trying to make sense of data by building models that can provide the insights necessary for organizations to grow and generate more value. However, these professionals face many challenges that prevent them from building powerful models.

In 2017, Kaggle did a study titled the “State of Data Science and Machine Learning”. One of the questions the survey asked was, “At work, which barriers or challenges have you faced this past year? (Select all that apply)”. Here are the top 10 results:

Here is a look at how often they encountered these problems:

  Most of the time Often Sometimes Rarely
Dirty Data 43% 40% 16% 1%
Lack of data science talent in the organization 31% 40% 27% 2%
Company politics / Lack of management/financial support for a data science team 26% 40% 30% 4%
Unavailability of/difficult access to data 28% 42% 27% 2%
The lack of a clear question to be answering or a clear direction to go in with the available data 29% 43% 27% 2%
Data Science results not used by business decision makers 16% 44% 37% 3%
Explaining data science to others 19% 41% 36% 3%
Privacy Issues 25% 36% 34% 5%
Lack of significant domain expert input 22% 46% 29% 3%
Organization is small and cannot afford a data science team 37% 36% 24% 3%

Data cleanliness is clearly a big issue, as data scientists spend 80% of their time cleaning data. However, challenges, like a lack of talent/expertise, company politics meaning results are not used, and data inaccessibility, are more difficult to solve as they require systemic changes within the organization.

To find how data professionals answered the other questions in the study, click here to visit Kaggle 2017 study.

Join our newsletter