Top 10 challenges data scientists face at work
We all have heard that “data is the new oil”. As with oil, data has to be transformed to be of real value to the society. The people in charge of this transformation are data professionals.
Data professionals are constantly trying to make sense of data by building models that can provide the insights necessary for organizations to grow and generate more value. However, these professionals face many challenges that prevent them from building powerful models.
In 2017, Kaggle did a study titled the “State of Data Science and Machine Learning”. One of the questions the survey asked was, “At work, which barriers or challenges have you faced this past year? (Select all that apply)”. Here are the top 10 results:
Here is a look at how often they encountered these problems:
|Most of the time||Often||Sometimes||Rarely|
|Lack of data science talent in the organization||31%||40%||27%||2%|
|Company politics / Lack of management/financial support for a data science team||26%||40%||30%||4%|
|Unavailability of/difficult access to data||28%||42%||27%||2%|
|The lack of a clear question to be answering or a clear direction to go in with the available data||29%||43%||27%||2%|
|Data Science results not used by business decision makers||16%||44%||37%||3%|
|Explaining data science to others||19%||41%||36%||3%|
|Lack of significant domain expert input||22%||46%||29%||3%|
|Organization is small and cannot afford a data science team||37%||36%||24%||3%|
Data cleanliness is clearly a big issue, as data scientists spend 80% of their time cleaning data. However challenges like a lack of talent/expertise, company politics meaning results are not used, and data inaccessibility, are more difficult to solve as they require systemic changes within the organization.
To find how data professionals answered the other questions in the study, click here to visit Kaggle 2017 study.