By Davide Fabrizio, Partner, Chief Analytics, IT & Consulting Officer at Conento
In this short post I analyze one of the main headaches of HR departments: the search for talent. This is intended as a reflection about the Data Scientist, currently one of the most highly valued profiles on the market. This professional must be able to combine technical skills with interpretive capacity and critical thinking, but the balance we can find is not always optimal or the desired one.
In search of critical thinking. The new Data Scientist between correlation and causality.
In Conento, where we are focused on Analytics and Big Data projects, we are always engaged in ongoing selection procedures for new profiles. Our focus is on finding talented Data Scientists. While it is true that every day there are more and more Data Scientists on the market, it is also true that the difficulty in finding profiles that fit our needs is increasing. Out of every 100 Data Scientists that enter our selection process, after making a first CV filter, only 2 are hired. We are talking about 2%, with a rate that has been decreasing over the last few years.
What’s going on? On the one hand, a more competitive market compels us to using more rigid selection criteria. On the other hand, there is the feeling that universities and institutions, with their different master programs in Big Data, Data Science and Machine Learning, are “generating” many Data Scientists who suffer from what I call “the correlation syndrome”.
This means that the new prototype of Data Scientist seems to have correlation as a priority, not causality. It seems that it is no longer interesting to analyze data asking the why of things or whether our results make sense. What matters is to get, as quickly as possible, a result with the most sophisticated Machine Learning algorithm: “hit the button” and see what comes out, without looking back. This situation is becoming commonplace in the practical tests that we provide to candidates in the selection processes, with our increasing amazement and disbelief. The lover of correlation has a blind faith in algorithmic logic -which, after all, is a nihilistic and totalitarian vision- relinquishing the “narration” of data and numbers.
It is curious to observe how there is an ever-increasing talk about an artificial intelligence with more and more efficient and precise algorithms, but which needs to be coupled with human intelligence, the only one -still- always able to analyze in depth the why of things, that is, cause-effect relationships. But, in the case we are analyzing, something different happens: it seems that the Data Scientist wants to follow the steps hand in hand with artificial intelligence, becoming a clone of it, that is, focusing his attention on mechanical and repetitive tasks and renouncing to bring real added value: critical thinking.
This deficiency actually reflects a new dynamic of modern society, which mixes new living and consumption habits, technology and educational models: the difficulty of having a vision of things that is not superficial is obvious, in a world of speed and continuous acceleration that leaves no time to look back, reflect and contemplate. Technology reduces distance and time, and this would allow us, theoretically, to free up time to think; but, instead of doing so, we prefer to fill this new space with “empty” activities, replicating indefinitely -like machines- mechanical processes with no real value: adding strangers to our social networks, reading and discussing about contents which do not contribute anything, checking our email compulsively…
Acceleration makes us lose the ability to follow a process of standard data analysis (as traditional statistics has always performed). Prior to launching the modelling stage, a thorough evaluation of the quality of the available data is necessary, a good construction of metrics and a descriptive analysis to capture first associations between variables. And, following the modelling, a careful evaluation of the results and calibrations in order to strike a balance between mathematics and logic. A progressive construction path of the model, with different stages that increase knowledge and understanding of the problem we are analyzing. It seems we are losing all this, and turning back is very difficult.
We are concerned because we believe that, beyond the technological revolution and social changes, something in the educational and training processes is failing. It is not easy to identify potential solutions (and this would be another debate), but the love of causality would have to be again the guide in our journey: the Data Scientist who combines this aspect with technical knowledge will succeed in the labor market of the future.
A surprisingly mild mid-November morning in Dublin welcomed the start of ESOMAR Fusion as delegates from over 40 countries descended upon the Irish capital to understand how as an industry, we can marry up the learnings from qualitative and big better to create a better tomorrow.
For someone that is new to the world of market research and the intricacies of big data and qualitative research, the event was perfect to really get under the skin of the topics, network and improve my general ignorance.
Before I get started on the content, one of the best things about FUSION from my perspective was the organisation of the event. Many conferences I have been to provide a 10-minute break for a coffee then straight back in. This wasn’t the case at FUSION, through all four days, delegates had ample time to network, chat and discuss the content they had just seen/ heard. Also, any conference that provides a hot lunch every day is winning in my eyes.
For many Big Data is still a topic that can bring out many into a cold sweat, as an industry it feels like we are still getting to grips with these big data sets, and how to get the best out of them. So, I was interested to discover what the future of big data fused with qualitative research looks like.
FUSION, however, set out it’s stall early. All of the speakers over the first two days provided real world analogies that really brought the data to life. One of the best examples of using big data to solve a real-world problem (first world) that stands out from the first two days for me was the Demystifying Machine Learning session with Sjoerd Koornstra of The House of Insight, and Wim Hamaekers of haystack International on how to use big data to pick a drink flavour in a specific market. This talk caused a lot of debate amongst the audience with many asking why not just use qual research. The response was simple – cost and time.
There were plenty of other examples of how big data is transforming our world, whether that be the “dreaded Blockchain” topic, in which Clint Taylor at RDM provided an extremely accessible way of explaining it, to Hans the clever horse, who was able to understand body language to solve complex mathematical problems. Along with Jonathan Mall of Neuroflash with his presentation (one of the highlights of the entire week), which demonstrated how by using big data, brands can understand the sentiment and resonance of each word on their website, to drive greater customer interaction.
After two days of APIs, coding and heavy tech, it was time to hear about qual. It was very interesting to see the distinctive change in styles of presentations and the way in which the qual researchers presented their information and papers.
The two days of qual papers called for a lot more audience participation with many of the sessions requiring the audience to split into small workshop groups to solve specific problems. One of which is how can a US cinema chain win the battle against streaming films at home and what research would need to be done in order to solve this challenge.
My group, which luckily for me consisted of many qual researchers decided to take a different approach to many of the other groups within the audience. Many felt that by fusing passive data such as social listening with qualitative research was the way forward (embracing the theme of the conference). We however, took a different path identifying how US cinema chain could collaborate with wider partners to use existing data to entice customers back, and marry this up by using qual to see which collaborations and partnerships consumers really wanted at their local cinema. All the points made for a really strong discussion in the networking break that followed the session.
One of the main highlights personally was the range and breadth of approaches that researchers can use to get qualitative insights. Shell for example, decided to pay homage to James Corden’s Carpool Karaoke as a way of getting to understand their customers better. While alcohol brand Suze proved that not all hipsters truly are unique, and many have the same opinions when it comes to individuality, a result that really shocked those within the panel.
So, after four days of presentations, debates and a couple too many Guinness’ with a few of the delegates – it was time to head back to London. The biggest thing that struck in my mind when sitting on the flight was actually how important qual is to big data and big data is to qual. Both are sides of the same coin. Researchers are starting to get to grips with the fusing of these two data points, but there is still a way to go before the potential is fully realised.
By Aiden Connolly
A short history of Agriculture
The agriculture industry is tried and true. From humans first ventures into farming, just 9,000 years ago, the industry has done things the same way for hundreds of years, family farms passing knowledge from generation to generation. Agriculture’s ability to innovate has been limited by the scarcity of data; even as the industrial revolution brought fertiliser, the green revolution brought irrigation and genetic selection has improved performance, the challenge is to measure performance improvements when variations caused by climate, soils and management are so great.
The number of people directly involved in agriculture and food production has significantly decreased all over the world, including the US. In the late 18th century, about 90% of Americans were involved in farming. Today, it’s about 2.5%. Even in China, where 35% of the population farm, the middle class is increasing in size and people are moving off the land and into cities and urban areas.
Clearly consumers are more and more removed from food production, creating a gap in understanding and knowledge with respect the business of agriculture and food. As a result, a myriad of challenges have ensued.
What challenges does agriculture face?
An Industry Misunderstood
The realm of food and agriculture is replete with misconceptions or even ignorance about how food is produced, where it comes from, what labels mean, etc. The lack of education on the part of the consumer causes a great deal of strife for the agriculture industry. Unfortunately, by the nature of being fragmented the industry has an uphill battle.
Animals start on one farm and often change hands at least once if not more before heading to a processing facility, a packaging facility, grocer or supermarket, to finally being purchased by the consumer. With some species, such as cattle, multiple actors are involved in their transfer and this can make tracking individual animals particularly difficult. This is another factor disconnecting producers from consumers.
If you’re an ESOMAR member you can read the full article in MyESOMAR in the digital copy of Research World. If you’re not a member of ESOMAR you can join and receive a free copy of Research World 6 times a year or alternatively you can sign up for a subscription of the magazine in our publications store.
By Kyle Findlay
ESOMAR Big Data World took place in Brooklyn, New York at the end of November 2017. It was a relatively intimate affair as far as ESOMAR events go but it was filled with the right people.