2 Responses

  1. Ray Poynter
    Ray Poynter at |

    Whilst I find the RIWI system for nano surveys very interesting, I was somewhat taken aback by this article. Partly because it seemed such a naked sales pitch for RIWI and their innovative business model, but mostly because the article seems full, IMHO, of mistakes, over-simplifications, and a general lack of understanding about the market research process.

    Here are a few of my key issues.
    1. At the top of the article it describes “Which of the following video games do you intend on buying in the next month?” as a system 1 decision, i.e. reflexive. However, as anybody who has researched video games knows, this is a field where people tend to read the reviews and the boxes, attempting to decide whether the cost of buying the game will be balanced by enough hours of playing. Kahneman describes buying a white good as a system 2 activity, by that reckoning, buying a video game is going to be system 2 for many people.

    2. The formulation “…buying in the next month” is also very odd for an attempt to get an instant, reflex, or system 1 response. Asking people to envisage the future is not something that can readily be done with a blunt survey question. This problem occurs in other suggested questions such as “Are you short of money each month to buy food?” – indeed as well as requiring reflexion, this question is flawed in that it is double-barrelled – no might mean I have plenty of money, or it might mean I sometimes don’t have enough money for food, or I have enough money for food, but I can’t pay my health bills.

    3. The article describes system 1 and system 2 as scientific fact based on the biochemistry of the brain. System 1 and System 2 are metaphors, the people who developed them have changed their mind and Kahneman says at the end of his book “This book has described the workings of the mind as an uneasy interaction between two fictitious characters: the automatic System 1 and the effortful System 2. You are now quite familiar with the personalities of the two systems and able to anticipate how they might respond in different situations. And of course you also remember that the two systems do not really exist in the brain or anywhere else. ”

    4. The author says that with RIWI all internet users have a relatively equal chance of being asked the survey. As I understand it, the sample will be skewed to people who use the internet more, amongst them to people who type the URL, and amongst those to people who make more mistakes. This skew may not matter, but it should at least be noted.

    5. The article suggests that the response rate for a survey will increase and decrease as public interest increases and decreases, talking about relevancy. This may be true, but is it a good thing and how will the norms the author talks about be created and who will have access to them?

    6. The author may well be correct when he recalls each one of his 200 students, in one course, on one occasion, doing better on their first quick setting of his paper than on their second and more considered setting, but what is the relevance to completing a market research survey, the process of passing a medical exam is different to that of completing a survey.

    7. The author gives the impression that Gladwell’s book Blink stresses the benefits of quick answers. It does not, it shows how in some situations, usually for experts, thin slicing works in surprising ways, but he also shows that first responses are very often wrong. Kahneman is critical of Gladwell’s Blink proposition, writing “Malcolm Gladwell definitely created in the public arenas the impression that intuition is magical… That belief is false.”

    I do not want this comment to come across as negative about innovation, or about RIWI. We need innovation and I think RIWI and nano surveys are well worth investigating. But in times of change we need to protect the basis on which market research makes its claims. I think this article is weak and I surprised that ESOMAR should have published it in the magazine. However, I do thank ESOMAR for posting in a location where others can assess it and add their comments. I look forward to hearing what others have to say.

    Reply
  2. Neil Seeman
    Neil Seeman at |

    Mr. Poynter may be surprised to learn that I agree with most of his points and am thrilled he raised them. Yet in some cases his points require nuance. Let me preface that the groundbreaking work of Kahneman and Tversky is about cognitive heuristics, and not, as he rightly observes, about an artificial distinction between ‘System 1’ and ‘System 2’ thinking.

    The latter divide, as Prof. Daniel Kahneman points out in every public lecture, and in his recent book — a summation and tribute to the genius of his dear friend and collaborator, the late Amos Tversky — is simply a way to help readers navigate the complex work of cognitive heuristics for which Tversky was celebrated. An example of such an heuristic is availability bias. Another bias is what is often referred to as ‘herd bias’, common in finance, which is the tendency to think as everyone else thinks. What is important is to be as aware as possible of one’s own biases.

    Having said this, I will briefly address each of Mr. Poynter’s important observations. Which questions constitute System 1 vs. System 2 is subjective; there has been, to the best of my knowledge, no case-controlled randomized research on this subject. RIWI’s clients define this since they know more about their clients’ insights on this matter than we do. As such, Mr. Poynter’s difference of opinion over what constitutes a System 1 vs. 2 decision, is, I would suggest, a difference of opinion. Admitting to the Internet nature of our data is well-documented in my own peer-reviewed research. It also resides on our website under the “methodology” tab. Although he is correct that this could well have been acknowledged in the article, one practical challenge is that it would take a longer article to dig into details on this matter; the overwhelming bias of the Internet is English, as well as the skew toward the younger demographic, notably the 18-34 year old population. These skews can be redressed to the best of our ability, and we do so with our clients.

    With respect to access to norms, in some cases we offer these at the client’s request; in others, we are building them (subject to the limitations to which we know and readily admit); in others, they will be made public, as we are shortly going to do in conjunction with Greenbook, in a major public release on hundreds of thousands of survey-takers around the world, releasing, for example, norms of Internet browser usage, operating system usage and aspects of the survey-taking experience in over 200 countries and regions of the world. Our respondent data norms (for example, in data captured in conjunction with the World Bank in Indonesia and Kenya on the public’s desire for open financial data from government) have been shared with the world and are readily accessible on our website.

    I will not comment too much on Kahneman’s nuanced interpretation of the populist writer Malcolm Gladwell; my own experience, having heard Prof. Kahneman respond to a question on this subject, was, in fact, laudatory of the writer. Yet with this point, too, I agree: Gladwell’s popularization of research does, at times, tend to avoid the nuance over what is, and is not, intuition. Most important, thanks to Mr. Poynter for acknowledging RIWI as being in the data innovation business. We are a data company, and we are considered authorities about rarefied parts of the Internet, on Internet software, and our objective is to provide our clients and research partners with a new and unique source of global respondents to answer the questions they deem important. Amos Tversky RIP.

    – Neil Seeman, Founder and CEO, The RIWI Corporation

    Reply

Leave a Reply