It just occurred to me why UX community often conflicts with traditional marketing point of view. Why stakeholders are perplexed when one study's finding seems to conflicts with another.

The reason is that the User Experience industry often makes generalisations based on statistically irrelevant samples.

You hear consultants say "Users don't use search, or users don't understand X..."  These statements are inferences from non-causational observations. We are erroneously saying the effect of X is caused by Y. The theory goes like this....

When we practice these pseudo-scientific reporting methods we run the risk of discrediting our findings and coming to false conclusions.

This article is aimed at research involving small amounts of users where there are insufficient numbers of participants to build reliable behavioural patterns. With these small samples the primary value is in the individual stories and experiences.

Of course where a empirical fact is established, such as a link takes users to the wrong page then it only takes a single user to establish this as a valid fact.

If we have spotted a pattern or even one occurrence of a behaviour we can form a hypotheses. This needs to be tested to prove or disprove that hypotheses. Alternatively, we could use published studies that have scientifically established behavioural patterns and quote these as evidence.

The goal of qualitative research is to answer the questions 'why'. As the answer to that question is subjective to the individuals we interview we should, therefore, be telling their subjective stories.

The reason that one person may reject a proposition may be fundamentally different from another persons'. Each separate reason may have design implications, which wouldn't have been manifest with a generalised report. For example, one user may reject a proposition because they think it's too technical, whereas another thinks it would be unaffordable. If the research company only generalises that 'the proposition was unattractive to users,' then no concrete steps can be taken to address the issue.

Only by understanding who each person is, what their motivations are, their mental models, contextual factors and internal & external cognitive variables can we qualify their statements and behaviour.

A friend of mine in a research and build company was questioned about a report finding that stated their users had disliked a specific function. Based on the research report, the design team had earmarked that function for deletion until client asked to see who each one of the participants were. One by one, he analysed what their financial role was to understand the reason they had rejected that function. As it turned out, these users were a-typical in this one respect. It was only by knowing the participants and the context of their particular jobs could they 'qualify' the finding and make an appropriate design decision.

The generalisation was correct in isolation, but in context it was invalid.

How does this translate into our reports? I suggest that agencies structure findings primarily by participant. They should tell us their name, show us a photo, give us a 'capsule' of that person. Only by knowing the person can we make sense of their behaviour.

The report shouldn't blend together an average of 10 people's experiences, but rather tell each individual journey.

Only at the end of the report should patterns be identified, as without the context and history of these patterns a reader would be unable to judge what they are reading.

Reports should allow the reader to trace findings back to the source material in a user-friendly manner.