Comparative Historical Research in a Design team by Miriam Manso on June 23, 2021

A heuristic evaluation case.

What is comparative historical research

We can define comparative historical research as a homogeneous set of techniques that facilitate comparative knowledge of national or international realities. With this type of research a broader and deeper knowledge of the subject matter is achieved. With the comparative methodology we will be able to understand other realities in their own contexts, comparing them with each other and with our own reality, and draw operational conclusions.

A priori, it may seem that such an academic type of research has no place in the business environment of a product team. But, we are going to analyze each of the steps that we must follow in an investigation like this. Then, we are going to illustrate it with a real example that we have carried out in the Liferay Design department.

Choose the topic

The first step that every researcher must take is the choice of the topic and its definition in space time.

For the choice of the topic, three criteria are usually discussed: relevance criterion, viability criterion and originality criterion.

• Relevance criterion: We speak of social relevance or scientific relevance. Social relevance comes to answer the question of whether the chosen topic will solve a problem of the present with the study of the past. In the second case, the important thing is the scientific relevance of the topic.
• Viability criterion: This criterion is easy to apply. We must ask ourselves if the documented resources are enough in terms of quantity and the quality and variety of the information they provide. We will also need to ask ourselves if we have enough time to undertake the investigation and if we have sufficient theoretical knowledge.
• Originality criterion: The purpose of all research is to fill a historiographic gap, reexamine already investigated topics taking into account new documentation, or review existing interpretations providing a new approach.

Our chosen topic was heuristic evaluation as a method of analysis in design.

We consider that it is relevant because by studying the past of the heuristics we solve present problems regarding their use in our team.

Regarding viability, we had to take into account that we did not have much time to carry out the research. This impacted certain decisions like the selection of the studied documentation.

Regarding the criterion of originality, despite being a rather hackneyed subject, for internal reasons of the design department, we wanted to make our own approach to the subject based on review of existing interpretations.

Bibliographic Review

In this next phase, reading the bibliography is essential. This allows us to delimit the topic based on what is already investigated.

Exhaustive research on heuristic evaluation from its inception would have taken years of dedication, time we did not have. Therefore, we made an incipient approach to the most relevant documentation to which we had access. We reviewed the contributions of John D. Gould and Clayton Lewis (1985), Ben Shneiderman (1992), Christian Bastien & Dominique Scapin (1992), Jakob Nielsen (1994), Gerhardt-Powals (1996), Weinschenk and Barker (2000), Connell & Hammon (2000), and Bruce Tognazzini’s (2003). Then we compiled a list of almost 30 documents including articles, conferences and doctoral theses of the most recent approaches to the subject (Granollers (2018), Suárez (2011); Paz (2018), Murillo (2018), Quiñones (2018); Kurosu (2018); Cooper (2007)).

Data collection

In this phase, it is best to summarize the most relevant findings from each document using cards. These cards can then be classified and ordered in a clear and simple way. They should allow a quick consultation. It is advisable that these cards contain both source information and content.

Cards can be made on paper or digital. We decided to work with Notion. Being a collaborative investigation between three people, we consider it to be a way of making information more accessible to everyone.

Data Analysis

Finally, once all the information was analyzed, we discovered an evolution in the approach to the topic from the past to the present. Traditional heuristic principles propose a more general evaluation. On the other hand, the most modern authors seek a more concrete definition of the evaluation criteria, break them down into easy-to-identify sub-principles and try to quantify, as much as possible, the evaluation results.

We did not want to stay in the comparison phase. We wanted to approach the creation of a new heuristic evaluation model that would adjust to the needs of our product teams.

Card Sorting

We used the card sorting method to group all the heuristic sub-principles that we had detected in the analysis stage into groups with common thematic characteristics. We reviewed each one, justifying its inclusion through references to different authors who talked about the subject.

After three iterations, we developed a heuristic evaluation list comprising 11 heuristic principles, with a total of 57 sub-principles.

Conclusions

We all know that everything happens very quickly in a product team and sometimes think doing such an exhaustive analysis of a topic is a waste of time. However, comparative research allows us to gain a deep knowledge of a topic, and only from this point can we start to grow. It is not a waste of time but an investment in knowledge that is later applied with practical tools such as a heuristic analysis table.

Patience, tenacity, dedication, and thoroughness are essential attitudes when undertaking an investigation of this type that, as you have observed, bears wonderful results.