Definitely Maybe: Hedges And Boosters in the HCI Literature

In this codebook we use the following definitions:

1/ A study report is a paper that reports research involving human participants. The report must be on an original study: surveys, discussions, and re-analyses of previous studies do not qualify.

• If the paper is a study report, write “Y” in the "Study" column
• Otherwise, write “N”, leave the "Type" column blank and proceed to the next paper.

2/ An inferential study report is a study paper that reports study results using measures of statistical uncertainty such as test statistics, p-values, interval estimates (confidence intervals, credible intervals, standard errors, but not standard deviations), Bayesian posteriors, or Bayes factors. All of these can be reported either numerically or graphically.  It is not necessary that the paper only report inferential statistics.

• If a paper is an inferential report, write “inferential” in the “Type" column.
• Otherwise, write “other”.

3/ An abstract is the first section of a paper and is titled “Abstract”.

4/ A result description is a sentence that describes or summarizes the results of a study, explains them, or discusses their implications. As a general rule, it is a sentence that may have been different had the results been different. Both sentences in the following excerpt are result-relevant: “We found that touch interaction is slower but more precise than tangible interaction. This suggests that designers and developers should consider this tradeoff when designing applications”. Counterexamples include generic statements that carry no information about the results or their implications (e.g., "We then discuss our results" or "We conclude with implications for design") and future work statements that do not clearly derive from the results (e.g., "In future work, we will investigate alternative designs"). Conversely, future work statements that are clearly informed by the results qualify, e.g., "Given this result, more complex designs should be investigated in the future". However, we exclude the description of further research steps that were carried out based on the results of a study and reported in the same paper, e.g., "This finding led us to design a second experiment where we tested the effect of task. We recruited 20 other participants [...]".

5/ A modal term is a term that is used either to tone down a statement, or to increase its strength. Examples are the word "may" in “The drug may have an effect on sleep” or the word "definitely" in "The drug definitely has an effect on sleep". However, the word "definitely" is not considered a modal term in the sentence "One participant said she would definitely use the tool in the future", because changing it would alter the meaning of the statement. As a general rule, if a term can be removed or replaced by another term in a way that alters the subjective strength of the statement made by the authors without altering its objective meaning, then it is a modal term. As another example, in the sentence "We found a strong correlation", the term "strong" partly reflects a subjective judgment by the authors, who could have easily used a term like "substantial" instead. Therefore, it is a modal term. We have already identified a number of potential modal terms in the sentences you will code, by searching within a dictionary of common modal terms. All matches will be highlighted in gray. However, there will be a good number of false positives, so we will ask you to indicate whether each of those terms highlighted in gray is indeed a modal term, and if so, to indicate its type. Modal terms come in two types: hedges and boosters, explained below. We will ask you to only mark terms that are evident hedges or evident boosters --- if you cannot decide between hedge and booster, leave the term in gray.

6/ A hedge is a term that is used to tone down a statement, for example by making it appear more fuzzy or more uncertain. Examples of hedges include "possibly", "approximately", "sometimes" or "speculate". For example, the sentence “The drug may have an effect on sleep” is weaker than the simpler sentence “The drug has an effect on sleep”, so "may" is a hedge in the first sentence. Other examples are "The technique was roughly twice as fast" (which increases fuzziness), or "The effect is somehow large" (which decreases strength). Hedges are not necessarily modifiers. For example, in the sentence "Our results suggest that the drug has an effect", "suggest" is a hedge because a stronger verb (like "show") could have been easily used instead. As a third example, the word "speculate" acts as a hedge in the sentence "We speculate that our findings are due to a difference in performance". An example of a clear false positive is the term "may" in the following sentence: "We conducted our study in May". Similarly, the word "speculate" is not a hedge in the sentence: “Our subjects often had to speculate about the actual goal of the task", because its role is not to reduce the force of the author's statement.

7/ A booster is a term that is used to increase the strength of a statement, for example by making it appear sharper or more certain. It has the opposite effect of a hedge. Examples of boosters include "definitely", "strongly", "exactly" or "prove". For example, the statement “The drug has a clear effect on sleep” appears stronger than “The drug has an effect on sleep”, so "clear" acts as a booster in the first sentence. Other examples are "The technique was exactly twice as fast" (which increases sharpness), or "The effect is extremely large" (which increases strength). Like hedges, boosters are not necessarily modifiers.  For example, in the sentence "Our results show that the drug has an effect", "show" is a booster because a weaker verb (like "suggest") could have been easily used instead.  As a third example, the word "demonstrated" is a booster in the sentence "We demonstrated that our findings are due to a difference in performance". However, the word "demonstrated" is not a booster in the sentence “Our subjects demonstrated their expertise", because its role is not to increase the force of the author's statement.

8/ You will see cases where it is difficult to decide whether a term is a modal term, and whether a modal term is a hedge or a booster. For example, in the sentence "We found that 72% of the participants were faster with A than B", you may find that the term "found" is not a clear hedge or a clear booster. In that case, leave the term in gray. Again, focus on terms that strike you as clear hedges or boosters. If you cannot decide whether a term is a hedge or a booster or it takes you too long to decide, leave it in gray. Keep in mind that there may be a good number of false positives among the terms highlighted in gray.

---

Pre-registration

The analysis for this study was planned in advance and pre-registered on OSF. The pre-registration document can be consulted here (external link).

---

Coding Interface

We used a custom coding interface for which we share the source code together with all other supplementary material on OSF. You can try out the coding yourself by clicking on the screenshot below.