It is impossible to cover all the issues that should be considered when designing an EMA study within this chapter, but we will briefly cover many of the most important considerations (more detail is available in Delespaul, 1998). In 2002, Stone and Shiffman published a paper on "reporting guidelines" for momentary studies that suggest which study design information should be reported in manuscripts. We use their outline as a convenient way of presenting design issues.
Sampling. We have already discussed basic types of sampling schemes used in EMA and the importance of careful planning about the data needed to address a study's hypotheses. Sampling density indicates the number of signals participants receive per day; with random and time interval prompting, this is decided in advance. Determining sampling density a priori with event-driven sampling can be difficult unless one has considerable information about the targeted thought or behavior's frequency of occurrence. This is an important consideration if the target behavior has the potential of occurring quite frequently. Some researchers have implemented ingenious schemes for sampling event-driven behaviors with electronic diaries to overcome this problem. Shiffman and colleagues studied antecedents of cigarette smoking, but felt that full momentary assessments of every cigarette smoked might prove too burdensome for participants. They therefore asked participants to indicate all instances of smoking, but randomly selected a portion of these for comprehensive assessment (Shiffman et al., 1997). When choosing a sampling strategy one must ensure that the proper time periods are sampled throughout the day. This maximizes validity of the results and minimizes bias.
Momentary procedures. Researchers face many decisions pertaining to the actual protocols used to collect momentary data, especially with EDs, given the great flexibility afforded by the programming (e.g., Delespaul, 1995; Hufford & Shields, 2002). A few examples show issues that need consideration. In the case of event-driven recording, participants need a very clear understanding of the "rules" for making a diary entry. To use a medically oriented example, researchers studying bowel movements would need to accentuate to participants exactly when they should make the recording (e.g. during, immediately after, within an hour) and if a recording should be made for each event. For discrete events and actions like this, the criteria for initiating a recording may be straightforward. However, when individuals must initiate a recording on the basis of having a certain intensity of a feeling or having a particular thought, then potential bias issues arise. Are individuals capable of detecting the threshold sought by the investigator (e.g., "moderately stressed"), and how can the reliability of these reports be verified? Extensive piloting with well-trained, extensively debriefed participants may help determine the success of such event-driven prompting schemes.
Another example of these reporting issues concerns how an investigator handles a missed prompt in a random or interval-contingent sampling study. Has a period of time after (or before) the targeted time been designated in advance as an acceptable period for completing a prompt? Does the ED administer another prompt a few minutes after a missed prompt? How many times? Thus, the development of a sampling protocol, which can impact the quality and validity of the data collected, is a complex process. In all cases, the electronic protocol needs extensive field testing prior to implementation in the trial—research staff must use the ED for several days, essentially exposing themselves to all kinds of circumstances that participants may encounter.
Data acquisition interface. The type and size of the data acquisition interface is another consideration for users of electronic diaries (Hufford & Shields, 2002). Most EDs have a small display screen, about 6 cm x 6 cm, making it a challenge to devise ways of presenting moderate length questions and responses in a legible, efficient manner. When transferring standardized questionnaires to ED platforms, it is typical to rewrite the questions to fit the display. Questions must retain their original meaning, however, and pretesting of significant rewrites may be necessary to ensure accuracy. Similarly, response options in questionnaires can take many forms ranging from "Yes/No" options to checklists of items to 5-point and visual analog scales (VAS). The "Yes/No" response option is usually not a problem given its brevity; however, lists of items can be problematic. One solution is to allow participants to scroll through a list of options where some of the options are not shown until the scrolling reaches them. In this way, many response options can be made available. Likewise, VAS scales in questionnaires are usually 10 cm in length, which is beyond the available presentation real estate of most PDA displays. Some have raised questions about whether or not a shorter VAS scale yields the same information as 10 cm scales; so far, the evidence is that they are equivalent (r = 0.97; Jamison et al., 2002).
Compliance. Compliance issues were discussed above and here we simply stress the importance of reporting the actual protocol compliance.
Training of participants. An important, yet often overlooked, feature of an EMA study is the procedures used to train participants in the use of the diaries and procedures to monitor and enhance compliance (Hufford & Shields, 2002). Participants need a thorough understanding of the study proto col and the data collection device prior to the onset of field recording. In many laboratories, participants are first trained in small groups about the purpose of the study and the procedures for making diary entries, and they can then practice completing the diary in the presence of the investigators. This ED training is especially important given the unfamil-iarity that some individuals have with PDAs and the complexity of the implementation. Many features of sophisticated EDs (e.g., sleep, nap, and delay) are perfectly understandable once fully explained, although not self-explanatory. Therefore, we strongly urge investigators using EDs to have participants practice with them during the training session. We find that 30 to 60 minutes is usually adequate for training a small group.
Apart from excellent training, obtaining good protocol compliance is really an ongoing process wherein participants are provided with performance feedback. Follow-up phone calls may be made after the training session. During this phone call the researcher can make sure the device is working properly in the field and ensure that the participant is comfortable using the ED for the study's duration. This troubleshoots problems during the early stages of data collection and sees that the participant is using the ED correctly. Another type of ongoing feedback involves providing participants with information about compliance or the quality of their data (e.g., missing entries) at regular intervals throughout a study. This sort of feedback may be especially important early in the study, when participants are still learning the protocol requirements; corrective feedback at this stage can largely impact data quality and compliance for the remainder of the study. Some implementations of EDs have onscreen compliance feedback, such that, when prompts are missed, information is provided about the missed prompt—perhaps with an encouraging word about the importance of timely diary completion. These and other clever ways of enhancing compliance are critical to achieving the goals of a momentary study.
Data management. These are rather technical, nevertheless important, issues for the overall success of a momentary study. However, we will not detail these issues here. Some things to remember when undertaking a study of this kind are (a) to have trained staff who are able to troubleshoot problems with the hardware and software of the device and (b) to have an adequate database program that can handle the amount of data generated from such a study.
Analytic issues. Momentary data present investigators with many challenges. The volume of data is often immense given the substantial number of recordings made by each participant. Most challenging, though, is the multilevel structure and repeated nature of the data. Unlike between-person studies where the individual is usually the smallest unit of analysis, in EMA research the moment is nested within persons, and this demands techniques specifically designed for this data structure. Multilevel analyses, hierarchical linear models, and random regression are all analysis techniques that can be used for analyzing momentary data. Several review papers including Schwartz and Stone (Schwartz & Stone, 1998, in press) and Tennen and Affleck (2002) discuss the pitfalls of using traditional analysis of variance procedures with momentary data and provide overviews and technical references for the other techniques. All novice researchers who are considering conducting a momentary study should familiarize themselves with these models or obtain expert consultation, as there are many subtleties to successfully conducting analyses of momentary data. Within the next year or so a book on conventional and alternative statistical approaches to real-time data, edited by Ted Walls, should be available.
Was this article helpful?