Many routinely raised concerns involve the lack of proper sampling and the lack of control in Internet-based studies. There are also issues of coverage, measurement, and nonresponse (Dillman, 2001). According to D. Dillman (personal communication, April 1, 2004) the situation gets worse, partly because of the ever increasing variety of media and differences in access to and knowledge about media. Along with other researchers (e.g., Brenner, 2002; Dillman, 2000), I have continuing concerns about potential problems in both Internet-based and laboratory studies. Many unresolved issues remain in traditional studies, including contaminated student samples, experimenter effects, demand characteristics, motivational confounding, low power, and generalizability (for an extensive discussion see Reips, 2000), and these issues can be alleviated or even resolved with Web-based methods. Experience has shown initial concerns regarding Web-based methods, like the frequency and detectability of multiple submissions, nonrepresen-tativeness of Internet users, dishonest or malicious behavior (false responses and "hacking"), are not as problematic as previously considered (Birnbaum, 2004; Birnbaum & Reips, 2005), and the real issues tend to be overlooked (Reips, 2002b, 2002c).
When designing a study one must find a balance between methodological advantages and disadvantages. From a multimethod perspective, the opportunity to validate findings with a new set of methods in a new setting is an exciting one: Design the study for the Web, and for comparison, run a subsample in the traditional way.
Was this article helpful?