Avoiding Pitfalls in Experimental Research in Marketing

Abstract

Experimentation is an important methodology in marketing research and is the subject of a significant amount of literature. These studies consider different ways of looking at the issue of experimentation and may be categorized as follows: (1) Publications asking why and how we carry out experiments in the first place: the fundamental idea behind experimentation. (2) Guidelines to be followed during the experimental process. (3) Statistical methods for analyzing experimental data. (4) Studies focusing on specific methodological problems. (5) Warnings against bad practice. In the current article we cover areas 2, 4 and 5. We attempt to make experimental researchers, especially those who are novices in the area of experimentation, aware of the possible pitfalls that occur during the process, from planning experiments to discussing results. We present a list of the questions frequently raised by our own students and discussed in the recent literature. Although it is beyond the scope of our paper to treat all of these questions in detail, we provide references to more detailed studies where appropriate. In particular, we examine the following issues: What constitutes a contribution to the marketing literature? What should I consider when formulating hypotheses? What are the different approaches to developing hypotheses? Where do hypotheses come from? How can I formulate hypotheses about moderation effects? How can I formulate mediation hypotheses? Can I combine moderation and mediation? What should I do if random assignment of participants to conditions is not possible? Should I use a between-subjects or a within-subjects design? Should I manipulate or measure variables? What should I take into consideration concerning the realism of the experiment? Which items and scales should I use for measurements? In which order should I arrange the constructs in the questionnaire? Must the independent variable be discrete and the dependent variable continuous? What should I take into account concerning the use of random samples vs. convenience samples? Should I tell respondents what the study is about? Which checks should I consider before testing the hypothesis? Can I remove cases if those checks indicate lower quality? When should I consider covariates? I have considered a moderator variable that has been measured as a continuous variable. Can I dichotomize in order to calculate ANOVA? Why should I include an effect size calculation? Which parameters of the findings should be reported? What should I do if the effect that I proposed turns out to be not what I expected? What should I take into consideration concerning the generalizability of the results? Our review of literature from the last five years shows that the discussion about experimental procedures and specific techniques is far from complete. Unsurprisingly, the replication crisis in social psychology and related areas has led to demand for more guidance by experienced researchers. Several journals feature regular tutorials or special issues on methodological problems. Novice researchers are advised to avoid merely citing authors of other experimental studies who use appealing procedures (“they did it that way, therefore so do I”) and instead provide arguments in favor of their procedures, drawing attention to recent developments in experimentation.