To learn to run an Ibex experiment, check out the following resources:
- The Ibex manual. This is the authoritative manual on Ibex.
- Brian and Rodica's 2018 LSA Minicourse. These slides contain much of the material here, and then some.
1: Basic acceptability judgment task: Basic_7pt_Likert.js.
2: Speeded acceptability judgment task: ROCOUT_Online_Binary. For speeded tasks, we typically disable the use of the mouse to respond to enforce some uniformity in responses. To do this we use Ethan Poole's EPDashedAcceptabilityJudgment. These must be uploaded to your experiment on Ibex for this script to work. If you want a version without the random assignment to groups, it is here: ROCOUT_Online_Simple.
3: Recall task: Basic_Expressives_Recall. This is useful for any task involving collection of written responses, such as free or cued recall tasks, or cloze tasks. This implements a version of Potter and Lombardi's 1990 free sentence recall task.
Example forms and instructions:¶
1: Consent form. (Note: This consent form cannot be used unless it is specifically the one approved for your study. It is here just as an example for format!)
2: Demographic collection form. This is a basic introduction form for collecting demographic information.
3: Debriefing form. This is a basic form for collecting debriefing information. The specific questions you will want to ask will vary experiment to experiment.
4: Instructions. These are a series of five separate instruction screens. They show typical instructions for an MTurk experiment, including comprehension questions on the instructions themselves.
Collecting data on Ibex¶
Recruiting participants: Mechanical Turk¶
We recommend that you start your own MTurk account to run studies. See the minicourse above for information on getting started with MTurk.
Recruiting participants: Lottery system¶
One system we use for recruiting participants from populations that are not well represented on Mechnical Turk or other crowdsourcing platforms is targeted recruitment (social media, mailing lists) along with a lottery system for paying participants (set up by Rodica Ivan and Thuy Bui).
- Give every participant a random code (see code-generating scripts above).
- Participants are directed to an external site to input code and email address.
- After experiment is done, choose winners, and compensate them.
Websites for collecting emails/codes¶
- One option is Google Forms; this option is no longer accepted by the UMass IRB.
- An acceptable option is Survey Monkey. Create a small survey that collects email addresses, random codes generated from your script, and no more.
Collecting participants: Other options¶
- Prolific Academic: This is like MTurk, but specifically for academic studies.
- Le RISC (Relais d’Information sur les Sciences de la Cognition): Free crowdsourcing platform for finding participants in France.
- Zhubajie: Crowdsourcing site active in China.
- Crowdflower: Another crowd-sourcing site. More limited than other options.
Tips and tricks for running experiments on Ibex¶
Ensuring a smooth Latin Square¶
One thing that can be difficult when running participants using Ibex Farm is to ensure a smooth Latin Square distribution (i.e. making sure that each experimental list is equally well represented). Ibex uses a counter variable that it stores on the server to determine which counterbalancing list any one participant is assigned to. As a researcher, you have two options for determining Ibex's behavior with respect to Latin Squaring.
Option 1: explicitly set the Latin Square Counter
var counterOverride = X. You may set the Latin Square counter explicitly in your Ibex script by defining the counterOverride variable somewhere in the script. When this line is present in the script, all participants who reach the experiment's URL address will be placed in the
Xth list. Note that list numbering begins at 0 (i.e. 0 usually means the first Latin Squares list).
Option 2: insert a hidden set counter event
set. If you do not set the counter explicitly using
var counterOverride, then Ibex Farm will increment the counter by one each time a participant finishes the experiment. In our experience this does not lead to the smoothest Latin Square, because if many participants begin the experiment at once they will all end up in the same list. Instead, we insert a hidden set counter event at the beginning of the experiment. This causes Ibex Farm to update the counter on the server immediately after a participant starts the experiment.
In theory, Option 2 should yield a perfect Latin Square if you run the appropriate number of participants. However, in practice, it is not always perfect, because some participants do not finish, some start at exactly the same time, etc.
For this reason, the usual way researchers in the CSL lab get a perfect Latin Square distribution is by running 80-85% of their participants using Option 2, checking to see how many participants are in each list, and then using Option 1 to manually fill in the counterbalancing lists for the remaining 15-20% of subjects.
No repeat subjects¶
If you are running on Mechanical Turk, you may want to exclude workers who have previously your experiments. To do this, please go to Unique Turker.
It is very difficult to guarantee that you are getting high quality data from internet participants, who typically take the experiment in less-than-ideal circumstances, and may not give their full attention to the experiment. Despite this, there is generally high correlation between lab data and internet data for judgment tasks (Sprouse, 2011), with identical qualitative patterns and only a minimal loss of power for the latter.
However, there are three options for quality control typically used in the lab:
- Catch trials: in your script, include a small number of labeled fillers whose responses are known. For example, you may use a small number of clearly grammatical and clearly ungrammatical fillers, and reject participants who do not consistently rate the grammatical fillers above the ungrammatical fillers. Or you might use a small number of unambiguous, easy-to-understand controls if you are doing a questionnaire.
- Attention check trials: Less often, researchers will include attention check trials. These are trials where you instruct the participant to do something very specific; for example, you might present the sentence "This is not a normal sentence to rate; please just click on 3 on the scale below" and reject participants who do not comply.
- Instruction question check: We often ask questions on the instructions themselves; participants who are unable to get perfect accuracy on these are rejected.
Note that in each of these cases, we recommend removing the data from analysis, but paying the worker (if MTurk).