Module: Inter-rater Agreement (Kappa)

Add a score that shows how far 2 coders agree on one video. It takes away the agreement luck alone would bring.

Before you start

• Cohen's kappa
• Give one module its own scope

Steps

1. Click Add module. The drawer opens on the right.
2. Click + next to Inter-rater Agreement. The module lands at the foot of the page. (Until the next update the palette greys this module out unless the workspace was opened from a single-video analysis link.)
3. Click the settings button in its title bar. A panel opens with Scope override.
4. Select one video under Video (required — single-video module). The cards draw.

Check it worked

The card says "Cohen's κ per annotator pair (1-second bins)". You get one card for each pair of coders, with an overall score and a score for each annotation type.

GRAPE MARS cuts the span into 1 second steps. For each step and each type it asks whether each coder marked it. The score comes from those answers, and it takes away the agreement that luck alone would bring.

The words on the card come from 4 bands: 0.8 and above is Excellent, 0.6 and above is Good, 0.4 and above is Moderate, and below that is Poor.

The module needs 3 things. It needs exactly one video, it needs 2 coders or more, and a fragment must last at least 10 seconds. Otherwise the card says what is missing.

The Excel export writes 2 sheets: one overall, one per type.

If something goes wrong

• The agreement module is empty
• The coder column shows a code, not a name

Related

• Cohen's kappa
• Give one module its own scope
• Export a workspace to Excel