# interrater reliabillity - but for any pair of raters?

11 messages
Open this post in threaded view
|

## interrater reliabillity - but for any pair of raters?

 Dear list members, I suspect I have a somewhat unusual questions - specifically, for a number of categorical judgements (on  a scale with judgements 1, 2, 3 or 4), I have data from 30 judges (j1, j2, j3 etc.). Judges are  in the columns and judgements in rows.   My task is to calculate an interrater reliability for any pair of the 30 judges. So this makes n*(n-1)/2 = 30*(29)/2 = 435 interrater coefficients, one for each dyad of raters. For any dyadic interrater coefficient, I could use e.g. the kappa coefficient available in the SPSS crosstabs procedure. But how could I tell SPSS to repeat the procedure for each combination of the 30 judges? So that ultimately, I receive as many kappa coefficients as there are pairwise comparisons (=435, in this example)? I would be very grateful for your suggestions how to implement such an automated procedure? Many thanks...Talma -- Sent from: http://spssx-discussion.1045642.n5.nabble.com/===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD
Open this post in threaded view
|

## Re: interrater reliabillity - but for any pair of raters?

Open this post in threaded view
|

## Re: interrater reliabillity - but for any pair of raters?

 In reply to this post by Talma Macro with nested !DO and CROSSTABS in body. Use OMS to capture output. AGGREGATE to get average. Or maybe Brian will post a link to his MATRIX program. UNTESTED. I'm on my phone... DEFINE ! AllPairsKappa( raters !CMDEND) !LET !CPY = !raters !DO ! j1 !IN ( !raters) !LET ! CPY= !TAIL( !CPY) !DO !j2 !IN (!CPY) CROSSTABS TABLES !j1 BY !j2 /STATISTICS KAPPA. !DOEND !DOEND !ENDDEFINE. SET UP OMS..RTFM FOR THAT... !AllPairsKappa raters= j1  j2  j3 ..... j30. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD
Open this post in threaded view
|

## Re: interrater reliabillity - but for any pair of raters?

 Looks like 2 of the ! chars have an illegal following blank.On Sat, Jan 20, 2018 at 7:04 PM David Marso <[hidden email]> wrote:Macro with nested !DO and CROSSTABS in body. Use OMS to capture output. AGGREGATE to get average. Or maybe Brian will post a link to his MATRIX program. UNTESTED. I'm on my phone... DEFINE ! AllPairsKappa( raters !CMDEND) !LET !CPY = !raters !DO ! j1 !IN ( !raters) !LET ! CPY= !TAIL( !CPY) !DO !j2 !IN (!CPY) CROSSTABS TABLES !j1 BY !j2 /STATISTICS KAPPA. !DOEND !DOEND !ENDDEFINE. SET UP OMS..RTFM FOR THAT... !AllPairsKappa raters= j1  j2  j3 ..... j30. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD -- Jon K Peck[hidden email] ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD
Open this post in threaded view
|

## Re: interrater reliabillity - but for any pair of raters?

 In reply to this post by Talma Note to self. Do not post code to SPSS List from phone. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD
Open this post in threaded view
|

## Re: interrater reliabillity - but for any pair of raters?

 In reply to this post by David Marso-2 This looks great. My syntax has all the matrix stuff to calculate Fleiss, so it wouldn't be as elegant as this, in which the OP just needs to figure out how to insert the OMS. I hope the OP from earlier this week is reading this thread, too. It's exactly what he wanted to do. My one concern is that the ordinal nature of the responses, or at least what looks ordinal, calls for the ICC rather than Fleiss. But that's not really our issue. Brian ________________________________________ From: SPSSX(r) Discussion [[hidden email]] on behalf of David Marso [[hidden email]] Sent: Saturday, January 20, 2018 9:04 PM To: [hidden email] Subject: Re: interrater reliabillity - but for any pair of raters? Macro with nested !DO and CROSSTABS in body. Use OMS to capture output. AGGREGATE to get average. Or maybe Brian will post a link to his MATRIX program. UNTESTED. I'm on my phone... DEFINE ! AllPairsKappa( raters !CMDEND) !LET !CPY = !raters !DO ! j1 !IN ( !raters) !LET ! CPY= !TAIL( !CPY) !DO !j2 !IN (!CPY) CROSSTABS TABLES !j1 BY !j2 /STATISTICS KAPPA. !DOEND !DOEND !ENDDEFINE. SET UP OMS..RTFM FOR THAT... !AllPairsKappa raters= j1  j2  j3 ..... j30. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD
Open this post in threaded view
|

## AW: interrater reliabillity - but for any pair of raters?

 Dear all, many thanks for your time + efforts! Because I am a bit slow I adapting the output management system (=SPSS novice, using it for the first time:) ) to David's syntax; I would also be interested in the matrix code Brian referred to? Best, Talma Gesendet von Outlook Von: SPSSX(r) Discussion <[hidden email]> im Auftrag von Dates, Brian <[hidden email]> Gesendet: Sonntag, 21. Januar 2018 18:57 An: [hidden email] Betreff: Re: interrater reliabillity - but for any pair of raters?   This looks great. My syntax has all the matrix stuff to calculate Fleiss, so it wouldn't be as elegant as this, in which the OP just needs to figure out how to insert the OMS. I hope the OP from earlier this week is reading this thread, too. It's exactly what he wanted to do. My one concern is that the ordinal nature of the responses, or at least what looks ordinal, calls for the ICC rather than Fleiss. But that's not really our issue. Brian ________________________________________ From: SPSSX(r) Discussion [[hidden email]] on behalf of David Marso [[hidden email]] Sent: Saturday, January 20, 2018 9:04 PM To: [hidden email] Subject: Re: interrater reliabillity - but for any pair of raters? Macro with nested !DO and CROSSTABS in body. Use OMS to capture output. AGGREGATE to get average. Or maybe Brian will post a link to his MATRIX program. UNTESTED. I'm on my phone... DEFINE ! AllPairsKappa( raters !CMDEND) !LET !CPY = !raters !DO ! j1 !IN ( !raters) !LET ! CPY= !TAIL( !CPY) !DO !j2 !IN (!CPY) CROSSTABS TABLES !j1 BY !j2 /STATISTICS KAPPA. !DOEND !DOEND !ENDDEFINE. SET UP OMS..RTFM FOR THAT... !AllPairsKappa raters= j1  j2  j3 ..... j30. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD
Open this post in threaded view
|

## Re: interrater reliabillity - but for any pair of raters?

 Talma, Before you abandon David's code, he inserted the OMS section to provide a file to average the pair-wise kappa's. If you want to calculate the overall kappa in addition to the pairwise kappa's, then pasting and inserting syntax for kappa from the built-in kappa analysis after !End Define will do the trick. This will be a whole lot easier than for me to adapt my syntax. Brian ________________________________________ From: Talma Claviger [[hidden email]] Sent: Sunday, January 21, 2018 2:55 PM To: [hidden email]; Dates, Brian Subject: AW: interrater reliabillity - but for any pair of raters? Dear all, many thanks for your time + efforts! Because I am a bit slow I adapting the output management system (=SPSS novice, using it for the first time:) ) to David's syntax; I would also be interested in the matrix code Brian referred to? Best, Talma Gesendet von Outlook ________________________________ Von: SPSSX(r) Discussion <[hidden email]> im Auftrag von Dates, Brian <[hidden email]> Gesendet: Sonntag, 21. Januar 2018 18:57 An: [hidden email] Betreff: Re: interrater reliabillity - but for any pair of raters? This looks great. My syntax has all the matrix stuff to calculate Fleiss, so it wouldn't be as elegant as this, in which the OP just needs to figure out how to insert the OMS. I hope the OP from earlier this week is reading this thread, too. It's exactly what he wanted to do. My one concern is that the ordinal nature of the responses, or at least what looks ordinal, calls for the ICC rather than Fleiss. But that's not really our issue. Brian ________________________________________ From: SPSSX(r) Discussion [[hidden email]] on behalf of David Marso [[hidden email]] Sent: Saturday, January 20, 2018 9:04 PM To: [hidden email] Subject: Re: interrater reliabillity - but for any pair of raters? Macro with nested !DO and CROSSTABS in body. Use OMS to capture output. AGGREGATE to get average. Or maybe Brian will post a link to his MATRIX program. UNTESTED. I'm on my phone... DEFINE ! AllPairsKappa( raters !CMDEND) !LET !CPY = !raters !DO ! j1 !IN ( !raters) !LET ! CPY= !TAIL( !CPY) !DO !j2 !IN (!CPY) CROSSTABS TABLES !j1 BY !j2 /STATISTICS KAPPA. !DOEND !DOEND !ENDDEFINE. SET UP OMS..RTFM FOR THAT... !AllPairsKappa raters= j1  j2  j3 ..... j30. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD
Open this post in threaded view
|