The Degree Of Agreement Among Several Trained Experts Refers To Group Of Answer Choices

To start, leave us on the same page about what we mean by “reliability.” Reliability is the degree to which a particular research method or tool is able to achieve consistent results from one test to the next. Third, the researcher must indicate the unit of analysis to which the CCI results apply, i.e. whether the CCI should quantify the reliability of ratings on the basis of average ratings provided by several coders or on the basis of the ratings of a single coder. In studies where all subjects are encoded multiple times and the average of their assessments is used for hypothesis testing, CCIs are appropriate with average measurements. However, in studies in which a subset of subjects is coded by several medical specialists and where the reliability of their assessments must be generalized to subjects evaluated by a programmer, it is necessary to use a single action ICC. Just as the average of several measurements tends to be more reliable than a single measure, CCIs with average measurements tend to be higher than CCIs with individual measurements. In cases where CCIs are low with individual dimensions, but ICCs are on average high, the researcher can report both ICCs to demonstrate this discrepancy (Shrout- Fleiss, 1979). Krippendorffs Alpha[16][17] is a versatile statistic that evaluates the agreement between observers who categorize, evaluate or measure a certain number of objects against the values of a variable. It generalizes several specialized agreement coefficients by accepting any number of observers applicable to nominal, ordinal, interval and proportional levels of measurement, capable of processing missing and corrected data for small sample sizes. The possible values for Kappa`s statistics range from 1 to 1, 1 indicating a perfect match, 0, indicating a totally random match, and 1, indicating a “perfect” divergence. Landis and Koch (1977) provide guidelines for interpreting Kappa`s values, values between 0.0 and 0.2 being slightly consistent, 0.21 to 0.40, indicating fair consent, from 0.41 to 0.60, indicating moderate support, from 0.61 to 0.80 and from 0.81 to 1.0, indicating near-perfect or perfect consistency. The use of these qualitative limit values, however, is under discussion and Krippendorff (1980) gives a more conservative interpretation that suggests that conclusions should be reduced for variables with values below 0.67, conclusions for values between 0.67 and 0.80, and final conclusions for values greater than 0.80.

However, in practice, Kappa coefficients below Krippendorff`s conservative Cutoff values are often retained in research studies, and Krippendorff proposes these cutoffs on the basis of his own content analysis work, while acknowledging that acceptable estimates of IRR vary according to study methods and the research issue.

This entry was posted in Uncategorized by admin. Bookmark the permalink.

Warning: count(): Parameter must be an array or an object that implements Countable in /homepages/7/d328811302/htdocs/teamalter/wp-includes/class-wp-comment-query.php on line 405

Comments are closed.