Two observers independently rate the same 30 pressure-injury photographs using a four-category scale. Their ratings agree on 18 photographs. Which conclusion about inter-rater reliability is most reasonable?
Free Applied Statistics in Healthcare practice question on Reliability and validity for BSN and Post-RN nursing students. Choose your answer, then open the answer and the explanation.
AAgreement should be ignored, because observers rated the same photographs rather than patients
BAgreement is high enough to accept the scale for routine use without further analysis
CAgreement shows the scale is valid, since consistent ratings reflect a true measure
DAgreement is modest and should be checked against chance agreement before accepting the scale
Show answer and explanation
✓ Correct answerD. Agreement is modest and should be checked against chance agreement before accepting the scale
Explanation
Inter-rater reliability is the degree to which two or more independent raters give the same score to the same thing. Raw agreement alone can look good just by chance, especially when one category is used very often, so kappa or another chance-corrected statistic is used to check how much agreement exceeds chance. Agreement between raters tells us the measurement is consistent, not that it measures the right thing.
Why each option is right or wrong
✗A. Agreement should be ignored, because observers rated the same photographs rather than patients — Rating the same photographs is exactly the correct design for inter-rater reliability, since both raters must judge identical material.
✗B. Agreement is high enough to accept the scale for routine use without further analysis — Raw agreement of 60% is not strong evidence, and consistency between raters still needs chance correction before routine use is justified.
✗C. Agreement shows the scale is valid, since consistent ratings reflect a true measure — Consistency between raters concerns reliability, while validity is whether the scale truly captures the intended construct.
✓D. Agreement is modest and should be checked against chance agreement before accepting the scale — Sixty percent raw agreement is modest, and because chance alone can produce some agreement, a chance-corrected index such as kappa is needed before accepting the scale.
Concept
What it measures
Typical statistic
Inter-rater reliability
Agreement between two or more raters
Cohen's kappa, ICC
Test-retest reliability
Stability of scores over time
Pearson r, ICC
Internal consistency
Consistency among items of a scale
Cronbach's alpha
Validity
Whether the tool measures the intended construct
Content, criterion, construct evidence
EXAM HINT Reliability = repeatable; validity = correct. Kappa corrects raw agreement for chance.
اردو وضاحت
Inter-rater reliability کیا ہے؟
Inter-rater reliability کا مطلب یہ ہے کہ دو یا زیادہ raters ایک ہی چیز کو دیکھ کر کتنے ملتے جلتے score دیتے ہیں۔ اگر دونوں کا score قریب قریب ہو تو کہا جاتا ہے کہ measurement consistent ہے۔ لیکن صرف خام agreement دیکھنا کافی نہیں، کیونکہ کبھی کبھی اتفاق محض اتفاق سے بھی ہو جاتا ہے۔
اس صورت میں کیا نتیجہ نکالنا چاہیے؟
جب 30 میں سے صرف 18 پر اتفاق ہو تو agreement درمیانی ہے، اور اسے chance agreement کے خلاف جانچنا چاہیے، جیسے kappa نکال کر۔ صرف خام اتفاق دیکھ کر scale کو قبول کر لینا یا اسے validity سمجھ لینا غلط ہے، کیونکہ reliability اور validity دو الگ چیزیں ہیں۔
Key terms
chance agreement — Agreement that happens just by luck rather than because the raters truly agree. (ایسا اتفاق جو محض اتفاق سے ہو، اصل رضا مندی سے نہیں۔)