Demonstration Session 544: In this demonstration we will provide a conceptual overview of current and common interrater reliability procedures and discuss ways to apply them in practice. Additionally, we will discuss pertinent concerns surrounding rater training, such as diagnostic methods and obtaining reliability without sacrificing validity. Observational measurement serves an integral role in program evaluation. Due to the unique individual observer inferences, it is pertinent that every rater observe and interpret the same events similarly. This lack of similarity can greatly distort the interpretation of the effectiveness of the target intervention or performance being measured. We aim to help evaluators with a basic understanding of measurement theory and statistics make competent decisions regarding assessing interrater reliability. Understanding the various methods for assessing agreement or disagreement at the conceptual and practical levels helps reduce the influence of subjective judgment. Assessing interrater reliability of observational measures is vital for drawing accurate inferences. #2009Conference #QuantitativeMethods-TheoryandDesign