Testing the reliability and validity of the Cluttering Severity Instrument
(2015) Testing the reliability and validity of the Cluttering Severity Instrument, no. 145.
Background Cluttering is a rather poorly defined fluency disorder, thought to be characterised by an accelerated and/or irregular speech rate with a large amount of disfluencies. As there is not yet a universally agreed upon definition of cluttering, clinicians have faced challenges identifying reliable and efficient assessment tools for the diagnosis of cluttered speech. A lack of diagnostics has also impacted upon the ability to develop treatment strategies and to monitor their success. A recently developed software programme called the Cluttering Severity Instrument (CSI; Bakker and Myers 2011) is anticipated to be the first acceptable, reliable and valid diagnostic tool for identifying cluttered speech. The CSI requires clinicians to provide perceptual ratings for eight speech characteristics, and to complete a task estimating the percentage of the sample that is cluttered. The CSI programme uses these two factors to calculate an overall CSI score. Aims The principle aim of the present study is to investigate whether the CSI programme can be regarded as a reliable and valid method of diagnosing cluttered speech. To achieve this, interrater and intra-rater reliabilities were determined. A second aim was to establish whether the CSI provides raters with the means to discriminate between impaired and unimpaired speech. A final aim is to identify whether raters' familiarity with speech analysis impacted upon rater reliability. Methodology Twelve raters provided perceptual ratings of eight speech characteristics and indicated the duration of the speech sample that they perceived to be cluttered. Raters recorded their judgements using the CSI programme and an overall cluttering percentage was calculated. Raters completed the experiment for five separate speech samples. Results Overall, inter-rater reliability was poor. The combined ratings of speakers and subtests found a high percentage of correlations, but not many with a strong coefficient value. Additionally, many of these correlations were determined to have significantly different means. Separate analysis of subtests indicated that although scores correlated, they often differed in range, with some raters giving consistently lower scores than others. These correlation analyses indicate that the CSI does not have strong inter-rater reliability. Intra-rater reliability was examined by re-testing four participants. Of these four, three had a strong correlation between their score sets, which provided some evidence of intra-rater reliability. When speakers were analysed separately, results indicated that raters were successfully discriminating between impaired and unimpaired speech. Correlation analysis of speech and language therapists (SLT) and laypersons groups did not determine a familiarity of speech analysis to have an influencing factor on perceptual judgements. Conclusions Based on the results of this study, we cannot conclude that the CSI programme is a reliable or valid diagnostic tool. Ratings provided by the SLT group were not found to be more reliable or less variable than the laypersons group. This indicated that a familiarity with speech analyses did not impact upon perceptual judgements of speech. Although the present study did not find significant evidence to support the CSI programme as a reliable and valid assessment tool, it did indicate that it provides raters with the means to successfully discriminate between impaired and unimpaired speech.