Assessing insomnia after stroke: a diagnostic validation of the Sleep Condition Indicator in self-reported stroke survivors

Introduction

A growing body of evidence indicates a bi-directional relationship between sleep and stroke.1 Insomnia, broadly defined as difficulty with initiating, maintaining or returning to sleep, which impacts daily functioning, is estimated to affect 32.21% to 50.4% of people affected by stroke; although estimates vary depending on diagnostic criteria and phase of recovery.2 3 Prior to an individual’s first stroke, insomnia symptoms are associated with a 30% increased risk of stroke4; and post-stroke are associated with poorer functional and cognitive outcomes,5 and psychological distress.6 Correspondingly, clinical guidelines and stroke organisations encourage post-stroke sleep assessment7 8 to facilitate stroke management and reduce the risk of recurrence.

With respect to the assessment and diagnosis of insomnia disorder in the general population, the use of clinical interviews investigating an individual’s sleep history is recommended.9 Psychometric tools commonly comprise part of this process, yet diagnostic validations of insomnia measures post-stroke have only recently been conducted. Chinese translations of the Insomnia Severity Index, Pittsburgh Sleep Quality Index and Athens Insomnia Scale have demonstrated ‘outstanding’ diagnostic accuracy post-stroke.10 However, none of these measures sufficiently capture the diagnostic criteria for insomnia disorder detailed in the Diagnostic and Statistical Manual of Mental Disorders—fifth edition (DSM-5).11

A valid diagnosis is dependent on specific diagnostic criteria,9 with which a scale intended for clinical use should be aligned. The Sleep Condition Indicator12 is a short self-report questionnaire, designed to screen for insomnia disorder in accordance with DSM-5 criteria.11 A recent validation of the Indonesian translation of the Sleep Condition Indicator (SCI)13 suggested the tool was valid for use post-stroke but recommended using a higher diagnostic threshold of ≤23 rather than ≤16 used in the general population.12 14 However, existing validations of SCI translations do demonstrate heterogeneity in estimates of optimal cut-offs,15–18 suggesting possible confounds from translation and/or nuanced cultural differences.

Accurate diagnoses are inextricably linked to appropriate treatment. Thus, a scale intended for clinical or research use must be able to discriminate accurately between individuals with and without the condition of interest. Therefore, this study investigated the discriminant validity and optimal threshold of the English language SCI when detecting insomnia disorder, and symptoms, in a sample of self-reported stroke survivors. Secondary objectives involved assessing convergent validity, examining relationships between the SCI and variables relating to mood and general demographics, and assessing the internal consistency of the SCI after stroke. Additionally, we explored the discriminant validity, optimal threshold and internal validity of the SCI-2,19 a two-item version of the SCI comprised of items 3 and 7 from the full scale.

Materials and methods

The pre-registered protocol for this cross-sectional prospective diagnostic validation study can be found at: https://doi.org/10.17605/OSF.IO/4DGXW.

Results

Participant characteristics

A total of 544 potential participants accessed the study platform (see figure 1), of which 246 exited before providing consent, and 118 did not complete the SCI and/or sleep history. The remaining 180 provided consent and all met inclusion criteria. The mean survey duration was 24.48 min (SD=10.56). Participant characteristics are summarised in table 1.

Request permission

Figure 1

Participant flowchart.

Table 1

View inline|View popup

: Participant characteristics

After classification based on the sleep history, 75 (41.67%) participants met criteria for insomnia disorder, 33 (18.33%) exhibited insomnia symptoms without meeting diagnostic criteria and 72 (40.0%) had no insomnia symptoms. A breakdown of participants by classification is available in figure 1 (a confusion matrix of index vs reference standard classifications can be found in the online supplemental material 2).

The distribution of total SCI scores by classification is displayed in figure 2. Skewness and kurtosis were 0.40 and 2.46, respectively. Visual inspection and Anderson-Darling tests (p=0.004) indicated non-normal distribution of total SCI scores.

Request permission

Figure 2

Violin and boxplot graphs displaying the distribution of total SCI scores by classification. SCI, Sleep Condition Indicator.

Sensitivity and specificity of the SCI

When detecting insomnia disorder post-stroke using the SCI, the AUC was 0.86 (95% CI (0.81, 0.91); figure 3), with an optimal cut-off of ≤13, yielding sensitivity, specificity and Youden’s J values of 88.0%, 71.43% and 0.59, respectively (see table 2). When detecting insomnia symptoms, the AUC was 0.85 (95% CI (0.80, 0.91); figure 3), with an optimal cut-off of ≤14, yielding sensitivity, specificity and Youden’s J values of 81.48%, 76.39% and 0.58, respectively (see table 2). Both AUC curves are indicative of ‘excellent’ diagnostic accuracy.32 Delong’s test for two paired ROC curves found no statistical difference (p=0.91) in the SCI’s ability to detect insomnia disorder vs symptoms. Using the sensitivity observed when detecting insomnia disorder, with all other parameters unchanged, returns a power of 0.71.

Request permission

Figure 3

ROC curves displaying the accuracy of the SCI when detecting Diagnostic and Statistical Manual of Mental Disorders—fifth edition (DSM-5) insomnia disorder and symptoms, including AUC values and 95% CIs. ROC, receiver operating characteristic; SCI, Sleep Condition Indicator; AUC, area under the curve.

Table 2

View inline|View popup

Sensitivity, specificity and Youden’s J statistic for select thresholds on the SCI when detecting DSM-5 insomnia disorder and insomnia symptoms

Sensitivity and specificity of the SCI-2

ROC analysis was performed on the summed totals for items 3 and 7 of the SCI (figure 4), forming the SCI-2.19 When detecting DSM-5 insomnia in stroke survivors, the AUC for the SCI-2 was 0.79 (95% CI (0.73, 0.85)), with an optimal cut-off of ≤2. Corresponding with a specificity of 68.57%, sensitivity of 76.0% and a Youden’s J of 0.45. Delong’s test provided statistical evidence (p<0.001) that the SCI outperformed the SCI-2. Sensitivity and specificity values at all thresholds can be found in online supplemental material 4. The positive and negative predictive values of the SCI in this sample, using the optimal threshold of ≤13, were 0.69 and 0.89, respectively. Predictive values at a range of hypothetical prevalence rates can be found in online supplemental material 5.

Request permission

Figure 4

ROC curves displaying the accuracy of the SCI-2 when detecting Diagnostic and Statistical Manual of Mental Disorders—fifth edition (DSM-5) insomnia disorder and symptoms, including AUC values and 95% CIs. ROC, receiver operating characteristic; AUC, area under the curve; SCI, Sleep Condition Indicator.

Convergent validity

Spearman’s rank correlation was computed to assess the relationship between total scores on the ISI and SCI, with ties handled by average ranks. Results revealed a large negative correlation between the two scales (r (178) = −0.86, 95% CI (−0.91, –0.80); figure 5).

Request permission

Figure 5

Scatter plot visualising the relationship between total SCI and Insomnia Severity Index (ISI) scores.

Discussion

This is the first study examining the SCI’s validity and reliability, and that of any insomnia screening tool, in a sample of English-speaking stroke survivors. Results from 180 participants indicated that 41.7% met DSM-5 criteria for insomnia disorder, and 60.0% exhibited insomnia symptoms (including those classified as insomnia disorder), consistent with meta-analytic findings on post-stroke insomnia prevalence (2,3), and supporting the representativeness of the sample. The remaining 40% had no insomnia symptoms.

The SCI demonstrated ‘excellent’ diagnostic accuracy when screening for DSM-5 insomnia disorder (AUC=0.86) and/or symptoms (AUC=0.85) post-stroke. Moreover, examination of internal consistency indicated acceptable reliability (Cronbach’s α=0.84). The SCI-2 demonstrated ‘acceptable’ accuracy, with an optimal cut-off of ≤2; paralleling validations in the general population.19 However, the full SCI demonstrated significantly higher accuracy and should be the preferred assessment method where possible.

Estimates of diagnostic accuracy in the current study are similar, although slightly lower, to recent validations in stroke populations of the SCI in Indonesian13 and the Insomnia Severity Index, Pittsburgh Sleep Quality Index and Athens Insomnia Index in Chinese.10 Nonetheless, comparisons should be drawn cautiously. Existing validations of SCI translations demonstrate heterogeneity in estimates of optimal cut-offs,15–18 suggesting possible confounds from translation and/or nuanced cultural differences. Additionally, a recent meta-analysis demonstrated that condition prevalence may influence the sensitivity and specificity of binary diagnostic tests.33 The prevalence of insomnia in the previous validations10 13 was 15.6% and 31.0%, respectively, both being lower than the current study (41.7%) and previous estimates of post-stroke insomnia prevalence.2 3 Thus, the variance in prevalence should be considered when comparing studies of diagnostic accuracy, and when considering a screening tool for clinical or research use.

The optimal cut-off for detecting insomnia disorder using the SCI in the current study was found to be ≤13. This is lower than both the conventional cut-off for the general population of ≤16,12 14 and the optimal cut-off of ≤23 observed in the validation of the Indonesian translation of the SCI post-stroke.13 Were the conventional cut-off adopted in the current sample, 96% of positive insomnia disorder cases would be correctly identified. However, approximately 45.7% of participants without insomnia disorder would be incorrectly classified as a positive case (ie, having insomnia disorder). Depending on the intended use for the scale, there may be instances where clinicians or researchers wish to maximise sensitivity at the expense of specificity (or vice versa) but should do so with caution. Use of an erroneously high threshold on the SCI will lead to an increased false-positive rate, potentially leading to inappropriate treatment and use of resources in clinical settings; and may cast doubt on the validity of findings in research.

Understanding the heterogeneity of optimal diagnostic thresholds is important for ensuring the SCI is appropriately used in different clinical samples, and future research should explore this further. One possibility is that poorer sleep post-stroke34 may necessitate a lower threshold. Mean SCI scores in the current study (x̄ = 13.74, SD=7.51) are slightly lower and display greater variance than results from a retrospective exploration of scores in 200 000 adults who completed the SCI (x̄ = 14.97, SD=5.93; 14). However, the results of the latter14 may be biased towards people with sleep impairment, and the true population mean SCI scores are likely higher. Variation in reference standards likely also contributes to variability in optimal cut-offs, and ideally studies seeking to validate measures of insomnia should select reference standards in accordance with guidelines on the assessment and diagnosis of insomnia disorder.9 In studies of diagnostic accuracy, the reference standard assumes infallibility. Yet, the index test may not always be solely responsible for the discrepancy.35 The conventional cut-off of ≤1612 14 was determined as the minimum score necessary for putative DSM-5 insomnia disorder, with convergent validity assessed against the ISI. The ISI boasts commendable accuracy but standalone is not a recommended tool for diagnosing DSM-5 insomnia disorder9 and is less specific than clinical interviews. Thus, when comparing the optimal diagnostic cut-offs between studies, one should be cognisant of the possibility of differences in reference standards explaining some or all the variation in optimal cut-offs.

Spectrum bias, a phenomenon whereby accuracy of a diagnostic test may be confounded by symptom severity,36 may further explain discrepancies in optimal diagnostic thresholds between studies. Scores closer to the cut-off are more prone to misclassification; therefore, studies where index test scores demonstrate a bimodal distribution have a greater risk of spectrum bias. Hasan et al13 excluded 164 participants with sleep or psychiatric disorders other than insomnia in their validation. Many of these participants would likely have scored lower on the SCI regardless of insomnia classification, as suggested by the relationships between SCI score and measures of mental health in the current study and corroborated by meta-analytic results from Hertenstein et al.37 Likely as a result of their inclusion criteria, mean SCI scores in Hasan et al13 for participants without insomnia were 29.55 (SD=1.96), with little variance. Compared with 19.33 (SD=6.81) in the current study, which is more akin to previous research.34 Such a skewed distribution of scores allows specificity to remain high at higher thresholds, contributing to potentially misleading estimates of accuracy and optimal cut-offs. Hasan and colleagues demonstrate that the Indonesian SCI can discriminate post-stroke insomnia disorder in the absence of potential confounds. The present study adds to this by validating the English SCI in self-reported stroke survivors and doing so while including participants with comorbid psychological or sleep disorders; thereby increasing generalisability of the findings. Results of the current study, when considered in the context of the existing literature, highlight the importance of considering spectrum bias in studies of diagnostic accuracy. When considering a screening tool, researchers and clinicians should consider whether a study’s selection criteria are clinically meaningful to the population they intend to use the test in.

This study has several strengths. A priori power analyses, a rarity in studies of diagnostic accuracy,38 ensured the study was suitably powered to obtain reliable estimates of accuracy. Consecutive completion of index and reference tests reduced the risk of temporal changes in disease state. Finally, stratifying analyses by insomnia classification reduced the risk of spectrum bias and prevalence influencing estimates of accuracy.33

Nonetheless, voluntary response sampling does carry a risk of response bias and may lead to over-representation of participants with an interest in sleep, or more severe sleep difficulties. However, prevalence in the current sample aligns with that of previous meta-analytic estimates.2 3 Recruiting participants via social media may have led to a younger than expected stroke sample (mean age=49.61 ± 12.41 years). Future research may wish to consult older stroke survivors to explore methods of increasing the accessibility of online stroke research to a wider age range. Nevertheless, stroke can affect people of any age, and research representative of younger stroke survivors is necessary, particularly to understand the challenges of improving life after stroke for working age individuals—perhaps with young families—compared with older adults. Results of a linear regression from a large cross-sectional study (n=200 000) by Espie and colleagues14 did demonstrate that SCI score was predicted to decrease 0.057 points per year of life. Findings from the current study demonstrated no significant relationship between age, or gender, and total SCI score; however, it is likely that a much larger sample size would be needed to detect an effect as small as that described in Espie et al.14 Thus, although the modulating effect of demographic characteristics on the accuracy of the SCI was not explored, current findings provide no evidence to suggest that such an effect exists. Due to the online nature of recruitment, participant self-reports of stroke were not verified by another means. Various efforts were made, however, to minimise the risk of individuals without a history of stroke volunteering to take part. First, recruitment adverts were shared explicitly by stroke researchers and third sector organisations supporting stroke survivors, targeting stroke peer support groups. Moreover, offering no incentive to participants removed the risk of individuals taking part without a history of stroke for financial gain. Furthermore, while the results of a systematic review by Woodfield et al39 demonstrate that the sensitivity and positive predictive value of self-reported stroke is low in samples with low prevalence, specificity (96% to 99.1%) and negative predictive value (88.2% to 99.9%) were consistently high. As one would expect, authors report that positive predictive values increased in line with prevalence. Thus, in samples with high prevalence of stroke, such as in the current study, the risk of false positives is likely to be small. We therefore believe self-reports of stroke in the present sample to be reliable. Nevertheless, where time and resources allow, future researchers may wish to verify self-reported stroke status by recruiting participants from stroke outpatient clinics or obtaining clinical records. Moreover, the number of incomplete responses (n=118) is worth noting. This may indicate systemic challenges related to conducting stroke research online, or difficulties faced by potential participants with language difficulties. Relatedly, the duration of the survey was relatively long, and future research may consider limiting survey length further to increase accessibility. Finally, participants were not asked which type of stroke they had. This was a methodological decision to avoid excluding participants who did not know or remember. Existing evidence does indicate potential relationships between insomnia and stroke type/location;40 therefore, future researchers may wish to consider purposive sampling methods that allow exploration of whether these phenomena may influence insomnia presentation and the accuracy of diagnostic measures.

In summary, this study confirms the validity and reliability of the English version of the SCI when detecting DSM-5 insomnia in self-reported stroke survivors and indicates that a lower threshold than is traditionally used should be considered when using the SCI in the stroke population. At the optimal diagnostic threshold of ≤13, the SCI demonstrates ‘excellent’ diagnostic accuracy. The brevity of the tool makes it attractive for clinical and research use and the SCI-2 does offer an acceptable shorter alternative. Nevertheless, the accuracy of the SCI-2 is outperformed by the full SCI, and use of the latter should be preferred where possible. In conclusion, the SCI should be considered a valid and reliable screening tool for DSM-5 insomnia disorder and insomnia symptoms post-stroke.