Improving the Quality of Clinical Assessments in Talking Therapies Through Machine Learning

Background

We developed a Clinical AI Assessment Assistant called Limbic Access, equipped with a clinical brain that uses specialised clinical machine learning models to inform clinical assessments and improve diagnostic quality. In this white paper, we collate three studies with a total of over 20,000 patients, and we show that Limbic Access' Machine Learning (ML) model identifies the appropriate additional outcome measures (known as Anxiety Disorder Specific Measures) with an accuracy of over 93% across the eight most common diagnostic categories, matching the performance of trained clinicians. Collecting additional, relevant outcome measures before assessment helps clinicians deliver a more informed assessment.

Methods

The ML model operates on a set of preprocessed input vectors (free text, questionnaire scores, behavioural indicators and demographic variables). The model transforms these preprocessed inputs into a set of probabilities across a Ranked Consideration Set over eight common diagnostic categories. The ranking is sorted in descending order of probability. The main evaluation metric of interest is the accuracy with which the algorithm would administer the relevant ADSM for a diagnosis, if that diagnosis was present. Formally that means the percentage of times with which the actual diagnosis is within the top two problems in Ranked Consideration Set selected and ranked by our machine learning model. With additional data collected by the selected outcome measures, Clinical Logic can be applied to generate a set of primary and secondary presenting problems, which can then be used by Clinicians to inform a more effective assessment.

Results

On historical data, the model achieved an overall accuracy of 93.5%. Moreover, it achieved good accuracy for each of the individual diagnostic categories as well, indicating that it has good performance. The model achieved a similar level of accuracy on our prospective evaluation - achieving an overall accuracy of 94.2%. On the live dataset, the Limbic Access correctly detected 92.47% of diagnoses, comparable to the performance on historical and prospective datasets.

‍

_{93% accurate}
Limbic Access machine learning accuracy when selecting the correct ADSMs

Conclusions

Together, this evidence supports our approach of using ML to

administer relevant ADSMs, and presenting clinicians with the entire set of model-informed Primary and Secondary Presenting Problems to assist in their assessments. By appropriately administering ADSMs pre-assessment and accurately identifying presenting problems, Limbic Access can save clinicians time and support the quality of their assessments, as well as offer patients an enhanced referral experience and potentially better outcomes.