Mhabitat Logo Mark Trans Rgb

NHS England Digital Development Lab and the creation of an effectiveness evaluation framework for digital tools

Process Map Example

Intro

The Digital Development Lab involved taking six existing digital tools for children and young people’s mental health through an extensive evaluation process similar to what is currently the Digital Assessment Questions (DAQ). One of the evaluation components was assessing the effectiveness of a digital tool. In collaboration with NIHR MindTech Health Technology Cooperative, mHabitat was commissioned to test current opinions on what is desirable and feasible in terms of effectiveness evaluation for digital tools, and to generalise this as an evaluation framework aimed at helping developers (the intended users of the DAQ) start their own evaluation studies, possibly in collaboration with clinical or academic partners.

To define the scope and content of the framework, we first organised a workshop to gather opinions from people from a variety of different backgrounds, including service users, clinicians, developers, third sector workers, academic researchers and regulators. Participants discussed a draft evaluation framework, considered how it would operate in practice and proposed ideas for improvement.

Based on the workshop feedback, we next developed a questionnaire to gather information about the digital tool. The aim of the questionnaire was two-fold. On one hand, developers could use the questionnaire to plan early stage evaluation studies or even (for new tools) build in evaluation from the very start of development. On the other hand, the responses to the questionnaire could also be used to provide the person who would do the expert evaluation with sufficient background information to create a report. This person could be for example an analyst or data scientist who would not necessarily know the prevalence or impact of a clinical condition, or current best practice and NHS pathways.

The 6 teams completed this questionnaire on their own and were then invited to provide feedback and discuss their evaluation plans in a subsequent meeting. The responses and feedback confirmed suggestions in the workshop that most developers feel they lack the skills, resources and incentive to run an evaluation. Tailored advice and support were offered to each team.

Based on all feedback in this project, the current state of research and our own evaluation experience, we created a user-friendly framework to help developers, especially those without research experience, evaluate the effectiveness of their digital tools. This framework is intended to be flexible and change over time as a result of user feedback as well as new policies and guidelines becoming available. The current version of the framework can be found in our open library.

Portrait Alicia RIdout

Alicia Ridout

Deputy Programme Director

Read bio
Portrait Francesca Quartarulli

Francesca Quartarulli

Digital Project Manager

Read bio
Portrait Steve Lloyd-Smart

Steve Lloyd-Smart

Technical Lead

Read bio
Portrait Natalie Nelissen

Natalie Nelissen

Research Fellow

Read bio
Open Library Framework for the effectiveness evaluation of mobile (mental) health tools

A Framework for the effectiveness evaluation of mobile (mental) health tools

If you are developing, commissioning or reviewing mobile health tools (apps and websites for health and wellbeing), this framework is here to make your job easier.

Download
Cropped Poster Kings Fund17 Digital Eval Frameworkm H

Developing a framework for the evaluation of digital mental health applications

A PDF version of a poster presented at the Kings Fund conference 2017

Download
Nhsddigital Innovation Map Fullgraphic 22 02 18

NHS Digital Diffusion

A review of the digital innovation landscape to identify gaps, and help the Partnerships and Innovation Team at NHS Digital understand how they could contribute to the digital ecosystem.

Learn more

Digital Innovation Pathway for Health and Care

Digital Innovation Pathway

Get in touch

Have a query or want a chat about what we offer? Call or email us.