Mhabitat Logo Mark Trans Rgb

NHS England Digital Development Lab and the creation of an effectiveness evaluation framework for digital tools

Process Map Example


The Digital Development Lab involved taking six existing digital tools for children and young people’s mental health through an extensive evaluation process similar to what is currently the Digital Assessment Questions (DAQ). One of the evaluation components was assessing the effectiveness of a digital tool. In collaboration with NIHR MindTech Health Technology Cooperative, mHabitat was commissioned to test current opinions on what is desirable and feasible in terms of effectiveness evaluation for digital tools, and to generalise this as an evaluation framework aimed at helping developers (the intended users of the DAQ) start their own evaluation studies, possibly in collaboration with clinical or academic partners.

To define the scope and content of the framework, we first organised a workshop to gather opinions from people from a variety of different backgrounds, including service users, clinicians, developers, third sector workers, academic researchers and regulators. Participants discussed a draft evaluation framework, considered how it would operate in practice and proposed ideas for improvement.

Based on the workshop feedback, we next developed a questionnaire to gather information about the digital tool. The aim of the questionnaire was two-fold. On one hand, developers could use the questionnaire to plan early stage evaluation studies or even (for new tools) build in evaluation from the very start of development. On the other hand, the responses to the questionnaire could also be used to provide the person who would do the expert evaluation with sufficient background information to create a report. This person could be for example an analyst or data scientist who would not necessarily know the prevalence or impact of a clinical condition, or current best practice and NHS pathways.

The 6 teams completed this questionnaire on their own and were then invited to provide feedback and discuss their evaluation plans in a subsequent meeting. The responses and feedback confirmed suggestions in the workshop that most developers feel they lack the skills, resources and incentive to run an evaluation. Tailored advice and support were offered to each team.

Based on all feedback in this project, the current state of research and our own evaluation experience, we created a user-friendly framework to help developers, especially those without research experience, evaluate the effectiveness of their digital tools. This framework is intended to be flexible and change over time as a result of user feedback as well as new policies and guidelines becoming available. The current version of the framework can be found in our open library.

Portrait Alicia RIdout

Alicia Ridout

Deputy Programme Director

Read bio
Portrait Francesca Quartarulli

Francesca Quartarulli

Digital Project Manager

Read bio
Portrait Steve Lloyd-Smart

Steve Lloyd-Smart

Technical Lead

Read bio
Portrait Natalie Nelissen

Natalie Nelissen

Research Fellow

Read bio
Img 9417


A co-design piece of work with young people, family members and clinicians to develop a new website and future concepts for DigiBete, a website designed to support and benefit young people self-managing Type 1 diabetes.

Learn more

Website, Content

Co-design, young people, Diabetes, Website, Design

Get in touch

Have a query or want a chat about what we offer? Call or email us.