Manchester Metropolitan University – Evaluating health technology systems
Switching to electronic health records can be extremely beneficial.
With electronic records, clinicians can provide accurate, up-to-date, and complete information about patients at the point of care. They can share electronic information with other health professionals. They can diagnose patients more effectively, reduce medical errors and provide safer care.
However, back in 2000, the NHS was still predominantly sticking with paper records, while other industries were gravitating towards IT systems. Why was this?
In short, systems were not designed to fit into clinicians’ daily working practices, and involved spending large amounts of time typing in data. Consequently, they were shunned in favour of tried-and-trusted methods.
In 2001, the NHS Information Authority initiated The Electronic Record Development and Implementation Programme (ERDIP). The new programme funded a number of beacon sites to explore the issues associated with the creation of electronic health records.
Heather Heathfield
Dr Heather Heathfield was part of a team commissioned to evaluate the programme. Dr Heathfield had designed electronic health records and decision support systems as part of her PhD work, and continued to explore this field both at IBM and later as a senior lecturer at Manchester Metropolitan University. She worked on the project with healthcare consultancy Secta and the Yorkshire Institute for Health Informatics at the University of Leeds.
“Having been involved in the design of a number of electronic health records and decision support systems, I was frustrated that despite the obvious benefits, these systems were not being used” she says. “I wanted to know what the barriers were and how we could overcome them.”
Their evaluation helped them to discover a great deal about the barriers that stopped electronic health records from being implemented, and gave them some insight into how to systematically identify and address them.
Following several publications on the subject, the NHS Information Authority went one step further, and encouraged them to work on a practical toolkit, suitable for an age in which clinicians were being asked to take a larger role in evaluating and commissioning systems. The Project Review and Objective Evaluation toolkit – known as PROBE – promoted a way of thinking that placed evaluation at the start of the process, and encouraged organisations to be more rigorous with their processes.
First released in 1996, PROBE was based on a six-step evaluation process, in which decision makers would be asked to agree on why an evaluation was needed, when, what and how to evaluate, analyse and report and finally to assess recommendations and decide on a course of action.
PROBE became a central part of NHS information strategy and policy. Since its revised publication in 2001, it has sparked the NHS to look at evaluation as a key strand of major IT projects.
Dr Heathfield says there are several “key principles of evaluation”, including advance planning, the establishment of clearly defined aims and objectives, and the inclusion of formative and summative elements. Evaluation needs to be closely integrated into the life-cycle of the project, and evaluators need to be able to compare the results to what went before by collecting both quantitative and qualitative data throughout.
“An evaluation framework needs to address structure, processes and outcomes”, says Dr Heathfield. “It should do this across five dimensions – strategy, operational, human, financial and technical.”
In 2012, NHS Connecting for Health’s head of patient and public partnerships said that Dr Heathfield’s work “greatly improved understanding of ICT projects” by focusing analysis on the important questions, and providing more information on the tools and techniques available for effective evaluation.
The PROBE methodology is still widely used in the UK. For example, it was employed by the National Patient Safety Agency to evaluate systems designed to improve the safety of blood transfusions.
A total of five patients died after receiving incompatible blood during transfusions between 1996 and 2004. The issue contributed to the deaths of nine others, and caused major illness in 54 more. The National Patient Safety Agency used PROBE to evaluate systems that could improve the process.
Using PROBE helped the National Patient Safety Agency develop a detailed picture of the views of stakeholders, and encouraged decision makers to think about areas such as installation of the system, any changes that were required, the response of staff, and its effectiveness and reliability. The final report in 2011 observed that the system chosen as a result of this process reduced the time to resolve problems with transfusion from 123 hours to around 30 minutes
Dr Heathfield is now a principal and Head of Evaluation at OPM. Her work focusses on evaluating complex transformational initiatives in health and social care, with a particular emphasis on integrated care and new models of commissioning.
“My key aim at the time was to raise awareness and promote debate about the role and appropriate methods for evaluating IT systems in healthcare”, Dr Heathfield says. “It certainly helped people design better evaluations. PROBE is still in used in teaching, by researchers and others involved in that process.
I still use the principles set out in PROBE in all my evaluations. It’s about being systematic and understanding the impact of an intervention or new model from a range of perspectives, including how it fits into local working practices.
Links to Additional Information
http://cs-academic-impact.uk/it-systems-in-healthcare/Case Study
You might also like to read:
Leave a Reply
You must be logged in to post a comment.