Gates Open Res Gates Open Research 2572-4754 F1000 Research Limited London, UK 10.12688/gatesopenres.13001.1 Research Article Articles Evaluation of the electronic health record system in maternal and child health centers of Marie Stopes International Ethiopia

[version 1; peer review: 2 not approved]

 Gizaw Tizita Conceptualization Formal Analysis Investigation Methodology Project Administration https://orcid.org/0000-0001-9943-9512 a 1 Bogale Mengistu Conceptualization Methodology Supervision 2 Alemayehu Tilahun Data Curation Resources Visualization Writing – Original Draft Preparation 3 Department of Social Franchising, Marie Stopes International Ethiopia, Addis Ababa, Ethiopia Research and Publication Department, Addis Ababa University School of Commerce, Addis Ababa, Ethiopia Department of Quality Assurance and Quality Control, Addis Pharmaceutical Factory, Addis Ababa, Ethiopia tizugiz@gmail.com

No competing interests were disclosed.

 7 11 2019 2019 3 1655 3 10 2019 Copyright: © 2019 Gizaw T et al. 2019 This is an open access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Background: The Health Information Management Systems Society (HIMSS) defines an Electronic Health Record (EHR) as a longitudinal electronic record of patient health information generated by one or more encounters in any care delivery setting”. EHR systems improve quality and adaptability of care, contributing to patient safety. The system is also efficient and provides real-time administration to patients. Driven by the need to facilitate clinical and administrative process, and to reduce medical errors, Marie Stopes international Ethiopia (MSIE) implemented EHR system in its maternal and child health (MCH) centers. This study sought to evaluate the EHR system in six departments (inpatient, outpatient, pharmacy, laboratory, reception cashiers and MCH managers) of these MCH centers in terms of ease of performing tasks and satisfaction with the system.

Methods: A cross-sectional method with formative evaluation was used to address the objective. A sample size of 54 were used for the investigation. The participants were from six departments (inpatient, outpatient, pharmacy, laboratory, reception cashiers and MCH managers) within three MCH centers. Data was collected using structured, self-administered questionnaires and interviews and analyzed using SPSS 20.

Results: In total, 83% of respondents found that performing tasks using EHR made tasks easier; the lowest score was from Pharmacy department (48%). The overall satisfaction with EHR was 87%. The lowest score is from the indicator “the system meets my needs” (80%).

Conclusions: Staffs in all department reported the EHR system as “very good” for both ease of performing their daily tasks and satisfaction with the system. In addition, the study revealed that ease of performing tasks and satisfaction with the system were varied by department and specific task within the department. It is essential to know the needs and requirements of each department before implementation of the system and user feedback for long lasting uptake and impact.

 Electronic health recording system user satisfaction Ethiopia and evaluation Department for International Development, UK Government B012 E2601 Gates Foundation OPP1181398 The Gates Foundation funded publication of this work (grant number OPP1181398). Funding for this work was provided to Marie Stopes Ethiopia by UKAID (B012,E2601), EKN (B002,E2578) and a large anonymous donor (E001,E2512). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Abbreviation

EHR, electronic health record system; HIMSS, health information management system society; MCH, Maternal and Child health; MSIE, Marie Stopes International Ethiopia

 Introduction

The Health Information Management Systems Society’s (HIMSS) define EHR as: “The Electronic Health Record (EHR) is a longitudinal electronic record of patient health information generated by one or more encounters in any care delivery setting. Included in this information, are patient demographics, progress notes, problems, medications, vital signs, past medical history, immunizations, laboratory data and radiology reports” ( Ahmed et al., 2013).

Recording of patient data on paper obstructs the coherence and nature of medical care offered to patients. Furthermore, paper-based frameworks have restricted usefulness, limiting reviewing of medical records by care givers simultaneously. Storage of paper records takes up more space and hinders easy access to patient information ( Carayon et al., 2009). By contrast, EHR can create a complete patient record of the entire clinical experience. The system automates the clinical work process and improves quality of care ( Ahmed et al., 2013). Patient records can also be easily retrieved using the patient’s name, phone number or registry number.

Currently, paper-based recordings are being migrated to electronic recording systems because electronic health recording (EHR) systems improves quality and adaptability of care recording, contributing to patient safety. The system is also efficient and provides real-time administration to patients ( Rezaeibagha et al., 2015). Therefore, using HERs is an essential part in health care delivery to improve quality and safety ( Ellsworth et al., 2017). The US Institute of Medicine has characterized EHRs as an essential technology for improving the safety, quality and efficiency of health care ( Chiang et al., 2013).

An assessment of nine hospitals that implemented a comprehensive electronic health record (EHR) system discovered an improvement on the speed and accuracy of communication leading to less duplicative tests, quicker responses to client’s needs, and a complete capture of charges ( Silow-Carroll et al., 2012).

The averted costs associated with efficiencies created by maintaining and availing patient data electronically contributes to the financial benefits of the EHR system. Some of the averted costs are related to reduced staff number required in patient management, reduced stationary and printing costs required to maintain paper records ( Menachemi & Collum, 2011).

On the other hand, a study done in a residency’s family medicine center in Birmingham, AL, USA, revealed that Physicians and residents are very dissatisfied with the amount of time required for documentation using the EHR system ( Bloom & Huntington, 2010). Following the implementation of EHR, both clinical staff and office staff spent more time on transcribing data to computer to maintain medical information system ( Carayon et al., 2009).

In this study we will evaluate the EHR system implemented in three of the MCH centers in Marie stopes international Ethiopia (MSIE). The study will evaluate ease of performing tasks and satisfaction with the system in six departments (inpatient, outpatient, pharmacy, laboratory, reception cashiers and MCH managers). Each department performs a unique task that contributes to the overall performance of the facility. Thus, we hypothesized that the ease of performing tasks and satisfaction with the EHR system varies by department and by specific task within a department.

 Methods Study approach

We used a cross-sectional study with formative evaluation method to address the objective. At the time of the investigation there were 134 staff members from three of the MCH centers using the EHR system in their daily activities. Of the 134 staff members, only 68 fulfilled the selection criteria of working 6 months in the MCH center prior to the implementation of the system. From the selected 68 staff members, only those trained by MISE about the system were selected as final study participants. Therefore, a total sample size of 54 participants were used for the investigation. The participants were from six departments (inpatient, outpatient, pharmacy, laboratory, reception cashiers and MCH managers) within three MCH centers.

Data was collected using structured, self-administered questionnaires and interviews (available as Extended data ( Sharew, 2019)). The interviews were conducted with two people from each department to find out their daily tasks. Then a unique set of task list was prepared for each department. Using these task lists, a different questionnaire for each of the six departments were developed. The questionnaires involved a section where the respondents answer by ticking and a section where respondents are asked to answer by writing.

 Data analysis

Data were then checked for completeness, consistency and analyzed using SPSS for Windows v20. The usefulness dimension was measured by how the participants found the system affecting their day to day activities and operations. The user satisfaction-dimension was measured for relevant aspects. Each dimension and their indicators were judged using a judgement criterion: >95% excellent, 80–94% very good, 65–79% good, 50–64% fair, <50% poor. Descriptive statistics tables and graphs will be used to describe the findings.

 Ethical approval

The study was conducted after ethical clearance was obtained from the Institutional Ethical Review Committee of Addis Ababa University School of Commerce. During the data collection period, the participants were well informed about the purpose of the study. Written informed consent of the respondents was obtained before the commencement of the study. Involvement was fully decided by the participants. The response was confidential. Monitoring and supervision of data collectors was done throughout the whole data collection period. To assess the completeness, clarity and consistency the collected data were checked daily after data collection.

 Results and discussion Demographic data of respondents

Of the total 54 respondents, 17 (31%) were from the inpatient department and 18 (33%) were from the outpatient department. Of all the participants, 31 (57%) worked between 4 and 5 years with the organization. Of the respondents, 14 (26%) were nurses and 11 (20%) were midwives ( Table 1). Results from all surveys and questionnaires are available as Underlying data ( Sharew, 2019).

Descriptive of socio-demographic of respondents (n=54).
Sociodemographic variables Frequency Percentage
Department of respondents Inpatient 17 31%
Outpatient 18 33%
Pharmacy 5 9%
Laboratory 4 7%
MCH Manager 3 6%
Reception Cashier 7 13%
Type of respondents Midwife 11 20%
Nurse 14 26%
Health officer 6 11%
Gynecologist 4 7%
Pharmacist/Pharmacy technicians 5 9%
Laboratory technicians 4 7%
Non-health professionals 10 19%
Work experience, years 4–5 31 57%
5–6 15 28%
7–10 8 15%
Ease of performing tasks using EHR Inpatient department

Most participants from the inpatient department agree that the EHR system made their daily tasks easier. Admitting patients, following admitted patients’ medical usage and monitoring consumables during procedure were indicated by all the participants from the department as activities made easier following EHR implantation ( Table 2). Similarly, a study done in a small family practice clinic in USA also indicated an increase in time spent on patient care and decrease in time spent in meeting and performing lab works ( Carayon et al., 2009).

Ease of performing tasks using EHR in the inpatient department.
How has the EHR system changed the performance of the following tasks? More difficult/slightly more difficult/no change Slightly easier/easier/ significantly easier
To admit patients has become 0% 100%
To refer clients to other Medical center/ specialists/departments has become 6% 94%
To follow admitted clients Medication usage 0% 100%
To monitor consumables during procedure 0% 100%
Billing for consumables 6% 94%
To follow 1 st stage labor care 12% 88%
To follow 2 nd stage labor care 12% 88%
To monitor labor using partograph 12% 88%
To follow post-delivery status 6% 94%
To monitor OR outcome 6% 94%
To monitor Neonate outcome 24% 76%
Overall ease of performing tasks using EHR/Inpatient 92%
Outpatient department

In total, 94% of those in the outpatient department felt EHR made performing tasks easier ( Table 3). All respondents agree that the system made it easier to seek out specific information from the client record, to produce data reviews for specific patient groups, to order clinical laboratory, to order and receive ultrasound results and to write prescription. However, only 61% of the respondents agree that the system made it easier to complete sick-leave forms.

Ease of performing tasks using EHR in the outpatient department.
How has the EHR system changed the performance of the following task? More difficult/slightly more difficult/No change Slightly easier/easier/ significantly easier
To review the client’s problems has become 6% 95%
To seek out specific information from the client record 0% 100%
To follow the results of a test or investigation has become 6% 94%
To enter daily notes has become 6% 94%
To answer questions concerning general medical knowledge (e.g. Concerning treatment, symptoms, complications etc.) has become 11% 89%
To produce data reviews for specific patient groups (EG. Complication rate) has become 0% 100%
To order clinical laboratory analysis has become 0% 100%
To obtain the results from clinical laboratory analysis has become 6% 94%
To order ultrasound investigation has become 0% 100%
To obtain results from ultrasound has become 0% 100%
To refer the patients to other departments or specialists has become 11% 89%
To order treatments directly (E.g. medicines, operations etc.,) has become 6% 94%
To write prescription has become 0% 100%
To complete sick-leave forms has become 39% 61%
To give written individual information to patients (E.g. About medications, disease status) 6% 94%
Over all ease of performing tasks using EHR/Outpatient 94%
Pharmacy department

The result showed that the system made it easier to know available drugs in the dispensing area and billing for medications. However, the respondents indicated that the system made the it difficult to rule out drug interactions and maintain allergy and active medication lists ( Table 4). The study revealed that the need and the requirements of pharmacy department had not been reflected in the system design. The agency of healthcare research and quality to support research policy information in the area of EHR indicated that, if effectively implemented, EHR system reduces the need to rely on memory alone for information required to complete a task such as medical history, allergies and formulations ( Armijo et al., 2009).

Ease of performing tasks using EHR in the pharmacy department.
How has the EHR system changed the performance of the following tasks? More difficult/slightly more difficult/no change Slightly easier/easier/ significantly easier
To clearly read prescription 40% 60%
To dispense medications 40% 60%
To dispense multiple drug from the prescription 60% 40%
To know available drugs in dispensing area 0% 100%
Monitor medications by expiry date 80% 20%
Serving clients in order 20% 80%
Billing for medication 0% 100%
Implement drug-drug and drug-allergy interaction checks 100% 0%
To maintain an active medications allergy list 100% 0%
To maintain an active medications list 100% 0%
To detect duplication of therapies 20% 80%
To monitor controlled drug prescription 60% 40%
Over all ease of performing tasks using EHR/Pharmacy 48%
Laboratory department

The overall ease of performing tasks using EHR in laboratory department was 82%. From the list of activities performed in laboratory department, only half of the agree that the system make processing lab packages by category of services easier ( Table 5).

Ease of performing tasks using EHR in the laboratory department.
How has the EHR system changed the performance of the following tasks? More difficult/slightly difficult/no change Significantly easier/ easier/slightly easier
Receiving lab orders 0% 100%
Serving clients in order 0% 100%
Avoiding errors in giving results 25% 75%
Billing for service 25% 75%
Ensure accuracy of lab results 25% 75%
Process reference laboratory results 0% 100%
Process lab packages by category of services (ANC, CAC) 50% 50%
Over all ease of performing tasks using EHR/Laboratory 82%
Reception cashiers

All the respondents, reception cashiers, showed that the system made their daily task easier ( Table 6).

Ease of performing tasks using EHR for reception cashiers.
How has the EHR system changed the performance of the following tasks? More difficult/slightly difficult/no change Significantly easier/ easier/slightly easier
To register patient/client information - 100%
Finding mistakes and missing information - 100%
Billing and change capture - 100%
Searching for patient file - 100%
Finding the exact client document searching for - 100%
Over all ease of performing tasks using EHR/Reception cashiers 100%
MCH managers

MCH managers found the system easy to use when performing most of their day to day activities. However, 67% of the respondents indicated that the system makes it difficult to do income to cost ration ( Table 7). Similarly, a survey done in Australian hospitals revealed that a majority of managers (82%) stated that EHR improved data quality by readily availing information and improving the legibility of records ( Mckenzie, 2003).

Ease of performing tasks using EHR system for MCH managers.
How has the EHR system changed the performance of the following tasks? More difficult/Slightly difficult/no change Significantly easier/ Easier/Slightly easier
Getting total number of clients as needed 0% 100%
Getting provider level performance 0% 100%
Getting number of clients/services 0% 100%
Producing data reviews for specific patient groups (e.g. complication rate, diagnosis) 0% 100%
Total cash collected daily/weekly and monthly 0% 100%
To calculate income to cost ration 67% 33%
To show service trend 0% 100%
Providing remote access to data 33% 67%
Providing live data for presentations 33% 67%
Controlling fraud 0% 100%
Over all ease of performing tasks using EHR/MCH Manager 81%
User satisfaction

When participants asked to rank their satisfaction with various aspects of use, total reported satisfaction with the EHR system (agree/strongly agree) was 87% ( Figure 1). With the work they do on EHR system, inpatient staff were least satisfied (72%) followed by outpatient (74%) and pharmacy department (76%).

Satisfaction reported by staff in each departments. Limitations

Limited literatures were available to compare and discuss the findings with, and social desirability bias.

 Conclusion and recommendations

Staff in all department reported the EHR system ease of use as ‘very good’ for ease of performing their daily tasks using EHR system. The lowest score for ease of performing tasks was from pharmacy department and highest score was from reception cashiers. The reported satisfaction with the system was also high. In addition, the study revealed that ease of performing tasks using EHR and satisfaction with the system varied by department and by task within a department. It is essential to know the needs and requirements of each department before implementation of the system and getting user feedback for long lasting uptake and impact. It is also imperative that the workflow and information needs of each unit are met to optimize system utilization resulting quality of care.

 Data availability Underlying data

Figshare: Evaluation of EHR system. https://doi.org/10.6084/m9.figshare.8203172 ( Sharew, 2019).

This project contains the following underlying data:

CM.sav (responses from MCH managers)

IP.sav (responses from inpatient staff)

laboratory.sav (responses from staff in the laboratory department)

OPD.sav (responses from outpatient staff)

pharmacy.sav (responses from pharmacy staff)

receptionists.sav (responses from cashiers/receptionists)

 Extended data

Figshare: Evaluation of EHR system. https://doi.org/10.6084/m9.figshare.8203172 ( Sharew, 2019).

The project contains the following extended data:

questionnaire.pdf (questionnaire used in the present study with consent form)

Interview guide.pdf (survey given to all participants)

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

 Acknowledgements

The authors take pride in acknowledging the insightful guidance of Teshager Mersha, and Mathias Tenaye. We wish to thank Gizaw Sharew and Wogayehu Mamo for their support. We also appreciate the hardworking and committed service providers in three of maternal and child health centers.

 Ahmed S Saad D Samie GA : Evaluation of Electronic health records adoption in Egypt. Journal of Engineering. 2013;3(1):1131–1134. Reference Source Armijo D McDonnell C Werner K : Electronic health record usability: evaluation and use case framework. AHRQ publication.2009;9(10):1–57. Reference Source Bloom MV Huntington MK : Faculty, resident, and clinic staff's evaluation of the effects of EHR implementation. Fam Med. 2010;42(8):562–566. 20830621 Carayon P Smith P Hundt AS : Implementation of an electronic health records system in a small clinic: the view point of clinic staff. Behav Inf Technol. 2009;28(1):5–20. 10.1080/01449290701628178 Chiang MF Read-Brown S Tu DC : Evaluation of electronic health record implementation in ophthalmology at an academic medical center (an American Ophthalmological Society thesis). Trans Am Ophthalmol Soc. 2013;111:70–92. 24167326 3797873 Ellsworth MA Dziadzko M O'Horo JC : An appraisal of published usability evaluations of electronic health records via systematic review. J Am Med Inform Assoc. 2017;24(1):218–226. 27107451 10.1093/jamia/ocw046 Mckenzie K W S : Building the bridge to E-coding. Health information management association of australin conference. Sidney.2003;1–6. Menachemi N Collum TH : Benefits and drawbacks of electronic health record systems. Risk Manag Healthc Policy. 2011;4:47–55. 22312227 10.2147/RMHP.S12985 3270933 Rezaeibagha F Win KT Susilo W : A systematic literature review on security and privacy of electronic health record systems: technical perspectives. Health Inf Manag. 2015;44(3):23–38. 26464299 10.1177/183335831504400304 , . Silow-Carroll S Edwards JN Rodin D : Using electronic health records to improve quality and efficiency: the experiences of leading hospitals. Issue Brief (Commonw Fund). 2012;17:1–40. 22826903 Sharew T : Evaluation of EHR system. figshare. Dataset.2019. http://www.doi.org/10.6084/m9.figshare.8203172.v2 10.21956/gatesopenres.14109.r29044 Reviewer response for version 1 Atkins Salla 1 Referee https://orcid.org/0000-0002-4116-893X New Social Research and Faculty of Social Sciences, Tampere University, Tampere, Finland

Competing interests: No competing interests were disclosed.

 29 9 2020 Copyright: © 2020 Atkins S 2020 This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. recommendation reject

Gizaw et al.’s study focused on an evaluation of the electronic health record system in maternal and child health centers of Marie Stopes International Ethiopia. Overall, electronic health records have several benefits over paper-based records, and are an essential part of safe, quality healthcare. I found this paper to be highly descriptive and would recommend further analysis and consideration of their findings. My more detailed comments are below:

The literature used in the article is rather dated, with fewer recent papers. The authors would benefit from adding more recent literature. For example there are several systematic reviews that the authors could include, and consider in their presentation of the paper.

The study design is a cross-sectional evaluation study, with interviews and surveys. Despite using mixed methods, the paper does not refer to the implications or methods of using mixed methods.

Another challenge of the paper is the highly descriptive nature of the paper, where results are reported by percentages in six departments in three different MCH centres. This, with 68 participants means that we don’t get a picture of what might have differed between different MCH centres, and we don’t know what data came from where. Overall, the results are presented by different departments, not taking into account different MCH centres in the study, and with no comparison across the settings.

I suggest the authors use SURGE – The Survey reporting guideline, to report their findings, and add detail of the interviews conducted 1 .

Sufficient detail is given of the analysis of the survey responses, but no information is given of the analysis of the 6 interviews that were part of this study.

In terms of statistical analysis and interpretation, again this refers to the descriptive nature of the study. Having three different centres is not considered in the analysis or results at all. I suggest the authors re-examine their data for patterns across the different MCH, and patterns across the interviews and self-administered questionnaires.

The limitations section is extremely limited, and does not discuss e.g. the potential limitation of self report, response rate of the survey (was it 100%)? and the limited sample size.

Is the work clearly and accurately presented and does it cite the current literature?

Partly

If applicable, is the statistical analysis and its interpretation appropriate?

Partly

Are all the source data underlying the results available to ensure full reproducibility?

Yes

Is the study design appropriate and is the work technically sound?

Partly

Are the conclusions drawn adequately supported by the results?

Yes

Are sufficient details of methods and analysis provided to allow replication by others?

Partly

Reviewer Expertise:

Public health policy.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to state that I do not consider it to be of an acceptable scientific standard, for reasons outlined above.

References : SURGE (The SUrvey Reporting GuidelinE).2014; 10.1002/9781118715598.ch20 206-213 10.1002/9781118715598.ch20 10.21956/gatesopenres.14109.r29469 Reviewer response for version 1 Syed Abdul Shabbir 1 Referee https://orcid.org/0000-0002-0412-767X International Center for Health Information Technology (ICHIT), Taipei Medical University, Taipei, Taiwan

Competing interests: No competing interests were disclosed.

 11 9 2020 Copyright: © 2020 Syed Abdul S 2020 This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. recommendation reject

The authors attempted to evaluate the EHR system in six departments of the MCH centres in terms of ease of performing tasks and satisfaction with the system. I have a few concerns with the methodology and results presented.

What new information this study is contributing to the research society? Already there are tons of research done on exactly the same topic.

Inclusion criteria are not clear, why authors need 6-month experience prior to the implementation of the system? I think they should include a participant with 6-months experience with EHR.

English has to be revised by the native speaker, many sentences are not clear.

The percentage of the respondents shown in table 1 is more than 100.

Citations are too few and very old, I strongly recommend authors to update the citations with the latest studies.

Is the work clearly and accurately presented and does it cite the current literature?

No

If applicable, is the statistical analysis and its interpretation appropriate?

I cannot comment. A qualified statistician is required.

Are all the source data underlying the results available to ensure full reproducibility?

Yes

Is the study design appropriate and is the work technically sound?

Partly

Are the conclusions drawn adequately supported by the results?

Partly

Are sufficient details of methods and analysis provided to allow replication by others?

Partly

Reviewer Expertise:

AI, Big data analytics, EHR.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to state that I do not consider it to be of an acceptable scientific standard, for reasons outlined above.