点击显示 收起
1 From the Department of General Practice (HJSM and JWVR), the Skillslab Department (J-JER), and the Department of Educational Development and Research (LWTS), Maastricht University, Maastricht, the Netherlands; and the Department of Gastroenterology, University of Amsterdam (LMHM-V).
2 Presented at the Third Heelsum International Workshop, held in Heelsum, the Netherlands, December 10–12, 2001. 3 Supported by research grants from the Dutch Sugar Bureau, the Dutch Dairy Foundation for Nutrition and Health, and the Dutch Meat Board. 4 Address reprint requests to HJS Maiburg, Maastricht University, Department of General Practice, Paviljoen DEB 1, PO Box 616, 6200 MD Maastricht, the Netherlands. E-mail: bas.maiburg{at}hag.unimaas.nl.
ABSTRACT
Background: Nutrition education is not an integral part of either undergraduate or postgraduate medical education. Computer-based instruction on nutrition might be an attractive and appropriate tool to fill this gap.
Objective: The study objective was to assess the degree to which computer-based instruction on nutrition improves factual knowledge and practice behavior of general practitioner (GP) trainees.
Design: We carried out a controlled experimental study, using a 79-item knowledge test and 3 incognito standardized patients’ visits in a pre- and posttest design with 49 first-year GP trainees. The experimental group (n = 25) received an average of 6 h of a newly developed computer-based instruction on nutrition. The control subjects (n = 24) took the standard vocational training program.
Results: The percentage of correct answers on the knowledge test increased from 30% at pretest to 42% at posttest in the experimental group, and from 36% to 37% in the control group. Analysis of covariance, with the pretest scores as covariate, showed a significant experimental versus control group difference at posttest: 9.2% (P = 0.002). The mean percentage of correctly performed items during the 3 standardized patients’ visits (assessed by checklists) showed an increase in the experimental group from 20% at pretest to 36% at posttest, whereas the control group changed from 20% to 22%. Analysis of covariance, with the pretest scores as covariate, revealed a significant group difference at posttest: 13.7% (P < 0.001).
Conclusion: The computer-based instruction proved its effectiveness, both by increasing factual knowledge and by substantially enhancing GP trainees’ practice behavior on the subject of nutrition.
Key Words: General practice • GP trainee • computer-based instruction • nutrition guidance • standardized patients • knowledge test
INTRODUCTION
In the Netherlands, general practitioners (GPs) and GP trainees often feel they lack the expertise necessary to deal with patients’ questions on nutrition because of insufficient education on this subject in both undergraduate and postgraduate medical courses (1). This phenomenon has also been reported in many other countries, and many attempts have been initiated to fill this gap in nutrition education (2–10). Recent research shows that GPs and GP trainees are interested in nutrition in general and welcome disease-related nutrition education (11, 12). It also found that patients have great expectations of GPs with regard to nutrition guidance (13). These findings motivated us to develop a new medical education program on nutrition. We chose computer-based instruction (CBI) to overcome certain flaws of previously developed educational programs (1). As CBI enables GP trainees to complete the program without a teacher, this method should avoid the major problem of faculty at vocational training departments not being acquainted with nutrition topics themselves (14). Moreover, CBI was developed as an Internet program, allowing easy updating and distribution of updates. As CBI is not currently used in the curriculum of GP vocational training in the Netherlands, opting for CBI also meant opting for innovation.
Although the value of CBI has been generally established and accepted (15, 16), we still had to prove the effectiveness of the newly developed CBI on nutrition. This assessment formed part of the NECTAR (Nutrition Education by Computerized Training and Research) study, whose main objective is to enhance the knowledge, skills, and attitudes of Dutch GP trainees on the subject of nutrition.
In terms of the effect of the nutrition CBI as an educational tool, we were interested in the improvement of both knowledge (competence) and performance of GP trainees (17, 18). Competence-based assessment measures what doctors can do in controlled representations of professional practice, whereas performance-based assessment measures what doctors actually do in their professional practice. Our first objective was to determine the degree to which GP trainees’ factual knowledge of nutrition would improve as a result of the CBI. Second, we wanted to examine whether the nutrition CBI improved the adequacy of consultations performed by GP trainees in real practice with patients with nutrition-related complaints.
The first objective refers to the "knows" level of facts and figures, as distinguished by Miller (19, 20). Many assessments of the value of educational tools are restricted to this level. The second question refers to the "does" or "performance" level, which implies transfer of knowledge and skills into daily practice.
SUBJECTS AND METHODS
Design
The study used a pretest/posttest design in which GP trainees’ knowledge and practice behavior were assessed before and after an educational intervention. Subjects were divided into an experimental group (using a newly developed CBI) and a control group (taking the regular program). Both pretest and posttest used a 79-item knowledge test and 3 visits by incognito standardized patients as measuring instruments. Standardized patients’ visits in the posttest were carried out an average of 1 mo after the conclusion of the intervention. The experiment took place from July 2000 to July 2001. The study was performed in accord with the Helsinki Declaration of 1975 as revised in 1983 (35th World Medical Assembly, Venice, Italy, October 1983).
Subjects
We included all 49 GP trainees who started their GP vocational training program in 2000 at Maastricht University. The selection committee of the Department of Vocational Training, in which the researchers were not represented, allocated these trainees to 4 groups. Two groups were to constitute the experimental group (n = 25), with the other 2 groups being control subjects (n = 24). Together with the selection committee, we stratified the admitted GP trainees according to their sex and the university where they had completed their undergraduate medical program (Maastricht vs elsewhere). Afterward, the selection committee reallocated some GP trainees, mainly for logistic reasons beyond the scope of our study.
At the start of their vocational training, GP trainees received information about the design of the study and the way they would be informed about its progress and results. During the same session, they completed a questionnaire on background variables (mainly focusing on previous experience with the subject of nutrition), providing us with information about the composition of the research groups. Participation in the experiment was obligatory.
All GP trainees worked in different practices with their own GP trainer (one trainee per trainer). Therefore, we also asked their GP trainers to fill out a questionnaire assessing their own attitude about nutrition, as GPs and as GP trainers. We argued that differences in the way GP trainers handle nutrition-related problems in practice, as well as differences in the numbers of patients with these problems seen by the GP trainers, might well influence GP trainees and thus to a degree confound the effect of the CBI.
Intervention
The GP trainees in the experimental group received a newly developed CBI, involving a simulation of a GP consultation consisting of 12 interactive streaming-video patient cases presenting with complaints related to nutrition (ie, food pyramid, obesity, diabetes mellitus, hypercholesterolemia, hypertension, irritable bowel syndrome, and constipation). The teaching objectives of the CBI are shown in Table 1. The GP trainees ran the program during their weekly vocational training days at the institute. We scheduled the CBI sessions at 2- to 3-wk intervals, with a maximum of 5 sessions, each session taking 1.5 h. After a plenary instruction session by the researcher, the GP trainees went through the CBI at their own pace during these sessions, each GP trainee being provided with a personal computer. The actual learning process with the CBI took an average of 6 h. The control group took the regular vocational training program, in which nutrition topics are sparsely incorporated and are subordinated to medical diagnosis and treatment.
View this table:
TABLE 1 . Teaching objectives of computer-based instruction on nutrition
Instruments: the knowledge test
We developed a 79-item knowledge test with open-ended questions, which was to be used in both the pre- and the posttest. The level at which we tried to design the questions was the "knows" level—that is, straightforward facts and figures about nutrition. We chose this format to prevent any "giving away" of the right answers, which multiple choice questions are likely to do (21). The general format of the questions is shown in Figure 1.
FIGURE 1. . Example of knowledge test question.
We ensured the content validity of the knowledge test by assessing the domain and making a blueprint (indicating the required chapters and the percentage of items for each chapter) for the test. We were careful to derive test items from the content of the CBI, which was based on a process of key-feature analysis by several nutrition experts (22, 23). An expert on knowledge testing revised the textual quality of the questions. Earlier studies had shown that the number of items to be used was sufficient to obtain good reliability in knowledge tests covering the broad GP domain (24).
We performed a pilot of the knowledge test in a GP trainee group not included in our study to assess the level of difficulty of the test and to avoid ceiling effects, as we planned to use the same test twice. The GP trainees took the test under exam conditions. The researcher graded all tests following a predefined grading protocol, being blinded for group allocation. The test score consisted of the number of correctly answered items (incorrect answers and items not answered scoring zero), converted to percentages.
Instruments: incognito standardized patients
To assess the practice behavior of GP trainees with regard to problems on the subject of nutrition, we decided to use incognito standardized patients (SPs). SPs are nonphysicians taught to portray patients in a standardized and consistent way. Such incognito SPs visit GP trainees as regular patients during daily practice and complete a checklist on the consultation afterward, thus assessing the "performance" level of the trainee (25, 26). We trained 14 newly acquired SPs, who matched the intended roles. Their instruction included standardized role-playing, reliable completion of checklists, and remedial training halfway through the planned visits. The total training time amounted to 16 h per role.
The content of the 3 SP roles in the pre- and posttest (intestinal problems, cardiovascular disease/hypertension, and overweight/ diabetes mellitus) covered the entire domain of nutrition as incorporated in the CBI. We developed the SP checklists with the experts who had contributed to the content of the CBI. The total number of checklist items in the pretest was 118; in the posttest it was 115. Scoring categories for SPs included "performed correctly" (= yes), "not performed/performed incorrectly" (= no), and "dubiously performed," as shown in Figure 2. SPs completed the checklists immediately after leaving the GP practice. In the case of a follow-up consultation, SPs used the same copy of the checklist again. The score for each case consisted of the number of correctly performed items (incorrectly, dubiously, or not performed items scoring zero), converted to percentage scores. We computed the total SP test score as the mean of the 3 checklist scores.
FIGURE 2. . Part of standardized patient checklist.
To ensure that SPs would remain incognito, we provided each GP practice that was to be visited by SPs with medical context information on each of the SPs. In addition, we provided the SPs with comprehensive information on each of the practices they were to visit (27, 28). We used different groups of SPs in the pre- and posttest. SPs were blinded for the group allocation (experimental vs control group) of the GP trainees. The GP trainees received a so-called SP detection form, to be sent to the researcher in case they thought they had identified an SP.
Statistics
To evaluate differences in baseline variables between research groups, we used an independent samples t test for age, Mann-Whitney tests for (skew-distributed) numeric variables and ordinal variables from the GP trainers’ questionnaires, and chi-square tests for data on sex and the university of undergraduate medical studies. To assess whether differences in knowledge test and SP test scores between experimental and control groups in the posttest were significant, taking into account the differences in the pretest, we carried out a linear regression analysis (analysis of covariance) (29). We also used the linear regression analysis to elicit and correct for confounding variables. To compute the significance of differences in knowledge test and SP test scores between the pre- and posttests, we performed paired samples t tests for both experimental and control groups.
Depending on the variable level, we used Pearson r or Spearman’s for correlations. We computed the reliability of the instruments as Cronbach’s . For the knowledge test, we entered the scores on all test items to compute Cronbach’s ; for the SP test, we entered the percentage scores of the 3 cases.
Two-sided P values 0.05 were considered significant. We carried out all analyses in SPSS for Windows 9.0 (SPSS Inc., Chicago, IL) (30).
RESULTS
Characteristics of experimental and control groups are reported in Table 2. The groups showed no significant differences for age, sex, or the university of their undergraduate medical studies. Previous theoretical and practice experience on the subject of nutrition did not differ between the 2 groups.
View this table:
TABLE 2 . Characteristics of general practitioner (GP) trainees and their GP trainers in control and experimental groups1
All GP trainees included in the study took part in the pretest (knowledge test and SP test). One GP trainee in the experimental group was visited by only 2 SPs, as he moved to another surgery during the pretest phase. One GP trainee in the control group, as well as one in the experimental group, abandoned the training program during the posttest phase for reasons not related to the experiment.
Knowledge test
The results of the knowledge test, including total scores and scores per chapter are shown in Table 3.
View this table:
TABLE 3 . Percentage of correct answers in 79-item knowledge test before and after educational intervention for control and experimental groups of general practitioner trainees in the Nutrition Education by Computerized Training and Research (NECTAR) study1
The relative improvement in the total score of the experimental group between the pre- and posttests amounted to 40.3%. Assessment per chapter revealed relative improvement rates in the experimental group of 54.4% for food pyramid, 67.7% for overweight and obesity, 67.8% for diabetes mellitus, 9.9% for cardiovascular disease, 29.4% for intestinal problems, and 37.3% for hypertension. Analysis of covariance, performed to correct for differences between the experimental and control groups in the pretest scores, showed a significant experimental versus control group difference at posttest of 9.2% (P = 0.002). Relative to the mean pretest score, this implied an improvement of 28.0% in the experimental group. Pretest reliability of the knowledge test was 0.70, whereas posttest reliability amounted to 0.83 (experimental group 0.84; control group 0.80).
Incognito SPs
Of the 294 SP visits planned, 287 were actually carried out (98%). The 7 missing visits were due to dropout by GP trainees, as mentioned above. Five of the 287 visits resulted in a follow-up consultation within several days, mainly to discuss a requested food record.
The results of the SPs’ pre- and posttest scores are displayed in Table 4. Based on all visits, the SP scores for the experimental group between the pre- and posttest showed a relative improvement of 79.6%. For the sake of consistency in our method of analysis, we once again used analysis of covariance to compute experimental versus control group differences on the posttest, although the pretest scores hardly differed. This analysis showed a significant difference of 13.7% (P < 0.001) between the groups at posttest. Relative to the mean pretest score, this implied an improvement of 68.2% in the experimental group. Reliability of the SP pretest was 0.67, and that of the posttest 0.72 (experimental group 0.45; control group 0.71).
View this table:
TABLE 4 . Percentage of correctly performed items during the 3 standardized patients’ visits before and after educational intervention for control and experimental groups of general practitioner trainees in the Nutrition Education by Computerized Training and Research (NECTAR) study1
In addition to several visits by genuine patients, GP trainees correctly reported 74 visits of presumed SPs (26%). During the pretest, the control group unmasked 16 visits of SPs, against 11 in the experimental group. The numbers of detected visits in the posttest were 15 and 32, respectively. As remaining incognito is an important feature of SPs in terms of validity, we subsequently computed the scores of undetected visits, which are also shown in Table 4. This resulted in a relative improvement of the SP score of the experimental group of 79.3%. Analysis of covariance showed a significant experimental versus control group difference of 12.8% (P < 0.001) at posttest, implying a 68.4% improvement in the experiment group relative to the mean pretest score.
Confounding variables
Analyzing for confounding variables, we found 2 items on the GP trainers’ questionnaires on which the trainers of GP trainees in the experimental group had scored significantly higher than the trainers of the control group GP trainees (Table 2). The first item, "perceived importance of nutrition in GP training," did not significantly correlate with posttest scores on the knowledge test (r = -0.186; P = 0.204) or the SP posttest score (r = 0.244; P = 0.099). "Use of obesity brochure" showed significant correlations with neither the knowledge test (r = 0.180; P = 0.221) nor the SP test (r = 0.223; P = 0.115).
We also found a significant difference of 5.8% (P = 0.001) in SP pretest scores between male and female GP trainees and a significant difference of 4.4% (P = 0.024) between GP trainees who had graduated from Maastricht University and those who had graduated from other institutions. Introducing sex and the university of undergraduate medical studies into the analysis of covariance showed that these variables were not confounding, as they hardly caused any change in the experimental versus control group difference on the SP posttest (31).
DISCUSSION
We carried out a controlled experiment with the objective of evaluating the degree to which our CBI on nutrition was able to change knowledge and practice behavior of GP trainees. At both the "knows" level and the "performance" level (19), the findings revealed differences between the group of GP trainees who had received the CBI and the trainees who had taken the standard vocational training program, the former showing considerably greater improvement.
The results of the knowledge test are encouraging, although it must be acknowledged that the questions in this test concentrated on straightforward facts. Facts and figures about nutrition are provided in the CBI in "pop-up" screens, which can be activated on demand during the session, and in concluding tutorials. It is debatable how eagerly GP trainees searched for these elements in the CBI. Some GP trainees in the experimental group stated their disappointment on being questioned on facts and figures in the posttest, as they had not paid much attention to these elements in the CBI in their search for practical tools. A semiannual Dutch national knowledge test for GP trainees provides some comparative material on the development of GP trainees’ knowledge about nutrition, as induced by the CBI (24). This national knowledge test, covering the total GP domain, shows an average growth in knowledge among all first-year GP trainees that is quite similar to the growth in knowledge about nutrition we found in our experimental group GP trainees. This suggests that knowledge about nutrition in our control group GP trainees fell short of their knowledge about the total GP domain.
The results of the performance test taken by our GP trainees are remarkable for more than one reason. First, we succeeded in establishing an improvement in consultation behavior, using incognito SPs as an instrument. Although several studies have been conducted using SPs in practice, effects on practice behavior were rather ambiguous (32, 33). Second, the practice behavior itself was found to improve, indicating that the nutrition CBI is suitable for achieving this relatively ambitious target. We attribute this outcome to the concept of the CBI being a simulation of GP consultations. As GP trainees seem to be focused on the acquisition of tools for daily practice, the program provides them with not only written guidelines for nutrition history taking and nutrition guidance but also video clips of GP experts as role models for good nutrition practice.
The SPs’ pretest scores, as a measure of current practice behavior on the subject of nutrition, confirmed our expectations. These expectations were based on previous research among GPs and GP trainees, indicating a lack of nutrition education and a lack of sufficient consultation time as important barriers to performing nutrition guidance (12). The CBI seemed to remove both barriers. Clearly, it offers sound nutrition education. Furthermore, our results suggest that GP trainees overcome the barrier of the lack of sufficient consultation time as soon as they have acquired the necessary practice skill of nutrition history taking and nutrition guidance.
The question may be raised whether the absolute level of practice behavior achieved after our intervention is relevant for daily practice. Some idea of this can be obtained by comparing the scores of our GP trainees, whose mean score was 36%, with those of experienced GPs found in previous studies with SPs, showing mean scores of 55% (17, 34). Of course, we have to take into account that these studies belonged to a different domain. Nevertheless, our findings show that the CBI did help GP trainees to make considerable progress toward reaching experience levels found among practicing GPs.
Whereas GPs likely take several consultations to deal with patients’ complaints thoroughly (35), the vast majority of the SPs’ visits turned out to be one consultation only, a situation partly accounted for by the limited time available in the study. This means that the results of our study have to be categorized as short term and based on one consultation.
With regard to the composition of the experimental and control groups, we conclude that we have managed to achieve adequate similarity in terms of characteristics and SP pretest scores. However, we have to take into account the relatively small number of subjects included, causing rather wide confidence intervals.
We wondered whether our study assessed the real performance of our GP trainees, as the detection rate of SPs was somewhat higher than that found in other SP studies (36). However, the scores found in the analysis of only undetected visits hardly differed from those based on all visits. The effect on practice behavior, within the experimental group as well as between the experimental and control groups, was firmly sustained in the subgroup of undetected visits. As GP trainees were visited by SPs in their own consultation rooms, during regular consultation hours, we may conclude that our assessment of GP trainees in practice, based on all visits, very closely reflects daily practice performance.
Finally, to return to the educational starting point of the study, the merits of the nutrition CBI in the education of GP trainees on the subject of nutrition are obvious. What is noteworthy is not only the effects of the program, as discussed above, but also the fact that the GP trainees used this program without any teacher intervention. The CBI permits self-directed learning by GP trainees, a learning method that is one of the basic elements of a new curriculum to be implemented in Dutch GP vocational training. Furthermore, as models of nutrition guidance do not differ between GPs and GP trainees (12), it should also be possible to use the CBI for refresher courses for practicing GPs. In terms of intrinsic motivation, similar or even better effects might be anticipated, given that GPs may voluntarily choose to attend the program, whereas GP trainees had the program imposed on them.
For many years, teachers and trainers have entertained great hopes for using information and computer technology for educational purposes. We hope our study will contribute to speeding up the changes in the educational landscape.
ACKNOWLEDGMENTS
We thank Arnold DM Kester of the Department of Methodology and Statistics at Maastricht University for his advice on the analysis of this study.
There are no conflicts of interest.
REFERENCES