点击显示 收起
1 From the Health Services Research Unit, University of Aberdeen, Aberdeen, United Kingdom, and the Department of Orthopedic Surgery, Princess Margaret Rose Orthopedic Hospital, Edinburgh, United Kingdom.
2 The Health Services Research Unit is funded by the Chief Scientist Office of the Scottish Executive Health Department; the views expressed herein are those of the authors. 3 Supported by the Medical Research Council (AA) and the Chief Scientist Office of the Scottish Executive Health Department (AA, HHGH, and AMG). 4 Address reprint requests to A Avenell, Health Services Research Unit, University of Aberdeen, Polwarth Building, Foresterhill, Aberdeen AB25 2ZD, United Kingdom. E-mail: a.avenell{at}abdn.ac.uk.
ABSTRACT
Background: A key aim when conducting systematic reviews of randomized controlled trials (RCTs) is to include all of the evidence, if possible. Serious bias may result if trials are missed through inadequate search strategies.
Objective: The objective was to evaluate the search plan for identifying RCTs in nutrition as part of a systematic review, in The Cochrane Library, of nutritional supplementation trials in patients after hip fracture.
Design: We identified potential studies by searching the electronic databases BIOSIS, CABNAR, CINAHL, EMBASE, HEALTHSTAR, and MEDLINE; reference lists in trial reports; and other relevant articles. We also contacted investigators and other experts for information and searched 4 nutrition journals by hand.
Results: We identified 15 RCTs that met the predefined inclusion criteria. The search plan identified 8 trials each in EMBASE, HEALTHSTAR, and MEDLINE and 7 in BIOSIS and CABNAR. BIOSIS was the only electronic database source of 2 trials. Eleven trials were identified by searching electronic databases and 2 unpublished trials were identified via experts in the field. We found one trial, published only as a conference abstract, by searching nutrition journals by hand. After publication of the protocol for the review in The Cochrane Library, we were informed of another unpublished trial.
Conclusions: We found that a limited search plan based on only MEDLINE or one of the other commonly available databases would have failed to locate nearly one-half of the studies. To protect against bias, the search plan for a systematic review of nutritional interventions should be comprehensive.
Key Words: Systematic reviews • meta-analysis • search strategy • randomized controlled trials • nutrition • databases • bibliography • bias • MEDLINE
INTRODUCTION
Systematic reviews of randomized controlled trials (RCTs), which often incorporate meta-analyses, are considered to provide the best evidence on which to base clinical practice. RCTs are preferred over other study designs because they are least likely to have serious bias (1). However, the variable quality of meta-analyses is a cause for concern. This concern led to the publication of the QUOROM statement—guidelines for the conduct and reporting of meta-analyses (2).
Bias can be introduced into a meta-analysis when RCTs that fit the criteria for the review are not identified. Publication bias results because studies reporting statistically significant results are more likely to be published than are those in which the results were not statistically significant (3), and hence are more likely to be identified by a reviewer. Location bias can also be introduced if relevant studies are not identified. Authors are more likely to get their trial reports published in an English-language journal if their results are statistically significant (4). Thus, bias may be introduced into a meta-analysis by restricting reports to those published in the English language or to databases that preferentially include English-language journals. Few journals published in less-developed countries are indexed by the major databases (3): MEDLINE (National Library of Medicine; Bethesda, MD), EMBASE (Elsevier Publishers BV, Amsterdam), BIOSIS (Biological Abstracts, Inc, Philadelphia), CABNAR (Commonwealth Agricultural Bureau Nutrition Abstracts and Reviews; CAB International Publishing, Wallingford, United Kingdom), CCTR (Cochrane Controlled Trials Register; The Cochrane Library, Oxford, United Kingdom), CINAHL (Cumulative Index of the Nursing and Allied Health Literature; CINAHL Information Systems, Glendale, CA), CURRENT CONTENTS (Institute for Scientific Information, Philadelphia), and HEALTHSTAR [Health Services, Technology, Administration and Research (National Library of Medicine) and the American Hospital Association (Chicago)]. Although the reference lists of published trials may be used to locate additional trials, these references may be prone to the same selectivity. In addition, there is clear evidence that authors of RCTs do not usually review their results in the context of the total evidence available from other RCTs (5).
As part of a regularly updated systematic review of nutritional supplementation trials for hip-fracture aftercare in the elderly (6), we searched for published and unpublished RCTs. We therefore needed to evaluate our search plan for identifying such nutrition trials. We also wanted to examine whether our strategy could provide the basis for constructing searches for other nutrition-related systematic reviews. Inspection of the 14 systematic reviews included in The Cochrane Library (2000, issue 1; Update Software, Oxford, United Kingdom) that assessed any form of vitamin supplementation showed that all of the reviewers used MEDLINE, 5 used EMBASE, 1 used BIOSIS, and none used the abstracting service provided by CABNAR. We thus decided to assess the utility of searching different databases, including CABNAR, of searching selected journals by hand (including conference proceedings), and of contacting authors and experts in the field.
METHODS
We first developed an electronic database search strategy for MEDLINE and searched this from 1966 to January 2000. This search strategy consisted of 3 sections: the first 2 levels of the standard Cochrane search strategy for RCTs (ie, the search strategy used for identification of RCTs in the Cochrane Review; 7) (Table 1), a nutrition section (general nutrition, vitamins, minerals, and trace elements), and an orthopedic section (Table 2). Our previous experience with MEDLINE showed that a search of nutrition-related MeSH (medical subject heading) terms identifies less than one-half of the trials identified by text word searching. We therefore decided to use text words in addition to MeSH headings in the nutrition strategy. Generally, we decided to make the literature search sensitive, rather than specific, to find as many trials as possible. We checked that the final MEDLINE strategy picked up trials already known to the Cochrane Musculoskeletal Injuries Group.
View this table:
TABLE 1. First 2 levels of The Cochrane Library search strategy for randomized controlled trials in MEDLINE1
View this table:
TABLE 2. Search terms used in MEDLINE (Ovid interface)1
The electronic database HEALTHSTAR (1975 to December 1999) uses the same search terms as MEDLINE, so we adopted the same strategy for this database. We also applied the MEDLINE search terms that were available to CINAHL (1982 to November 1999).
The electronic database EMBASE (1980 to January 2000) uses search terms similar to those used in MEDLINE; thus, the search strategy for this database is similar to that used for MEDLINE (C Lefebvre, S McDonald, unpublished observations, 1996). The nutrition and orthopedic terms, which are also similar to those used in MEDLINE, were added to the strategy for RCTs. Incidentally, the coverage of European and pharmaceutical journals is greater in EMBASE than in MEDLINE.
Neither BIOSIS (1985 to December 1999) nor CABNAR (1983 to December 1999) have a standard Cochrane search strategy for RCTs, nor do these databases readily identify studies by their research methodology. For BIOSIS, we searched for the orthopedic text words and the concept term "nutrition." For CABNAR (1973 to December 1999), we searched for only the orthopedic text words because the database already selects nutrition studies. However, because CABNAR covers agriculture and animal nutrition, we excluded references with the following text words: dog$, bird$, horse$, soil$, wood$, and freeze fracture$. (The symbol $ is a truncation symbol, allowing retrieval of all possible variations of a term.) The resulting references were then combined with text words (random$ or trial$ or placebo$) to yield the final citations for consideration. CABNAR is the only one of these databases to index theses (up until 1994), although it is unclear how these are obtained for indexing.
All the electronic databases were searched by using an Ovid interface (Ovid Technologies, New York), except for BIOSIS, which was accessed by using BASIS software (version 8.2.3; Opentext, Beaconsfield, United Kingdom). We checked the reference lists of trials included in our review, in other relevant reviews, and in epidemiologic and other trial reports. Review, epidemiologic, and other trial reports on the subject of nutrition and hip fracture were also identified from 6 of the electronic databases (MEDLINE, EMBASE, BIOSIS, CABNAR, CINAHL, and HEALTHSTAR) by using only the nutrition and orthopedic search terms.
One researcher screened all titles and, when available, abstracts obtained with the above methods. Those citations that appeared to be of trials or potential trials were obtained as complete articles. Each article was read in its entirety and a shortlist of potential RCTs was prepared. Two researchers then reviewed the shortlist and determined which trials were RCTs on the basis of the inclusion criteria prespecified in the published protocol for the review (6). We contacted the first authors of the RCTs obtained to request information on additional relevant trials. Individuals known to the reviewers who had a particular interest in the field of inquiry were also contacted for details of additional trials.
Additionally, we searched 4 journals by hand: The American Journal of Clinical Nutrition (volumes 2–71; 1954 to January 2000), Clinical Nutrition (volumes 1–18; 1982 to December 1999), JPEN The Journal of Parenteral and Enteral Nutrition (volumes 1–23; 1977 to December 1999), and Proceedings of the Nutrition Society (volumes 1–58; 1944 to August 1999). We chose these journals because they contain abstracts of conference proceedings and studies of clinical nutrition and were available locally. We considered this procedure to be useful because early volumes of journals as well as conference proceedings are generally not included in electronic databases and because the indexing of articles for databases is not always reliable.
Once all the trials were identified, we searched the electronic databases again to see whether the trials were indexed by the database but had been missed by our search strategy. If this was the case, the reason the trial was missed was identified. The yields from the different databases and strategies were calculated and compared. The end date for the RCT search for the first update of our review was January 2000. The end dates given above for the various databases and journals were those available to the first author at that time.
RESULTS
Our updated review included 15 RCTs, identified from 15 main trial reports (8–22). These trials were grouped into categories on the basis of the type (multinutrients, protein only, vitamins, or other) and route (oral, nasogastric, or other) of nutritional supplementation used. Three of these trials (9, 10, 14) were identified as conference abstracts only, although one author subsequently provided a reference (23) to a journal article that was not found in the databases searched. The databases and sources of the trials discovered in our search strategies are given in Table 3.
View this table:
TABLE 3. Sources of randomized controlled trials (RCTs) identified by search strategies developed for nutrition and hip fracture review in The Cochrane Library1
The search strategies in the electronic databases initially yielded for consideration 1095 citations in BIOSIS, 390 citations in CABNAR, 57 citations in CINAHL, 3977 citations in EMBASE, 858 citations in HEALTHSTAR, and 956 citations in MEDLINE. The search of CABNAR was the most specific and yielded the most trials in the least amount of search time. Despite this laborious task, we did not fail to identify a trial when screening full citations in these databases. Eleven trials were identified by searching electronic databases. The EMBASE, HEALTHSTAR, and MEDLINE strategies produced the most trials (n = 8 each). One trial (20) was not retrieved from MEDLINE with the Cochrane search strategy for RCTs, but actually was in MEDLINE. Similarly, another trial (17) was not identified by the nutrition search strategy for EMBASE, but actually was in EMBASE. Our strategy also failed to identify one trial in CABNAR because the abstract did not contain the text words random$, trial$, or placebo$ (20). The search of CABNAR did not recover any trials additional to those found in the other electronic databases. The search of BIOSIS yielded 7 trials, 2 of which were not found in any of the other databases (both were conference abstracts; 9, 10). These 2 trials were also located by searching conference abstracts in the Proceedings of the Nutrition Society by hand. Because none of the 15 RCTs identified were indexed in CINAHL, we were unable to assess our search strategy with this database.
In addition to the 11 trials found by searching electronic databases, 1 trial was found only after a search by hand of JPEN The Journal of Parenteral and Enteral Nutrition (14). Additionally, a search of Clinical Nutrition by hand produced one trial (16) before it was indexed by the databases. After the publication of the protocol for our review in The Cochrane Library, we were notified by the investigators of an additional trial (13) that was reportedly in press. Two unpublished trials were identified via contact with an expert in the field.
Eleven of the 15 trials were found by searching electronic databases with the strategies used here. A search of the reference lists of known trials identified only 5 of the trials, 4 of which had been found electronically and 1 of which had been found by hand. Only 7 of these RCTs were referenced in review, epidemiologic, and other trial reports.
All of the 15 RCTs were published in English-language journals. Five additional trials published in French, German, Italian, and Russian did not meet the inclusion criteria after examination of limited translations.
A subsidiary analysis examined the degree to which the available evidence would be deficient if only 1 of the 6 electronic databases had been used. Of the 1054 participants recruited into the 15 trials, only 58% would have been included if EMBASE only were used, 56% if MEDLINE or HEALTHSTAR only were used, 48% if CABNAR only were used, 42% if BIOSIS only were used, and 0% if CINAHL only were used. In one of our main review comparisons we examined the effect of protein and energy supplementation. If we had used MEDLINE alone, only 30% of the participants evaluated in the studies identified from all sources would have been included.
DISCUSSION
The systematic and comprehensive search for trials is an important component of any systematic review. We decided to monitor and record the results of a search of different sources of trials to get an idea of what method retrieved the greatest number of relevant trials within practical constraints. Given that this is just one review topic, there is bound to be variation in the yields of various databases and other sources for other topics, and over time for all topics. Because we do not know which trials we failed to identify, we cannot propose an optimal search plan. However, some pointers to guide future searches can still be made.
It is clear from our study that systematic reviews in the field of nutrition should not rely on the use of a single electronic database, such as MEDLINE. Published systematic reviews of protein and energy supplementation (24), total parenteral nutrition (25), and immunonutrition (26, 27), which all used MEDLINE as the sole electronic database, may be subject to significant location bias. In each case we identified RCTs that should have been included in the meta-analysis on the basis of the criteria set by the reviewers.
Others have also concluded that a search of MEDLINE alone results in the risk that some published studies will be missed. In a published study of search strategies, including strategies for nutrition trials, Kleijnen and Knipschild (28) evaluated the yield of RCTs for vitamin C for the common cold from MEDLINE and EMBASE only compared with systematic review articles with extensive reference searching. A search of MEDLINE produced 46% of the published RCTs and a search of EMBASE produced 31% of the published RCTs. However, these yields would probably have been slightly higher if text words had been incorporated into the search strategy.
We were particularly interested to see how well CABNAR performed. We already considered this database to be useful for locating articles in nutrition journals not included in other databases. CABNAR performed well in comparison with other databases alone. Although CABNAR did not provide any trials additional to those from the other databases, it was specific; therefore, we suggest that CABNAR be evaluated further in other systematic reviews of nutritional interventions.
Farriol et al (29) examined the yield from 149 bibliographic databases after a search for articles on artificial nutrition (enteral and parenteral nutrition). However, the search was for all published works and not specifically for RCTs. Fifteen databases were found to have appreciable numbers of articles on artificial nutrition. MEDLINE provided the largest number of titles, followed by EMBASE and CURRENT CONTENTS. Because CURRENT CONTENTS (clinical medicine) was searched as part of the overall search strategy of the Cochrane Musculoskeletal Injuries Group, we did not search CURRENT CONTENTS specifically for this review.
The findings of the present study also showed that trials are likely to be missed if the search plan involves a search of electronic databases only. For instance, a search of JPEN The Journal of Parenteral and Enteral Nutrition by hand yielded one trial that was not indexed in any of the 6 electronic databases. This shows the value of searching by hand to identify reports not found with other strategies, particularly conference proceedings. In addition, one trial report was found after a search by hand before it had been indexed in the databases examined here (16). These examples illustrate the adverse effect on retrieval of appropriate trials when there is a delay in indexing in these databases. In addition to a search of current issues of journals by hand, recent publications can be identified electronically, such as in CURRENT CONTENTS.
Personal contact with investigators in the field was also an important source of trial identification and moreover provided information on ongoing and planned trials. Bibliographic searches of reports of RCTs showed that such reports in this area of nutrition usually failed to include a discussion of the results in light of the evidence available from other trials (5). This lack of discussion means that additional relevant trials are not identified and indicates that some authors fail to realize the importance of discussing their results in the context of the results of others.
Searching for trials is time consuming and costly. We believe that the identification of nutrition-related RCTs would be greatly aided if BIOSIS and CABNAR indexed RCTs as a publication type, as recently became the standard procedure in MEDLINE, HEALTHSTAR, and EMBASE. We recognize that our search plan was not exhaustive; however, there was a need to balance the potential for further searches to influence the results against the delay to the publication of the review and subsequent update.
One database not mentioned above is the CCTR, published in The Cochrane Library and updated quarterly. The CCTR is compiled from the results of extensive and systematic searches of various sources for randomized and controlled clinical trials by members of the Cochrane Collaboration and is a valuable source of controlled trials. Because study types are already limited to RCTs and other controlled trials and because various databases such as MEDLINE, and to some extent EMBASE, have been centrally searched by the Cochrane Collaboration, the CCTR should be searched for reports of trials. In time, it may be the prime source of such trials for nutrition reviews. However, the addition of new reports of trials into the CCTR usually occurs later than it does for the other databases reviewed here. Thus, all searches need to be complemented by other searches of current materials. We did not assess the CCTR in this review because we had entered many of the trials into the CCTR ourselves.
In conclusion, although the search plan adopted will vary depending on the nutrition topic, we recommend that several electronic databases, including the CCTR, be searched in conjunction with hand searching of journals covering relevant aspects of nutrition (particularly recent issues and those that contain conference proceedings), consulting with knowledgeable experts in the field, and searching the reference lists of trial reports. We also strongly recommend that advice be sought from an information specialist early in the process. Without such a comprehensive approach, systematic reviews of nutritional interventions may be prone to serious bias.
ACKNOWLEDGMENTS
We thank the following persons for their advice on literature searching: Julie Bruce, Margaret Buchan, Susan McCourt, Mary Mowat, Moira Napper, Máire O'Donnell, and Sheila Wallace. We particularly thank Cynthia Fraser and Carol Lefevbre for their comments, Ronald Koretz for his help with the search by hand of the JPEN Journal of Parenteral and Enteral Nutrition, Heidi Guyer and Ian Cameron for their provision of information about 3 unpublished trials, and the anonymous referees for their helpful comments.
REFERENCES