For a systematic review of psychometric properties, the specific aim of the search strategy is to locate studies that have described the development or validation process of a measurement instrument or those that have evaluated a measurement instrument’s psychometric properties. Research has found that a poor search strategy is a common and major methodological weakness of systematic reviews of measurement instruments (Mokkink et al., 2009; Terwee et al., 2016a), which can threaten the validity of a reviews’ findings and its role as a reliable source of evidence-based guidance for choosing appropriate instruments for use in research or clinical practice (Mokkink et al., 2009; Prinsen et al., 2018). Accordingly, the development of a comprehensive search that locates all the relevant and available literature on the topic of interest is fundamental to the conduct of a complete and high quality review. Unfortunately, locating studies that have reported or evaluated the psychometric properties of a measurement instrument can be challenging due to the poor indexing of such studies; the heterogeneity in terminology used to describe measurement properties; and poor reporting by authors, who frequently omit commonly used measurement property terms from the titles and abstracts of their published studies (Terwee et al., 2009).
Search filters
The COSMIN initiative have developed two validated search filters to aid researchers in finding studies of measurement properties in PubMed (a sensitive search and a precise search) (Terwee et al., 2009). COSMIN suggest researchers conducting systematic reviews of psychometric properties use the sensitive search filter, as it was designed to retrieve a high number of relevant articles (sensitivity: 97.4%; precision: 4.4%) (Terwee et al., 2009). Although the precise search filter is more specific, it is more likely to miss relevant studies (sensitivity of 93.1%; precision of 9.4%) (Terwee et al., 2009). The sensitive search filter has been translated for use in EMBASE, for MEDLINE using OVID, and two translations have been developed for CINAHL; however, none of translations have been validated. All search filters can be found on the COSMIN website (https://www.cosmin.nl/tools/pubmed-search-filters/). The search filters contain a combination of search terms (free text and index terms) that capture relevant measurement properties and should be used together with search terms defined by the review team for the population, construct and/or instrument/type of instrument of interest (Terwee et al., 2009). A search filter for use with PubMed and Ovid to find studies evaluating patient-reported outcome measures (PROMs) has also been developed by the Patient-Reported Outcomes Measurement Group (PROM Group), at the University of Oxford (University of Oxford), and can be used for the ‘type of instrument’ concept in the search strategy.
Search strategy development
The development of a search strategy involves identifying key search terms and synonyms for each major concept in the inclusion criteria mnemonic for psychometric reviews (Population; Instrument name/Type of Instrument; Construct, Outcomes [measurement properties]). The selection of search terms for each concept (i.e. search string) should be developed using an iterative process that involves adjusting terms and performing test searches. The final search should contain a combination of both free text words and index terms (e.g. MeSH) to improve the sensitivity of the search, i.e. its ability to recall relevant studies. The type and number of key search terms to be included will depend on the topic of interest and the size of its evidence base. For example, the population or construct of interest may involve very specific or very broad domains of interest, which will impact on the number of relevant records retrieved (sensitivity/specificity of the search). How the search strings for the major concepts (Population, Instrument name/Type of instrument, Construct, Measurement properties) are combined also depends on the type of systematic review of measurement instruments being conducted (Mokkink et al., 2018b). In their user manual, COSMIN provide a schematic that shows how search strategies for different types of reviews of patient-related outcome research should be constructed (Mokkink et al., 2018b). For example, if a systematic reviews seeks to evaluate all PROMS (validated or not), search terms related to the ‘Measurement properties’ concept should not be used in the search strategy (Mokkink et al., 2018b).
As a wide variety of terms related to the ‘Type of instrument’ concept exist and are not always reported in the abstract, these terms should typically be excluded from the search to avoid the high risk that relevant studies will be missed; however, if the systematic review seeks to locate PROMS, the PROMS filter can be included in the ‘Type of instrument’ concept in the search strategy. If possible, reviewers should select their search terms and develop their strategy in consultation with an expert research librarian or information specialist, whose contribution should be acknowledged (with permission) in any related publications. When considering possible search terms, reviewers may wish to familiarize themselves with the different types of instruments available, which differ in content and in their intended purpose or application. The PROM Group (University of Oxford) has classified measurement instruments into seven major categories with examples (disease-specific; population-specific; dimension-specific; generic; individualized; summary items; and utility measures).
The JBI recommend a three-phase search process that should be undertaken in the development of a comprehensive search strategy:
Phase one involves conducting an initial limited search in a selected database (e.g. PubMed) to find articles on the topic of interest. The keywords (i.e. text words) used in the titles and abstracts of these articles, and the index terms used to describe and categorize them, should be identified, and subsequently, used to develop a full search strategy. The search strategy must be adapted and individualized for every selected database as each one uses its own controlled vocabulary (i.e. index terms).
Phase two involves running the database-specific searches in each of the bibliographic databases and information sources selected and reported in the protocol.
Phase three involves scanning the reference lists of all studies selected for critical appraisal to identify any additional relevant studies.
Information sources
The review protocol should list all the information sources that will be searched for the systematic review: electronic bibliographic databases; search engines; relevant websites; references lists of eligible studies; pre-selected journals, and direct contact with experts in the field who may help to identify measures under development or articles reporting on instruments that assess the construct of interest. The search should be conducted across a comprehensive range of relevant and content-specific (i.e. construct or population of interest) information sources appropriate for the topic and the type of studies being sought. The search should include MEDLINE and EMBASE at a minimum and should include sources of both published and gray or unpublished literature. Examples of other commonly searched databases include CINAHL, PsycINFO, Web of Science, Scopus, Dissertations and Theses Global (gray literature), and WorldCat (gray literature). The PROMS Group has also collated a list of information sources specific to PROMS (e.g. organizations and research groups; journals; royal colleges and relevant links) on their website (University of Oxford) and the Health and Psychosocial Instruments (HaPI) database contains information on behavioral measurement instruments from journal abstracts covering the health and psychosocial sciences. Databases should be searched from the date of inception until the present time unless a valid justification for placing a limit of the publication date can be provided.
Reporting a search strategy
It is important that the search strategy used to find eligible studies is reported in a detailed and transparent manner, such that other researchers could repeat it, if required. The review protocol should describe in detail the proposed search strategy (three phase approach), the complete list of information sources to be searched, the timeframe for the search, and any language and date restrictions with appropriate justifications. The full search strategy for at least one major electronic database (such as PubMed) should be provided in an appendix and should report the name of the information source and the platform or service provider, for example, CINAHL (via Ovid); all search terms (both free text and index terms) and how they are to be combined using Boolean logic (if applicable); the use of database specific truncation and wild cards; all limits placed on the search (e.g. publication date, publication type, etc.); and the number of records retrieved by the search.