In addition to systematic reviews, the Task Force uses modeling to inform the recommendation process. The Task Force uses modeling only when there is direct evidence of the benefit of a preventive service on health outcomes or when there is evidence for each of the linkages in the analytic framework. While the latter situation is considered indirect evidence, it is sufficient basis for using modeling, subject to the considerations described below.
Topics meeting the direct evidence criterion and for which modeling has previously been applied include cervical cancer screening (very strong ecological evidence of mortality benefit), colorectal cancer screening, lung cancer screening, and breast cancer screening (RCT evidence of screening and mortality benefit). Other examples of USPSTF recommendations based on direct evidence are limited, but include topics such as abdominal aortic aneurysm screening and breast cancer chemoprevention. Candidates for decision modeling are generally A, B, and some C recommendations. While models may integrate sufficient evidence across an analytic framework, they should not be used to bridge a gap in the analytic framework where evidence is insufficient by using assumptions or unreliable data in order to provide the basis for a USPSTF recommendation. However, decision models can be critical to the USPSTF when there is insufficient empirical evidence to determine when to start and how long to continue delivering a clinical preventive service, how frequently to repeat the service, and the best or equally appropriate choices among different ways of delivering the service. Thus, decision modeling is primarily warranted when: 1) there are outstanding clinical questions about how best to target the clinical preventive service at the individual and the program level; and 2) it is highly unlikely that the systematic review can confidently determine magnitude of net benefit, particularly as it varies among important prespecified subpopulations.
Models used by the Task Force include simple calculations using probabilities for each sequential step from screening to health outcomes (historically known as "outcomes tables") and more formal decision models. The purpose of either of these is to estimate the population impact of a given preventive service.
Outcomes tables derived from systematic reviews apply evidence from the systematic review alone to estimate the number of persons in a hypothetical population who would be affected in specific ways from implementation of the preventive service over a given time horizon (often 5 to 10 years). Further details about systematic review–derived outcomes tables are provided in Appendix VIII. A decision model is a more formal exercise using complex mathematical simulation to project the health outcomes that result from alternative interventions for screening, diagnosis, prevention, and treatment. Decision models incorporate evidence about natural history, disease-specific epidemiology, and other topic-related factors in addition to evidence from the systematic review.
There are often limitations in the evidence on important aspects of clinical preventive services. Critical aspects such as starting and stopping ages or preventive service delivery intervals are almost never directly addressed in trials or studies. How to define the best (or comparable) options among different approaches to preventive service delivery, particularly in the context of a screening program, may also have limited empirical evidence. There is often limited evidence to determine when a new or different screening test can be substituted for one that was initially recommended based on RCT evidence; available evidence on new screening tests is usually about test performance characteristics. When necessary, a decision model can link natural history, population characteristics, and screening or treatment effectiveness to estimate the relative impact and comparative effectiveness of varying screening or treatment approaches. As the USPSTF methods on modeling evolve details will be published and made available on the Task Force Methods Webpage.
Similarly, the usual information available through systematic review may not adequately summarize net benefit with the level of precision required for the USPSTF recommendation process. This is particularly likely when the clinical preventive service is one characterized by multiple benefits and harms, such as when the benefits and harms vary substantially between subpopulations; when there is moderate certainty of net benefit, but the magnitude is difficult to determine; or when the empirical evidence provides an estimate based on intermediate outcomes or using a limited time horizon that is clearly insufficient to determine net health benefits.
Inputs into decision models include best estimates from evidence on the linkages between screening and outcomes of interest and explicit assumptions about natural history, epidemiology, and other topic-related factors. It is important that modelers make their assumptions explicit so that Task Force members can assess the applicability of the model and its results to Task Force deliberations.
While acknowledging the demonstrated value of decision modeling for selected USPSTF recommendations, the Task Force also recognizes that there are opportunity costs and resource implications.
To maintain consistency and transparency across topics, the Task Force has developed a framework and criteria for determining whether decision modeling will be added to topics that could benefit from decision modeling along with the systematic review. This framework is based on the following six sequential questions that the Task Force considers, to the extent possible, during the scoping phase of an eligible topic's systematic review.
- Has benefit for this clinical preventive service been established?
Based on past experience and its evidence-based philosophy, the USPSTF intends to primarily apply decision modeling in cases where the benefit of the clinical preventive service is satisfactorily established, yet decisionmakers could benefit from further specification. As stated above, for the USPSTF, this implies that there is either: 1) direct evidence of a health benefit from a randomized (or otherwise very robust) comparison of delivery of the preventive service versus not delivering this service; or 2) evidence of benefit that can be achieved through strong linkages between multiple separate bodies of evidence.
- Which of the primary reasons for adding decision modeling are important to address for this clinical preventive service?
In planning for decision modeling, the USPSTF specifies one or more of these primary reasons to add decision modeling to the systematic review in its recommendation development: 1) to assess appropriate starting or stopping ages or compare alternative intervals of preventive service delivery; 2) to compare alternative technologies, such as different screening tests; 3) to assess the impact of a newly developed substitute test in an established screening program; 4) to quantify net benefit more precisely or specifically than can be done based on systematic evidence review alone; 5) to extend time horizons beyond those available from studies; 6) to assess net benefit for population groups that represent combinations of higher and lower risk for benefits and/or harms; and 7) to assess net benefit stratified by sex or other demographic characteristics, such as race or ethnicity.
- Is the information gained from modeling or reviewing existing models likely to be worth the opportunity cost of modeling?
This question represents a judgment and offers a potential stopping point in the process of considering the addition of decision modeling to a topic. Clearly articulating the expected advantages of using a decision model can enhance transparency and consistency and focus efforts on the most important topics for potential decision modeling, given that resource constraints are likely to limit its incorporation in all instances where it may be desired. Deciding to not commission a decision model at this stage can also reduce effort that would be required to complete steps 4, 5, and 6 below.
The Task Force is continuing to determine a full set of clear criteria, using its past experience with decision modeling. Until that work is complete, the following considerations may help to determine whether to commission a decision model.
- When decision modeling will primarily assist in specifying screening program components (i.e., stopping and starting ages, intervals, comparative modalities or programs) for an effective clinical preventive service, the usual criteria of public health burden will apply. In addition, the Task Force may want to consider the potential for enhancing benefit through extending the preventive service to unstudied populations, reducing harms, or enhancing efficiency through more targeted service delivery.
- When decision modeling is primarily intended to assist in net benefit or subpopulation determinations, USPSTF members offer the following questions to help project expected gains from incorporating modeling:
- Would the model provide important information for addressing uncertainty when making a recommendation for this preventive service?
- If one could predict the ultimate recommendation based on the systematic review evidence, would the results of a decision model inform one's deliberation on a grade (e.g., from an I statement to a letter grade) or change the messaging (i.e., to enhance public understanding of the USPSTF recommendation, especially if it differs from those of other organizations or community practice)?
- Would modeling enhance appropriate uptake of a recommendation by reducing under- or over-utilization in a subpopulation?
- Would modeling enhance appropriate uptake of a recommendation by providing clarity for measurement within quality improvement activities?
- Can the desired modeling approach be clearly outlined at this point, or is it contingent on additional information not known at the outset of the systematic review?
There is tension between the need to commence modeling as early as possible to coordinate the modeling and systematic review work, and wasting effort when it is too early in the process to determine an appropriate focus for the decision model. The USPSTF must determine when it is possible in the overall process to define the decision problem/objective, the decision-important modeling outputs, and any approaches to be compared through modeling. The "Decision Problem/Objective" is a template approach (Appendix XIII) that outlines the modeling objectives for the purposes of the recommendation statement, using the PICOTS (population, intervention, comparator, outcome, timing, and setting) framework to specify the scenarios to be modeled, required outcome measures, and the relevant modeling time horizon. If the main constructs of relevance cannot be defined, it is premature to proceed with decision modeling. Instead, if possible, the USPSTF delineates what modeling-relevant information is still uncertain and at what point in the systematic review process it will likely become clearer, in order to plan when to reconsider if a decision model is appropriate.
- What is the decision problem/objective to be addressed through decision modeling?
Taking into consideration the prior USPSTF recommendation statement (and its evidence review approach), public response to the previous recommendation, current issues for clinical practice, and the expected state of the science, the USPSTF defines the parameters of the decision framework relevant to this topic, including: the rationale for decision modeling, the desired approach (e.g., type of model, one model or comparative models), scenarios to be modeled, populations and settings to be targeted and/or compared, desired outcome measures, and modeling time horizon needed. When searching for existing decision models, recency and setting should be considered.
- What is the most expedient approach for needed decision modeling?
The most efficient mechanism for considering decision models alongside systematic reviews would be to use an already published analysis. While this is an attractive concept, it is uncommon that an existing decision model will match the decision problem outlined by the USPSTF well enough to be of use to the USPSTF. With more defined USPSTF guidance, however, opportunities for using existing models may increase. Further, iterative exchange between modelers and decisionmakers has been proposed as a critical element of the process by those with prior experience. Nonetheless, identifying existing models may be important if only to identify existing modeling groups and candidate modelers for collaboration.
Searching for existing decision models is most efficient if it is focused on the specific decision problem outlined by the USPSTF. Unlike a systematic review search, which is highly sensitive to avoid missing any relevant primary research, searches for existing models should emphasize specificity, since the purpose is to find one or a few excellent candidates for use or adaptation. The proposed process is outlined in Appendix XIV.
These six questions frame the considerations around adding a decision model to the systematic review for a specific topic, but do not make that decision. In the case of multiple competing priorities and resource constraints, the USPSTF Chairs make the final decision about whether to add decision modeling to a specific topic. The decision of the USPSTF Chairs is based on the findings from these six questions and their judgment about relative needs and priorities, considering the overall USPSTF portfolio.
Current as of: July 2017
Internet Citation: Section 5. Modeling. U.S. Preventive Services Task Force. July 2017.