Appendix D: Summary of Evaluation Designs' Strengths and Weaknesses

Design Strengths Weaknesses
STATIC COMPARISON
  • Better than nothing
  • Can be useful for preliminary examination
  • Pre-existing differences between groups are likely to be the cause of the different levels of the problem
  • Cannot determine if the response came before or after the pre-existing differences
  • Cannot eliminate the possibility that pre-existing trends created the results
PRE-POST
  • Simple and quick to implement
  • Can easily be used with surveys
  • Can provide a reasonable estimate of the change in a problem following a response
  • Can only show short-term changes in problems
  • Cannot account for pre-existing trends
  • Very weak at eliminating alternative explanations for the change in the problem
  • Cannot account for the possibility that some other factor occurred at the same time as the response, and caused the problem to change
INTERRUPTED TIME SERIES
  • Very easy to use with data routinely collected over many time periods
  • Can eliminate pre-existing trends and many other alternative explanations
  • Very hard to use if special data-collection efforts, such as surveys, are used to measure the problem
  • Cannot account for the possibility that some other factor occurred at the same time as the response, and caused the problem to change
  • Results take a long time to be established
  • Difficult to interpret when there are few problem events per time period before the response
PRE-POST WITH CONTROL
  • Can easily be used with surveys
  • Can account for the possibility that some other factor occurred at the same time as the response, and caused the problem to change
  • Can only show short-term changes in problems
  • Requires a problem-troubled control group that will not get the response and is similar to the response group
MULTIPLE TIME SERIES
  • Easy to use with data routinely collected over many time periods
  • Can eliminate pre-existing trends and many other alternative explanations
  • Can account for the possibility that some other factor occurred at the same time as the response, and caused the problem to change
  • Very hard to use if special data-collection efforts, such as surveys, are used to measure the problem
  • Requires a problem-troubled control group that will not get the response and is similar to the response group
  • Results take a long time to be established
  • Difficult to interpret when there are few problem events per time period before the response