Center for Problem-Oriented Policing

Appendix D: Summary Of Evaluation Designs' Strengths and Weaknesses

Design

Strengths

Weaknesses

Pre-Post

  • Is simple and quick to implement
  • Can easily be used with surveys
  • Can provide a reasonable estimate ofthe post-response change in theproblem
  • Can show only short-term changes inthe problem
  • Cannot account for preexisting trends
  • Cannot account for the possibility that some other factor occurred at the same time as the response, and caused the problem to change
  • Is very weak at ruling out other alternative explanations

Interrupted Time Series

  • Is easy to use with data routinely collected over many time periods
  • Can rule out preexisting trends and many other alternative explanations
  • Is very hard to use if special data collection methods, such as surveys, are used to measure the problem
  • Cannot account for the possibility that some other factor occurred at the same time as the response, and caused the problem to change
  • Takes a long time to establish results
  • Is hard to interpret when there are few problem events per time period before the response

Pre-Post With a Control Group

  • Can easily be used with surveys
  • Can account for the possibility thatsome other factor occurred at thesame time as the response, and caused the problem to change
  • Can show only short-term changes in the problem
  • Requires a control group that is similar to the response group

Multiple Time Series

  • Is easy to use with data routinely collected over many time periods
  • Can rule out preexisting trends and many other alternative explanations
  • Can account for the possibility that some other factor occurred at the same time as the response, and caused the problem to change
  • Is very hard to use if special data collection methods, such as surveys, are used to measure the problem
  • Requires a control group that is similar to the response group
  • Takes a long time to establish results
  • Is hard to interpret when there are few problem events per time period before the response