All analyses conducted for the Mercyhurst Project on Chinese Crime Syndicates and Film Piracy used 10-point scales to quantify both source reliability and analytic confidence. The scores are determined by the author or authors, and can be found near the bottom of each report.

Source Reliability

Uses a 1-10 point scale to convey the reliability of the body of sources used for that particular report, 1 being the lowest and 10 the highest. Frequent references to U.S. government reports may contribute to a higher source reliability, for example, while a reliance on more speculative international news sources may decrease the score.

Analytic Confidence

Uses a 1-10 point scale to express the analyst's confidence in his analysis, or the analysis of the group if it is a group estimate. A score of 1 is the lowest possible while 10 is the highest.

Additional Information

Source Reliability is a necessary but not sufficient condition for Analytic Confidence. Ideally, an analyst wants all of his sources to be of the highest quality; above reproach. Practically, this is rarely the case. In this project the analysts used a number scale, from 1-10, designed to give the reader a sense of how reliable the analyst thought the sources he used were overall.

There is, admittedly, a good deal of subjectivity inherent in this appraisal. Some sources, such as reports on the U.S. State Department's website were considered generally reliable. Other sources might only be considered reliable with regard to the facts that they recorded. Finally, some sources had to be considered speculative, suspect or unreliable. In addition, it is possible that an analyst might misjudge the reliability of a source due to a general lack of subject matter expertise or to the time constraints implicit in attempting to cover such a vast topic in such a short time frame. Because of these caveats, the project analysts thought it particularly important to give policymakers and other readers of this product a general sense of the analyst's perception of the overall reliability of the sources used in the analysis.

Analytic Confidence reflects the level of confidence an analyst has in his or her estimates. Analytic Confidence, as a concept, is often confused with the estimative judgment itself (captured in these documents using Words Of Estimative Probability such as "likely" or "unlikely"). For example, an analyst might state, as an estimative conclusion, that he thinks it is "likely" or "highly likely" that an event will occur in the future. An analyst's confidence in that same assessment, however, can vary from very high to very low depending on a variety of different elements. The appropriate elements of Analytic Confidence are a subject of some debate. Generally speaking, however, high quality sources reporting a consistent message should increase analytic confidence. In addition, utilizing structured methods that eliminate or limit the impact of bias are legitimate ways to improve analytic confidence. Experience and access to specialized bodies of information are traditionally thought to increase analytic confidence though recent research suggests that, when it comes to forecasting, these are not particularly advantageous.

Typically, analysts are loathe to make estimative evaluations without some reasonable degree of confidence in them. However, such situations do happen with regular frequency. It is easy to imagine, for example, a crisis situation where a decisionmaker needs an analytic estimate but does not have time to wait for the analyst to do the kinds of things that might increase the analyst's confidence. Under such a scenario, it is intellectually dishonest to communicate an estimate ("I think X is likely to happen...") without also communicating the analyst's level of confidence ("...but my confidence is low."). An analyst must communicate the level of confidence