From the September/October 2004 issue of Foreign Affairs.

As the war on terrorism continues, statistics on terrorist attacks are becoming as important as the unemployment rate or the GDP. Yet the terrorism reports produced by the U.S. government do not have nearly as much credibility as its economic statistics, because there are no safeguards to ensure that the data are as accurate as possible and free from political manipulation. The flap over the error-ridden 2003 Patterns of Global Terrorism report, which Secretary of State Colin Powell called "a big mistake" and which had to be corrected and re-released, recently brought these issues to the fore. But they still have not been adequately addressed.

Now-common practices used to collect and disseminate vital economic statistics could offer the State Department valuable guidance. Not long ago, economic statistics were also subject to manipulation. In 1971, President Richard Nixon attempted to spin unemployment data released by the Bureau of Labor Statistics (BLS) and transferred officials who defied him. This meddling prompted the establishment of a series of safeguards for collecting and disseminating economic statistics. Since 1971, the Joint Economic Committee of Congress has held regular hearings at which the commissioner of the BLS discusses the unemployment report. More important, in the 1980s, the Office of Management and Budget issued a directive that permits a statistical agency's staff to "provide technical explanations of the data" in the first hour after principal economic indicators are released and forbids "employees of the Executive Branch" from commenting publicly on the data during that time.

The State Department should adopt similar protections in the preparation and dissemination of its reports. In addition to the global terrorism report, the State Department is required by Congress to report annually on international bribery, human rights practices, narcotics control, and religious freedom. Gathering and reporting data for congressional oversight is presently a low-level function at the State Department. The department rarely relies on high-quality, objective data or on modern diagnostic tests to distinguish meaningful trends from chance associations. Adopting safeguards against bias, both statistical and political, would enable Congress to better perform its constitutional role as the White House's overseer and allow the American public to assess the government's foreign policy achievements.

A PATTERN OF ERRORS

Congress requires that the State Department provide each year "a full and complete report" that includes "detailed assessments with respect to each foreign country ... in which acts of international terrorism occurred which were, in the opinion of the Secretary, of major significance." The global terrorism reports are intended to satisfy this requirement, but, over time, they have become glossy advertisements of Washington's achievements in combating terrorism, aimed as much at the public and the press as at congressional overseers.

The 2003 global terrorism report was launched at a celebratory news conference in April. Deputy Secretary of State Richard Armitage and Ambassador J. Cofer Black, the State Department coordinator for counterterrorism, outlined some remaining challenges, but principally they announced the Bush administration's success in turning the terrorist tide. Black called the report "good news," and Armitage introduced it by saying, "You will find in these pages clear evidence that we are prevailing in the fight." The document's first paragraph claimed that worldwide terrorism dropped by 45 percent between 2001 and 2003 and that the number of acts committed last year "represents the lowest annual total of international terrorist attacks since 1969." The report was transmitted to Congress with a cover letter that interpreted the data as "an indication of the great progress that has been made in fighting terrorism" after the horrific events of September 11.

But we immediately spotted errors in the report and evidence contradicting the administration's claims. For example, the chronology in Appendix A, which lists each significant terrorist incident occurring in the year, stopped on November 11-an unusual end to the calendar year. Clearly, this was a mistake, as four terrorist attacks occurred in Turkey between November 12 and the end of 2003. Yet it was impossible to tell whether the post- November 11 incidents were inadvertently dropped off the chronology and included in figures in the body of the report or completely overlooked.

More important, even with the incomplete data, the number of significant incidents listed in the chronology was very high. It tallied a total of 169 significant events for 2003 alone, the highest annual count in 20 years; the annual average over the previous five years was 131. How could the number of significant attacks be at a record high, when the State Department was claiming the lowest total number of attacks since 1969? The answer is that the implied number of "nonsignificant" attacks has declined sharply in recent years. But because nonsignificant events were not listed in the chronology, the drop could not be verified. And if, by definition, they were not significant, it is unclear why their decrease should merit attention.

On June 10, after a critical op-ed we wrote in The Washington Post, a follow-up letter to Powell from Representative Henry Waxman (D-Calif.), and a call for review from the Congressional Research Service, the State Department acknowledged errors in the report. "We did not check and verify the data sufficiently," spokesman Richard Boucher said. "... [T]he figures for the number of attacks and casualties will be up sharply from what was published."

At first, Waxman accused the administration of manipulating the data to "serve the Administration's political interests." Powell denied the allegation, insisting that "there's nothing political about it. It was a data collection and reporting error." Although there is no reason to doubt Powell's explanation, if the errors had gone in the opposite direction-making the rise in terrorism on President George W. Bush's watch look even greater than it has been-it is a safe bet that the administration would have caught them before releasing the report. And such asymmetric vetting is a form of political manipulation.

Critical deficiencies in the way the report was prepared and presented compromised its accuracy and credibility. Chief among these were the opaque procedures used to assemble the report, the inconsistent application of definitions, insufficient review, and the partisan release of the report. These deficiencies resulted in a misleading and unverifiable report that appeared to be tainted by political manipulation.

It is unclear exactly how the report was assembled. The report notes that the U.S. government's Incident Review Panel (IRP) is responsible for determining which terrorist events are significant. It says little, however, about the panel's members: how many there are, whether they are career employees or political appointees, or what affiliations they have. Nor does it describe how they decide whether an event is significant. Do they work by consensus or majority rule? What universe of events do they consider?

The State Department announced a decline in total terrorist attacks, which resulted from a decline in nonsignificant events. But without information about the nonsignificant events, readers were essentially asked to blindly trust the nameless experts who prepared it.

The report's broad definitions, moreover, are sometimes too blunt to help classification. Terrorism is defined as "premeditated, politically motivated violence perpetrated against noncombatant targets by subnational groups or clandestine agents, usually intended to influence an audience." The report specifies that an international terrorist attack is an act committed by substate actors from one nation against citizens or property of another. An incident "is judged significant if it results in loss of life or serious injury to persons, major property damage, and/or is an act or attempted act that could reasonably be expected to create the conditions noted."

But hardly any explanation was provided about how the IRP distinguishes significant from nonsignificant events. When is property damage too minor for an event to be significant? How are nonsignificant events identified? Is the IRP responsible for making these determinations too? Has the source and scope of their information changed over time? The corrected 2003 report, the first to list individual nonsignificant acts, defines as "major" property damage that exceeds $10,000. It does not indicate, however, whether that criterion applied to previous reports.

Admittedly, measuring international terrorism is no easy task. Even scholarly reckonings are not free from subjective judgment, and there are inevitably close calls to be made. The most one can hope for in many cases is consistent application of ambiguous definitions.

Unfortunately, in the global terrorism reports the rules have been applied inconsistently. Many cross-border attacks on civilians in Africa have not been included in the reports, for example, even though similar attacks in other regions have been. The report for 2002, moreover, counts as significant a suicide attack by Chechen shaheeds (Islamist martyrs) against a government building in Moscow that killed 72 people. Yet none of the numerous suicide attacks by the Chechen "black widows" that terrorized Russia and killed scores in 2003 was tallied as an international terrorist attack in the latest report. After one such attack, Russian President Vladimir Putin said, "Today, after a series of recent terrorist attacks, we can say that the bandits active in Chechnya are not just linked with international terrorism, they are an integral part of it." If the State Department considers such attacks domestic, rather than international, it should do so consistently from one year to the next.

Another problem is that the staff that prepared the 2003 global terrorism report did not participate in releasing it; in fact, they have yet to be identified. High-level Bush administration officials presented the report to the media, using it to support White House policies and take credit for the alleged decline in terrorism. Even after the report's flaws were recognized, they continued to spin the figures. When the corrected version was released, Black repeated that "we have made significant progress," despite being pressed to acknowledge that last year the number of significant attacks reached a 20-year high. Given the war on terrorism's central role in the upcoming presidential election, such presentation gives the appearance that the report is being manipulated for political gain.

The State Department has tried to explain the report's flaws using language eerily reminiscent of the Bush administration's justification of the failure to find weapons of mass destruction in Iraq. Spokesman Boucher told reporters that previous claims that the war on terrorism was succeeding had been based "on the facts as we had them at the time [and] the facts that we had were wrong." Even Powell partook in the spinning. On the one hand, he announced that "the [original] narrative is sound and we're not changing any of the narrative." On the other hand, he acknowledged, "We will change the narrative wherever the narrative relates to the data."

To his credit, Powell instructed those responsible for preparing the report to brief Waxman's staff on the procedures they had used and the origins of their mistakes. Based on a summary of the briefing by Waxman's staff, much has come to light. Authority for compiling the list of attacks was shifted from the CIA to the Terrorist Threat Integration Center (TTIC), an organization created in May 2003 to "merge and analyze all threat information in a single location." The TTIC provided information to the IRP, which, it was disclosed, consists of representatives from the CIA, the Defense Intelligence Agency, the National Security Agency, and the State Department's Bureau of Intelligence and Research. A TTIC representative chaired the meetings and could cast a vote to break ties on the classification of an event as significant or nonsignificant.

At least this year, chaos prevailed. The IRP's members changed from meeting to meeting-when they attended the meetings at all. The CIA employee responsible for the database left but was never replaced; in mid-process, an outside contractor who entered data was replaced by another contractor. Because of technical incompetence, the report relied on the wrong cutoff date.

Arithmetic errors were rampant. Larry Johnson, a retired CIA and State Department professional, discovered that the total number of fatalities in the chronology exceeded the number listed in the statistical review in Appendix G. According to Black, the errors resulted from "a combination of things: inattention, personnel shortages and database that is awkward and is antiquated and needs to have very proficient input be made in order for to be sure that the numbers will spill then to the different categories that are being captured [sic]." The debacle is more like an episode of the Keystone Kops than a chapter from Machiavelli, but even that analogy is not very comforting.

SETTING THE RECORD STRAIGHT

Despite the data's limitations, the chronology of significant events in the 2003 global terrorism report yields important information about terrorism's trends, its geographical characteristics, and its magnitude.

Time-series analysis, which seeks to discern trends in given phenomena over time, requires a consistent approach to collecting data. The State Department's terrorism report presents time-series analysis, but by focusing on the total number of attacks it misleadingly combines verifiable data on significant events with nonverifiable data on insignificant ones. And because, as TTIC director John Brennan admitted, "many nonsignificant events occur throughout the world that are not counted in the report," one must also be concerned about consistency in the measurement of the total number of terrorist events. Even if the nonsignificant events were listed (and thus could be verified), trends in significant events are more relevant because they track events that, by definition, are more important. Accurately measuring these trends is a prerequisite for understanding the factors that underlie them and the policies that shape them. In fact, an analysis of the revised report reveals that the number of significant attacks increased from 124 to 175, or by 41 percent, from 2001 to 2003-a significant fact indeed.

The detailed chronology also allows analysts to cumulate terrorist events for each country and cross-classify them according to the country where they occurred and the perpetrators' country of origin. These figures can then be related to the countries' characteristics, yielding information that can help policymakers devise strategies to address terrorism's root causes. Using the global terrorism reports for the years 1997-2002, the authors of this article have previously found that terrorists tend to come from nondemocratic countries, both rich and poor, and generally target nationals from rich, democratic countries.

The State Department has rightly emphasized that the threat of terrorism remains serious, but a close examination of its data helps put the magnitude of the threat in perspective. In 2003, a total of 625 people--including 35 Americans--were killed in international terrorist incidents worldwide. Meanwhile, 43,220 died in automobile accidents in the United States alone, and three million died from AIDS around the world. Comparative figures, particularly when combined with forecasts of future terrorism trends, can help focus debate on the real costs people are willing to bear--in foregone civil liberties and treasure--to reduce the risk posed by terrorism.

CHANGING TRACKS

The State Department currently uses, and Congress accepts, nineteenth-century methods to analyze a twenty-first-century problem. To prevent errors of the type that riddled the 2003 global terrorism report, Congress has two alternatives. It could reassign the State Department's reporting responsibilities to a neutral research agency, such as the GAO (the General Accounting Office, recently renamed the Government Accountability Office) which routinely uses appropriate statistical practices. The problem is that the GAO has little foreign policy expertise and does not necessarily have access to the (sometimes classified) information that goes into the reports. Alternatively, Congress could keep the reports within the State Department's purview but demand that its practices for data collection and analysis be improved and that the reports be insulated from partisan manipulation.

If responsibility remains within the State Department, Congress should establish a statistical bureau in the department to ensure that scientific standards are respected in all reports, thereby elevating the status of data-gathering and statistics there. The bureau would promote consistency, statistical rigor, and transparency. When appropriate, it could seek input from the scientific community. And, while respecting classified sources, it could also insist that sufficient information be released to independent analysts for verification.

To overcome conflicts of interest facing political appointees who issue government reports, the State Department should adopt rules similar to those that govern the production and dissemination of key economic indicators. Career staff who prepare the reports should be given an hour to brief the media on technical aspects of the data, during which time political appointees would be precluded from making public comments. (After the hour elapses, it is expected that political appointees would offer their interpretations.) Career staff should be protected so they can prepare mandated reports without interference from political appointees and then present them for review by the statistics bureau. Once the reports are finalized, but before they are publicly released, they should be circulated to designated political appointees who need to prepare for their release. Disclosure dates should be announced long in advance to prevent opportunistic timing by political appointees.

Last October, in a candid memorandum to top aides that was leaked to the press, Secretary of Defense Donald Rumsfeld admitted, "Today, we lack metrics to know if we are winning or losing the global war on terror. Are we capturing, killing, or deterring and dissuading more terrorists every day than the madrassas [Islamic schools] and the radical clerics are recruiting, training, and deploying against us?" The statement was a stinging acknowledgment that the government lacks both classified and unclassified data to make critical policy decisions. It is also a reminder that only accurate information, presented without political spin, can help the public and decision-makers know where the United States stands in the war on terrorism and how best to fight it.

Even when the European-Iranian agreement to halt Iran's uranium-enrichment program looked solid, the United States was blunt in its disapproval. The ink was barely dry on the accord when the Bush administration, it appears, began trying to derail it.

First, rather than endorse the accord, Secretary of State Colin Powell essentially accused the Iranians of lying when they said their nuclear program was for peaceful purposes. He announced that new intelligence showed Iran is developing a nuclear warhead to arm its Shahab-3 ballistic missiles. Then, at a Nov. 20 meeting of heads of state in Santiago, Chile, President Bush stated unequivocally that Iran is trying to build a nuclear weapon.

Why would the administration take such a combative stance? Because hard-liners within the administration thought Tehran would use the settlement to buy time for building nuclear weapons, and that the United States would be better off bombing Iran's suspected weapons sites.

Proponents of using military force against Iran have not yet won the argument within the Bush administration. But the past two weeks of strong pronouncements about the threat Iran poses suggest that the military option may be gaining ground. And Iran's last-minute attempts to maintain some enrichment capabilities -- which by press time Friday were threatening to kill the European agreement -- no doubt strengthened the hard-liners' hand.

Before the United States even considers such a drastic step as airstrikes against suspected nuclear weapons sites -- or even trying to compel the United Nations to endorse new economic sanctions against Iran -- it is essential that our leaders be clear about what they are trying to accomplish in Iran and whether such actions will help or hurt.

If the ultimate goal is to create a democracy -- one that would not fear the United States and therefore have less use for the bomb -- then dual-track diplomacy with Iran's government and with its people is more likely to work than military action.

Probably the most important question the administration's leaders should ask themselves is whether Iran, even a nuclear-armed Iran, poses a direct threat to the United States and its allies.

The answer, we believe, is no.

The mullahs who rule Tehran long ago gave up their ideological quest to "export'' revolution. Like the last generation of octogenarians who ruled the Soviet Union, Iran's leaders today want nuclear weapons as a means to help them preserve their power, not to help them spread their model of theocratic rule to other countries.

Deterrence works

In other words, even if Iran's rulers succeeded in building nuclear bombs, they would be very unlikely to take on the United States and its vast nuclear arsenal or to attack Israel. (The mullahs in Tehran understand that any nuclear attack against Israel would trigger full retaliation from the United States.) In dealing with Iran, deterrence works.

Tehran would also be unlikely to pass a bomb to Islamist terrorists, despite its support of Arab terrorist organizations that continue to attack Israel. One reason, again, is deterrence. Iran's rulers know that the United States would probably be able to trace the weapon back to them and retaliate.

The threat of a nuclear Iran comes, instead, from the reaction it is almost sure to spark in the region and the world, possibly sending Egypt and Saudi Arabia on their own quests for nuclear weapons.

Such an arms race would undermine the longstanding Treaty on the Non-Proliferation of Nuclear Weapons, an agreement signed by nearly 190 countries, that has proved indispensable in preventing the spread of nuclear weapons.

Bush administration hard-liners want to save that arms-control treaty by using arms. In advocating a "surgical'' military strike against Iran's most important nuclear facilities, including the once hidden enrichment plant in Natanz, they cite Israel's airstrike against Iraq's nuclear complex at Osirak in 1981 as a model of success. They argue that an American (or Israeli) strike would not end Iran's nuclear aspirations, but would dramatically slow its program and make the mullahs reconsider the costs of trying to restart it.

Attack would backfire

But a pre-emptive military strike would instead do just what the hard-liners in Tehran hope for: It would unite their people behind them.

Even a precise bombing campaign would kill hundreds if not thousands of innocent Iranians; destroy ancient buildings of historical and religious importance; trigger an Iranian counterstrike, however feeble, against American targets and friends in the region, and spur the mullahs to increase their direct support for American enemies in the Shiite part of Iraq.

Even more important, an attack would only encourage Tehran to redouble its efforts to build a bomb, just as Saddam Hussein sped up his efforts after the 1981 strike. It would also hurt the democratic opposition movement inside Iran, which is already in retreat and cannot afford another setback. After an attack, Iranians, not unlike Americans, are sure to rally around the flag and their government.

If the administration decides, in the end, that American military options are limited and counterproductive, the only serious way to impede the development of Iranian nuclear weapons is through negotiation. Iran's recent accord with France, Britain and Germany is only temporary, and negotiations are expected to continue.

If the United States were to jump in now, it could try to ensure that our European allies accept nothing less than a permanent and verifiable dismantling of Iran's enrichment capabilities, as well as banning any plutonium production.

Allowing the Iranians to enrich even some uranium, which they say will be used merely to feed their nuclear power plant, makes it too easy to cheat. To make the deal work, the United States would need to join with Europe, Russia and China in pledging to guarantee Iran a permanent and continuous supply of enriched uranium. To make the deal even more attractive, the fuel could be offered at reduced prices.

Even under the strictest inspection regime, Iran's leaders will cheat, as they have often done in the past, and they will eventually divert enriched uranium from peaceful to military purposes. But the harder and more transparent the allies can make it, the longer it will take Iran to begin building bombs.

In the long run, the world's only serious hope for stopping Iran from developing nuclear weapons is the development of a democratic government in Tehran. A democratic Iran will become an ally of the Western world no longer in need of a deterrent threat against the United States.

Democracy in Iran therefore obviously serves U.S. national interests. Yet Bush administration officials (as well as their predecessors in the Clinton, Bush and Reagan administrations) have not succeeded in developing a strategy for advancing the cause of Iranian democracy.

New strategy

What is needed is a radical new approach that would nurture change from within the country, in alliance with Iran's democratic movement, rather than impose change from without.

A first step would be to establish an American presence in Tehran, as many in Iran's democratic opposition have proposed. Now decades old, the U.S. policy of isolating Iran has not weakened but instead strengthened its autocratic government.

Of course, we are not suggesting that the United States open an embassy in Tehran and turn a blind eye to human rights abuses; that would only contribute to the further consolidation of the mullahs' hold on power. But we are suggesting a new strategy that would allow American government officials, as well as civic leaders, academics and business people, to engage directly with Iranian society.

This engagement cannot occur on a widespread scale without some level of diplomatic relations and some revision of the American sanctions against Iran. Then, more Western foundations would be able to make grants to pro-democracy Iranian organizations, while business people -- and especially the Iranian-American business community in the United States -- would be able to leverage their capital and know-how to influence economic and political change inside Iran. A U.S. presence in Iran would, not incidentally, also enhance the West's ability to monitor Iran's nuclear program.

Critics of engagement argue that diplomatic relations with Iran will reward this "axis of evil'' member for years of supporting terrorism and pursuing nuclear weapons. In fact, an American presence in Iran is the mullahs' worst nightmare.

Iran's government has long used its ongoing tensions with the United States, as well as the embargo, as an excuse for the economic difficulties that are, in fact, the direct results of the regime's incompetence and corruption. Tehran's leaders have conveniently labeled nearly all of their opponents as "agents of America.''

Most important, part of the regime's self-declared legitimacy lies in its claim to be the only Muslim country fighting what it sees as U.S. imperialism. If the United States could prove it's not an enemy of the Iranian people, the legitimacy of Iran's leaders would diminish.

Reagan's course

In the first years of his presidency, Ronald Reagan labeled the Soviet Union the "evil empire'' and went out of his way to avoid contact with such a regime.

Over time, however, Reagan charted a new course of dual-track diplomacy. He engaged Kremlin leaders (well before Gorbachev) in arms control, while also fostering contacts and information flow between the West and the Soviet people in the hope of opening them up to the possibilities of democracy.

In the long run, it was not arms control with the Soviets, but democratization within the Soviet Union, that made the United States safer.

If George W. Bush desires a foreign-policy legacy as grand as Reagan's, now is the time to think big and change course as dramatically as Reagan did.