Why North Korean intelligence is so hard to read

(CNN) -- The latest controversy over whether North Korea has the technology to fit a nuclear warhead on a missile is not the first time American intelligence agencies have been at odds in assessing Pyongyang's capabilities. And it is not the first time the Pentagon's intelligence arm, the Defense Intelligence Agency (DIA) has been out front in reaching the most alarmist conclusions about North Korea -- one with which other U.S. agencies have disagreed.

The DIA assessment, disclosed by a Congressman at a hearing on April 11, was that the DIA has "moderate confidence" North Korea has developed a nuclear warhead that can be delivered by a ballistic missile. Within hours, however, James Clapper, Director of National Intelligence, released a statement saying the DIA report did not represent the consensus of the intelligence community, and that "North Korea has not yet demonstrated the full range of capabilities necessary for a nuclear armed missile," -- a view echoed by a Pentagon spokesman and the South Korean Defense Ministry.The sharply different judgments about Pyongyang's capabilities recall a similar episode in 1998, when a fierce debate erupted within the American intelligence community after U. S. spy satellites discovered an underground complex at Kumchangri, not far from North Korea's main nuclear facility at Yongbyon.

The DIA insisted that Kumchangri was a construction site for a secret, underground nuclear weapons facility and that its existence proved Pyongyang was cheating on the Agreed Framework, the Clinton administration's 1994 deal under which the North froze operations at Yongbyon in return for economic and diplomatic concessions from Washington."The DIA was adamant," then-National Security Adviser Sandy Berger recalled in a 2007 interview for my book, "Meltdown: The Inside Story of the North Korean Nuclear Crisis."

"They said they could see the roads that were characteristic of this kind of facility, they could actually see that this hole was perfectly suited for a nuclear reactor," he said.

Other intelligence agencies at the time, including the CIA, were more skeptical. "I looked at the evidence," one senior Clinton administration official told me during research for my book, "and I could easily see innocent explanations for virtually everything they pointed to. And others in the intel community looked at exactly the same stuff and heard the DIA's arguments about it, and said essentially, this is really flimsy stuff."

On August 17, 1998, the New York Times published a story that revealed the intelligence findings, turning what had been an internal intelligence debate into a public firestorm of controversy -- just as the congressman's revelations did on Thursday.

The Times report, with headline "North Korea Site An A-Bomb Plant, US Agencies Say" created major problems. In Washington, the Clinton administration came under attack for Pyongyang's alleged violations of the Agreed Framework. And, unless the issue was handled with great skill, the tenuous link between the U.S. and North Korea that has existed since the 1994 deal would collapse.

In the spring of 1999, Chuck Kartman, Washington's Special Envoy for Peace Talks with North Korea -- nicknamed "Iron Butt" for his ability to sit patiently and listen to North Korean envoys spew out venom, bombast and threats across the negotiating table -- engaged in protracted talks with his North Korean counterparts.In May, Kartman convinced the North to agree to permit U.S. inspectors to visit Kumchangri. In addition, after initially demanding a payment of $300 million dollars for a one-time visit, North Korea eventually accepted a shipment of 100,000 tons of potatoes as the "price" of permitting the inspection.

Accompanied by a team of experts, Kartman flew to Pyongyang and the North Koreans took them to Kumchangri. To the Americans' surprise and embarrassment, they found a large, empty underground cave. There was no evidence of a secret nuclear site.

The crisis passed. According to Gary Samore, then the National Security Council's Senior Director for Non-Proliferation and subsequently President Obama's Coordinator for Weapons of Mass Destruction, Counter-Terrorism and Arms Control: "The conclusion was that Kumchangri was not configured in a way that could be used either for plutonium or for enrichment. So, whatever we thought Kunchangri was, it wasn't. It was not nuclear-related."

Nearly two decades later, the purpose of the construction site at Kumchangri remains unknown. But the episode underscores the complexities of assessing intelligence on North Korea. The rush to judgment in 1998, driven by the DIA's aggressive briefings and the deliberate leak of the story to the media, left many officials at the time cautious, if not downright skeptical, about subsequent intelligence findings.

"DIA hyped their findings unbelievably," one former senior Clinton administration official said to me. "They mis-stated their level of certainty, and the further up the food chain they went, the more they dropped the caveats, and ended up putting us in a terrible position."

Gary Samore had accepted the DIA's analysis, and lobbied strongly for the Clinton administration to make Kumchangri a major issue. "Sandy Berger [President Clinton's National Security Adviser] depended pretty heavily on my judgment," he told me, "and I feel that my recommendation to him that it was a serious issue led the administration to deal with it more seriously than it ever turns out is justified. I would say that was the biggest mistake I made in my career as a civil servant."

The Kumchangri episode does not mean the DIA is necessarily wrong in its current assessment of North Korean nuclear capabilities. But the skepticism expressed by other agencies after Thursday's revelation carries a lesson. In a country which the former deputy head of the CIA, John McLaughlin, once described to me as an "intelligence black hole," serious analysts can reach sharply differing conclusions about the same evidence, and the public should be careful in taking every ala