No I'm not talking about al Qaeda, Osama bin Laden or even radicalized Muslims. For those who may not remember or have forgotten, according to the findings of the 9/11 Commission the events on that day were caused by an intelligence failure. Agencies that all had most or part of the picture did not share the information with one another allowing the hijackers to complete their plan.
The whole purpose of the Uniting and Strengthening America by Providing Appropriate Tools Required to Intercept and Obstruct Terrorism Act, US PATRIOT ACT, was to combine all of the right agencies to prevent this type of failure from ever happening again and thus was born the US Dept of Homeland Security. After reshuffling 22 agencies, plus several hundred thousand employees, to form the new DHS, two glaring exceptions didn't make the new team…the CIA and the FBI. So the two biggest intelligence gathering agencies in the US, along with the NSA and DIA, were not folded into the new DHS. Basically, DHS was sent into a gunfight without any bullets.
But even if the CIA and FBI had been rolled in, things might not be much different if what former MI6 Chief Sir John Sawers says here is correct. The CIA and FBI have increasingly shifted their focus away from "HUMINT" or human intelligence (spies) and relied more on more on technology. Part of this is budget driven but part of it is a legacy from Robert McNamara and his "whiz kids".
McNamara and the "Whiz Kids" wanted to turnaround the management of the Department of Defense and the conduct of the Vietnam War through quantitative analysis. Basically it meant taking the human out of the process and turning everything into data. Hence McNamara's view that the success of the war could be tracked by "body count". McNamara and his Whiz Kids forgot to consider that data was still being reported and collected by humans so fraud was rampant.
Even though Vietnam disproved "body count" as a way of assessing the success of a military campaign, quantitative analysis became the ideal way for the military and federal bureaucracy to hide behind numbers. Intelligence became a fertile ground for quantitative analysis mavens.
Up until the end of the Vietnam War, intelligence gathering had been the forte of spies (human intelligence). Spies were people who seemingly worked for the other side but in reality worked for us. Spies would smuggle out plans, documents, hardware that the enemy had been working on for analysts stateside to dissect and analyze. But the problem is it takes a long time to develop a spy. They have to have a cover story, be planted into just the right spot and hope that they can feed stateside analysts with some real information. Often this costly assets took years before they could produce something important. To the McNamara's of the intelligence world, this was an antiquated and costly way of doing business.
Instead, McNamara and his friends wanted to see scientific methods and quantitative analysis used. Hence the shift during the Cold War years from warm-blooded spies to spy planes and spy satellites. These assets removed much of the human error that McNamara hated and were immediately available at a moments notice.
The problem with spy satellites is you only see or hear what the satellite sees or hears. You get none of what is going on in the mind of the target. For example, you can see troop build up and monitor the radio frequencies indicating there is an invasion. But only a spy in the headquarters can tell you that this is all a feint by the target.
Regardless, by the 1990s the CIA HUMINT programs were being slashed in favor of cyber and overhead surveillance. Once the wars began in Iraq and Afghanistan, drones showed the overhead collection game wasn't just for the National Reconnaissance Office anymore!
But even with the incredible advancements with drones, the ability intercept any data transmission at will, is always the same as before. You cannot always gauge intent simply from data. You need someone to tell you what they are thinking.
I've written about this before. Say you do an assessment that says an attack will happen within two weeks. You have imagery and chatter to back this up. Your higher ups concur and the appropriate forces are activated. Then on the day in question nothing happens. Instead of saying, "Good job, we avoided an attack". The analyst gets, "WTF with is wrong with you? Do you know how many assets we wasted on your assessment?" You won't ever get the attention of your higher ups again.
What doesn't happen enough is a retro-analysis of why that attack didn't happen. Only a spy could fill in the blanks for you. Was it a lack of political support or funding? Did the opposition threaten to kill the leader if the attack went through? Didn't a key player suddenly end up in jail? Often you find answers to these questions in you quantitative analysis. You need a spy.
Spies have another advantage over overhead collection (including cyber collection). You have to target your drones, satellites and software to look for a particular threat. A spy can find out about a new, unheard of threat much more easily and redirect their efforts accordingly.
Alas, the love affair with quantity over quality, continues to permeate the intelligence world. The US is also losing its ability to recruit and train spies (since funding is being diverted to newer toys). The problem is terrorism is increasingly carried out by small, unassociated cells using small weapons. These tactics are much easier to disguise from the eyes of satellites and drones. And as long as the group stays off the Internet, software programs won't find them.