OpenSource Connections
Category: Lucene
I just came back from the 2011 DI2E conference in Dallas, Texas and certainly left with a lot of impressions. I want to state up front that my impressions were very positive about the state of U.S. intelligence activities. Weve come a long way since we first started inserting Green Berets into Afghanistan. Whereas we once faced an intelligence gathering problem, we no longer do. Instead, we have trouble indexing, sorting, filtering, and analyzing the glut of intelligence that we have gathered. I did notice that Solr and Lucene are in use pretty robustly in the intelligence community, with Solr a part of the ICDL and JDL (intelligence community data layer and joint data layer respectively). Thats fantastic news, but theres a long way to go.
- Going to the cloud helps, but is not the panacea. One of the big issues that the intelligence community faces is the ability to index what it calls “big data.” Granted, it was a public conference, so they werent going to reveal all of the secrets, but I didnt hear anything which led me to believe that the data indexing issue was insurmountable. Certainly throwing enough hardware at the problem can help solve it, but whose cloud rules? So, instead of waiting for the cloud to deliver a magical elixir, the intelligence community can also look at improving indexing, caching strategies, and leveraging other systems. One common complaint that I heard was that Hadoop causes read time isues – its too slow. To answer that, look at Katta or Cassandra, or be willing to crack into Hadoops code and do modifications. The government cant just rely on COTS to solve its problems.
- A glut of money led to inefficient development and stovepipes. Because each service got a pile of money, they each developed their own solutions. Most of them dont integrate well with other systems and solutions. They also use proprietary systems and protocols. At least the military recognizes the problem, as one speaker said that they can no longer add twenty different license costs for software that all does essentially the same thing.
- Even with all of that money, search user experience got left behind. The military employs people, called Knowledge Managers, to teach people how to use the tools that are there. However, the tools are, at best, clunky, and at worst, a terrible user experience. As a result, as one of the speakers noted, over 90% of the tools are not used. I am certain that Netflix does not have a cadre of trainers to teach its users how to find movies. Zappos does not have its payroll stacked with people who can tell you how to find shoes on the Zappos website. Why? Because the user experience is well conceived and well executed. The military should look at innovators in industry and adapt their models. The money saved spending a little on vastly improved user experience will be more than made up for in the reduction of knowledge management requirements. Improving search and the search user experience has a HIGHLY positive ROI in this environment. It begs for attention and improvement.
- You cannot just Google for intelligence information. Googles strength is that it has the wisdom of the masses. It has algorithms designed over the years to identify what the average person wants to see and when a website has legitimate information rather than being a spam or link farm. The intelligence community is vastly different. First, how many regular Google users are trying to do an IPB for a commander via Google? The sample size from which to draw the algorithms is incredibly small. Secondly, while analysts are interested in knowing the provenance of intelligence information, there is no spam in the intelligence community dataset. Thus, the real need is to tune for relevancy – get relevant results for the specific user set that the search identifies. When the intelligence community buys a license for a search engine which doesnt make relevancy tuning easy, intuitive, and comprehensive, it is wasting taxpayer money.
- Acquisition hinders innovation. Because requirements and contracting have to be so specific with most government contracts, the cycle time from need to fielding is often over two years. That does not keep up with the pace of change. This is not a new jet fighter. New, category killer technology is unlikely to pop up overnight. However, in a theater where information dominance is paramount, that lack of speed is going to kill soldiers. Risk aversion and an acquisition strategy shackled by a 1970s Cold War mindset means that we dont get the best information technology available out to the soldiers who need it most in a timely fashion.
As GEN Odierno stated, the intelligence community is going to need to get more bang for the buck. Defense budgets have been reduced and will continue to see declines in the intermediate future. This funding reduction means that intelligence community technology users will need to demand increased interoperability, open standards, and control. They will need to be willing to look at new, innovative, and alternative solutions, not the same old ones trotted out by the same old integrators, and they will need to demand alacrity and agility from both providers and the acquisition community, because the rate of growth of the amount of collected data will only continue to increase. The tooling and capability to search, retrieve, and do something with that information needs to catch up quickly.