Subscribe / Unsubscribe Enewsletters | Login | Register

Pencil Banner

DARPA: Big data needs Big search, Big analytics

Michael Cooney | May 28, 2014
Defense Advanced Research Projects Agency programs to look at changing search, analytics technology

How you might one day most effectively negotiate the enormous amounts of data and video around the Internet was on display recently as DARPA's Information Innovation Office hosted Demo Day 2014 to highlight the agency's ongoing contributions to what it called "preserving and expanding IT superiority."

Among the over 100 products on display were a couple looking to address future issues such as vastly improving search functions and analyzing tons of data.

DARPA says the existing methods of searching the Internet for all manner of information just won't cut it in the future as the "one-size-fits-all approach to indexing and search of web content limits use to the business case of web-scale commercial providers."

To go beyond that DARPA is working on a program known as Memex which looks to devise "better methods for interacting with and sharing information, so users can quickly and thoroughly organize and search subsets of information relevant to their individual interests. The technologies developed in the program would provide the mechanisms for improved content discovery, information extraction, information retrieval, user collaboration and other key search functions." DARPA says.

In its announcement of the program DARPA said it expects Memex to:

  • Develop of next-generation of search technologies to revolutionize the discovery, organization and presentation of domain-specific content
  • Create of a new domain-specific search paradigm to discover relevant content and organize it in ways that are more immediately useful to specific task
  • Extend of current search capabilities to the deep web and nontraditional content
  • Improve interfaces for military, government and commercial enterprises to find and organize publically available information on the Internet

While Memex technology will focus on search, DARPA's Big Mechanism program aims to change the way tons of data are analyzed. 

DARPA Big Mechanism Program Manager Dr. Paul Cohen said the program looks to develop a system that could autonomously gather all exiting data about a particular topic, keep it up-to-date and develop new conclusions or research directions.

"We are looking to change the way people develop knowledge," Cohen said.

For example, today researchers struggle to keep up with relentless streams of relevant publications. To stay current, a researcher must specialize, becoming expert in a small part of something much bigger. The vision for the Big Mechanism program is fundamentally different:  Every publication would immediately become part of a public, computer-maintained, causal model of a complicated system-a big mechanism-and every aspect of a big mechanism would be tied to the data that supports it or contradicts it.  To the extent that we can automate the construction of Big Mechanisms, we can change how science is done," DARPA said.

In a nutshell the Big Mechanism program will develop technology to read research abstracts and papers to extract fragments of causal mechanisms, assemble these fragments into more complete causal models, and reason over these models to produce explanations.


1  2  Next Page 

Sign up for Computerworld eNewsletters.