Stream Processing for Earth Science Data

From MilcordWiki

Overview

Timely processing of raw Earth science data for calibration and validation in a highly distributed and networked environment, and its storage at Distributed Active Archive Centers (DAACs) for presentation to the global scientific community is critical in NASA’s mission for Earth Sciences. Our solution to this this challenge is to use the emerging stream processing engine technology to build Earth science data processing applications.

Needs

Traditionally stream processing applications have been built using customized DBMS, which tend to be costly, and hard to change by non-specialist end-users. Given the ever increasing space and Earth based science data streams, there is a need for a more efficient way to develop streaming applications to distribute the information to its consumers.

Approach

Our innovation leverages the StreamBase Stream Processing Engine technology platform. A stream processing engine provides in memory inbound processing and real-time stream-oriented processing in a scalable and distributed environment, enabling the development team to focus on the application as opposed to performance optimization.

Benefits

Our solution offers several significant benefits. First, an SPE developed application enables the refinement of stream filtering, the rapid development of new stream filtering capability faster than any other database or middleware based solution using StreamSQL , thus improving the maintainability and adaptivity of the system especially by non-specialist end-users. Second, by design, SPE technology offers inherent fault tolerance against asynchronous data input with attendant drop-outs.

Applications

Our innovation can serve the NASA Terra SDP Toolkit initiative, Raytheon ECS (EOSDIS Core System) Maintenance and Development Project, and Distributed Active Archive Centers (DAAC) computing facilities. In general, our solution will enable the faster design, development and modification of Level 1 through 3 algorithms used in direct broadcast of NASA Earth Science data. Ultimately, our application has the potential to be used by consumers of EOS products to access and manipulate data structures. Our innovation can serve several DoD applications such as the Army Battle Command System (ABCS), and the surveillance monitoring programs at NGA, the Navy and Coast Guard. In the civilian sector, target applications include coastal management programs at NOAA, water management programs at DOI, and disaster management programs at EPA, and DHS.

References

• Abadi,D. J. et al, "The Design of the Borealis Stream Processing Engine", 2nd Biennial Conference on Innovative Data Systems Research (CIDR'05), Asilomar, CA, January 2005.
• Windholz, T., Nittel, S., Thomas, A. C. and Hoffman, R. A. (2006) “A Stream Processing Engine Approach to Earth Science Data Processing”, Technical Report, Contract No. NNC06CB50C, NASA Stennis Space Research Center, Stennis, MS, July 2006.