Several months after publishing the SOS 4.x developments, the Sensor Web community is pleased to announce the
A downloadable zip archive of the release contains the sources, as well as the binaries. Release requirements, installation and configuration information are documented in the 52°North wiki.
Many thanks to all 52°North SOS contributors for their enhancements and improvements, but also to all users for their feedback!more >
As I mentioned in a previous post (please read it before continuing), I intended to extend the Open Geospatial Consortium (OGC) Web Processing Service (WPS) enabling it to process continuous data streams, as well as to send intermediate results back to the client. These two capabilities are realized with Full Streaming WPS, a new approach that brings near real-time geoprocessing to WPS. In this post, the foundations of such an approach are given, as well as implementation details and pointers to public resources. It also includes a demo.
I intend to implement a streaming based WPS on the top of the 52°North WPS framework for enabling WPS basic processes to handle input and output spatial data streams. This post shows the current state of the work, as well as further steps.
The Open Geospatial Consortium (OGC) Web Processing Service (WPS) specification lacks a way to deal with continuous spatial data streams. Think about sensors providing potentially unbounded spatial data streams that must be processed before being meaningful to end-users. Real-time geoprocessing (McCullough et al., 2011) or Live Geoinformation (Foerster et al., 2012) have been described recently, but we still miss an open implementation that can be seamlessly integrated with existing OGC standards, namely WPS. My work at 52°North deals with solving this issue.
Multimedia streaming
The idea comes from multimedia streaming. It allows users to consume media without the need to download the entire content (Timmerer and Müller, 2010). Thus, the user accesses and consumes chunks of media. In the spatial context, we can think of these chunks as feature collections (for vectors) or tiles (for rasters). Early consumption of spatial data chunks makes sense since it enables rapid assessment of process results at an early stage, which could allow, for instance, one to cancel the process if results are not as expected.
Multimedia streaming has two main modalities: on demand streaming and live streaming. The former provides output streaming for content stored in the server (e.g., Dailymotion, Vimeo), whereas the latter does not intend to store media but to transfer it as soon as it is available (e.g., Ustream, Skype). Both modalities can be implemented for spatial data with some caveats. On demand streaming is implemented as “Output Streaming” and live streaming as “Full (input/output) Streaming.”
Below is a comparison of the streaming modalities, adapted from (Foerster et al., 2012): Current WPS (at the top), Output Streaming (in the middle), and Full Streaming (at the bottom). Where tasks are parallel, initial response time gets reduced.
Output Streaming
Output Streaming WPS receives input data as a whole, splits it into chunks, and starts processing each chunk for publishing intermediate results. This reduces latency, i.e., time to access process results. A prototype of output streaming for both vector and raster data has been already implemented, see the videos.
Output Streaming WPS on vector data
Output Streaming WPS on raster data
Intermediate results are published by means of a plain text file called ‘playlist’, in which URLs pointing to new available results are appended by the server. The client constantly checks the playlist looking for new content in order to download, decode (if needed), and display it. At the end, the client assembles all chunks in a single layer.
It is important to note that not all algorithms could benefit from Output Streaming. Particularly, algorithms that require focal or global knowledge are not supported. Nevertheless, alternative ways of splitting spatial data for parallel processing could enable support for more algorithms. This is an open research field in Geographic Information Science.
Full Streaming
On the other hand, Full Streaming WPS is yet to be implemented. It will receive spatial data streams and meanwhile, start processing them for publishing results. Full Streaming WPS will enable one to process unbounded data streams because it receives, processes, and publishes spatial data simultaneously. This is a potential new field of action for WPS.
Conclusion
In summary, streaming based WPS brings new capabilities for WPS: first, it allows one to access intermediate process results and, second, it enables processing of unbounded data streams (Borsutzky, 2011). There is a prototype of the former, whereas the latter is still to be implemented. At the end of the project the source code will be available at the 52°North Geoprocessing Community’s repository. Stay tuned!
References