52°North Spatial Information Research GmbH

I intend to implement a streaming based WPS on the top of the 52°North WPS framework for enabling WPS basic processes to handle input and output spatial data streams. This post shows the current state of the work, as well as further steps.

The Open Geospatial Consortium (OGC) Web Processing Service (WPS) specification lacks a way to deal with continuous spatial data streams. Think about sensors providing potentially unbounded spatial data streams that must be processed before being meaningful to end-users. Real-time geoprocessing (McCullough et al., 2011) or Live Geoinformation (Foerster et al., 2012) have been described recently, but we still miss an open implementation that can be seamlessly integrated with existing OGC standards, namely WPS. My work at 52°North deals with solving this issue.

Multimedia streaming

The idea comes from multimedia streaming. It allows users to consume media without the need to download the entire content (Timmerer and Müller, 2010). Thus, the user accesses and consumes chunks of media. In the spatial context, we can think of these chunks as feature collections (for vectors) or tiles (for rasters). Early consumption of spatial data chunks makes sense since it enables rapid assessment of process results at an early stage, which could allow, for instance, one to cancel the process if results are not as expected.

Multimedia streaming has two main modalities: on demand streaming and live streaming. The former provides output streaming for content stored in the server (e.g., Dailymotion, Vimeo), whereas the latter does not intend to store media but to transfer it as soon as it is available (e.g., Ustream, Skype). Both modalities can be implemented for spatial data with some caveats. On demand streaming is implemented as “Output Streaming” and live streaming as “Full (input/output) Streaming.”

Below is a comparison of the streaming modalities, adapted from (Foerster et al., 2012): Current WPS (at the top), Output Streaming (in the middle), and Full Streaming (at the bottom). Where tasks are parallel, initial response time gets reduced.

Output Streaming

Output Streaming WPS receives input data as a whole, splits it into chunks, and starts processing each chunk for publishing intermediate results. This reduces latency, i.e., time to access process results. A prototype of output streaming for both vector and raster data has been already implemented, see the videos.

Output Streaming WPS on vector data

 

Output Streaming WPS on raster data

Intermediate results are published by means of a plain text file called ‘playlist’, in which URLs pointing to new available results are appended by the server. The client constantly checks the playlist looking for new content in order to download, decode (if needed), and display it. At the end, the client assembles all chunks in a single layer.

It is important to note that not all algorithms could benefit from Output Streaming. Particularly, algorithms that require focal or global knowledge are not supported. Nevertheless, alternative ways of splitting spatial data for parallel processing could enable support for more algorithms. This is an open research field in Geographic Information Science.

Full Streaming

On the other hand, Full Streaming WPS is yet to be implemented. It will receive spatial data streams and meanwhile, start processing them for publishing results. Full Streaming WPS will enable one to process unbounded data streams because it receives, processes, and publishes spatial data simultaneously. This is a potential new field of action for WPS.

Conclusion

In summary, streaming based WPS brings new capabilities for WPS: first, it allows one to access intermediate process results and, second, it enables processing of unbounded data streams (Borsutzky, 2011). There is a prototype of the former, whereas the latter is still to be implemented. At the end of the project the source code will be available at the 52°North Geoprocessing Community’s repository. Stay tuned!

References

• H. Borsutzky. (2011). Streaming von Geodaten mit webbasierten Prozessierungsdiensten. (Unpublished diploma thesis). University of Münster, Germany.
• T. Foerster, B. Baranski, and H. Borsutzky. (2012). Live Geoinformation with Standardized Geoprocessing Services. In J. Gensel, D. Josselin, and D. Vandenbroucke (Eds.), Bridging the Geographic Information Sciences (pp. 99–118). Berlin, Heidelberg: Springer Berlin Heidelberg. 2012. Retrieved from http://www.springerlink.com/index/10.1007/978-3-642-29063-3_6
• A. McCullough, S. Barr, and P. James. (2011). A Typology of Real-Time Parallel Geoprocessing for the Sensor Web Era. In T. Foerster, A. Bröring, B. Baranski, B. Pross, C. Stasch, T. Everding, and S. Maes. (Eds.) (2011). Integrating Sensor Web and Web-based Geoprocessing. CEUR Workshop proceedings (Vol. 712). Utrecht, Netherlands: CEUR. Retrieved from http://ceur-ws.org/Vol-712/
• C. Timmerer and C. Müller. (2010). HTTP Streaming of MPEG Media. Proceedings of the Streaming Day 2010, Udine, Italy, September 16-17, 2010.