data formats Archive - Blog

Introduction

The enviroCar SPARQL endpoint is a feature that enables the enviroCar data to be queried in RDF. This feature creates an open endpoint for crawlers and other bots and allows them to crawl the data and let the data be part of the LOD cloud. With the enviroCar server’s current architecture, it is possible to query the data by converting the entities on the fly into RDF when a particular kind of request is made. Please find a more detailed description about the project’s evolution on the SPARQL EndPoint wiki page.

Over the past few weeks, I’ve embarked on the task of moving the data from mongoDB (the database of choice for the enviroCar server) and cloning the entire database content to a raw data type, independent of database formats. My task also involved solving the problem of trying to remove the tight coupling of frameworks with the generation and retrieval of data. This would make more data interchanging processes more commonly known as ETL processes. This is a design for future implementation and can be found here.

more >

The culmination of 13 long and arduous weeks as part of Google Summer of Code 2012 has produced a new prototype feature within Web Processing Service: the ability to take OpenStreetMap data and transform it on the fly into other GIS formats.

What does this mean?

We can now directly interface with OpenStreetMap servers (Overpass API) such that we can send custom queries for map data retrieval, and have it returned not as OSM XML files, but rather any GIS format that is supported by WPS. This is demonstrable through the WPS test client on the demo server. Here is a breakdown of the features implemented:

Standard conversion of existing OSM XML files into other GIS formats (basic GML, KML, and Shapefiles)
On-the-fly conversion of Overpass API output into other GIS formats
- Option for OSM passthrough
- Option for any of the GIS formats supported by WPS as output
- Option to show original Overpass query

Currently there are two sample algorithms that are bundled in this project:

A sample implementation of a Bounding Box algorithm for retrieving specific areas of the OpenStreetMap planet.osm
A generic implementation that takes any string query and sends it to the Overpass server, and retrieves the result for processing

This allows for adventurous developers to go in and create their own custom algorithms for WPS that interface directly with OpenStreetMap.

Some videos of this in action are available here:

More details on installation, configuration, and development can be found on the wiki page for this project.

Comments and Future work

Of course, this is probably not the end. There exist many avenues of future work, some of which I have thought of during the process of development, and some which have been suggested to me. A short list of what would be useful next steps:more >

SPARQL endpoint and Interoperability of enviroCar data : Midterm Report

Introduction

On-demand transformation of OpenStreetMap Data into common GIS formats