My GSoC project in 2014 is split into two important parts to better track the project’s evolution and for easier task management. For “Sensor Data Access for Rasdaman”, the two parts of the project were significantly different but aimed at the same result: to integrate Rasdaman as a data storage backend to the 52°North Sensor Observation Service (SOS) implementation. The evolution and results of the first part of the Sensor Data Access for Rasdaman project were presented in the mid-term blog post and on the project’s wiki page. The content of the wiki page contains all the weekly reports, not only the ones corresponding to the first seven weeks. This final blog post gives a brief presentation of the results of this last GSoC phase, but also provides all the resources for the project’s installation and describes some of the problems met along the way.

Achievements

The first step in adding Rasdaman to the SOS was to display it as an option in the web interface, i.e. write the methods for opening the connection, configure the default credentials, configure the default database location and provide the appropriate hibernate dialect for ASQLDB. Image 1 is a screen shot of the configuration screen displayed by the SOS web application.

The user can choose any password and any username as long as the database was not previously created using different login credentials. If the database already exists when you are trying to configure it for the SOS, you can use one of the existing users or create a new one using ASQLDB’s GUI, command line or the JDBC driver. Adding a new user is nothing new, so it can be done using the HSQLDB’s create user statement. The HSQLDB query language is generally valid for ASQLDB. The only differences occur for the SELECT statements when array columns are involved.

The full database path can be configured – a previously defined path for the database would be a problem especially if the SOS would run on Windows instead of Linux. But taking into account that Rasdaman can’t be installed on Windows, chances are slim that someone wants to configure ASQLDB without Rasdaman on Windows.

We found a way to use Hibernate’s Geometry type with ASQLDB. Since ASQLDB does not support this type, we had to find a way to store it in our database. The solution we found was inspired by the geomajas project and it uses the JTSLibrary to serialize the Geometry Object to a String and back. This way we can store the Geometry type as a VARCHAR and receive a Geometry Object when executing queries using Hibernate. The solution to this problem resulted in a successfully configured database (see Image 2).

Once the admin username and password has been chosen, the ASQLDB’s tables are created. The multidimensional array support has not been added yet. It was important to make the datasource configuration work before adding other sources of error, like creating new tables or adding extra mappings for the multidimensional array support. In Image 3 you can see a screenshot with the tables created using the Rasdaman data source configuration. The screenshot presents the configured database opened in ASQLDB’s GUI.

We created a Java class – RasdamanTestData – that inserts all the data that was available in the batch example. This class can be run for data insertion, because the batch example from the Test Client is not working yet. The tables must be created in the database before running the program. In addition to populating the tables, this class creates the “arrayValue” table to store the array value and populates it using Rasdaman. In Image 4, you can see what an array column looks like in the ASQLDB’s GUI. It displays the name of the file in which the array data is stored because the array is usually too large to be shown on a GUI. Only the GUI has this feature. The JDBC driver returns the array itself as a result and not a file name.

We also added an “arrayValue” observation type to the SOS data model in order to start using ASQLDB with Rasdaman and to store multidimensional arrays. After adding this new type and configuring new hibernate mappings for this new table, the “arrayValue” table is created simultaneously with the other tables – at the data source configuration time. This new table is not independent from Rasdaman and if you try to configure another database, like PostgreSQL or MySQL, you will get an error because other databases can’t create this table.

ASQLDB – which uses Rasdaman in order to store array values – accepts queries that ask for a CSV version of the stored array value. We decided to make this type of queries and then transform the CSV result to a SWEDataArray. This conversion is important because the SOS implements encoders and decoders for the SWEDataArray. This means we don’t need to work on encoding anymore, we only make minor changes and specify the use of the converter from Rasdaman CSV to SWEDataArray when an Array Observation response is returned as a result of a request.

Problems

• The default user for HSQLDB is “SA” with an empty password. These were the credentials I used at the beginning for the Rasdaman datasource configuration; however, the SOS web interface does not accept an empty password field. So, be careful and avoid confusing situations like this one.
• ASQLDB’s Hibernate dialect does not support the Hibernate’s Geometry type and the datasource configuration could not create one of the tables. This threw a dialect mapping exception and stopped the configuration.
• A previously configured database can’t be configured a second time. This is a problem that occurs because the SOS can’t load the metadata from the database. We couldn’t solve this problem and for the moment you can only configure an existent database that doesn’t contain any of the SOS tables or create a new database automatically.
• The batch example is not running. At some point during the execution, an exception is thrown. We do not know the concrete reason for this exception. As a result, the problem was not solved. We do have an alternative – run a java class after the tables are created. This class populates all the tables created by the datasource configuration with exactly the same data that is in the batch example. Moreover, it creates a new table – arrayValue – and, as the name suggests, populates this table with array values using Rasdaman as a storage backend for the array columns.
• I have tested the GetObservation request on the database, but I run into other problems. It seems that the SOS does not load the data inserted into the database and the information from the request can not be found in the database.

Future work

• Future work should include fixing the problems listed above. Most of it is debugging work, which may also require code changes for ASQLDB.
• The ASQLDB is still in developement and as soon as other features are implemented, we should integrate and test these within the SOS.

This is the current state of the project. Unfortunately, this is not a final version and it requires additional work to get to a version that can be used by general users. It is less than the GSoC application proposed, but I found myself in many blocking situations which were not easy to debug and solve. I do believe; however, that we are not too far away from the initial expectations.

The Sensor Data Access for Rasdaman is a complex and very challenging project, which requires a good understanding of two (almost three, now that ASQLDB came as a solution) different projects. This is fun to work on, sometimes it may be annoying, but all in all you will enjoy working on it and you will be very pleased with the results. You cann see the reuslts in the project but also in the knowledge gained, especially if you are working on a big project for the first time, like me. All in all, the application successfully demonstrates a possible architecture for connecting Rasdaman with the Sensor Web. Many thanks to my mentors Dimitar Misev (Jacobs University), Daniel Nüst and Carsten Hollman (both 52°North)!

Additional information about the project

• Step-by-step installation guide
• Ubuntu 14.04 VM that includes the source code and a working installation of Rasdaman and SOS. Please find the login information and the installation paths VM documentation file.
• Screen cast demonstrating ASQLDB driver
• Screen cast demonstrating Rasdaman + SOS

Please find all these files in the google drive folder.