The aim of the OpenSensorSearch project is to provide an implementation of a sensor discovery server with the high performance of a Solr back-end to give users results fast. In addition, we are working on harvesting mechanisms for OpenSensorSearch (OSS) to allow users and developers to write harvesters for their own sensor data sources.
In the previous blog post, we discussed the work done in the first few weeks related to the database back-end and how we used Apache Solr for indexing and fast searching. Today we present the work done in the second half of the project. It focussed on harvesting metadata from other sources and the user interface. I mainly discuss
- “harvest callback” technique,
- user interface implementation with Spring MVC and
- general technical details.
The full mechanism is outlined in the following diagram:
A web API allows the submission and scheduling of remote servers supporting this method. For detailed information check the wiki. The mechanism was implemented and tested using both integration tests and unit tests using frameworks like wiremock and mockito.
New User Interface
A whole new Spring MVC web application was implemented using bootstrap for layout and styling. It features login/registration of users, and users can schedule and submit their harvesting scripts or harvest callback URLs. The start page was enhanced with the autocomplete search based on the Solr back-end. On top of that, the new web service APIs are documented for developers with new HTML documentation pages.
The main interface
Sign in for script submission
Harvest callback with auth_token returned
Developers API developed using Swagger UI
Screenshots of testing the UI on major browsers can be found at https://wiki.52north.org/bin/view/SensorWeb/OpenSensorSearchResponsiveUI.
Technical Details and Future Tasks
- Because we take integration tests seriously, we must admit that some weren’t written very well and some were missed, so we added all of those for reference and future use in our backlog system which uses easybacklog.com.
- As indicated in the previous blog post, the results were not decisive as to whether SOLR results were better than PGSQL or not, so we restrict it to autocomplete for now and wait for real data sensors for better testing.
- All of the app with it’s API and OSS-UI relied on DAO and pure SQL for query implementation. This got very complicated as the size of the project increased. So I think that a shift to an ORM tool like Hibernate would improve the project considerably.
- I think the data structures that are used need some working, i.e. we should implement a whole new classes of data structures to be used by script authors.
It was nice working on OpenSensorSearch even though it was tough at some times. To me, the most annoying things about it were the cumbersome configuration and sticking to a set of standards – that we have to – that tends to complicate things. For next summer, I would encourage students to work on this project and extend it, enhance it’s back-end and test it thoroughly, change the database access to use Hibernate and maybe extend the UI to allow more user levels and more capabilities.
Concerning my goals, I was able to achieve most of what I intended to do. I was delayed a bit by the time required to understand the system and it’s standards. Also I spent more time writing test cases than I expected, but the overall progress was satisfying. My mentors were very helpful and spent considerable time in managing the system and giving help and direction.
- The original post that explain what I was intended to do at the beginning of summer can be found here.
- The midterm report that contain detailed explanation of the database backend can be found here.
- My fork of OpenSensorSearch at GitHub can be found here.
- The demo server where soon the described features can be tested is here.