The Sensor Web community announces a new version of the SOS (Sensor Observation Service) Importer . It is a tool for importing observations from CSV (Character Separated Values) files to a SOS instance. An eight step wizard guides a user through the often cumbersome process of data transformation.
You can download the binaries from our Maven release repository:
Alternatively, you can build the binary files from the source code.
Feel free to share your experience with the Sensor Web community in our forum or mailing list (choose which you want, they are synchronized ;-)).
Version 0.4 offers the following new features:
- Support for sensors with multiple outputs
- Introduced import strategies:
- SingleObservation: Default strategy
- SweArrayObservationWithSplitExtension: Reads hunksize# lines and imports each time series using a SWEArrayObservation in combination with the SplitExtension of the 52°North SOS implementation. Hence, this strategy works only in combination with a 52°North implementation. Other implementaions might work too, but not as expected. Hunksize and import strategy are both optional <AdditionalMetadata><Metadata> elements.
- Support for date information extraction from file name using two new OPTIONAL attributes in element <DataFile>:
- “regExDateInfoInFileName” for extracting date information from file names
- “dateInfoPattern” for parsing the date information into a java.util.Date.
- Date information extraction from last modified date using two new OPTIONAL attributes:
- “useDateFromLastModifiedDate” for enabling this feature
- “lastModifiedDelta” for moving the date n days back (this attribute is OPTIONAL for this feature, too).
- Ignore lines with regular expressions feature: 0..infinity elements can be added to the element. Each element will be used as a regular expression and applied to each line of the data file before parsing.
- Handling of data files containing several sample runs. A sample run contains additional metadata, such as its size (number of performed measurements) and a date. The required attributes are:
- “sampleStartRegEx” – the start of a new sample (MUST match the whole line).
- “sampleDateOffset” – the offset of the line containing the date of the sample from the start line.
- “sampleDateExtractionRegEx” – the regular expression to extract the date information from the line containing the date information of the current sample. The expression MUST result in ONE group. This group will be parsed to a java.util.Date using “sampleDatePattern” attribute.
- “sampleDatePattern” – the pattern used to parse the date information of the current pattern.
- “sampleDataOffset” – the offset in lines from sample beginning until the first lines with data.
- “sampleSizeOffset” – the offset in lines from sample beginning untl the line containing the sample size in lines with data.
- “sampleSizeRegEx” – the regular expression to extract the sample size. The regular expression MUST result in ONE group which contains an integer value.
- Setting of timeout buffer for the insertion of SweArrayObservations: With the attribute “insertSweArrayObservationTimeoutBuffer” of “SosMetadata”, it is possible to define an additional timeout buffer for connect and socket timeout when using import strategy “SweArrayObservationWithSplitExtension”. Scale is in milliseconds, e.g. 1000 => 1s more connect and socket timeout. The size of this value is related to the set-up of the SOS server, importer, and the HUNK_SIZE value. The current OX-F SimpleHttpClient implementation uses a default value of 5s, hence setting this to 25,000 results in 30s connection and socket timeout.
More details can be found on the Wiki page of the SOS importer.
Leave a Reply