So what can we expect for lets say the next month of ILWIS. It’s nice that the 3.8 has been released but that doesn’t mean that the work has suddenly stopped. I have been rather busy the last few weeks with a few developments that I would like to share with you.
- Tables. The internal design of tables in ILWIS is quite silly (imo). Echoes from a very distant past of Ilwis that were never properly redesigned. The basic assumption of the ( internal) structure is that everything is based on columns instead of records. Now basically there is nothing wrong with that. Most operations you do (in Ilwis) are column based not record based. Yet internally there is a big flip-flop continuously happening between column based and record based handling of things. This causes a big performance hit. Furthermore because all columns are owners of their own data (and not the table), data access is often going through many layers before getting at the ‘real’ data.
I redesigned the table structure to a unified system. In the Ilwis 3.8 this already used for loading the vector maps which are basically tables when stored. But these tables are very isolated and don’t have any interaction with tables in general. I am extending that now to the rest of the tables but have to be very careful not to disrupt existing functionality.
So what do I gain as end-user? Well, tables still work the same as they used to do. But… They load 20x times as fast(realy) and have an access time that is 2-3x as fast. So , yes. You do gain something.
- Openstreemap tile server. For those who don’t know it. Openstreetmap is an Open database where geographical vector data is stored free for access by the outside world(see http://wiki.openstreetmap.org/wiki/Main_Page). An Openstreetmap tileserver is a renderer of this data onto tiles of many different zoom levels so that others can use it as layers in their application. I need to support this in Ilwis as it is needed for one of sub-projects of Ilwis and is useful in general. Of course this only works if you have a network connection where there is a tile server.
- Geonetcast data as OpenDap service. This is work I already started late last year (with success) and I need to polish it up somewhat. Geonetcast is a successful toolbox based on Ilwis, which exposes (see also https://52north.org/communities/earth-observation/ of 52N) loads of RS data in a consistent way. There are whishes from another project I am working with to be able to access this data as a service. The problem is that the data stream is rather large, continuously updating ( from 5 minutes to one day) and in the usual zoo of different raster formats. The toolbox translates this to an Ilwis format to have a consistent and accessible format to use the data in analyses. The data stream is so large though that this can’t be done on a continuous basis. It has to be done per request for a specific dataset(client side). This is the work of the toolbox. For this project I had to do about the same but this time server side(but still per request) and then translate the Ilwis data to netcdf as OpenDap is built around netcdf. What you now basically do is sent a WPS command to an ILWIS or 52N WPS server, it will do the necessary processing server side and then send a OpenDap link back to the client. This seems to work ok; I now included some extra subsetting to reduces the workload for the server.
- Debugging. As usual after a release you encounter some issues that should have been addressed ( import shape, grrr). Ah well, such is life. I will not make a new 3.8.1 or something but simply replace the current download. The about box will probably show 3.8.01 I suppose.
- Orfeo. Already for some time I eyed the orfeo toolbox (http://www.orfeo-toolbox.org/otb/) as a possible extension for Ilwis functionality. But during the making of 3.8 I simply didn’t have time for it. I will make an Ilwis shell for it (both on UI level as on command level) and the include it in Ilwis. If all works well Ilwis will be seamlessly be integrated with the Orfeo functionality. Probably this will not be part of the standard Ilwis download but available as separate plug-in from the earth observation community of 52n.
- And last but not least Ilwis NG. This is continuous process and progress has been made there.
So what is de planning of all this? Well point 1), 3) and 4) are in the last stages. Have still to do some work there but the bulk is done. I plan the 3.8.01 for end of this month or possibly the first week of June. This will probably not include the table stuff (yet) as I need to do some real testing there if I didn’t mess things up, tables are quite important after all. Orfeo will probably start somewhere mid June. I am unsure at the moment how long this will take but my first guess would be about a month. And Ilwis NG? Hmm well, difficult question. I suppose that somewhere in 2013 there will be a version but if that already has the same functionality as Ilwis 3.8 I am unsure. Anyway NG will be very modular so I can release it piece by piece. We will see, it is a bit premature to speculate too much about that.