1. Tables. The internal design of tables in ILWIS is quite silly (imo). Echoes from a very distant past of Ilwis that were never properly redesigned. The basic assumption of the ( internal) structure is that everything is based on columns instead of records.  Now basically there is nothing wrong with that. Most operations you do (in Ilwis) are column based not record based. Yet internally there is a big flip-flop continuously happening between column based and record based handling of things. This causes a big performance hit. Furthermore because all columns are owners of their own data (and not the table), data access is often going through many layers before getting at the ‘real’ data.
   I redesigned the table structure to a unified system. In the Ilwis 3.8 this already used for loading the vector maps which are basically tables when stored. But these tables are very isolated and don’t have any interaction with tables in general. I am extending that now to the rest of the tables but have to be very careful not to disrupt existing functionality.
   So what do I gain as end-user? Well, tables still work the same as they used to do. But… They load 20x times as fast(realy) and have an access time that is 2-3x as fast. So , yes. You do gain something.
2. Openstreemap tile server. For those who don’t know it. Openstreetmap is an Open database  where geographical vector data is stored free for access by the outside world(see http://wiki.openstreetmap.org/wiki/Main_Page). An Openstreetmap tileserver is a renderer of this data onto tiles of many different zoom levels so that others can use it as layers in their application. I need to support this in Ilwis as it is needed for one of sub-projects of Ilwis and is useful in general. Of course this only works if you have a network connection where there is a tile server.
3. Geonetcast data as OpenDap service. This is work I already started late last year (with success) and I need to polish it up somewhat. Geonetcast is a successful toolbox based on Ilwis, which exposes (see also http://52north.org/communities/earth-observation/ of 52N) loads of RS data in a consistent way. There are whishes from another project I am working with to be able to access this data as a service. The problem is that the data stream is rather large, continuously updating ( from 5 minutes to one day) and in the usual zoo of different raster formats. The toolbox translates this to an Ilwis format to have a consistent and accessible format to use the data in analyses. The data stream is so large though that this can’t be done on a continuous basis. It has to be done per request for a specific dataset(client side). This is the work of the toolbox. For this project I had to do about the same but this time server side(but still per request) and then translate the Ilwis data to netcdf as OpenDap is built around netcdf. What you now basically do is sent a WPS command to an ILWIS or 52N WPS server, it will do the necessary processing server side and then send a OpenDap link back to the client. This seems to work ok; I now included some extra subsetting to reduces the workload for the server.
4. Debugging. As usual after a release you encounter some issues that should have been addressed ( import shape, grrr). Ah well, such is life. I will not make a new 3.8.1 or something but simply replace the current download. The about box will probably show 3.8.01 I suppose.
5. Orfeo.  Already for some time I eyed the orfeo toolbox (http://www.orfeo-toolbox.org/otb/) as a possible extension for Ilwis functionality. But during the making of 3.8 I simply didn’t have time for it. I will make an Ilwis shell for it (both on UI level as on command level) and the include it in Ilwis. If all works well Ilwis will be seamlessly be integrated with the Orfeo functionality. Probably this will not be part of the standard Ilwis download but available as separate plug-in from the earth observation community of 52n.
6. And last but not least Ilwis NG. This is continuous process and progress has been made there.

So what is de planning of all this? Well point 1), 3) and 4) are in the last stages. Have still to do some work there but the bulk is done. I plan the 3.8.01 for end of this month or possibly the first week of June. This will probably not include the table stuff (yet) as I need to do some real testing there if I didn’t mess things up, tables are quite important after all. Orfeo will probably start somewhere mid June. I am unsure at the moment how long this will take but my first guess would be about a month.  And Ilwis NG? Hmm well, difficult question. I suppose that somewhere in 2013 there will be a version but if that already has the same functionality as Ilwis 3.8 I am unsure. Anyway NG will be very modular so I can release it piece by piece. We will see, it is a bit premature to speculate too much about that.

Daniel demonstrates proper throwing techniques.

attentive listeners…

Ready?

Action!

Registration is open from April 6 – May 6, 2012 at www.app-camp.eu.

So, are we done? Of course not. Undoubtedly there will be a 3.8.1 (no, I don’t say when) and a 3.8.2 is also likely. Maintenance is always needed. There are a few things I have on the menu (in random order)

• Polygon editor. Missing from 3.8 . I now have the technology to make a “real” polygon editor and not that half assed thing that was there in 3.3 and later. But this costs some time.
• Sample set editor. Maybe. I am waiting a bit until people start complaining . I didn’t find the time to finish it ( a very buggy version exists) and until now nobody seems to be missing it. Since I am reconsidering Ilwis’ whole classification stuff  anyway (it has to be improved) I might need a full redesign. This may be related too.
• The Orfeo toolbox. I am considering  integrating the Orfea toolbox (http://www.orfeo-toolbox.org/otb/) in Ilwis as a plug-in. This would extend the functionlity of Ilwis with a lot of cool stuff.
• Improving the speed of tables. Tables are internally based on a silly concept (not my design ) that costs a lot of performance. I already made a prototype for a new design (it is used at a few isolated places in Ilwis 3.8) that loads data files 15-20x as fast (really) and gives access to data about 10x as fast. I need this for big data sets. I need it fast.
• And of course QA. That is always there. There are always new bugs to solve.

So enough to do.

And then there is of course Ilwis NG. The next version of Ilwis. This will probably occupy more and more of my time in the rest of the year. A proposal for a general architecture has been created

These are the picture programmers like to stare at, forgetting all the work that is involved. Most people couldn’t care less about all the squares and arrows, but I thought it might be useful to show that Ilwis NG is no pipe dream. For me the goal is to create an Ilwis version that can function as a kind of connection point between hetrogeneous data sources and diverse processing services while facilitating a good UI – Ilwis as a kind of integrator that delivers a consistent and good UI, be it desktop or browser (something that is sorely lacking in many open source software projects). I also want to create a source base that is much more open for outside developers and much more modular.

In a pilot, I already created a base version of Loader and part of de core data structures for Ilwis NG. This doesn’t do much (apart from loading data) but it is a proof of concept that the ideas that are represented in the diagram are workable. It would be nice to have more developers on board as this would speed up the work and improve the design process.

52°North WPS 3.0,

as well as the corresponding releases

52°North WPS Process Development Kit 3.0
WPS Client API 3.0

The basic changes and extensions from the last version are:

- R support
R is a language and environment for statistical computing is a language and environment for statistical computing. Over the past 10 years, the developers of spatial statistical methodology in R have agreed on a common set of classes and methods for spatial data in R, and thus made the set of 70+ add-on packages for spatial statistical analysis in R a coherent undertaking. The WPS framework allows to upload and expose R scripts dynamically as WPS processes.

- stability and performance improved on a large scale
- 85 Unit tests added covering all parsers, generatros and WPS requests modes
- more parsers, databindings and generators
- redesigned config
- “offical” mime types and schemas used
- several bugfixes
- extended Tutorials

See the tutorials section for setting up the 52°North WPS, creating a new Process, executing and exporting the process [2].

The source code is taged as “WPS-3.0″ and can be found in our svn repository [3]
The binary distribution can be found here [4].

Test instances can be invoked here [5].

A tutorial for the WPS Process Development Kit can be found here [6].

A tutorial for the WPS Client API can be found here [7].

Regards,
52N WPS development Team

[1] http://security.demo.52north.org:8080/view/geoprocessing/job/52n-wps%20complete%20Release/changes
[2] http://52north.org/communities/geoprocessing/wps/tutorials.html
[3] http://52north.org/communities/geoprocessing/wps/source_setup.html
[4] http://52north.org/index.php?option=com_jdownloads&view=finish&cid=431&catid=14
[5] http://geoprocessing.demo.52north.org:8080/wps/
[6] http://52north.org/communities/geoprocessing/wps/tutorials/ProcessDevlopmentKit.html
[7] http://52north.org/communities/geoprocessing/wps/tutorials/ClientAPI.html

In the next days, the mentors will review and rank the applications.

(Thanks to Olly Betts for the post and the spreadsheet.)

In the Animation management form there had always been a somewhat mysterious tab called “progress control”. In it you could see a graph and a marker going at the speed of the animation over graph that was basically a straight line. There was a combobox called “reference attribute” but what it all meant was rather unclear. It was unclear because it was unfinished. I had some ideas about it but never found the time to act on those ideas. Fortunately Ms Dewi came along and with her help the unfinished part became finished.

The idea behind this tab was to create a system to enable a user to visualize when during an animation the (numerical) values in a map exceed a certain meaning full number. During an animation the image is constantly changing and it is sometimes difficult to track the values on the map to see exactly what is happening. To use this system a user has to create an attribute table attached to the maplist that contains a column with marker values. To assist you in this there has been created an application “TableMapListAttributeFrom” that creates a table with statistical marker values for each index in the maplist (e.g. average, median etc..). But you may also create a column by any other means with significant values.

The column can then be used as “reference attribute”. The tab might then look like this

A graph that shows the average of 72 maps and a time marker moving over those maps. Furthermore there is a blue threshold marker (see below) that gives an indication were the average of a map passes the treshold. ‘Fine’ you may think, maybe slightly useful but not overly so. Things get more interesting when you also look what is happening in the animation.

On the map red pixels have appeared. These are pixels where the pixel value has exceeded the treshold. During the animations you constantely see the areas change and it is easier to track changes this way. The properties of the treshold can be set by the pressing the button. You then get small form with some properties.

The tool becomes realy interesting to use when running two synchronized animations with threshold values, making the top animation partly transparent. It is then easy to track if changes in one dataset have possibly some relation to changes in another one.

Note, that you have related functionality through the “interactive representation” but here there is a clear link with statistical significant markers values.

It’s fun to explore new horizons! We look forward to the next frisbee challenge on April 20th.

Martin Reckmann demonstrates throwing techniques.

Conny works her way around bothersome obstacles…

hmmm…

Yeah!

Refreshments after hard work!