Map/TabCalc was and will be a central part of ILWIS. In ILWIS 3 it is a very compact way to do calculations on whole data sets at once. It is easy to learn and forms the basis for many scripts that access ILWIS 3 functionality. However, it is far from perfect.

• There is a confusing set of interfaces with different names and different subsets of the command set. You have the commandline, simplecalc, mapcalc, tabcalc, script; not easy to explain the differences between them. Is this really needed? I don’t think so.
• There is a severe lack of conditionals and looping in the script. That’s ok if you have the IFF constructs, but they only work in a single call. There is no IF/THEN/ELSE. There is no WHILE. This severely limits the control you need for a script. Now you might make a case that you don’t need that since you can integrate ILWIS with the ILWIS-C commandline construct in almost any existing script environment. I do sympathize somewhat with this argument, but I don’t like a mandatory dependency on some external scripting environment (e.g. python). I do need to make a scripting environment for backward compatibility anyway, so improving it is not that big a deal.
• The implementation of the current scripting is a mess. Its performance is ok. It does things reasonably efficiently, but it is one of the very few parts of the code in which I would rather not make any changes. The inner workings of it are rather obscure. As far as I know (the core of it was written before I started with ILWIS), there is no formal syntax with nice EBNF notation that runs through a parser/lexer generator combination to generate some abstract syntax tree or equivalent. It is a meticulously, hand crafted parser/lexer that does all the (very) complicated stuff that is normally obscured from the programmer and you must understand it if you want to make changes. I don’t expect that many readers have ever made custom languages (I have), but that is a horrible way to do it. As programmer, I only want to deal with the EBNF and (in this case) the abstract syntax tree. The unreadable code that connects these two is automatically generated with well know techniques and proven tools (that was also the case 18 years ago. I am a bit puzzled why those tools were not used).

There is of course the case of changes in Ilwis-Objects themselves that forces some things to be changed in the scripting language.

• Where ILWIS 3 used to have one file format, Ilwis-Objects will have no file format and as many data formats as the connectors support. What will be the output format of the final script products? Basically anything in the script (in Ilwis-Objects) exists  as a logical view on the data and not as a physical view. There is only a physical view when some permanent storage is needed. This requires a format.
• The iterators over the data sets support all kinds of sub-setting. It would be a shame if this is not reflected in the script. What do I mean by that? The Ilwis-Object script supports, for example, a construct such as “map1=map2[20,25,440,330] + map3” , which only would add the rectangle  indicated to the values of map3. I will probably extend this to a polygon notation but you get the idea.

And there are some changes I myself find necessary.

• Functions. Though ILWIS 3 has some function objects (I wonder how many people use them), the new script will support a more traditional function implementation.
• Variables. A more general implementation of the notion of variable is necessary. It is now somewhat limited to ILWIS data objects. This makes it less useful.
• In the current mapcalc, maps need to have the same georeference to be usuable in a number of operations. I probably will lift this limitation. Of course you could always resample by hand, but I probably will do it automatically when needed.

Apart from my wish to improve the current scripting, I also needed a tool to describe the process flow when I need to hand craft a custom operation. I looked at the Orfeo library to see how they do it. They have a quite usable system of building chains of basic operations (they call them filters, a bit confusing for an ILWIS user 🙂 ) in c++. After the chain has been built, they do some execute (‘update’ they call it) on it that realizes the whole chain. As such cool, but you need to program in c++. I quickly realized that what they are basically doing is very similar to running a script. If I added some command and control functionality to ILWIS script I could basically do the same, but on a more accessible level. So I added them.

One last point. I probably will remove the notion of dependent maps. It will certainly be removed in its current form and might be resurrected in an alternate form. In its current form it complicates the code in a significant way (and you know my preference for the KISS principle). For some years I wondered why dependent maps were not implemented as specialized scripts, which is what they basically are. It would have simplified the code in a major way. But they weren’t, so now is my chance :). I am uncertain how many people actually use dependencies. From anecdotal evidence I know that some people hate them, some people use them and some don’t care. So I will probably skip it while I have an easy backup available. Let me know if you feel differently.

The next blog will not be about the the next generation of ILWIS but on current developments for ILWIS 3.8.4. This version is still sometime in the future (guesstimate end of August/ beginning of September, summer holidays always postpone things), but significant work has already been done on the Orfeo side, as well as some other stuff.