I have a scripting engine I just published as an open source project. It’s been sitting on my harddrive waiting for about a year. My engine of course isn’t complete in any way, but it does work for simple scripts. It has a javascript-ish feel to it, but I don’t wish to abide by the ECMA spec or anything.
Now, the big thing I’m working on is improving code quality while leaving the language working as it is(which I have a few regression tests to “prove”). It doesn’t have a formal grammar at all and works like so:
- Preprocess/Tokenize. At this point it removes whitespace and cuts everything into “tokens”, which is basically just a structure containing a string and a rough “hint” as to what the token is (Number, Identifier, Operation, etc) and some debugging info such as line number
- A ScriptingEngine class which takes the list of tokens and actually parses them and executes them
- An “ExpressionEvaluator” class which will take a subset of the tokens list and build a specific tree of operations, values, and then execute operations and such and collapse the tree down into a single value
My engine has the goals of being portable(works everywhere .Net does) and self-contained. So far, this “works”, but the code is terrible and I’m pretty sure that I’m going about it the wrong way.
I’m wondering if a formal grammar and everything that goes with it might help
Some benefits I’ve heard of being more formal with grammar
- Unambiguous specification of the language
- Easier to maintain/change
- More traditional/Bigger community support?
And some of the disadvantages
- Some languages can be very difficult to reduce to a formal grammar, ie Perl.
- A learning curve for someone not in the know(ie, me)
- Generally rely on tools such as yacc and ANTLR, which introduce another step in your workflow and/or add dependencies(which I’d like to avoid)
Although this project is in .Net, it could equally apply to any other implementing language. Should I use a formal grammar? Can someone expand on the pros/cons of both sides?
3
First of all, a formal grammar doesn’t imply you have to use a parser generator. You can write a recursive descent parser or whatever floats your boat.
Formal grammars are good as documentation. They help programmers decide what foreign code means and tells you what is allowed / mandatory in the language.
For example, in javascript, do I always have to wrap a function literal in parens if I want to call it right away? That is, is this allowed:
function(a1,a2) { ... }(42, "bacon").
Formal grammars are also very helpful when you want to
- Make another implementation of the language,
- Do code generation targeting the language,
- Or for doing static analysis of the language.
Of course, if it’s a quick and dirty scripting language these points are less important.