If one would to write auto-suggest for an IDE/code editor (like IntelliSense), does one generally need to write a parser for any language that should be supported, or do compilers/runtimes provide these things?
Neither the PHP or Python binaries seem to support just parsing files and generating some kind of raw output.
Eclipse seems to roll their own parser, but is this the only way to do it? Using the “native” parser would be awesome on so many levels 🙂
4
PHP has token_get_all to parse a file and generate the list of tokens you can use to work with code, like doing static analysis, checking the style or implementing an auto-complete feature.
Python has Abstract Syntax Trees (AST), which seems even more helpful, and has also some neat features like the compile function which compiles the tree itself.
In general, it’s a very bad idea to implement your own parser (aside from learning). It’s hugely difficult and error prone and will become invalid as soon as the language specification changes. Such changes are unusual with well-designed languages like C#, but are not so unusual in languages like PHP which have lots of flaws and missing features (example: the recent implementation of namespaces in PHP). Also, by reinventing your own parser, you’re reinventing the wheel already invented for a compiler: instead of doing exclusively your work (the auto-complete feature), you spend time writing lots of code related to parsing, the code you have to test and maintain later.
Some hints
You may be interested in a term of “compiler as a service”. For example, Microsoft is working on a compiler as a service for C#, which will let you to extract programmatically the information from the compiler; such scenario might be helpful for an auto-complete feature.
You may also search for static checkers for the language of your interest. Many are open source, so looking at how they process the code source may give you some suggestions about the parsing.
Finally, some compilers are themselves open sourced. Depending on the license they use and the one you will use for your auto-complete-enabled product, you may be unable to reuse the code, but still it can give some hints too.
2