Which programming idiom is easier to use for beginner developers writing concrete file parsing classes?
I’m developing an open source library, which one of the main functionality is to parse plain text files and get structured information from them. All of the files contain the same kind of information, but can be in different formats like XML, plain text (each of them is structured differently), etc. There are a common set of information pieces which is the same in all (e.g. player names, table names, some id numbers)
There are formats which are very similar to each other, so it’s possible to define a common Base class for them to facilitate concrete format parser implementations. So I can clearly define base classes like SplittablePlainTextFormat, XMLFormat, SeparateSummaryFormat, etc. Each of them hints the kind of structure they aim to parse. All of the concrete classes should have the same information pieces, no matter what.
To be useful at all, this library needs to define at least 30-40 of these parsers. A couple of them are more important than others (obviously the more popular formats).
Now my question is, which is the best programming idiom to choose to facilitate the development of these concrete classes? Let me explain:
I think imperative programming is easy to follow even for beginners, because the flow is fixed, the statements just come one after another. Right now, I have this:
class SplittableBaseFormat:
def parse(self):
"Parses the body of the hand history, but first parse header if not yet parsed."
if not self.header_parsed:
self.parse_header()
self._parse_table()
self._parse_players()
self._parse_button()
self._parse_hero()
self._parse_preflop()
self._parse_street('flop')
self._parse_street('turn')
self._parse_street('river')
self._parse_showdown()
self._parse_pot()
self._parse_board()
self._parse_winners()
self._parse_extra()
self.parsed = True
So the concrete parser need to define these methods in order in any way they want. Easy to follow, but takes longer to implement each individual concrete parser.
So what about declarative? In this case Base classes (like SplittableFormat and XMLFormat) would do the heavy lifting based on regex and line/node number declarations in the concrete class, and concrete classes have no code at all, just line numbers and regexes, maybe other kind of rules.
Like this:
class SplittableFormat:
def parse_table():
"Parses TABLE_REGEX and get information"
# set attributes here
def parse_players():
"parses PLAYER_REGEX and get information"
# set attributes here
class SpecificFormat1(SplittableFormat):
TABLE_REGEX = re.compile('^(?P<table_name>.*) other info d* etc')
TABLE_LINE = 1
PLAYER_REGEX = re.compile('^Player d: (?P<player_name>.*) has (.*) in chips.')
PLAYER_LINE = 16
class SpecificFormat2(SplittableFormat):
TABLE_REGEX = re.compile(r'^Tournament #(d*) (?P<table_name>.*) other info2 d* etc')
TABLE_LINE = 2
PLAYER_REGEX = re.compile(r'^Seat d: (?P<player_name>.*) has a stack of (d*)')
PLAYER_LINE = 14
So if I want to make it possible for non-developers to write these classes the way to go seems to be the declarative way, however, I’m almost certain I can’t eliminate the declarations of regexes, which clearly needs (senior :D) programmers, so should I care about this at all? Do you think it matters to choose one over another or doesn’t matter at all? Maybe if somebody wants to work on this project, they will, if not, no matter which idiom I choose. Can I “convert” non-programmers to help developing these? What are your observations?
Other considerations:
Imperative will allow any kind of work; there is a simple flow, which they can follow but inside that, they can do whatever they want.
It would be harder to force a common interface with imperative because of this arbitrary implementations.
Declarative will be much more rigid, which is a bad thing, because formats might change over time without any notice.
Declarative will be harder for me to develop and takes longer time. Imperative is already ready to release.
I’m looking for answers about which idioms to use when, which is better for open source projects with different scenarios, which is better for wide range of developer skills.
TL; DR:
- Parsing different file formats (plain text, XML)
- They contains same kind of information
- Target audience: non-developers, beginners
- Regex probably cannot be avoided
- 30-40 concrete parser classes needed
- Facilitate coding these concrete classes
- Which idiom is better?
2
I think you can go the declarative minilanguage route, but you’re unlikely to find one that works for all those formats.
Consider grouping your formats. For example:
- for parsing XML, or anything similarly structured, using anything other than an XML parser is probably asking for trouble. You might be able to provide a general XML-to-whatever engine that less technical users can configure, maybe by just specifying which tags they want mapped to which fields.
-
for parsing text, full regular expressions look like overkill for your examples – you could handle those with scanf. You could probably provide a translation from something like Python format strings to regex, while still exposing REs directly to users who want them
eg. convert the slightly more user-friendly
'Player {player_num:d}: {player_name:s} has {player_chips:d} in chips.'to
'Player (?P<player_num>d+): (?P<player_name>S+) has (?<player_chips>d+) in chips.'NB. the original string isn’t much simpler than the RE, although there’s a bit less noise, but there are many fewer opportunities for a non-technical user to confuse themselves than with an RE
-
for parsing CSV, fixed-width or binary formats, some general parser can probably be configured with delimiter, field width, or type+size info respectively. Then we’re back to specifying the mapping from columns to fields, similar to the XML.
Since none of these have anything much in common, trying to fit them all into a single DSL is hard. It’ll end up -like regular expressions – being too general for a beginner to learn easily.
The downside is, it will be hard to provide any uniformity or consistency between formats in different groups. Still, each individual group should be much easier to learn and support.
You should look into parser combinator libraries, which when set up properly, can give you declarative syntax that’s easier to read than either your recursive descent example or your regex example. Done correctly, it sort of looks like concatenating a string, but it works in reverse. With some work on your part, it might look something like:
parser = "Player" + playerNumber + ":" + playerName + "has" + chipCount + "in chips"
That being said, if you truly want non-programmers to contribute, you’re going to want to create some sort of guided graphical extractor. Think of the wizard you get when you open a text file in Excel, or the way AdBlock lets you choose an element on the page to block. The code does some autodetect on the basic structure, the user answers some questions to get the columns looking right, labels the different fields, then the code highlights all the fields and the user confirms. This system would be difficult to create, but will make maintenance of changes to the formats very nice.
1