How to parse HTML with Regular Expressions
I’ll be using JavaScript here. With it we can write this in <100 lines of code.
Harness
We’ll start by laying out our code. I’ll be using closures, but you could — and my first version did — use a class instead. We’ll need a parse_html function, inside we’ll put a helper function called pull and a parse_content function which we’ll fill in later.
If you’d like to follow along, the finished code is here: https://github.com/evan-brass/regex-html
The cursor will be a reference to a node in our DOM tree. Tag nodes will have a tag property for their name, an attributes map, and a children array. Our root node just has a children array so that it looks like a tag to the parse_content function, but then we’ll just return the children array because we can have multiple top level nodes in HTML.
Tags
The first regular expression we’ll write will parse opening HTML tags. Open tags look something like this <tag-name> so we’ll use /^<([a-zA-Z][a-zA-Z0-9\-]*)>/. Of course, we’ll also need to match closing tags, which are the same but with a slash: /^\/([a-zA-Z][a-zA-Z0-9\-]*)>/. Since we’re matching on the beginning of the input, we need the ‘^’ to anchor to the start of the string.
Since our pull function returns a boolean, we can use short circuit evaluation to chain calls to pull and stop when one consumes some input. It’s kinda like choice in parser combinators.
One complication of using this pull function is not being able to break out of the loop, which is why there’s this silly run variable.
This gets us past our first test which doesn’t have any text nodes: <p><b></b><i></i></p><div><span></span></div>
Text Nodes
Let’s add text nodes next. For simplicity, we’ll assume that ‘<’ characters don’t exist in text.
This gets us past our second test which looks like this:
The parsed tree for that test looks like this:
Attributes
While we could probably extend our open tag regular expression to accommodate attributes, that might require a variable number of captures so I’d rather put it in its own function. We’ll need to adjust our open tag handler to not require a closing ‘>’:
And then we’ll write a parse_attributes function. (Please excuse the wacky while loop):
Comment Nodes
We won’t go into parsing script tags, style tags, or cdata, but let’s add comments real quick. We’ll put another regular expression between open tags and closing tags like so:
Void Tags
The last thing we’ll do for our little parser is to correctly handle void tags. Tags like img and source cannot have children and can’t have a closing tag. For them we’ll just skip parsing content. I found a list of void tags here that we’ll use:
Conclusion
Congratulations! You just built a recursive descent parser for a simplified version of HTML!
The moral of this story is that parsing non-regular languages is a pretty small step once you understand regular languages. The first time I heard “You can’t parse HTML with regular expressions” I thought I must need some completely different thing. I think there’s value in a tutorial that is a minimal extension on top of regular expressions and HTML is not syntactically complex.
I wouldn’t recommend using this parser in anything real, but for a little project it might work. I’d recommend adding helpful error messages at the very least.
