I’m writing a crawler in Python that needs to extract the content of all <div> tags with a specific class name (e.g. “class-name”) from an HTML document. I’ve learned that regular expressions are usually not the best tool for parsing HTML because it can fail due to the complexity and nested structure of HTML. However, in this particular case, the HTML structure is relatively simple and predictable, so I would like to try using regular expressions for this task. I have tried the following code, but it does not seem to work as expected:
My question is:
Is my regular expression correct? If there is an error, how should it be modified to ensure that it only captures the content of<div>tags with the class name “class name”?
If regular expressions are indeed not the best way to handle this situation, can you recommend a more suitable Python library (such as BeautifulSoup) to handle this problem and provide a brief example code?
-
The
re.DOTALLflag is used to make the.character match any character including newlines. -
The
re.IGNORECASEflag is optional but can be useful if you’re not sure about the case sensitivity of the class name. -
This regex assumes that there are no nested
<div>tags with the same class name inside the target<div>tags. Nested tags can break this regex. -
HTML attributes can be in any order, and there can be additional attributes or whitespace, which can make regex solutions fragile.
-
import re html_content = """ <html> <body> <div class="unwanted-class">Don't want this content</div> <div class="class-name">Need this content</div> <div class="class-name">Also need this content</div> </body> </html> """ pattern = r'<div class="class-name">(.*?)</div>' matches = re.findall(pattern, html_content, re.DOTALL) for match in matches: print(match.strip())
shit nige is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.