Stop Fighting Regex: Clean XML in Python by Mutating the BeautifulSoup Tree and Removing Unwanted Nodes

Learn how to clean XML in Python: remove unwanted nodes by mutating the BeautifulSoup parse tree, avoid brittle regex, and serialize once for reliable results.

Cleaning XML by removing unwanted nodes seems trivial until state management gets in the way. A common approach is to parse the document, iterate matched elements, and apply regex replacements to a string buffer. That mix of techniques is exactly where things go sideways: you iterate over a parsed tree but mutate a raw string, so your loop appears to "overwrite" changes instead of accumulating them. The reliable approach is to operate on the parsed XML tree itself.

Problem setup

Below is a minimal example that tries to drop every EmbeddedImage except the one with Name="Brand_1" by repeatedly applying a wildcard regex to the XML string.

import re
from bs4 import BeautifulSoup as Soup
raw_doc = '''
 <Images>
    <EmbeddedImage Name="Brand_1">
      <ImageData>/9j/4AAQSkZJB//2Q==</ImageData>
    </EmbeddedImage>
        <EmbeddedImage Name="Brand__2XX">
              <ImageData>/9j/4AAQSkZJB//2Q==JB//2</ImageData>
            </EmbeddedImage>
      <EmbeddedImage Name="Brand___3XX">
      <ImageData>/9j/4AAQSkZAAQSkkB//2Q=AAQSk=</ImageData>
    </EmbeddedImage>
 <Images>'''
parsed = Soup(raw_doc, 'xml')
out_text = raw_doc
for node in parsed.select('EmbeddedImage'):
    if node.get('Name') != 'Brand_1':
        expr = '<EmbeddedImage Name="' + node.get('Name') + '">.*</EmbeddedImage>'
        out_text = re.sub(expr, '', out_text, flags=re.DOTALL)
        print('updated with pattern:', expr)
print(out_text)

What actually goes wrong

The loop enumerates elements from the parsed structure, but each removal happens on a separate string variable. The parse tree is never updated, so the iteration context and the mutation target are out of sync. That is why the behavior feels like changes are being overwritten rather than accumulated. The fix follows directly: perform deletions on the parsed XML itself and serialize once at the end.

It is also worth noting that manipulating XML with a wildcard regex is inherently brittle. Pattern-based deletion with a greedy wildcard can swallow more than intended, especially across nested tags. Parsing first and then mutating the tree avoids that class of problems entirely.

Solution: mutate the parse tree

Work with BeautifulSoup’s tree directly and drop nodes you do not want. The method decompose removes a tag from the tree along with its contents, which is exactly what we need here.

from bs4 import BeautifulSoup as Soup
xml_doc = '''
<Images>
    <EmbeddedImage Name="Brand_1">
      <ImageData>/9j/4AAQSkZJB//2Q==</ImageData>
    </EmbeddedImage>
    <EmbeddedImage Name="Brand__2XX">
      <ImageData>/9j/4AAQSkZJB//2Q==JB//2</ImageData>
    </EmbeddedImage>
    <EmbeddedImage Name="Brand___3XX">
      <ImageData>/9j/4AAQSkZAAQSkkB//2Q=AAQSk=</ImageData>
    </EmbeddedImage>
</Images>
'''
tree = Soup(xml_doc, 'xml')
for tag in tree.find_all('EmbeddedImage'):
    if tag.get('Name') != 'Brand_1':
        tag.decompose()
print(tree.prettify())

Why this matters

When cleaning XML, consistency of state is everything. Iterating one representation while mutating another invites subtle bugs and unexpected output. By modifying the parsed document, you maintain a single source of truth and get deterministic results. The final serialization will contain only the nodes that survived the in-memory transformation—in this case, the single EmbeddedImage with Name="Brand_1".

There are other ways to express the same intent. For example, selecting EmbeddedImage[@Name='Brand_1'] is a concise XPath or XQuery expression. But if Python is the environment of choice, mutating the BeautifulSoup tree is simple and robust.

Takeaways

If you need to keep only one EmbeddedImage (Brand_1) and drop the rest, operate on the DOM-like structure produced by your parser and use decompose to remove unwanted nodes. Avoid regex-based wildcard deletion on raw XML strings; it decouples iteration from mutation and easily leads to confusing outcomes. Parse first, transform in place, serialize once.