2025, Oct 19 06:00

Why Headless Selenium in Chrome Breaks on Element Lookup and How a Desktop User-Agent Unblocks It

Headless Selenium in Chrome may hit Access Denied and fail on element IDs. Learn to detect blocked DOMs and fix headless runs by setting a desktop User-Agent.

Headless Selenium often behaves differently from a regular browser window. A common symptom: the same script that works in a visible Chrome session fails in headless mode at the very first element lookup. Below is a practical case where locating an input by ID succeeds with a visible browser but throws an error in headless Chrome, and how to make both modes behave consistently.

Working example in a visible browser

The following snippet opens the FAA runway safety diagrams page, enters a code in the input with id=ident, submits the form, scans the results table, and returns the first link matching the expected domain.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
import webbrowser
def locate_faa_diagram_url(icao):
    browser = webdriver.Chrome()
    browser.get("https://www.faa.gov/airports/runway_safety/diagrams/")
    search_box = browser.find_element(By.ID, "ident")
    search_box.send_keys(icao)
    search_box.send_keys(Keys.RETURN)
    grid = browser.find_element(By.XPATH, "//table")
    anchors = grid.find_elements(By.TAG_NAME, "a")
    for a in anchors:
        link_url = a.get_attribute("href")
        if link_url:
            if "aeronav.faa" in link_url:
                print(link_url)
                return link_url
webbrowser.open(locate_faa_diagram_url("ATL"))

The same flow in headless mode, but it errors

Switching to headless mode, the script fails on locating the input field by its ID. Minimizing the window or setting a window size does not address the failure.

from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
from selenium import webdriver
import webbrowser
def locate_faa_diagram_url_headless(icao):
    opts = Options()
    opts.add_argument("--headless")
    opts.add_argument("--window-size=1920x1080")
    browser = webdriver.Chrome(options=opts)
    browser.get("https://www.faa.gov/airports/runway_safety/diagrams/")
    search_box = browser.find_element(By.ID, "ident")  # errors here in headless
    search_box.send_keys(Keys.RETURN)
    grid = browser.find_element(By.XPATH, "//table")
    anchors = grid.find_elements(By.TAG_NAME, "a")
    for a in anchors:
        link_url = a.get_attribute("href")
        if link_url:
            if "aeronav.faa" in link_url:
                print(link_url)
                return link_url
webbrowser.open(locate_faa_diagram_url_headless("ATL"))

The keystroke that enters the airport code also belongs in this flow, but its absence is not the cause of the element lookup failure. The failure happens earlier.

What’s actually going on

In headless mode the site blocks access and serves an Access Denied page instead of the expected content. The driver attempts to find an element by ID in a different document, so the lookup fails.

<html>
 <head>
  <title>Access Denied</title>
 </head>
 <body>
  <h1>Access Denied</h1>
  You don't have permission to access "http://www.faa.gov/airports/runway_safety/diagrams/" on this server.
  <p>Reference #18.56633b8.1754209447.14b9f2c8</p>
  <p>https://errors.edgesuite.net/18.56633b8.1754209447.14b9f2c8</p>
 </body>
</html>

This discrepancy stems from how headless Chrome identifies itself and is handled by anti-bot/CDN layers. Headless mode typically sends a slightly different User-Agent string and defaults (like viewport), and some infrastructures react by blocking or serving reduced markup. Passing a mainstream User-Agent minimizes the differences between headed and headless runs.

How to verify you’re blocked

Dump the page source right after navigation and inspect what you actually received. If you see an Access Denied document instead of the expected page, element lookups will fail because the DOM is not the same.

from bs4 import BeautifulSoup
# place this immediately after browser.get(...)
dom = BeautifulSoup(browser.page_source)
with open("faa_dump.html", "w", encoding="utf-8") as fh:
    fh.writelines(dom.prettify())

The fix: set a User-Agent in headless Chrome

Add a User-Agent string to Chrome options. This aligns headless with a typical desktop Chrome and allows the page to load as it does in a visible session.

from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
from selenium import webdriver
import webbrowser
def fetch_diagram_link_headless(icao):
    opts = Options()
    opts.add_argument("--headless")
    opts.add_argument("--window-size=1920x1080")
    opts.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36")
    browser = webdriver.Chrome(options=opts)
    browser.get("https://www.faa.gov/airports/runway_safety/diagrams/")
    search_box = browser.find_element(By.ID, "ident")
    search_box.send_keys(icao)
    search_box.send_keys(Keys.RETURN)
    grid = browser.find_element(By.XPATH, "//table")
    anchors = grid.find_elements(By.TAG_NAME, "a")
    for a in anchors:
        link_url = a.get_attribute("href")
        if link_url:
            if "aeronav.faa" in link_url:
                print(link_url)
                return link_url
webbrowser.open(fetch_diagram_link_headless("ATL"))

If you rely on a specific viewport, be aware that standard window sizing switches do not always apply in modern headless Chrome. An alternative flag can be used to define screen metrics explicitly: --screen-info={0,0 1920x1080}.

Why this matters

Headless automation is foundational for CI pipelines and server-side scraping. When a site detects or treats headless sessions differently, your tests or data collection silently diverge from real-world behavior. Aligning the User-Agent reduces those discrepancies and prevents brittle scripts that pass locally but fail in automation. Verifying the actual DOM you received guards against chasing phantom selector issues when the real problem is upstream access control.

Takeaways

If a script works in a visible browser but fails in headless mode on the same selectors, confirm the actual HTML you received. If the site serves an Access Denied document or stripped markup, adjust the headless session to look like a normal browser by setting a mainstream User-Agent. Keep the functional flow intact, including entering values before submitting forms, and avoid relying solely on window-size switches in headless mode; when necessary, define screen metrics explicitly. This keeps your headless runs predictable and closer to real-user behavior.

The article is based on a question from StackOverflow by RJTTAZ and an answer by Ajeet Verma.

headless-browser python selenium-webdriver