Fix Selenium driver.get stuck on New Tab: avoid Chrome Default profile and the DevTools non-default data directory error

Selenium 4.25 + Chrome 136 can leave driver.get on New Tab with the Default profile. Use an isolated user data directory and skip --profile-directory option.

Running Selenium against an already-signed-in Chrome profile sounds convenient until the browser refuses to leave chrome://newtab/. If you are on Selenium 4.25.0 with Chrome 136 and point ChromeDriver at your real Default profile, you may see the browser launch with your theme and bookmarks but ignore driver.get(). The critical clue is the console warning about DevTools demanding a non-default data directory.

Reproducing the behavior

The following Python snippet spins up Chrome with an existing Default profile and then tries to navigate to https://aistudio.google.com/prompts/new_chat. Chrome opens, shows the personalized New Tab page, and stalls there.

import time
import os
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.common.exceptions import WebDriverException
DRIVER_BIN = r'C:\path\to\your\chromedriver-win64\chromedriver.exe'
USERDATA_ROOT = r'C:\Users\YourUserName\AppData\Local\Google\Chrome\User Data'
PROFILE_LABEL = "Default"
DEST_URL = "https://aistudio.google.com/prompts/new_chat"
def boot_driver():
    print(f"Driver Path: {DRIVER_BIN}")
    print(f"User Data Dir: {USERDATA_ROOT}")
    print(f"Profile Dir Name: {PROFILE_LABEL}")
    if not os.path.exists(DRIVER_BIN):
        print(f"FATAL: ChromeDriver not found at '{DRIVER_BIN}'.")
        return None
    if not os.path.isdir(USERDATA_ROOT):
        print(f"FATAL: Chrome User Data dir not found at '{USERDATA_ROOT}'.")
        return None
    cfg = Options()
    cfg.add_argument(f"--user-data-dir={USERDATA_ROOT}")
    cfg.add_argument(f"--profile-directory={PROFILE_LABEL}")
    cfg.add_experimental_option("excludeSwitches", ["enable-automation", "load-extension"]) 
    cfg.add_experimental_option("useAutomationExtension", False)
    cfg.add_argument("--start-maximized")
    try:
        svc = Service(executable_path=DRIVER_BIN)
        drv = webdriver.Chrome(service=svc, options=cfg)
        print("WebDriver initialized.")
        return drv
    except WebDriverException as e:
        print(f"FATAL WebDriverException during setup: {e}")
        if "user data directory is already in use" in str(e).lower():
            print(">>> Ensure ALL Chrome instances are closed via Task Manager.")
        return None
    except Exception as other:
        print(f"Unexpected FATAL error during setup: {other}")
        return None
def run():
    print("IMPORTANT: Ensure ALL Google Chrome instances are FULLY CLOSED before running this script.")
    input("Press Enter to confirm and continue...")
    drv = boot_driver()
    if not drv:
        print("Driver setup failed. Exiting.")
        return
    try:
        print("Browser launched. Waiting a few seconds for it to settle...")
        print(f"Initial URL: '{drv.current_url}', Initial Title: '{drv.title}'")
        time.sleep(4)
        print(f"Attempting to navigate to: {DEST_URL}")
        drv.get(DEST_URL)
        print("Called driver.get(). Waiting for navigation...")
        time.sleep(7)
        after_url = drv.current_url
        after_title = drv.title
        print(f"After 7s wait - Current URL: '{after_url}', Title: '{after_title}'")
        if DEST_URL not in after_url:
            print(
                f"NAVIGATION FAILED: Browser did not navigate to '{DEST_URL}'. It's still on '{after_url}'."
            )
        else:
            print(f"NAVIGATION SUCCESSFUL to: {after_url}")
    except Exception as e:
        print(f"An error occurred during main execution: {e}")
    finally:
        print("Script execution finished or errored.")
        input("Browser will remain open for inspection. Press Enter to close...")
        if drv:
            drv.quit()
if __name__ == "__main__":
    if r"C:\path\to\your\chromedriver-win64\chromedriver.exe" == DRIVER_BIN or \
       r"C:\Users\YourUserName\AppData\Local\Google\Chrome\User Data" == USERDATA_ROOT:
        print("ERROR: Default placeholder paths are still in the script.")
        print("Please update DRIVER_BIN and USERDATA_ROOT with your actual system paths.")
    else:
        run()

What is actually blocking navigation

The key signal appears in the console right before navigation stalls.

"DevTools remote debugging requires a non-default data directory"

ChromeDriver (version 113 and newer, as distributed via Chrome for Testing) places guardrails around regular Chrome profiles. When you pass --user-data-dir that points at Chrome’s real Default profile and pair it with --profile-directory, DevTools may not attach fully. As a result, Selenium commands that rely on the DevTools protocol, such as driver.get(), may not take effect and the browser stays on the New Tab page.

Even though the browser looks correctly personalized, that is only the UI level. Under the hood the DevTools session is restricted, which prevents reliable navigation and control.

How to make driver.get() win over the New Tab page

The practical workaround is to avoid the real Default profile and use a dedicated test profile located in its own data directory. The easy path is to copy the Default profile once into a separate folder and point --user-data-dir there, without using --profile-directory. This sidesteps the protections around Default and lets DevTools attach properly.

You can create a profile copy like this on Windows:

mkdir "C:\SeleniumChromeProfile"
xcopy /E /I "%LOCALAPPDATA%\Google\Chrome\User Data\Default" "C:\SeleniumChromeProfile\Default"

Then update the driver options so Chrome uses that directory directly, omitting the profile selector. The rest of the script remains the same.

import time
import os
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.common.exceptions import WebDriverException
DRIVER_BIN = r'C:\path\to\your\chromedriver-win64\chromedriver.exe'
ISOLATED_PROFILE_HOME = r'C:\SeleniumChromeProfile'  # points to the copied profile root
DEST_URL = "https://aistudio.google.com/prompts/new_chat"
def boot_driver():
    print(f"Driver Path: {DRIVER_BIN}")
    print(f"Using isolated user data dir: {ISOLATED_PROFILE_HOME}")
    if not os.path.exists(DRIVER_BIN):
        print(f"FATAL: ChromeDriver not found at '{DRIVER_BIN}'.")
        return None
    if not os.path.isdir(ISOLATED_PROFILE_HOME):
        print(f"FATAL: Isolated Chrome User Data dir not found at '{ISOLATED_PROFILE_HOME}'.")
        return None
    cfg = Options()
    cfg.add_argument(f"--user-data-dir={ISOLATED_PROFILE_HOME}")
    # Intentionally no --profile-directory here
    cfg.add_experimental_option("excludeSwitches", ["enable-automation", "load-extension"]) 
    cfg.add_experimental_option("useAutomationExtension", False)
    cfg.add_argument("--start-maximized")
    try:
        svc = Service(executable_path=DRIVER_BIN)
        drv = webdriver.Chrome(service=svc, options=cfg)
        print("WebDriver initialized.")
        return drv
    except WebDriverException as e:
        print(f"FATAL WebDriverException during setup: {e}")
        if "user data directory is already in use" in str(e).lower():
            print(">>> Ensure ALL Chrome instances are closed via Task Manager.")
        return None
    except Exception as other:
        print(f"Unexpected FATAL error during setup: {other}")
        return None
def run():
    print("IMPORTANT: Ensure ALL Google Chrome instances are FULLY CLOSED before running this script.")
    input("Press Enter to confirm and continue...")
    drv = boot_driver()
    if not drv:
        print("Driver setup failed. Exiting.")
        return
    try:
        print("Browser launched. Waiting a few seconds for it to settle...")
        print(f"Initial URL: '{drv.current_url}', Initial Title: '{drv.title}'")
        time.sleep(4)
        print(f"Attempting to navigate to: {DEST_URL}")
        drv.get(DEST_URL)
        print("Called driver.get(). Waiting for navigation...")
        time.sleep(7)
        after_url = drv.current_url
        after_title = drv.title
        print(f"After 7s wait - Current URL: '{after_url}', Title: '{after_title}'")
        if DEST_URL not in after_url:
            print(
                f"NAVIGATION FAILED: Browser did not navigate to '{DEST_URL}'. It's still on '{after_url}'."
            )
        else:
            print(f"NAVIGATION SUCCESSFUL to: {after_url}")
    except Exception as e:
        print(f"An error occurred during main execution: {e}")
    finally:
        print("Script execution finished or errored.")
        input("Browser will remain open for inspection. Press Enter to close...")
        if drv:
            drv.quit()
if __name__ == "__main__":
    if r"C:\path\to\your\chromedriver-win64\chromedriver.exe" == DRIVER_BIN or \
       r"C:\SeleniumChromeProfile" == ISOLATED_PROFILE_HOME:
        print("ERROR: Default placeholder paths are still in the script.")
        print("Please update DRIVER_BIN and ISOLATED_PROFILE_HOME with your actual system paths.")
    else:
        run()

This adjustment removes the Default profile from the equation. With a dedicated user data directory, driver.get() reliably takes control instead of yielding to the New Tab logic.

Why you want to know this

Automation against everyday browsing profiles is fragile. Modern ChromeDriver builds bundled via Chrome for Testing emphasize stability and security, which means regular profiles are not a safe target for DevTools-driven automation. Using a dedicated profile guarantees that the automation channel can attach cleanly and that core commands like navigation, script execution, and element interaction behave deterministically. In practice, this also isolates your tests from extensions, startup pages, and profile-specific quirks.

In the reported scenario, moving to a copied profile immediately unblocked navigation. That is a strong signal that the profile boundary, not the site or the code timing, was the limiting factor.

Wrap-up

If Chrome opens with your familiar UI but ignores driver.get(), look for the DevTools warning about a non-default data directory. Point ChromeDriver at a dedicated user data dir, ideally a copy of your Default profile placed in a separate folder, and omit --profile-directory. Ensure Chrome is fully closed before starting, keep ChromeDriver aligned with the browser build, and run your automation on that isolated profile going forward. This straightforward change makes navigation work as expected and restores predictable control over the browser.