Reliable Selenium click on Magicbricks: wait for all 'See all Properties' links, then switch tabs

Fix flaky Selenium clicks on Magicbricks. Wait for all three 'See all Properties' links, then switch to the new tab. Uses WebDriverWait and robust locators.

Clicking a seemingly simple link with Selenium can turn into a moving target when the DOM is assembled dynamically. On magicbricks.com, the “See all Properties” link exists early, but the page goes through an intermediate state where this link isn’t fully wired up. A naive wait-and-click works intermittently and fails often enough to break scraping workflows.

Problem recap

The target markup looks like this and opens a new tab when clicked:

<a href="javascript:void(0);" class="mb-home__section__title--anchor-see-all push-right" onclick="fireDynamicPropSeeAllGTM(event,'Owner properties see all','ownerProp');seeAppPropertiesOpenUrl('ownerProp');">
    See all Properties
</a>

A straightforward Selenium attempt waits for visibility using an absolute XPath, triggers a click via JavaScript, then switches to the newly opened tab:

try:
    ui_wait = WebDriverWait(browser, 10)
    link_see_all = ui_wait.until(
        EC.visibility_of_element_located((By.XPATH, "/html/body/div[5]/div[1]/div[10]/div/section/div[1]/a"))
    )

    base_tab = browser.current_window_handle
    browser.execute_script("arguments[0].click();", link_see_all)

    ui_wait.until(EC.number_of_windows_to_be(2))

    for handle in browser.window_handles:
        if handle != base_tab:
            browser.switch_to.window(handle)
            break

except Exception as err:
    print(f"->Error: {err}")

Despite the waits, this approach can throw driver errors. Manually clicking in the browser works reliably and opens a new tab, which hints the issue is timing rather than locators alone.

Why the click fails intermittently

The page is dynamically created and the link appears during an intermediate phase where it isn’t fully set up. A basic wait like waiting for visibility or even element-to-be-clickable can still race with the page finishing its dynamic wiring. Complicating things further, there are multiple “See all Properties” links on the page. Clicking the first one that appears doesn’t guarantee the page is ready.

A minimal clickable wait such as the following can therefore succeed or fail depending on timing:

hold.until(EC.element_to_be_clickable((By.XPATH, "//a[text()='See all Properties']"))).click()

Working approach

The reliable path is to wait for all occurrences of the target link to exist before clicking. On this page there are three links containing the exact text “See all Properties”. Waiting until the count reaches three stabilizes the state and makes the click consistent. Afterward, wait for a second window, switch to it, and proceed with scraping.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait

site_url = 'https://www.magicbricks.com/'
browser = webdriver.Chrome()
browser.maximize_window()
browser.get(site_url)

pause = WebDriverWait(browser, 10)
pause.until(lambda d: len(d.find_elements(By.XPATH, "//a[text()='See all Properties']")) == 3)
browser.find_element(By.XPATH, "//a[text()='See all Properties']").click()

pause.until(EC.number_of_windows_to_be(2))

for win in browser.window_handles:
    if win != browser.current_window_handle:
        browser.switch_to.window(win)
        break

for hdr in pause.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "h2"))):
    print(hdr.text)

This flow clicks the first matching link by visible text after confirming that all three are present. It then waits for the new tab before switching and extracting headings.

Why this matters

Dynamic pages often expose elements before attaching all behaviors or before other dependent components finish loading. If you synchronize on a single element too early, any of the following can happen: the element is visible but not truly clickable; another script is about to replace it; a competing overlay intercepts the click. Waiting for a stable condition tied to the page’s real readiness avoids flakiness.

There are a few additional practical takeaways. A single WebDriverWait instance can be reused throughout the script instead of reinstantiating it. Running without --headless helps observe what actually happens during races. In some environments a native click works, while in others a JavaScript click such as arguments[0].click() behaves differently; for example, it works on Chrome here while Firefox has issues. Keeping a minimal working script makes it easier to test small variations quickly.

Conclusion

When a page is assembled dynamically, synchronize with the page’s real ready state rather than with the first visible match. On magicbricks.com that means waiting until all three “See all Properties” links appear, clicking the intended one, and then switching to the newly opened tab. Reuse a single wait object, prefer stable locators by visible text over brittle absolute XPaths when possible, and validate behavior in a visible browser session before automating headless runs. This combination eliminates the intermittent failures and gives you a dependable path to the listings page for scraping.