How to Stabilize PyAutoGUI Game UI Automation: Use Cropped Templates, Constrained Regions, and Higher Confidence

Fix PyAutoGUI misclicks in game UI. Crop templates, limit the search region, and use OpenCV confidence to get faster, more accurate, repeatable clicks.

Automating a game UI with image search can be deceptively tricky. A common pitfall appears when multiple on-screen elements share nearly identical size and shape. In Cornerpond, a pink quests button and an orange shop button are both 34×34 and sit in fixed positions. With pyautogui.locateCenterOnScreen at around 0.85 confidence, the match may intermittently land on the wrong one, especially when the background (biome) changes. There are no matches at 0.9, and locateAllOnScreen reveals that not only the pink target but also the orange and even a green equipment icon are flagged as candidates in the same region.

Reproducing the issue

The following snippet searches a bottom-right region and clicks the first match. At 0.85 confidence it may click the orange shop button instead of the intended pink quests icon.

import pyautogui
import win32gui

def seek_icon():
    hwnd = win32gui.FindWindow(None, 'Cornerpond')
    hit_point = pyautogui.locateCenterOnScreen(r'C:\path\image.png', region=(1532, 718, 388, 322), confidence=0.85)
    if hit_point:
        print(hit_point)
        pyautogui.click(hit_point)

seek_icon()

To understand what is being detected, inspecting all matches shows several rectangles clustered around the buttons, all sized 34×34, with tiny coordinate variance of about ±1 pixel.

import pyautogui

spots = list(pyautogui.locateAllOnScreen(r'C:\path\image.png', confidence=0.85))

for box in spots:
    print(box)

What’s really going on

The template image used for matching contained the button with a black border and bits of background in its corners. That extra context is the culprit. When the biome/background behind the UI changes, the overall patch becomes less unique, and visually similar buttons of the same size can meet the similarity threshold. The result is that one biome steers the match to the orange shop icon, while another steers it to the intended pink quests icon, despite the buttons themselves not moving.

The fix: make the template unique

Cropping the template to the button content and removing the border and any background made the target sufficiently distinctive. With the trimmed template, matching worked correctly and even allowed a stricter threshold such as confidence=0.99.

Here is a compact way to verify results against a screenshot. It uses locateAll to collect matches and Pillow to draw outlines around them for inspection.

import pyautogui
from PIL import Image, ImageDraw

print("pyautogui:", pyautogui.__version__)
print("OpenCV:", pyautogui.pyscreeze._useOpenCV)

snapshot_file = "screen.png"
# template_file = "button.png"       # with border
template_file = "button-min.png"     # cropped: no border, no background

hits = pyautogui.locateAll(template_file, snapshot_file, confidence=0.85)

canvas = Image.open(snapshot_file)
pen = ImageDraw.Draw(canvas)

for rect in hits:
    x, y, w, h = rect
    pen.rectangle((x, y, x + w, y + h), outline="lime", width=2)

canvas.save("output.png")
canvas.show()

To use the improved template at runtime, keep your search region and raise the confidence now that the template is tighter.

import pyautogui
import win32gui

def click_cropped_template():
    hwnd = win32gui.FindWindow(None, 'Cornerpond')
    center_xy = pyautogui.locateCenterOnScreen('button-min.png', region=(1532, 718, 388, 322), confidence=0.99)
    if center_xy:
        print(center_xy)
        pyautogui.click(center_xy)

click_cropped_template()

Why this matters

GUI automation hinges on consistent, reproducible recognition. Templates that include borders and background pick up noise from the environment, and small UI similarities can cause false positives. A clean, minimal template reduces ambiguity, allows higher confidence, and stabilizes behavior across visual variations like changing biomes.

Practical guidance and takeaways

Keep template images as tight as possible: remove borders and strip any background so the visual signal is unique to the control you intend to click. Validate matches by enumerating them with locateAll and, if needed, draw rectangles on a saved screenshot to confirm what is being detected. If the target never moves, cache or focus on a smaller region rather than scanning a large area; shrinking the search window, for example to something close to the button’s bounding box, lowers the chance of spurious hits. The pink quests button’s distinctive color also makes pixel-level search a viable tactic; pyautogui includes functions that can look up pixels by color. Finally, confidence control depends on OpenCV being available, and with a properly cropped template you can push the threshold higher, which further reduces accidental matches.

Clean templates, constrained regions, and careful verification will give you predictable, fast, and accurate automation in scenarios where multiple UI elements look alike.