Stop Chaining gt().lt() in Pandas: Correct Range Filtering with & and Series.between()

Learn why chaining gt().lt() in pandas compares booleans to numbers and flips results. See correct range checks using & and Series.between(), with examples.

Chaining comparison helpers in pandas looks convenient until it suddenly returns the opposite of what you expect. A common trap is calling .gt().lt() in sequence to filter a Series to a range, only to get results that match values on the wrong side of the boundary. Let’s unpack why this happens and show the reliable way to express range checks.

Reproducing the issue

Consider a numeric column with values between 0.00 and 1.00. The intent is to select values greater than 0.7 and less than 0.9. Direct comparisons with & and operators like .gt() and .lt() work fine, but chaining .gt().lt() leads to a surprising outcome.

import pandas as pd
payload = {"rate": [0.5, 0.8, 1.0]}
frame = pd.DataFrame(payload)
# First step: greater than 0.7
mask_gt = frame["rate"].gt(0.7)
print(mask_gt)
# Chained: .gt(0.7).lt(0.9)
mask_chained = frame["rate"].gt(0.7).lt(0.9)
print(mask_chained)

The first mask correctly yields False, True, True. The chained version, however, produces True, False, False, effectively selecting values less than or equal to 0.7 instead of those between 0.7 and 0.9. Reversing the order to .lt(0.9).gt(0.7) flips the effect and selects values less than or equal to 0.9.

What’s really going on

The key lies in how the second comparison is applied. In Python, True == 1 and False == 0. After the first step, .gt(0.7) returns a boolean Series. Chaining .lt(0.9) compares those booleans to 0.9, not the original numbers. Since False is 0 and True is 1, the check becomes 0 < 0.9 (True) and 1 < 0.9 (False), which explains the inverted-looking result.

For clarity: s.gt(0.7).lt(0.9) doesn’t mean “items in s that are both greater than 0.7 and less than 0.9”, it means “compute s.gt(0.7), then take that result and pass it as r.lt(0.9)”.

Correct approach

To express a range condition, use a conjunction of two comparisons or rely on a dedicated helper that understands bounds. Both options below keep the comparison on the original numeric values.

The explicit conjunction reads cleanly and does exactly what it says:

valid_range = frame["rate"].gt(0.7) & frame["rate"].lt(0.9)
print(valid_range)

There’s also a concise method purpose-built for this use case. Series.between lets you specify the lower and upper bounds and choose inclusivity. To exclude both endpoints, set inclusive='neither'.

inside_bounds = frame["rate"].between(0.7, 0.9, inclusive="neither")
print(inside_bounds)

Why this matters

Data filtering is foundational in analytics and feature engineering. A subtle misuse of boolean operations can silently distort a dataset, especially when working with thresholds and confidence intervals. Understanding that a chained comparison applies to the result of the previous step prevents logic bugs that are hard to spot by eye.

Takeaways

If the goal is a range filter, don’t chain .gt().lt() expecting “a < x < b”. Either combine comparisons with & so they operate on the original values or use Series.between with the desired inclusivity. Keeping comparisons anchored to the numeric Series avoids accidentally comparing booleans to numbers and ensures the mask reflects your intent.