How to Detect Row-Wise Duplicate Values (>=1) Across Columns in Pandas, with a Fast NumPy Alternative

Learn a concise, vectorized method to detect row-wise duplicate values >=1 across columns in pandas, plus a fast NumPy approach for financial time series.

Detecting repeated values within a single row sounds trivial until you introduce business rules like “ignore everything below 1” and “match only when the same number occurs at least twice.” In financial time series that span many columns per timestamp, this quickly becomes a row-wise deduping problem across selected columns. Below is a concise way to solve it in pandas, plus a NumPy variant that scales well.

Sample dataset and target

We’re looking across columns B to I for each timestamp. A row qualifies if there are at least two identical values and those values are not 0 and are not below 1.

                A       B       C       D       E       F       G       H       I
5/7/2025 21:00  0   0   0   0   0   0   0   0
5/7/2025 21:15  0   0   19598.8 0   19598.8 0   0   0
5/7/2025 21:30  0   0   0   0   0   0   0   0
5/7/2025 21:45  0   0   0   19823.35    0   0   0   0
5/7/2025 22:00  0   0   0   0   0   0   0   0
5/7/2025 22:15  0   0   0   0   0   0   0   0
5/7/2025 22:30  0   0   0   19975.95    0   19975.95    0   19975.95
5/7/2025 23:45  0   0   0   0   0   0   0   0
5/8/2025 1:00   0   0   19830.2 0   0   0   0   0
5/8/2025 1:15   0   0   0   0   0   0   0   0
5/8/2025 1:30   0   0   0   0   0   0   0   0
5/8/2025 1:45   0   0   0   0   0   0   0   0

The expected rows are those where a value appears at least twice with value ≥ 1. For the sample above, that’s:

                A       B       C       D       E       F       G       H       I
5/7/2025 21:15  0   0   19598.8 0   19598.8 0   0   0
5/7/2025 22:30  0   0   0   19975.95    0   19975.95    0   19975.95

Why this is tricky

Typical duplicate detection is column-wise or across the entire DataFrame. Here the requirement is row-wise across a subset of columns, combined with a threshold that excludes 0 and anything less than 1. This means you can’t just use a global duplicated() or drop_duplicates(); you need per-row logic across B:I.

Solution in pandas

The idea is to slice the columns of interest, work row-wise, ignore values below 1, and check whether any value repeats within that row. One practical way is to transpose the slice so rows become columns, mark duplicates with duplicated(keep=False), mask values below the threshold, then reduce with any() to get the qualifying rows.

import pandas as pd

# assume df_main is your DataFrame and columns A..I exist
# 1) take B..I and transpose so we can apply duplicated per "original row"
col_block = df_main.loc[:, 'B':].T

# 2) mark duplicates within each original row, ignore values < 1, then reduce to a boolean index
row_selector = col_block.apply(lambda s: s.duplicated(keep=False)).where(col_block >= 1).any()

# 3) filter original DataFrame
filtered_rows = df_main[row_selector]

This returns the rows where at least two values in B..I are the same and are ≥ 1.

Alternative with NumPy

A compact NumPy approach masks values below 1, sorts values within each row, and uses diff + isclose to detect equal neighbors after sorting. The boolean result then selects the corresponding rows.

import numpy as np

# 1) select B..I and mask values below 1
vals = df_main.loc[:, 'B':].where(lambda d: d >= 1).values

# 2) sort values within each row so equal numbers become adjacent
vals.sort()

# 3) diff adjacent values and detect zeros (or near-zeros) with isclose
hit = np.isclose(np.diff(vals), 0).any(axis=1)

# 4) filter
filtered_rows_np = df_main[hit]

Using isclose is helpful for floating-point values where exact equality might be brittle.

What you get as output

For the sample data, the result is the two rows where values repeat at least twice with value ≥ 1:

                A  B  C        D         E        F         G  H         I
1  5/7/2025 21:15  0  0  19598.8      0.00  19598.8      0.00  0      0.00
6  5/7/2025 22:30  0  0      0.0  19975.95      0.0  19975.95  0  19975.95

Why this matters

Row-wise duplicate detection under constraints is a common pattern in financial and telemetry data quality checks. It helps surface repeated fills, mirrored values, or copied entries that pass type checks but violate domain expectations. Doing this purely with vectorized operations keeps the workflow efficient and easier to reason about than iterative scanning.

Takeaways

When you need to find rows with repeated values across multiple columns, define the slice explicitly, apply the threshold to filter out irrelevant numbers, and compute duplicates within each row. In pandas, duplicated with a transpose and boolean indexing gives a concise solution. With NumPy, masking, sorting, and diff plus isclose provides a tight, often faster alternative for large blocks of numeric data. Keep the threshold logic close to the selection to avoid accidental matches on zeros or sub-threshold values, and prefer vectorized operations to stay performant and maintainable.