2025, Dec 22 01:00
Add a Column in pandas DataFrame by Concatenating Current and Previous Rows with shift (No Loops, Adjustable Window)
Learn how to add a column to a pandas DataFrame that concatenates the current row with previous rows using shift—no loops. Includes adjustable rolling window.
When you need to add a column to a pandas DataFrame that collects values from the current row together with values from previous rows, it’s tempting to reach for a manual loop. It works, but in this case pandas already provides a direct path. Below is a compact guide to turning a list-of-lists into a DataFrame and building a new column that concatenates the previous row with the current row, or even a wider window if needed.
Reproducing the setup with a straightforward loop
The following snippet demonstrates a working approach that iterates through the rows and manually constructs the target structure. It produces a list of rows, where the first one ends with an empty list, and each subsequent row ends with the concatenation of the previous and current row values.
import pandas as pds
import numpy as npx
# Source data
base = pds.DataFrame(npx.array([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]]))
# Container for the final rows
assembled = []
# Row-wise construction
for idx in range(len(base)):
if idx == 0:
assembled.append(list(base.iloc[idx]) + [[]])
else:
curr = list(base.iloc[idx])
prev = list(base.iloc[idx - 1])
glued = curr + [prev + curr]
assembled.append(glued)
print(assembled)
What’s really going on
There are two core operations in play. First, each row must be turned into a Python list. Second, the new column should store either an empty list for the first row or the concatenation of the immediately preceding row with the current one. For a two-row combination, this isn’t a rolling aggregation in the strict sense; it can be expressed as a simple shift of the row-wise lists, aligned and concatenated per index.
A concise pandas solution using shift
You can accomplish the same result without explicit loops by converting each row to a list, then shifting those lists by one position and adding them to the current row lists. This yields a DataFrame with an added column that matches the intended output.
import pandas as pds
import numpy as npx
# Create the DataFrame
grid = pds.DataFrame(npx.array([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]]))
# Convert each row to a list
row_lists = grid.apply(lambda s: s.tolist(), axis=1)
# Add concatenation of previous and current row as a new column
grid[3] = (row_lists.shift() + row_lists)
print(grid)
Output:
0 1 2 3
0 1 2 3 NaN
1 4 5 6 [1, 2, 3, 4, 5, 6]
2 7 8 9 [4, 5, 6, 7, 8, 9]
3 10 11 12 [7, 8, 9, 10, 11, 12]
The first row has no previous row to combine with, so the result naturally shows NaN there.
Controlling the window size
If instead of just the previous row you need a larger window, you can shift by multiple offsets and concatenate them into a single list per row. The following example shows how to parameterize the window and build the combined lists by reducing a sequence of shifted Series.
import pandas as pds
import numpy as npx
from functools import reduce
# Example dataset
wide = pds.DataFrame(npx.arange(99).reshape(-1, 3))
# Turn each row into a list
as_lists = wide.apply(lambda s: s.tolist(), axis=1)
# Choose the window size (number of previous rows to include)
win = 3
# Build the concatenation of lists from shifts win..0
wide[3] = reduce(lambda a, b: a + b, [as_lists.shift(i) for i in range(win, -1, -1)])
print(wide)
Output:
0 1 2 3
0 0 1 2 NaN
1 3 4 5 NaN
2 6 7 8 NaN
3 9 10 11 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
4 12 13 14 [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]
5 15 16 17 [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17]
6 18 19 20 [9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]
7 21 22 23 [12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]
8 24 25 26 [15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26]
9 27 28 29 [18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29]
10 30 31 32 [21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32]
11 33 34 35 [24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35]
12 36 37 38 [27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38]
13 39 40 41 [30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41]
14 42 43 44 [33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44]
15 45 46 47 [36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47]
16 48 49 50 [39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50]
17 51 52 53 [42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53]
18 54 55 56 [45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56]
19 57 58 59 [48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59]
20 60 61 62 [51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62]
21 63 64 65 [54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65]
22 66 67 68 [57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68]
23 69 70 71 [60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71]
24 72 73 74 [63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74]
25 75 76 77 [66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77]
26 78 79 80 [69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80]
27 81 82 83 [72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83]
28 84 85 86 [75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86]
29 87 88 89 [78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89]
30 90 91 92 [81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92]
31 93 94 95 [84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95]
32 96 97 98 [87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98]
Why this is worth knowing
The requirement is to keep the result as a DataFrame with an added column. Expressing the transformation with shift does exactly that while staying close to how the data are aligned in pandas. It also makes the window size explicit and easy to adjust. As shown by the outputs, positions without enough history hold NaN until the window can be fully formed.
Takeaways
If the objective is to append a column that merges the current row with values from prior rows, starting with two rows, use row-wise lists and shift. For larger windows, shift multiple times and concatenate the resulting Series in the desired order. This keeps the logic clear, produces a DataFrame with the extra column, and matches the expected shape and semantics demonstrated above.