Fixing MultiLabelBinarizer errors in pandas: preserve index to prevent NaN from misaligned concat when one-hot encoding multi-valued columns

Fix pandas MultiLabelBinarizer TypeError: float object is not iterable. Index misalignment from concat injects NaN; preserving index fixes one-hot encoding.

One-hot encoding multi-valued fields in the Stack Overflow 2024 survey sounds straightforward until it suddenly breaks on a perfectly similar column. A common scenario: encoding Employment works, encoding LanguageAdmired throws TypeError: float object is not iterable. The trap is not in MultiLabelBinarizer itself, but in how pandas aligns rows when you concatenate encoded frames after filtering and deduplication.

Problem reproduction

The dataset contains semicolon-separated values in Employment and LanguageAdmired. The goal is to split these into lists and apply MultiLabelBinarizer. The following snippet encodes Employment, concatenates the result, and then tries to encode LanguageAdmired, where the error appears.

import pandas as pd
from sklearn.preprocessing import MultiLabelBinarizer
src = 'survey_results_public.csv'
data = pd.read_csv(src)
data.drop('ResponseId', axis=1, inplace=True)
data = data[~data.duplicated(keep='first')].copy()
data['LanguageAdmired'] = data['LanguageAdmired'].fillna('Other')
data['LanguageAdmired'] = data['LanguageAdmired'].str.split(';')
data['Employment'] = data['Employment'].str.split(';')
enc_emp = MultiLabelBinarizer()
emp_mat = enc_emp.fit_transform(data['Employment'])
emp_ohe = pd.DataFrame(emp_mat, columns=['Employment_' + v for v in enc_emp.classes_])
data = pd.concat([data, emp_ohe], axis=1).copy()
enc_lang = MultiLabelBinarizer()
lang_mat = enc_lang.fit_transform(data['LanguageAdmired'])  # TypeError appears here

Why it fails

Checking data.shape before and after concatenation reveals that new rows appear. The sequence drop followed by removing duplicates changes the original index. When the first encoded frame is created without passing the original index, it gets a fresh RangeIndex. Concatenating this with the filtered data makes pandas align by index, not by row order. Any indices that don’t match produce new rows with NaN in the rest of the columns, including LanguageAdmired. The next MultiLabelBinarizer call receives NaN in place of a list, and since NaN is a float in pandas, the transformer raises TypeError: float object is not iterable.

There is a second observation that matches this: removing the first concatenation step altogether makes the workflow succeed, because there is no misaligned join in the middle of the pipeline that would introduce NaN into LanguageAdmired.

The fix

The solution is to preserve the original index when you build the encoded DataFrames. That way, concat aligns rows correctly and does not create extra rows or inject NaN into subsequent inputs.

import pandas as pd
from sklearn.preprocessing import MultiLabelBinarizer
src = 'survey_results_public.csv'
data = pd.read_csv(src)
data.drop('ResponseId', axis=1, inplace=True)
data = data[~data.duplicated(keep='first')].copy()
data['LanguageAdmired'] = data['LanguageAdmired'].fillna('Other')
data['LanguageAdmired'] = data['LanguageAdmired'].str.split(';')
data['Employment'] = data['Employment'].str.split(';')
enc_emp = MultiLabelBinarizer()
emp_mat = enc_emp.fit_transform(data['Employment'])
emp_ohe = pd.DataFrame(
    emp_mat,
    columns=['Employment_' + v for v in enc_emp.classes_],
    index=data.index,
)
enc_lang = MultiLabelBinarizer()
lang_mat = enc_lang.fit_transform(data['LanguageAdmired'])
lang_ohe = pd.DataFrame(
    lang_mat,
    columns=['LanguageAdmired_' + v for v in enc_lang.classes_],
    index=data.index,
)
data = pd.concat([data, emp_ohe, lang_ohe], axis=1)

With the index preserved for both encoded frames, concatenation does not add extra rows and LanguageAdmired remains a list-valued column for every existing row, so MultiLabelBinarizer operates without errors.

Why this matters

pandas aligns on index by design. Any time rows are dropped, filtered, or deduplicated, the index can become non-consecutive or shift relative to newly created frames. If you then concatenate frames that use a fresh RangeIndex, pandas will happily generate new rows, which often surface later as unexpected NaN and type issues. In this workflow, a single misaligned concat converted list inputs into floats for some rows, surfacing in the transformer as a TypeError.

It’s also useful to verify the pipeline with quick shape checks before and after concat and to use print debugging to inspect what exactly lives in a column before passing it to a transformer. Simple prints of type, length, and head can quickly reveal where NaN or unexpected types appear.

Takeaways

When one-hot encoding multi-valued columns with MultiLabelBinarizer, keep row alignment under control. If you’ve filtered or deduplicated a DataFrame, preserve the original index when you convert encoded arrays into DataFrames. Concatenate after both encodings are ready or ensure every intermediate DataFrame carries index=data.index. If a similar error shows up, first compare shapes across steps and look for NaN introduced by misaligned concat—fixing the index usually resolves it cleanly.