From d61c6aebcb78a496edf9a02242e7e700f0423da7 Mon Sep 17 00:00:00 2001 From: FAQ Bot Date: Mon, 5 Oct 2026 15:04:40 +0000 Subject: [PATCH 1/2] NEW: Is there a faster alternative to manually encoding categorical columns t --- ...-categorical-encoding-performance-warning.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 _questions/machine-learning-zoomcamp/module-3/033_359acb3362_faster-categorical-encoding-performance-warning.md diff --git a/_questions/machine-learning-zoomcamp/module-3/033_359acb3362_faster-categorical-encoding-performance-warning.md b/_questions/machine-learning-zoomcamp/module-3/033_359acb3362_faster-categorical-encoding-performance-warning.md new file mode 100644 index 00000000..746f872b --- /dev/null +++ b/_questions/machine-learning-zoomcamp/module-3/033_359acb3362_faster-categorical-encoding-performance-warning.md @@ -0,0 +1,17 @@ +--- +id: 359acb3362 +question: Is there a faster alternative to manually encoding categorical columns that + triggers a pandas performance warning when repeatedly adding many columns? +sort_order: 33 +--- + +That warning usually appears when you build the encoded DataFrame by repeatedly writing new columns into an existing `DataFrame` inside loops, e.g. `df[col] = ...` for many unique categories. Each assignment can force pandas to re-handle a larger frame, which is slow and memory-heavy. + +Better options: +1) Use `DictVectorizer` (recommended in this course for classification feature engineering). Instead of manually creating one column per category, convert each row to a dictionary of categorical features and let scikit-learn do the one-hot expansion efficiently. + +2) Use `pandas.get_dummies` (one-hot encoding) on the categorical columns in a single operation, then concatenate once with your numeric columns. + +Key idea: avoid per-category/per-loop assignments like `df["%s" % j] = ...`. Build the encoded representation first (with `DictVectorizer` or `get_dummies`), then concatenate to `df_num` one time. + +(Your approach idea—splitting object vs numeric columns, encoding the object part, and `pd.concat` once—matches this key idea. The main performance bottleneck is the repeated column assignment inside the nested loops, not the final merge.) \ No newline at end of file From 4de6663306490094cbc07a404961076d0c1d944c Mon Sep 17 00:00:00 2001 From: Alexey Grigorev Date: Wed, 14 Oct 2026 07:17:17 +0200 Subject: [PATCH 2/2] Rewrite PerformanceWarning one-hot encoding FAQ --- ...ategorical-encoding-performance-warning.md | 30 ++++++++++++++----- 1 file changed, 22 insertions(+), 8 deletions(-) diff --git a/_questions/machine-learning-zoomcamp/module-3/033_359acb3362_faster-categorical-encoding-performance-warning.md b/_questions/machine-learning-zoomcamp/module-3/033_359acb3362_faster-categorical-encoding-performance-warning.md index 746f872b..77863667 100644 --- a/_questions/machine-learning-zoomcamp/module-3/033_359acb3362_faster-categorical-encoding-performance-warning.md +++ b/_questions/machine-learning-zoomcamp/module-3/033_359acb3362_faster-categorical-encoding-performance-warning.md @@ -1,17 +1,31 @@ --- id: 359acb3362 -question: Is there a faster alternative to manually encoding categorical columns that - triggers a pandas performance warning when repeatedly adding many columns? +question: 'PerformanceWarning: DataFrame is highly fragmented when one-hot encoding + categorical columns in a loop' sort_order: 33 --- -That warning usually appears when you build the encoded DataFrame by repeatedly writing new columns into an existing `DataFrame` inside loops, e.g. `df[col] = ...` for many unique categories. Each assignment can force pandas to re-handle a larger frame, which is slow and memory-heavy. +pandas raises `PerformanceWarning: DataFrame is highly fragmented` when you add many columns one by one, for example `df[name] = (df[col] == value).astype(int)` inside nested loops. Each insert adds a separate internal block. -Better options: -1) Use `DictVectorizer` (recommended in this course for classification feature engineering). Instead of manually creating one column per category, convert each row to a dictionary of categorical features and let scikit-learn do the one-hot expansion efficiently. +Don't build one-hot columns by hand. Encode all categorical columns in one call, as in Module 3: -2) Use `pandas.get_dummies` (one-hot encoding) on the categorical columns in a single operation, then concatenate once with your numeric columns. +```python +from sklearn.feature_extraction import DictVectorizer -Key idea: avoid per-category/per-loop assignments like `df["%s" % j] = ...`. Build the encoded representation first (with `DictVectorizer` or `get_dummies`), then concatenate to `df_num` one time. +dv = DictVectorizer(sparse=False) +train_dicts = df_train[categorical + numerical].to_dict(orient='records') +X_train = dv.fit_transform(train_dicts) +``` -(Your approach idea—splitting object vs numeric columns, encoding the object part, and `pd.concat` once—matches this key idea. The main performance bottleneck is the repeated column assignment inside the nested loops, not the final merge.) \ No newline at end of file +`pd.get_dummies(df, columns=categorical)` also works in a single call. + +If you do need a loop, collect the new columns in a dict and concatenate once: + +```python +new_cols = {} +for col in categorical: + for value in df[col].unique(): + new_cols[f'{col}_{value}'] = (df[col] == value).astype(int) + +df = pd.concat([df, pd.DataFrame(new_cols)], axis=1) +```