Meteorite logoMeteorite
Back to videos

Pandas Data Cleaning: Fix Messy Real-World Data

Pandas data cleaning, done properly: one messy form export turned the first of July into the seventh of January without an error. Here is how to catch that and every other silent bug before it reaches your model. Real data lies quietly: numbers stored as text, five spellings of one branch, hidden duplicates and missing values in disguise. In this video you'll learn: - How to find missing values in disguise (dash, question mark, N/A) with na_values and isna - How to fix "could not convert string to float" with to_numeric and errors="coerce", and audit what failed - How to standardise text categories with str.strip, str.lower and map - Why drop_duplicates finds nothing until you normalise, and how subset and keep work - How to parse dates with an explicit format in to_datetime - Drop or fill? Missing completely at random, at random, and not at random, with fillna, ffill and a group median - pandas 3 changes: why fillna with inplace on a column does nothing, and fillna(method=) is gone - How to fill with scikit-learn's SimpleImputer without data leakage Every snippet runs on pandas 3.1 with real output, on a small form export and the real Auto MPG dataset. Who it's for: engineering students from any branch and working professionals learning Python for AI and data science. It assumes the previous lesson, Pandas Basics: DataFrames and Series. Next in this playlist: Lesson 9, Pandas: GroupBy, Merge and Pivot. ⏱️ Chapters 0:00 1 July became 7 January 0:24 Cleaning Messy Real-World Data 0:48 Five kinds of mess 1:17 The cleaning order 1:43 First look 2:14 Text to numbers 2:44 Coerce safely 3:10 Standardise text 3:42 Duplicates 4:10 Normalise first 4:30 Dates 4:58 Impossible values 5:26 Math in words: mean vs median 5:53 Why data goes missing 6:25 Drop or fill? 6:48 Fill by group 7:20 pandas 3 traps 7:52 Leakage rule 8:11 No leakage 8:38 Where cleaning happens at work 9:06 Learning, working, earning 9:34 Recap 9:56 Quiz 10:20 Next lesson 📖 Sources and further reading - pandas user guide: Working with missing data (doc source): https://github.com/pandas-dev/pandas/blob/main/doc/source/user_guide/missing_data.rst - pandas 3.0.0 release notes (removed fillna method=, errors='ignore'): https://github.com/pandas-dev/pandas/blob/main/doc/source/whatsnew/v3.0.0.rst - pandas source: to_datetime and to_numeric docstrings: https://github.com/pandas-dev/pandas/blob/main/pandas/core/tools/datetimes.py - scikit-learn user guide: Imputation of missing values: https://github.com/scikit-learn/scikit-learn/blob/main/doc/modules/impute.rst - Auto MPG (UCI), CSV copy in seaborn-data: https://github.com/mwaskom/seaborn-data/blob/master/mpg.csv ▶️ Watch the whole playlist, in order: https://www.youtube.com/playlist?list=PLSvS9Zafwd8s 📚 All learning paths - AI Foundations: https://www.youtube.com/playlist?list=PLCS9GxefUfTc 🔊 Sound effects - "Deep Cinematic Impact" (trimmed) by MeijstroAudio, licensed CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/): https://freesound.org/people/MeijstroAudio/sounds/377303 - "Keyboard Typing Sounds" (trimmed) by crz1990, licensed CC BY 3.0 (https://creativecommons.org/licenses/by/3.0/): https://freesound.org/people/crz1990/sounds/135872 🔔 Subscribe so the next lesson finds you: https://www.youtube.com/channel/UCR4xqYrDt9VJnsklAkP-lTQ?sub_confirmation=1 #Python #Pandas #DataCleaning #DataScience #MachineLearning

Suggested videos