How to Leverage Data Wrangler for Automated Data Cleaning in Fabric Notebooks

You can use Data Wrangler in Fabric Notebooks to clean data faster. This tool can save you many hours every week. A survey shared by Forbes says data scientists spend 60% of their time cleaning and sorting data.

With Data Wrangler, you can use simple language to explain what changes you want. You can see each step and watch your progress. You can also export your cleaning steps as Python code. This helps you work faster and repeat your steps easily. Try these features to see how they can help your work.

Key Takeaways

• Data Wrangler lets you clean data quickly with easy steps. It shows every change so you can see what happens. You can load data into Fabric Notebooks without trouble. You can check for missing or repeated values in your data. The tool helps you fix missing data and change columns. You can also remove mistakes without hard coding. Data Wrangler makes Python code for each cleaning step. This makes your work easy to do again and share with others. You can save your cleaned data for later. You can export it as Delta tables or parquet files.

Data Wrangler Overview

What It Does

Data Wrangler helps you clean and get your data ready in Microsoft Fabric Notebooks. When you put your data into a DataFrame, you will see a dropdown menu. This menu shows all the DataFrames you have open. You can pick one and open a screen that looks like a grid. This screen lets you look at your data and see some quick facts about it. You can also spot patterns in your data. You can filter rows, sort columns, and do other changes with just a few clicks.

Here is a table that lists some things you can do with Data Wrangler:

You can look at every change before you make it final. Data Wrangler also writes Python code for each step you take. You can put this code back into your notebook. This makes your work easy to repeat and keeps things simple.

Why Use It

Data Wrangler makes cleaning data quicker and simpler. You do not have to write long code or use different tools. You can watch your changes happen right away and fix mistakes fast. Data Wrangler shows you clear facts and pictures to help you learn about your data.

• Data Wrangler makes it easier to see and understand hard datasets.

• It keeps your data the same and helps stop mistakes.

• You can save time by letting the tool do boring jobs for you.

• The tool works with big datasets and many types of data.

• You can show your steps and results to your team without trouble.

Note: Data Wrangler does a lot for you, but you should still check your results. Sometimes, special data needs you to look at it yourself. Automated tools can miss things or need you to double-check for the best results.

Start in Fabric Notebooks

Load Data

You begin by loading data into your Fabric Notebook. Microsoft Fabric lets you get data from many places. You can use files like CSV or Excel. You can also connect to cloud storage, like Azure Data Lake Storage Gen2. Most people start by uploading CSV files to the Lakehouse. This step matters because Data Wrangler works with pandas or Spark DataFrames.

To load your data, do these steps:

1. Get your data file ready in a format like CSV.

2. Go to your Microsoft Fabric workspace.

3. Make or pick a Lakehouse for your data.

4. Upload your file to the Lakehouse.

5. In your notebook, load the file into a pandas or Spark DataFrame.

Here is a simple example using pandas:

import pandas as pd
df = pd.read_csv('/lakehouse/default/files/my_data.csv')

Tip: If your file is huge, try loading just part of it first. This helps stop memory errors and makes checking your data easier.

Prepare Data Frame

After you load your data, you need to get your DataFrame ready for Data Cleaning. Start by looking at your data. Check for missing values, duplicates, or odd numbers. You can use the Data Wrangler screen to see quick facts and find problems.

Good ways to get your DataFrame ready are:

• Look at your data to find missing values or outliers.

• Write down any problems you notice.

• Fix easy things first, like removing duplicates or fixing dates.

• Try your cleaning steps on a small sample before using them on all your data.

• Make text, dates, and categories the same to keep your data neat.

You should also think about keeping your data safe. Microsoft Fabric uses roles and permissions to protect private data. Data Loss Prevention rules help keep information safe by blocking access until you fix problems.

If you follow these steps, your data will be ready for easy and good Data Cleaning in Fabric Notebooks.

Automate Data Cleaning

Automating Data Cleaning in Fabric Notebooks with Data Wrangler helps you save time and avoid mistakes. You can follow a clear process to clean your data, handle missing values, transform columns, and make your work repeatable. Here is how you can do it step by step.

Cleaning Steps

You can use Data Wrangler’s visual interface to clean your data without writing code. Follow these steps to automate your Data Cleaning tasks:

1. Load your data into a Pandas DataFrame from your Lakehouse file. You can use pd.read_csv() or pd.read_excel().

2. In your Fabric Notebook, open the Data tab and launch Data Wrangler. Select the DataFrame you want to clean.

3. Use the Data Wrangler interface to pick cleaning actions. You can sort columns, filter rows, remove duplicates, and group data.

4. Preview your changes. Data Wrangler shows you a summary and lets you see how your data looks after each step.

5. Review the data quality and profiling information. This helps you spot problems like missing values or outliers.

6. Export the generated Python code for all your cleaning steps back into your Notebook.

7. Edit or add to the code if you need more changes. You can also save the cleaned data back to your Lakehouse.

8. Schedule your Notebook or add the code to a data pipeline. This lets you run your Data Cleaning process automatically whenever you need.

Tip: Data Wrangler uses AI-driven features to suggest cleaning steps. You can reuse these steps for new data, making your process faster each time.

Handle Missing Data

Missing data can cause problems in your analysis. Data Wrangler gives you tools to find and fix missing values. Here is how you can handle missing data:

• Visualize missing data with built-in charts. This helps you see where data is missing.

• Decide how to handle missing values. You can:

  • Fill missing values with a static value like "Unknown" or a number.

  • Use the mean or median of the column to fill gaps.

  • Drop rows or columns if too much data is missing.

  • Use interpolation for time-series data to estimate missing points.

• Preview the results before you apply them. This helps you keep your data accurate.

Data Wrangler uses data profiling to show you summary statistics like mean, median, and standard deviation. You can choose the best method for your data. For example, you might fill missing ages with the average age or drop rows with too many missing fields.

Note: Always check your data after handling missing values. This keeps your dataset reliable and ready for analysis.

Transform Columns

You often need to change columns to prepare your data for analysis. Data Wrangler supports many types of column transformations:

• Change date and time formats, such as turning "01/02/2024" into "2024-02-01".

• Split text columns by a comma or other delimiter.

• Replace text strings, like fixing typos or standardizing names.

• Delete columns you do not need.

• Add new columns with custom formulas or calculations.

• Change data types, such as turning a string into a number.

• Write your own Python formulas for complex changes.

For example, you can standardize state names to USPS codes or create a new column that shows the length of a text field. Data Wrangler shows you a preview and the Python code for each transformation. You can review and adjust these steps before applying them.

Tip: Use the Operations Panel to search for actions like "Fill Missing Values" or "Change Data Type." Preview changes in the data diff view before saving.

Insert Generated Code

One of the best features of Data Wrangler is that it generates Python code for every cleaning step you take. You can insert this code into your Fabric Notebook to make your Data Cleaning process repeatable and shareable.

• Each action you perform in Data Wrangler creates a matching Python code snippet.

• You can export all the code at once and paste it into a new cell in your Notebook.

• The code documents every step, so you and your team can see exactly what changes you made.

• You can edit the code to add more steps or adjust existing ones.

• Saving the code in your Notebook lets you rerun the cleaning process on new data or schedule it to run automatically.

Using generated code in Fabric Notebooks helps you build reliable workflows. You can test, document, and share your Data Cleaning steps with others. This makes your work easier to repeat and improves teamwork.

By following these steps, you can automate Data Cleaning in Fabric Notebooks. You use AI-driven suggestions, reusable functions, and clear code to keep your data clean and ready for analysis.

Save and Export

Save in Notebook

After you finish cleaning your data with Data Wrangler, you need to save your work. You can add the code that Data Wrangler creates right into your Fabric Notebook. When you run this code, it makes a new cleaned dataframe, such as df_clean. You can then save this cleaned data so you do not lose your changes.

To save your cleaned dataframe in Fabric, use Spark’s write method. You can pick the format you want, like "delta", and set the mode to "overwrite" if you want to replace old data. Then, save your data as a table with a name you choose. Here is an example:

df_clean.write.format("delta").mode("overwrite").saveAsTable("Products")

This code saves your cleaned data as a Delta table called "Products". You can find this table in your Lakehouse. Saving your data this way keeps it safe and easy to use for future work.

Tip: Saving as a Delta table lets you use your cleaned data in other notebooks or share it with your team.

Export Data

You may want to export your cleaned data for use outside Fabric Notebooks. The best format for this is parquet. Parquet files work well with Spark and the Fabric Lakehouse. They store data in a way that saves space and makes it easy to search.

Exporting your data as a parquet file also works smoothly with OneLake. OneLake acts as a big storage place for your organization. You can use the file explorer in OneLake, which connects to Windows File Explorer, to see and manage your parquet files. This makes it simple to move, copy, or share your data.

To export your dataframe as a parquet file, use code like this:

df_clean.write.parquet('/lakehouse/default/files/cleaned_data.parquet')

This command saves your cleaned data in the parquet format. You can then use this file in other tools or share it with others. Parquet files also let you save your data as Delta tables later if you want to do more analytics.

Note: Parquet files help you keep your data organized and ready for fast analysis.

---

You can get more done and do better work by letting Data Wrangler automate tasks in Fabric Notebooks. This tool helps you keep your data correct and the same every time. It also saves you a lot of time. Many people think the tool is simple to use. They like that it can export code, which helps them learn. If you want to do even more, you can look at open-source libraries, AI tools, or big data platforms for more ways to automate.

Use these features in your next project to get results that are faster and cleaner.

FAQ

How do you launch Data Wrangler in Fabric Notebooks?

You open your notebook, load your data into a DataFrame, then click the Data tab. You select your DataFrame and choose “Open in Data Wrangler.” The tool opens in a new window for you to start cleaning.

Can you undo a step in Data Wrangler?

Yes, you can undo any step before you export the code. Use the step history panel to remove or change actions. This helps you fix mistakes quickly.

What file formats does Data Wrangler support?

You can work with CSV, Excel, and parquet files. Data Wrangler also supports data from Lakehouse tables. Always check your file format before loading.

How do you reuse cleaning steps for new data?

Export the generated Python code from Data Wrangler. Paste it into your notebook. When you load new data, run the same code. This keeps your cleaning process consistent and saves time.

Is Data Wrangler free to use in Microsoft Fabric?

You can use Data Wrangler as part of your Microsoft Fabric subscription. Some advanced features may need extra licensing. Check your plan for details.