2023-03-10

Progress Bars with TQDM in Python

What is TQDM

TQDM is a popular Python library that provides an easy way to add progress bars to your code. It is a simple and efficient way to monitor the progress of your code's execution and estimate the remaining time to completion. This can be especially useful when working with large datasets or when running long and complex computations.

Installing TQDM

Installing TQDM is very straightforward. You can install it using pip, the Python package manager, by running the following command:

bash

$ pip install tqdm

Alternatively, you can install it using conda, the Anaconda package manager, by running the following command:

bash

$ conda install tqdm

Once installed, you can import the tqdm module into your Python script or notebook and start using it to add progress bars to your code. TQDM provides a simple and intuitive API that makes it easy to add progress bars to loops, iterators, and other iterable objects.

Basic Usage of TQDM

We will explore some basic usage examples of TQDM, including how to use it with iterables, loops, and how to customize the appearance of the progress bar. We will also explore how to use nested progress bars with TQDM.

Using TQDM with Iterables

One of the most common use cases for TQDM is to add progress bars to iterable objects, such as lists, tuples, and ranges. This can be done by passing the iterable object to the tqdm function, as shown in the following example:

python

from tqdm import tqdm

my_list = [1, 2, 3, 4, 5]

for item in tqdm(my_list): # do something with item

This will create a progress bar that tracks the progress of the loop through the elements of the list. The progress bar will be updated in real-time, and it will display the current progress, estimated remaining time, and the speed of the loop's execution.

Using TQDM with Loops

In addition to iterable objects, TQDM can also be used with regular loops. This can be done by wrapping the loop in a tqdm function call, as shown in the following example:

python

from tqdm import tqdm

for i in tqdm(range(100)):
    # do something with i

This will create a progress bar that tracks the progress of the loop through the range of values from 0 to 99. The progress bar will be updated in real-time, and it will display the current progress, estimated remaining time, and the speed of the loop's execution.

Customizing the TQDM Bar

TQDM provides several options for customizing the appearance of the progress bar. These options include changing the color of the progress bar, setting the width of the progress bar, and displaying custom messages. The following example demonstrates how to customize the appearance of the progress bar:

python

from tqdm import tqdm

my_list = [1, 2, 3, 4, 5]

for item in tqdm(my_list, desc="Processing items", bar_format="{l_bar}{bar}| {n_fmt}/{total_fmt} [{elapsed}<{remaining}, {rate_fmt}]"):
    # do something with item

In this example, we pass a custom format string to the bar_format parameter. This format string defines the layout of the progress bar and includes placeholders for various progress bar parameters, such as the current progress, estimated remaining time, and the loop's execution speed.

Nested Progress Bars with TQDM

TQDM also supports nested progress bars, which can be useful when working with complex computations that involve multiple levels of iteration. The following example demonstrates how to use nested progress bars with TQDM:

python

from tqdm import tqdm

outer_list = [1, 2, 3, 4, 5]
inner_list = [10, 20, 30, 40, 50]

for outer_item in tqdm(outer_list, desc="Outer loop"):
    for inner_item in tqdm(inner_list, desc="Inner loop", leave=False):
        # do something with outer_item and inner_item

In this example, we have two nested loops, and we use TQDM to create progress bars for both of them. We also set the leave parameter to False for the inner loop, which causes the progress bar for the outer loop to stay visible while the inner loop is running. This can be helpful in keeping track of the overall progress of the computation.

Advanced Usage of TQDM

We will explore some advanced usage examples of TQDM, including how to use it with Pandas, Multiprocessing, Jupyter Notebooks, and AsyncIO.

Using TQDM with Pandas

Pandas is a popular Python library for data analysis, and TQDM can be used to add progress bars to Pandas operations that involve large datasets. The following example demonstrates how to use TQDM with Pandas:

python

import pandas as pd
from tqdm import tqdm

df = pd.read_csv("my_large_dataset.csv")

for index, row in tqdm(df.iterrows(), total=len(df)):
    # do something with row

In this example, we use the Pandas DataFrame.iterrows() method to iterate over the rows of the DataFrame, and we use TQDM to create a progress bar that tracks the progress of the iteration. The total parameter is set to the length of the DataFrame, which ensures that the progress bar accurately reflects the progress of the iteration.

Using TQDM with Multiprocessing

Multiprocessing is a Python module that allows you to run multiple processes concurrently. TQDM can be used to add progress bars to multiprocessing operations, which can be helpful in keeping track of the progress of complex computations. The following example demonstrates how to use TQDM with Multiprocessing:

python

import multiprocessing as mp
from tqdm import tqdm

def process_item(item):
    # do something with item

pool = mp.Pool(processes=4)

results = []

for item in tqdm(pool.imap(process_item, my_list), total=len(my_list)):
    results.append(item)

In this example, we create a multiprocessing pool with four processes, and we use the pool.imap() method to apply the process_item() function to each item in my_list. We use TQDM to create a progress bar that tracks the progress of the multiprocessing operation.

Using TQDM with Jupyter Notebooks

Jupyter Notebooks are a popular tool for data analysis and scientific computing in Python. TQDM can be used to add progress bars to Jupyter Notebooks, which can be helpful in keeping track of the progress of long-running computations. The following example demonstrates how to use TQDM with Jupyter Notebooks:

python

from tqdm.notebook import tqdm

for i in tqdm(range(100)):
    # do something with i

In this example, we use the tqdm.notebook module to create a progress bar that is compatible with Jupyter Notebooks. The progress bar will be displayed inline with the notebook output, and it will update in real-time as the loop progresses.

Using TQDM with AsyncIO

AsyncIO is a Python module that allows you to write asynchronous code using coroutines and event loops. TQDM can be used to add progress bars to AsyncIO operations, which can be helpful in keeping track of the progress of asynchronous computations. The following example demonstrates how to use TQDM with AsyncIO:

python

import asyncio
from tqdm.asyncio import tqdm

async def process_item(item):
    # do something with item

async def main():
    tasks = [asyncio.create_task(process_item(item)) for item in my_list]
    for task in tqdm(asyncio.as_completed(tasks), total=len(tasks)):
        await task

await main()

In this example, we create an AsyncIO coroutine called process_item() that performs some asynchronous computation on an item. We use the asyncio.as_completed() function to create a list of coroutines that we want to execute, and we use TQDM to create a progress bar that tracks the progress of the coroutines as they complete. The total parameter is set to the length of the tasks list, which ensures that the progress bar accurately reflects the progress of the asynchronous computation.