Stop Using Python Lists and Start Using These Instead

Have you ever found yourself waiting far too long for a Python script to process your data? If you’ve relied on Python lists for heavy number crunching, you’ve likely hit some performance roadblocks. Rather than using lists to process your data, consider using NumPy arrays—a faster, more efficient way to handle large datasets—and their secret weapon: vectorization.

What Are NumPy Arrays?

At their core, NumPy arrays are like Python lists, but better suited for numerical computations. While Python lists can store mixed data types, NumPy arrays store data in a consistent, tightly packed format. This allows them to:

1. Consume Less Memory: Arrays are more memory-efficient compared to lists.
2. Process Data Faster: Operations on arrays are implemented in C, making them lightning-fast.
3. Enable Advanced Operations: Arrays allow you to perform mathematical operations on entire datasets with minimal code.

Let’s compare how a list of lists and a NumPy array handle data.

Processing a List of Lists

Imagine summing the elements in a list found within another list. As you can see below, each sublist_sum requires an iteration in a for loop, one step at a time:

my_numbers = [[6, 5], [1, 3], 
              [5, 6], [1, 4], 
              [3, 7], [5, 8], 
              [3, 5], [8, 4]]
result = []
for sublist in my_numbers:
    sublist_sum = sublist[0] + sublist[1]
    result.append(sublist_sum)
print(result)
# Output: [11, 4, 11, 5, 10, 13, 8, 12]

While this approach does work, it’s not very efficient—especially with large datasets.

Processing a NumPy Array

Using a NumPy array, you can sum all elements in a single line:

import numpy as np

arr = np.array([[6, 5], [1, 3], 
                [5, 6], [1, 4], 
                [3, 7], [5, 8], 
                [3, 5], [8, 4]])
result = np.sum(arr, axis=1)
print(result)
# Output: [11  4 11  5 10 13  8 12]

No for loop, no fuss! And the best part? It’s significantly faster.

The Vectorization Advantage

The concept of vectorization is best understood by comparing how Python lists and NumPy arrays handle operations. We’ll use the coding examples from above to demonstrate.

Summation with Python Lists

In Python lists, summation involves iterating through each sublist and manually computing the total for each. The animation below shows how each number is processed one step at a time, with significant overhead introduced by Python loops:

Summation with NumPy Arrays

With NumPy, summation is performed as an optimized batch operation. The animation below illustrates how NumPy processes chunks of the operation at once, leveraging optimized backend implementations:

By avoiding iteration and using efficient underlying libraries, NumPy achieves both simplicity and speed. This comparison shows us the power of vectorization, making NumPy arrays indispensable for large-scale computations.

A Quick Performance Comparison

To see the actual difference, let’s benchmark summing 10 million numbers using both methods:

Python Lists:

import time

numbers = list(range(10_000_000))
start = time.time()
sum(numbers)
stop = time.time()
print("Time taken:", stop - start)
# Time taken: 0.044 seconds

NumPy Arrays:

arr = np.arange(10_000_000)
start = time.time()
np.sum(arr)
stop = time.time()
print("Time taken:", stop - start)
# Time taken: 0.0039 seconds

NumPy arrays are often an order of magnitude faster than Python lists for numerical computations due to their optimized C implementation and the elimination of Python loops. The specific times may vary based on your system's hardware and environment, but the NumPy version will consistently outperform the Python list version.

Real-World Applications of NumPy Arrays

NumPy arrays and vectorization are indispensable in fields like:

• Data Analysis: Quickly compute summary statistics or apply transformations.
• Machine Learning: Efficiently manipulate feature matrices.
• Scientific Simulations: Model complex systems with large datasets.

For example, imagine you’re analyzing sensor data with millions of numerical readings. Using NumPy arrays, you can clean, transform, and analyze the data in seconds rather than minutes.

Explore More with NumPy

NumPy arrays and vectorization are just the beginning. You’ll learn this and much more in our Introduction to NumPy lesson. If you want a more comprehensive learning experience, consider enrolling in the Junior Data Analyst Career Path to master these essential tools and more.

Happy coding, and keep experimenting!