Polars: Past, Present and Future, with Polars Creator Ritchie Vink

Because of it's stunningly fast speed, Polars is an extremely popular open-source library for DataFrame operations in Python. Kinda unreal to have Ritchie Vink, Polars' creator, as today's guest!

Ritchie:

• Is CEO and Co-Founder of Polars, Inc., a startup that has raised $4m in seed funding to support his Polars open-source project.

• Previously worked as an ML Engineer, Data Scientist and Data Engineer at companies like adidas and KLM Royal Dutch Airlines.

• Holds a Master’s in Structural Engineering and worked as a civil engineer prior to catching the data-science bug.

Today’s episode will appeal most to hands-on practitioners like data scientists and ML engineers. In it, Ritchie details:

• How Polars regularly achieves 5-20x (sometimes 100x!) speed improvements over Pandas for most DataFrame operations.

• The Eager and Lazy execution APIs Polars offers and when you should use one or the other.

• Ritchie's vision for scaling Polars to handle massive distributed datasets.

• How we can continue to make data-processing efficiency gains even as Moore's Law slows down.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

DataFrame Operations 100x Faster than Pandas, with Marco Gorelli

Today's episode is all about Polars — the hot library for Python that offers up to 100x speedups for DataFrame operations relative to pandas. Marco Gorelli, a core Polars developer, is our gifted guide.

Marco is a tremendously talented communicator of complex technical topics, making him the perfect guest for this highly technical episode. He:

• Is a core developer of the popular Python libraries pandas and Polars.

• Is the creator of the Narwhals library.

• Has spoken at several major Python conferences (such as PyData), taught Polars professionally, and wrote the first complete Polars plugins tutorial.

• Currently works as Senior Software Engineer at Quansight Labs.

• Previously, worked as a data scientist and was one of the prize winners (from amongst >100,000 entrants!) of the M6 forecasting competition.

• Holds a Master’s in Mathematics and the Foundations of Computer Science from the University of Oxford.

Today’s episode will appeal primarily to hands-on technical folks like data scientists, ML engineers and software developers.

In today’s episode, Marco details:

• What the hot, fast-growing Polars library for working with DataFrames in Python is (it already has 65m downloads and 28k GitHub stars).

• How Polars offers up to 100x speed-ups relative to Pandas on DataFrame operations.

• How the lightweight, dependency-free Narwhals package he created allows for easy compatibility between different DataFrame libraries such as Polars and Pandas.

• How he got addicted to open-source development.

• The simple trick he used to be a prize-winner in super-popular forecasting competitions.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.