LLM Benchmarks Are Lying to You (And What to Do Instead), with Sinan Ozdemir

Sensational episode for you today with the illustrious A.I. author, educator and entrepreneur Sinan Ozdemir on how LLM benchmarks are lying to you... and what you can do about it.

Sinan:

• Is Founder and CTO of LoopGenius, a generative A.I. startup.

• Authored several excellent books, including, most recently, the bestselling "Quick Start Guide to Large Language Models".

• Hosts the "Practically Intelligent" podcast.

• Was previously adjunct faculty at The Johns Hopkins University, now teaches several times a month within the O'Reilly platform.

• Serial A.I. entrepreneur, including founding a Y Combinator-backed generative A.I. startup way back in 2015 that was later acquired.

• Holds a Master’s in Pure Math from Johns Hopkins.

Today’s episode skews slightly toward our more technical listeners but Sinan excels at explaining complex concepts in a clear way so today’s episode may appeal to any listener of this podcast.

In today’s episode, Sinan details:

Why the A.I. benchmarks everyone relies on might be lying to you.

• How the leading A.I. labs are gaming the benchmark system.

• Tricks to actually effectively evaluate LLMs’ capabilities for your use cases.

• What the future of benchmarking will involve, including how to benchmark agentic and multimodal models.

• How a simple question about watermelon seeds reveals the 40% failure rate of even today’s most advanced A.I. models.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

In Case You Missed It in June 2025

We had exceptional guests on my podcast in June. In today's "In Case You Missed It" episode, hear the best parts of all my June convos. Here's a quick summary to tantalize you:

1. Strategy consultant Diane Hare with five tricks for gaining buy-in on A.I. transformation in your organization.

2. Renowned data-career educator Avery Smith on the two portfolio projects every aspiring data analyst should build.

3. SuperDataScience founder Kirill Eremenko on what you're missing if you're struggling to land an A.I. job.

4. San Fran-based venture capitalist Shaun Johnson on the traits that make a great A.I. startup founder.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

Automating Legal Work with Data-Centric ML (feat. Lilith Bat-Leah)

Today, exceptional communicator Lilith Bat-Leah explains why "Data-Centric ML Research" trumps our typical focus on model capability, with examples from her extensive Legal A.I. background.

Lilith:

• Has over a decade of experience specializing in the application of ML to legal tech.

• Is Senior Director of A.I. Labs at Epiq, a leading LegalTech firm that has over 6000 employees.

• Has published work on evaluation methods for the use of ML in legal discovery as well as on Data-centric ML Research (DMLR).

• Is co-chair of the DMLR working group MLCommons and has organized DMLR workshops at [ICML] Int'l Conference on Machine Learning and ICLR, two of the most important A.I. conferences.

• Holds a degree from Northwestern University, in which she focused on statistics.

Today’s episode will appeal primarily to hands-on practitioners like data scientists, AI/ML engineers and software developers.

In today’s episode, Lilith details:

How A.I. is revolutionizing the legal industry by automating up to 80% of traditional discovery processes.

• Why 'elusion' is a critical metric that only exists in LegalTech — and what it reveals about machine learning evaluation.

• The surprising reason why we should stop obsessing over model improvements and focus on something that takes up 80% data scientists’ time instead.

• How she grew from being a temp receptionist to an A.I. lab director by falling in love with statistics.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

95-Year-Old Annie on How to Stay Healthy and Happy

Our 900th Episode! It almost goes by as quickly as 95 years! By popular demand, my grandmother Annie returns to the podcast with wisdom on staying happy and healthy at any age.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

Landing $200k+ AI Roles: Real Cases from the SuperDataScience Community, with Kirill Eremenko

As we approach episode #900, the original SuperDataScience Podcast host Kirill Eremenko returns to reflect on what leads to the highest-paying opportunities in AI. This is a special one; enjoy!

Many of you will already know Kirill:

• Founder and CEO of SuperDataScience.com, the eponymous e-learning platform.

• Founded the SuperDataScience Podcast nine years ago and hosted the show until he passed me the reins five years ago.

• With over 3 million students, he’s the most popular data science and A.I. instructor on Udemy.

• He holds a Master’s from The University of Queensland in Australia and a Bachelor’s in Applied Physics and Mathematics from the Moscow Institute of Physics and Technology.

Today’s episode is ideal for anyone looking to advance their data science or A.I. career — or looking to break into a career in this field for the first time.

In today’s episode, Kirill details:

• Why employers are still testing A.I. engineers on basic machine learning fundamentals — even for LLM-focused roles.

• The surprising reason why staying in data science (as opposed to developing an A.I. specialization) could be the right career move for you.

• How one developer discovered the hidden age bias in tech recruiting — and the simple hack to beat it.

• The two critical skill areas that separate amateur A.I. engineers from the pros commanding huge salaries.

• Why the "back to office" movement could give you a competitive advantage in landing a top A.I. role.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

How to Enable Enterprise AI Transformation, with Strategy Consultant Diane Hare

People, not technical capability, are holding back A.I.'s impact in organizations. In today's episode, Diane Hare explains how to overcome friction and enable strategic A.I. transformation.

Diane:

• Founder and CEO of the New York-based strategic consulting firm BizLove, which has been mobilizing key stakeholders to deliver on enterprise-wide priorities (like A.I. initiatives!) at Fortune 100 companies for seven years.

• Prior to her seven years leading BizLove, spent seven years at EY, the global professional services giant (they have nearly 400,000 employees) formerly known as Ernst & Young.

• Board Member at NANO Nuclear Energy Inc. (NASDAQ: NNE)

• Holds and MBA and was captain of a semi-professional women’s soccer team in New York City!

Today’s episode is well-suited to anyone looking to make an impact with A.I. and automation, which I suspect is about every listener to my podcast!

In today’s episode, Diane details:

• Why people, not technical capability, are holding back A.I.’s transformative power in organizations.

• How to prioritize the items on an enterprise A.I. roadmap.

• Why storytelling is essential for gaining buy-in from stakeholders on an A.I. initiative.

• Her top five tips for enabling A.I. transformation.

This was a super-cool episode for me because Diane's consultancy, BizLove, is a formal partner of my own consultancy, Y Carrot 🥕. While Y Carrot brings rich technical expertise on A.I. (from development through to production deployment), BizLove naturally complements us with their deep experience enabling digital and A.I. transformations of enterprises. Together, we offer every service organizations need to make lasting, impactful improvements with A.I.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

The Future of Enterprise AI: Investor Shaun Johnson Reveals What Actually Works

What are the biggest opportunities for A.I. startups? Find out in today's episode with the trailblazing venture capitalist Shaun Johnson, including tricks for gaining enterprise A.I. adoption.

Shaun:

• Co-founder and general partner at AIX Ventures in San Francisco, where he’s led deals into companies including Perplexity, Chroma, and Workhelix.

• He is a former VP of Engineering, Product and Design at Lilt; and a former VP of Product and Design at NimbleRx.

• Holds a Master’s in Electrical Engineering from Stanford University and an MBA from the University of California, Berkeley.

Today’s episode is well-suited to any listener to this podcast. In it, Shaun details:

How having investment partners like Richard Socher and Christopher Manning, who are practitioners actively building at the cutting edge of A.I., gives AIX Ventures an edge.

• What it takes to become one of the few thousand people in the world pushing the A.I. frontier.

• The surprising strategy that makes enterprise A.I. adoption 10x easier.

• Why some A.I. startups are better off building in 'red oceans' full of competition rather than seeking blue-ocean opportunities.

• The reason big tech companies are buying A.I. talent without acquiring the actual startups.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

In Case You Missed It in May 2025

We had stellar guests in May, including one episode that had the most positive social-media response of any episode ever. In today's "In Case You Missed It" episode, hear the best parts of all my May convos.

The specific conversation highlights included in today's episode are:

1. John Roese, Dell Technologies' global CTO and Chief A.I. Officer, on the biggest A.I. opportunities for enterprises in the coming months/years. (This is the episode that received an unprecedented social-media response.)

2. The authors of the brand-new O'Reilly book "Python Polars: The Definitive Guide", Jeroen Janssens and Thijs Nieuwdorp, on a real-world Polars success story.

3. Space engineer and entrepreneur Mary Spio on solving global talent shortages with A.I.-infused virtual reality hardware.

4. Martin Brunthaler, serial entrepreneur/CTO, on how platforms like Adverity allow you to talk with your data in natural language.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

How to Jumpstart Your Data Career (by Applying Like a Scientist), with Avery Smith

Today's fun episode with superstar Avery Smith (>140k LinkedIn and >40k YouTube subscribers) is for folks looking to jumpstart their data career — either landing your first data role or advancing your career. Enjoy!

Avery:

• Is the creator of Data Career Jumpstart — a platform to help working professionals break into, well, data careers (like data analyst or data scientist roles).

• Hosts the popular Data Career Podcast.

• Runs Snow Data Science, an analytics and data-solutions consultancy with clients including the Utah Jazz 🏀

• Previously held data scientist roles at ExxonMobil and Vaporsens.

• Holds a Master’s in Data Analytics from Georgia Tech.

Today’s episode contains helpful tips for anyone looking to advance their career but is particularly intended for listeners who are seeking their first role working with data.

In today’s episode, Avery details:

How spilling acid on himself led him to becoming a data professional.

• His "Every Turtle Swims Past" learning ladder for breaking into data careers.

• What’s even more important than skills or experience for landing a job.

• How one of his bootcamp students went from delivery driver to data analyst by AB testing her text messages.

• Which job boards are killing your data career applications.

• Why GitHub is not a portfolio, but what you can use instead.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

Conversational AI is Overhauling Data Analytics, with Martin Brunthaler

Fascinating new episode for you from serial entrepreneur/CTO Martin Brunthaler on how GenAI and Agentic A.I. are transforming data analytics today... and how analytics will continue to evolve in the coming years.

Martin Brunthaler:

• CTO of Adverity, an Austrian data analytics platform he co-founded a decade ago and that has since raised over $160m in venture capital.

• Before Adverity, Martin was co-founder and CTO at two other European tech start-ups, giving him over 20 years of combined experience in starting, scaling and exiting companies across multiple industries including eCommerce, media and mobile.

• Holds an engineering diploma (equivalent to a Bachelor's degree) from the Salzburg University of Applied Sciences in Austria.

Today’s episode should be of interest to just about anyone who’d be interested in this podcast because it touches on data analytics, transforming user experiences with modern AI capabilities and growing tech businesses.

In today’s episode, Martin details:

How a childhood fascination with computer programming evolved into founding a globally leading platform for marketing data analytics.

• What "data democratization" really means and how the traditional dashboard-based approach to data reporting is failing businesses.

• Why data analysts are spending too much time on "busy work" instead of delivering business value.

• How conversational AI is overhauling how data insights are gleaned for hands-on data practitioners and business users alike.

• His no-nonsense tips for tech startup success.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

AI-Powered Virtual Reality: The Future of Education and Entertainment, with Mary Spio

In today's episode, the deep-space engineer and visionary entrepreneur Mary Spio takes us on a journey into the A.I.-powered virtual reality that is transforming education, entertainment and more.

Mary:

• Is CEO and CTO of CEEK INC, a platform pioneering A.I.-powered virtual-reality experiences featuring the likes of Lady Gaga, Bon Jovi and Dwayne Wade.

• Holds 10+ patents across A.I., digital cinema, spatial audio, and extended-reality technologies.

• Was a deep space engineer at Boeing and, before ever even going to university, was a satellite technician for the US Air Force.

• Her innovations have been used by Xbox, Lucasfilm, and Universal Music Group.

• Holds a Masters in Electrical Engineering, Computer Science and Innovation Management from the Georgia Institute of Technology.

• Today’s episode is fascinating and relatively high-level and should be of interest to any listener.

In today’s episode, Mary details:

• How a childhood in Ghana during a military coup led to a career as a deep space engineer and A.I. entrepreneur.

• The neuroscience of how VR training can create memories that are indistinguishable from reality in your brain.

• The shocking discovery about why VR headsets were making women violently ill (and how Mary fixed it).

• How A.I. music is revolutionizing the industry and giving artists unexpected new powers.

• How blockchain verification might be our only defense against an impending tsunami of A.I.-generated deepfakes.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

Teams of Agents: The Next Frontier in AI Collaboration, with Mike Pell

Special episode for you today (filmed in front of a live audience!) with the inventor and exceptional communicator, Mike Pell. Hear his vision for the way teams of A.I. agents will change work and life for the better.

Today’s episode features a session I hosted a couple weeks ago in Brooklyn at the inaugural "A.I. & Creativity Summit", which was run by Artist and the Machine. It was an excellent full-day event on a gloriously sunny day.

My guest for an on-stage conversation in front of a live audience was the extraordinary Mike Pell:

• Inventor of the PDF and Adobe Acrobat.

• Director of The Microsoft Garage, a global innovation program.

• Holds over 20 US Patents.

• Author of three books.

Today’s episode is entertaining, optimistic and forward-looking and will be of interest to any listener of my podcast.

In the episode, Mike details:

• Why A.I. agents are like an exoskeleton that gives you capabilities you never had time to master.

• The coming shift from passive to proactive A.I. that will interject like a trusted coworker.

• Why he believes we're "getting to the good part" in the A.I. revolution and what that means for the future of work.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

Multi-Agent Teams, Quantum Computing and the Future of Work, with Dell’s Global CTO John Roese

Collosal guest today: Dell's global CTO and Chief A.I. Officer John Roese, who eloquently describes mind-blowingly fascinating topics including multi-agent teams in enterprises and quantum computing.

John Roese:

• Is Global CTO and Chief A.I. Officer at Dell Technologies, the giant Texas-based corporation with over 100,000 employees and $88 billion dollars of revenue in 2024.

• Responsible for Dell’s future-looking technology strategy and accelerating A.I. adoption for Dell and its customers.

• With an unreal career stretching back several decades, he was previously Global CTO at EMC, Global CTO at Nortel and CTO at Broadcom, amongst many other top roles at world-leading tech companies, board memberships and deep involvement with the private equity and venture capital ecosystems.

• Holds a degree in Electrical and Computer Engineering from the University of New Hampshire.

Despite John being such a deep technical expert, today’s episode stays relatively high level and so should be of great value to any listener.

In today’s episode, John details:

How Dell narrowed 800 generative A.I. ideas down to 8 high-impact projects.

• Proof-of-Concept Prison and his strategy for escaping it.

• Where multi-agent teams will make the biggest impacts in enterprises first.

• The unexpected way A.I. is creating more construction jobs than any other sector as well as new careers that will emerge in the coming years because of A.I.

• How quantum computing and A.I. advances are entangled in a way that will dramatically change the future.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

In Case You Missed It in April 2025

We had a record number of guests on my podcast in April — and they were spectacular! In today's "In Case You Missed It" episode, hear the best parts of my conversations with each of them.

The specific conversation highlights included in today's episode are:

1. Sama Bali from NVIDIA and Logan Lawler from Dell Technologies fill us in on the AI software stack on NVIDIA GPUs, including libraries like CUDA.

2. Continuing on the A.I. hardware topic, Emily Webber details Amazon Web Services (AWS)'s own A.I. accelerator chips.

3. Zerve AI's co-founder Dr. Greg Michaelson describes how data scientists can deploy A.I. models to production without needing to call on an engineering team.

4. Kai Beckmann, CEO of Merck KGaA's semiconductor business, describes intricate details of the semiconductors that make A.I. systems hum.

5. Finally, Shirish Gupta explains his "A-I-P-C" framework for finding out if you should be using edge compute for local A.I. inference instead of relying on cloud compute.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

Python Polars: The Definitive Guide, with Jeroen Janssens and Thijs Nieuwdorp

Today's episode on Polars is in equal parts hilarious and informative with Jeroen and Thijs, who co-authored the brand-new O'Reilly book "Python Polars: The Definitive Guide". Enjoy this one!

More on Dr. Jeroen Janssens:

• Senior Developer Relations Engineer at Posit PBC (iconic creators of RStudio and much more).

• Previously, was Senior Machine Learning Engineer at Xomnia.

• Wrote the invaluable O’Reilly book "Data Science at the Command Line".

• Holds a PhD in machine learning from Tilburg University.

...and on Thijs Nieuwdorp:

• Lead Data Scientist at Xomnia, the largest Dutch data and A.I. consulting company.

• Holds a degree in A.I. from Radboud University.

Today’s episode will be particularly appealing to hands-on data science, machine learning and A.I. practitioners but Jeroen and Thijs are tremendous storytellers and frankly very funny so this episode can probably be enjoyed by anyone interested in data and A.I.

In today’s episode, Jeroen and Thijs detail:

• Why pandas users are rapidly switching to Polars for dataframe operations in Python.

• The inside story of how O'Reilly rejected four book proposals on Polars before accepting the fifth.

• The moment when an innocuous GitHub pull request forced a complete rewrite of an entire book chapter.

• A previously secret collaboration with NVIDIA and Dell that revealed remarkable GPU acceleration benchmarks by Polars.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.