LLM Benchmarks Are Lying to You (And What to Do Instead), with Sinan Ozdemir

Sensational episode for you today with the illustrious A.I. author, educator and entrepreneur Sinan Ozdemir on how LLM benchmarks are lying to you... and what you can do about it.

Sinan:

• Is Founder and CTO of LoopGenius, a generative A.I. startup.

• Authored several excellent books, including, most recently, the bestselling "Quick Start Guide to Large Language Models".

• Hosts the "Practically Intelligent" podcast.

• Was previously adjunct faculty at The Johns Hopkins University, now teaches several times a month within the O'Reilly platform.

• Serial A.I. entrepreneur, including founding a Y Combinator-backed generative A.I. startup way back in 2015 that was later acquired.

• Holds a Master’s in Pure Math from Johns Hopkins.

Today’s episode skews slightly toward our more technical listeners but Sinan excels at explaining complex concepts in a clear way so today’s episode may appeal to any listener of this podcast.

In today’s episode, Sinan details:

Why the A.I. benchmarks everyone relies on might be lying to you.

• How the leading A.I. labs are gaming the benchmark system.

• Tricks to actually effectively evaluate LLMs’ capabilities for your use cases.

• What the future of benchmarking will involve, including how to benchmark agentic and multimodal models.

• How a simple question about watermelon seeds reveals the 40% failure rate of even today’s most advanced A.I. models.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

In Case You Missed It in June 2025

We had exceptional guests on my podcast in June. In today's "In Case You Missed It" episode, hear the best parts of all my June convos. Here's a quick summary to tantalize you:

1. Strategy consultant Diane Hare with five tricks for gaining buy-in on A.I. transformation in your organization.

2. Renowned data-career educator Avery Smith on the two portfolio projects every aspiring data analyst should build.

3. SuperDataScience founder Kirill Eremenko on what you're missing if you're struggling to land an A.I. job.

4. San Fran-based venture capitalist Shaun Johnson on the traits that make a great A.I. startup founder.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

Automating Legal Work with Data-Centric ML (feat. Lilith Bat-Leah)

Today, exceptional communicator Lilith Bat-Leah explains why "Data-Centric ML Research" trumps our typical focus on model capability, with examples from her extensive Legal A.I. background.

Lilith:

• Has over a decade of experience specializing in the application of ML to legal tech.

• Is Senior Director of A.I. Labs at Epiq, a leading LegalTech firm that has over 6000 employees.

• Has published work on evaluation methods for the use of ML in legal discovery as well as on Data-centric ML Research (DMLR).

• Is co-chair of the DMLR working group MLCommons and has organized DMLR workshops at [ICML] Int'l Conference on Machine Learning and ICLR, two of the most important A.I. conferences.

• Holds a degree from Northwestern University, in which she focused on statistics.

Today’s episode will appeal primarily to hands-on practitioners like data scientists, AI/ML engineers and software developers.

In today’s episode, Lilith details:

How A.I. is revolutionizing the legal industry by automating up to 80% of traditional discovery processes.

• Why 'elusion' is a critical metric that only exists in LegalTech — and what it reveals about machine learning evaluation.

• The surprising reason why we should stop obsessing over model improvements and focus on something that takes up 80% data scientists’ time instead.

• How she grew from being a temp receptionist to an A.I. lab director by falling in love with statistics.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

95-Year-Old Annie on How to Stay Healthy and Happy

Our 900th Episode! It almost goes by as quickly as 95 years! By popular demand, my grandmother Annie returns to the podcast with wisdom on staying happy and healthy at any age.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

Landing $200k+ AI Roles: Real Cases from the SuperDataScience Community, with Kirill Eremenko

As we approach episode #900, the original SuperDataScience Podcast host Kirill Eremenko returns to reflect on what leads to the highest-paying opportunities in AI. This is a special one; enjoy!

Many of you will already know Kirill:

• Founder and CEO of SuperDataScience.com, the eponymous e-learning platform.

• Founded the SuperDataScience Podcast nine years ago and hosted the show until he passed me the reins five years ago.

• With over 3 million students, he’s the most popular data science and A.I. instructor on Udemy.

• He holds a Master’s from The University of Queensland in Australia and a Bachelor’s in Applied Physics and Mathematics from the Moscow Institute of Physics and Technology.

Today’s episode is ideal for anyone looking to advance their data science or A.I. career — or looking to break into a career in this field for the first time.

In today’s episode, Kirill details:

• Why employers are still testing A.I. engineers on basic machine learning fundamentals — even for LLM-focused roles.

• The surprising reason why staying in data science (as opposed to developing an A.I. specialization) could be the right career move for you.

• How one developer discovered the hidden age bias in tech recruiting — and the simple hack to beat it.

• The two critical skill areas that separate amateur A.I. engineers from the pros commanding huge salaries.

• Why the "back to office" movement could give you a competitive advantage in landing a top A.I. role.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

How to Enable Enterprise AI Transformation, with Strategy Consultant Diane Hare

People, not technical capability, are holding back A.I.'s impact in organizations. In today's episode, Diane Hare explains how to overcome friction and enable strategic A.I. transformation.

Diane:

• Founder and CEO of the New York-based strategic consulting firm BizLove, which has been mobilizing key stakeholders to deliver on enterprise-wide priorities (like A.I. initiatives!) at Fortune 100 companies for seven years.

• Prior to her seven years leading BizLove, spent seven years at EY, the global professional services giant (they have nearly 400,000 employees) formerly known as Ernst & Young.

• Board Member at NANO Nuclear Energy Inc. (NASDAQ: NNE)

• Holds and MBA and was captain of a semi-professional women’s soccer team in New York City!

Today’s episode is well-suited to anyone looking to make an impact with A.I. and automation, which I suspect is about every listener to my podcast!

In today’s episode, Diane details:

• Why people, not technical capability, are holding back A.I.’s transformative power in organizations.

• How to prioritize the items on an enterprise A.I. roadmap.

• Why storytelling is essential for gaining buy-in from stakeholders on an A.I. initiative.

• Her top five tips for enabling A.I. transformation.

This was a super-cool episode for me because Diane's consultancy, BizLove, is a formal partner of my own consultancy, Y Carrot 🥕. While Y Carrot brings rich technical expertise on A.I. (from development through to production deployment), BizLove naturally complements us with their deep experience enabling digital and A.I. transformations of enterprises. Together, we offer every service organizations need to make lasting, impactful improvements with A.I.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

Agentic AI Hands-On in Python: MCP, CrewAI and OpenAI Agents SDK (by Jon Krohn and Ed Donner)

Now live! Four hours long and 100% free, this hands-on workshop covers all the Agentic A.I. theory and tools you need to develop and deploy multi-agent teams with Python.

Beautifully shot by a professional film crew (led by the exceptional Lucie McCormick) at the Open Data Science Conference (ODSC) East in Boston a few weeks ago and then meticulously edited by SuperDataScience's inimitable Mario Pombo, this training (within the GenAI-forward Cursor IDE) features all of today's essential agent frameworks:

• OpenAI Agents SDK

• CrewAI

• Anthropic's Model Context Protocol (MCP)

From design considerations through to practical implementation tips, by completing all four modules in this video, you will have all the knowledge and skills needed to create effective multi-agent systems. The four modules are:

1. Defining Agents

2. Designing Agents

3. Developing Agents

4. The Future of Agents

The coding elements are led by the wonderful Ed Donner, whom many of you will already know as one of the very best in the world at creating and teaching hands-on A.I. content.

We received rave reviews for the session at ODSC East and the lecture hall was standing-room only for the entire duration, so I anticipate that you'll love it too!

Watch the full training here: youtu.be/LSk5KaEGVk4

The Future of Enterprise AI: Investor Shaun Johnson Reveals What Actually Works

What are the biggest opportunities for A.I. startups? Find out in today's episode with the trailblazing venture capitalist Shaun Johnson, including tricks for gaining enterprise A.I. adoption.

Shaun:

• Co-founder and general partner at AIX Ventures in San Francisco, where he’s led deals into companies including Perplexity, Chroma, and Workhelix.

• He is a former VP of Engineering, Product and Design at Lilt; and a former VP of Product and Design at NimbleRx.

• Holds a Master’s in Electrical Engineering from Stanford University and an MBA from the University of California, Berkeley.

Today’s episode is well-suited to any listener to this podcast. In it, Shaun details:

How having investment partners like Richard Socher and Christopher Manning, who are practitioners actively building at the cutting edge of A.I., gives AIX Ventures an edge.

• What it takes to become one of the few thousand people in the world pushing the A.I. frontier.

• The surprising strategy that makes enterprise A.I. adoption 10x easier.

• Why some A.I. startups are better off building in 'red oceans' full of competition rather than seeking blue-ocean opportunities.

• The reason big tech companies are buying A.I. talent without acquiring the actual startups.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

In Case You Missed It in May 2025

We had stellar guests in May, including one episode that had the most positive social-media response of any episode ever. In today's "In Case You Missed It" episode, hear the best parts of all my May convos.

The specific conversation highlights included in today's episode are:

1. John Roese, Dell Technologies' global CTO and Chief A.I. Officer, on the biggest A.I. opportunities for enterprises in the coming months/years. (This is the episode that received an unprecedented social-media response.)

2. The authors of the brand-new O'Reilly book "Python Polars: The Definitive Guide", Jeroen Janssens and Thijs Nieuwdorp, on a real-world Polars success story.

3. Space engineer and entrepreneur Mary Spio on solving global talent shortages with A.I.-infused virtual reality hardware.

4. Martin Brunthaler, serial entrepreneur/CTO, on how platforms like Adverity allow you to talk with your data in natural language.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

How to Jumpstart Your Data Career (by Applying Like a Scientist), with Avery Smith

Today's fun episode with superstar Avery Smith (>140k LinkedIn and >40k YouTube subscribers) is for folks looking to jumpstart their data career — either landing your first data role or advancing your career. Enjoy!

Avery:

• Is the creator of Data Career Jumpstart — a platform to help working professionals break into, well, data careers (like data analyst or data scientist roles).

• Hosts the popular Data Career Podcast.

• Runs Snow Data Science, an analytics and data-solutions consultancy with clients including the Utah Jazz 🏀

• Previously held data scientist roles at ExxonMobil and Vaporsens.

• Holds a Master’s in Data Analytics from Georgia Tech.

Today’s episode contains helpful tips for anyone looking to advance their career but is particularly intended for listeners who are seeking their first role working with data.

In today’s episode, Avery details:

How spilling acid on himself led him to becoming a data professional.

• His "Every Turtle Swims Past" learning ladder for breaking into data careers.

• What’s even more important than skills or experience for landing a job.

• How one of his bootcamp students went from delivery driver to data analyst by AB testing her text messages.

• Which job boards are killing your data career applications.

• Why GitHub is not a portfolio, but what you can use instead.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

Conversational AI is Overhauling Data Analytics, with Martin Brunthaler

Fascinating new episode for you from serial entrepreneur/CTO Martin Brunthaler on how GenAI and Agentic A.I. are transforming data analytics today... and how analytics will continue to evolve in the coming years.

Martin Brunthaler:

• CTO of Adverity, an Austrian data analytics platform he co-founded a decade ago and that has since raised over $160m in venture capital.

• Before Adverity, Martin was co-founder and CTO at two other European tech start-ups, giving him over 20 years of combined experience in starting, scaling and exiting companies across multiple industries including eCommerce, media and mobile.

• Holds an engineering diploma (equivalent to a Bachelor's degree) from the Salzburg University of Applied Sciences in Austria.

Today’s episode should be of interest to just about anyone who’d be interested in this podcast because it touches on data analytics, transforming user experiences with modern AI capabilities and growing tech businesses.

In today’s episode, Martin details:

How a childhood fascination with computer programming evolved into founding a globally leading platform for marketing data analytics.

• What "data democratization" really means and how the traditional dashboard-based approach to data reporting is failing businesses.

• Why data analysts are spending too much time on "busy work" instead of delivering business value.

• How conversational AI is overhauling how data insights are gleaned for hands-on data practitioners and business users alike.

• His no-nonsense tips for tech startup success.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

ODSC Speaker Impact Award

Surreal to have ODSC, the biggest conference organizers in my field, make a statement like “As our most prolific and consistently top-rated instructor, Jon has been a driving force behind many of [our] most popular courses” about me, whoa:

Celebrating 5 Years with ODSC: An Award, A Workshop, and What’s Ahead

Last week in Boston, the Open Data Science Conference (ODSC) surprised me with their "Speaker Impact Award" to recognize the years of training I've been providing at ODSC conferences.

Thank you Sheamus McGovern (pictured) and the whole ODSC team (Alex, Alina, Anna, Deepti, Elen, Paula, Ruby) for the honor and for putting on such stellar technical conferences.

I first lectured at ODSC New York in June 2019, when I provided a half-day workshop that introduced Deep Learning. (By great chance, the now-legendary Serg Masís emceed my session!)

Since then, I've enjoyed both ODSC East (held each spring in Boston) and ODSC West (held each autumn in San Francisco) most years, delivering (typically full-day) workshops on:

• Deep Learning

• The mathematical foundations of Machine Learning (e.g., linear algebra, partial-derivative calculus)

• Training and deploying Large Language Models (with Lightning AI and Hugging Face)

This year at ODSC East, Ed Donner and I delivered a full-day training on developing and deploying Agentic A.I. featuring the open-source tools CrewAI, OpenAI Agents SDK, and Anthropic's Model Context Protocol (MCP). The session was jam-packed for the entire day and received rave reviews.

If you couldn't make it to Boston last week, I have good news for you! I hired a film crew to capture our entire Agentic A.I. training and am currently having the footage professionally edited. In the coming weeks (as soon as possible!), we'll be publishing this on YouTube so that it's freely available to everyone worldwide. Watch this space :)

AI-Powered Virtual Reality: The Future of Education and Entertainment, with Mary Spio

In today's episode, the deep-space engineer and visionary entrepreneur Mary Spio takes us on a journey into the A.I.-powered virtual reality that is transforming education, entertainment and more.

Mary:

• Is CEO and CTO of CEEK INC, a platform pioneering A.I.-powered virtual-reality experiences featuring the likes of Lady Gaga, Bon Jovi and Dwayne Wade.

• Holds 10+ patents across A.I., digital cinema, spatial audio, and extended-reality technologies.

• Was a deep space engineer at Boeing and, before ever even going to university, was a satellite technician for the US Air Force.

• Her innovations have been used by Xbox, Lucasfilm, and Universal Music Group.

• Holds a Masters in Electrical Engineering, Computer Science and Innovation Management from the Georgia Institute of Technology.

• Today’s episode is fascinating and relatively high-level and should be of interest to any listener.

In today’s episode, Mary details:

• How a childhood in Ghana during a military coup led to a career as a deep space engineer and A.I. entrepreneur.

• The neuroscience of how VR training can create memories that are indistinguishable from reality in your brain.

• The shocking discovery about why VR headsets were making women violently ill (and how Mary fixed it).

• How A.I. music is revolutionizing the industry and giving artists unexpected new powers.

• How blockchain verification might be our only defense against an impending tsunami of A.I.-generated deepfakes.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

Teams of Agents: The Next Frontier in AI Collaboration, with Mike Pell

Special episode for you today (filmed in front of a live audience!) with the inventor and exceptional communicator, Mike Pell. Hear his vision for the way teams of A.I. agents will change work and life for the better.

Today’s episode features a session I hosted a couple weeks ago in Brooklyn at the inaugural "A.I. & Creativity Summit", which was run by Artist and the Machine. It was an excellent full-day event on a gloriously sunny day.

My guest for an on-stage conversation in front of a live audience was the extraordinary Mike Pell:

• Inventor of the PDF and Adobe Acrobat.

• Director of The Microsoft Garage, a global innovation program.

• Holds over 20 US Patents.

• Author of three books.

Today’s episode is entertaining, optimistic and forward-looking and will be of interest to any listener of my podcast.

In the episode, Mike details:

• Why A.I. agents are like an exoskeleton that gives you capabilities you never had time to master.

• The coming shift from passive to proactive A.I. that will interject like a trusted coworker.

• Why he believes we're "getting to the good part" in the A.I. revolution and what that means for the future of work.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.