TabPFN: Deep Learning for Tabular Data (That Actually Works!), with Prof. Frank Hutter

I've been teaching Deep Learning for a decade. In that time, countless students have been disappointed by applying DL to tabular and time-series data. Finally, thanks to Prof. Frank Hutter, that will no longer be the case!

Frank:

• Is a tenured professor of machine learning and head of the Machine Learning Lab at the University of Freiburg, although he has been on leave since May to focus on…

• His fellowship on AutoML and Tabular Foundation Models at the ELLIS Institute Tübingen in Germany…

• As well as becoming Co-Founder and CEO of Prior Labs, a German startup that provides a commercial counterpart to his tabular deep-learning model research and open-source projects… and that has just announced a huge €9m pre-seed funding round.

• Holds a PhD in Computer Science from The University of British Columbia and his research has been extremely impactful: It has been cited over 87,000 times!

Today’s episode is on the technical side and will largely appeal to hands-on practitioners like data scientists, AI/ML engineers, software developers and statisticians (especially Bayesian statisticians)!

For a bit of context: Pretty much everyone works with tabular data, either primarily or occasionally. Tabular data are data stored in a table format, so structured into rows and columns, where the columns might be different data types, say, some numeric, some categorical and some text. For a decade, deep learning has ushered in the A.I. era by making huge advancements across many kinds of data — pixels from cameras, sound from microphones and of course natural language — but through all of this revolution, deep learning has struggled to be impactful on highly ubiquitous tabular data… until now.

In today’s episode, Prof. Hutter details:

• How his revolutionary transformer architecture, TabPFN, has finally cracked the code on using deep learning for tabular data and is outperforming traditionally leading approaches like gradient-boosted trees on tabular datasets.

• How version 2 of TabPFN, released last month to much fanfare thanks to its publication in the prestigious journal Nature, is a massive advancement, allowing it to handle orders of magnitude more training data.

• How embracing Bayesian principles allowed TabPFN v2 to work "out of the box" on time-series data, beating specialized models and setting a new state of the art on the key time-series analysis benchmark.

• The breadth of verticals that TabPFN has already been applied to and how you can now get started with this (conveniently!) open-source project on your tabular data today.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

Deep Learning Classics and Trends, with Dr. Rosanne Liu

Today's guest is the amazing Google DeepMind research scientist, Dr. Rosanne Liu!

Rosanne:

• Is a Research Scientist at Google DeepMind in California.

• Is Co-Founder and Executive Director of ML Collective, a non-profit that provides global ML research training and mentorship.

• Was a founding member of Uber AI Labs, where she served as a Senior Research Scientist.

• She has published deep learning research in top academic venues such as NeurIPS, ICLR, ICML and Science, and her work has been covered in publications like WIRED and the MIT Tech Review.

• Holds a PhD in Computer Science from Northwestern University.

Today’s episode, particularly in the second half when we dig into Rosanne’s fascinating research, is relatively technical so will probably appeal most to hands-on practitioners like data scientists and ML engineers.

In today’s episode, Rosanne details:

• The problem she founded the ML Collective to solve.

• How her work on the “intrinsic dimension” of deep learning models inspired the now-standard LoRA approach to fine-tuning LLMs.

• The thorny problems with LLM evaluation benchmarks and how they might be solved.

• The pros and cons of curiosity- vs goal-driven ML research.

• The positive impacts of diversity, equity and inclusion in the ML community.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

Full Encoder-Decoder Transformers Fully Explained, with Kirill Eremenko

Last month, Kirill Eremenko was on the show to detail Decoder-Only Transformers (like the GPT series). It was our most popular episode ever, so he's come right back today to detail an even more sophisticated architecture: Encoder-Decoder Transformers.

If you don’t already know him, Kirill:

• Is Founder and CEO of SuperDataScience, an e-learning platform that is the namesake of this podcast.

• Founded the Super Data Science Podcast in 2016 and hosted the show until he passed me the reins a little over three years ago.

• Has reached more than 2.7 million students through the courses he’s published on Udemy, making him Udemy’s most popular data science instructor.

Kirill was most recently on the show for Episode #747 to provide a technical introduction to the Transformer module that underpins all the major modern Large Language Models (LLMs) like the GPT, Gemini, Llama and BERT architectures. We received an unprecedented amount of positive feedback from that episode, demanding more! So here we are.

That episode, #747, as well as today’s, are perhaps the two most technical episodes of this podcast ever so they probably appeal mostly to hands-on practitioners like data scientists and ML engineers, particularly those who already have some understanding of deep neural networks.

In this episode, Kirill:

• Reviews the key Transformer theory that we covered in Episode #747, namely the individual neural-network components of the Decoder-Only architecture that prevails in generative LLMs like the GPT series models.

• Builds on that to detail the full, Encoder-Decoder Transformer architecture that is used in the original Transformer by Google, in their “Attention is All You Need” paper, as well as in other models that excel at both natural-language understanding and generation such as T5 and BART.

• Discusses the performance and capability pros and cons of full Encoder-Decoder architectures relative to Decoder-Only architectures like GPT and Encoder-Only architectures like BERT.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

AlphaGeometry: AI is Suddenly as Capable as the Brightest Math Minds

Google DeepMind's open-sourced AlphaGeometry blends "fast thinking" (like intuition) with "slow thinking" (like careful, conscious reasoning) to enable a big leap forward in A.I. capability and match human Math Olympiad gold medalists on geometry problems.

KEY CONTEXT
• A couple weeks ago, DeepMind published on AlphaGeometry in the prestigious journal peer-reviewed Nature.
• DeepMind focused on geometry due to its demand for high-level reasoning and logical deduction, posing a unique challenge that traditional ML models struggle with.

MASSIVE RESULTS
• AlphaGeometry tackled 30 International Mathematical Olympiad problems, solving 25. This outperforms human Olympiad bronze and silver medalists' averages (who solved 19.3 and 22.9, respectively) and closely rivals gold medalists (who solved 25.9).
• This new system crushes the previous state-of-the-art A.I., which solved only 10 out of 30 problems.
• Beyond solving problems, AlphaGeometry also generates understandable proofs, making A.I.-generated solutions more accessible to humans.

HOW?
• AlphaGeometry uses a new method of generating synthetic theorems and proofs, simulating 100 million unique examples to overcome the limitations of (expensive, laborious) human-generated proofs.
• It combines a neural (deep learning) language model for intuitive guesswork with a symbolic deduction engine for logical problem-solving, mirroring "fast" and "slow thinking" processes akin to human cognition (per Daniel Kahneman's "Thinking, Fast and Slow" book).

IMPACT
• A.I. that can "think fast and slow" like AlphaGeometry could generalize across mathematical fields and potentially other scientific disciplines, pushing the boundaries of human knowledge and problem-solving capabilities.

The SuperDataScience podcast is available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

Upskilling in Data Science and Machine Learning

This week, iconic Stanford University Deep Learning instructor and entrepreneur Kian Katanforoosh details how ML powers his EdTech platform Workera, enabling you to systematically fill gaps in your data science skills.

Kian:
• Is Co-Founder and CEO of Workera, a Bay Area education technology company that has raised $21m in venture capital to upskill workers, with a particular early focus on upskilling technologists like data scientists, software developers, and machine learning specialists.
• Is a lecturer of computer science at Stanford University (specifically, he teaches the extremely popular CS230 Deep Learning course alongside Prof. Andrew Ng, one of the world’s best-known data scientists).
• Was awarded Stanford’s highest teaching award.
• Is also a founding member of DeepLearning.AI, a platform through which he’s taught over three million students deep learning.
• Holds a Masters in Math and Computer Science from CentraleSupélec.
• Holds a Masters in Management Science and Engineering from Stanford.

By and large, today’s episode will appeal to any listener who’s keen to understand the latest in education technology, but there are parts here and there that will specifically appeal to practicing technologists like data scientists and software developers.

In this episode, Kian details:
• What a skills intelligence platform is.
• Four ways that machine learning drives his skills intelligence platform.
• What frameworks and software languages they selected for building their platform and why.
• What he looks for in the data scientists and software engineers he hires.

The SuperDataScience show's available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

Optimizing Computer Hardware with Deep Learning

The polymath Dr. Magnus Ekman joins me from NVIDIA today to explain how machine learning is used to guide *hardware* architecture design and to provide an overview of his brilliant book "Learning Deep Learning".

Magnus:
• Is a Director of Architecture at NVIDIA (he's been there 12 years!)
• Previously worked at Samsung and Sun Microsystems.
• Was co-founder/CTO of the start-up SKOUT (acquired for $55m).
• Authored the epic, 700-page "Learning Deep Learning".
• Holds a Ph.D. in computer engineering from the Chalmers University of Technology and a masters in economics from Göteborg University.

Today’s episode has technical elements here and there but should largely be interesting to anyone who’s interested in hearing the latest trends in A.I., particularly deep learning, software and hardware.

In the episode, Magnus details:
• What hardware architects do.
• How ML can be used to optimize the design of computer hardware.
• The pedagogical approach of his exceptional deep learning book.
• Which ML users need to understand how ML models work.
• Algorithms inspired by biological evolution.
• Why Artificial General Intelligence won’t be obtained by increasing model parameters alone.
• Whether transformer models will entirely displace other deep learning architectures such as CNNs and RNNs.

The SuperDataScience show's available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

TEDxDrexelU on Deep Learning

I'm giving my first TED-format talk at TEDxDrexelU in Philadelphia on May 21. I'll provide a visual intro to Deep Learning and to the momentous opportunity we have to shape a bewilderingly prosperous world with A.I.

There are only 100 tickets available for sale (for $15!) but my understanding is that my talk will eventually be made available on the TED YouTube channel if you can't make it in-person.

The other compelling speakers are:
• Ebony White, PhD
• Adit Gupta
• Dale Moss
• Nadia Christina Jagessar, MBA
• Dr. Nyree Dardarian
• Raja Schaar, IDSA

Event and ticket details available here.

GPT-3 for Natural Language Processing

With its human-level capacity on tasks as diverse as question-answering, translation, and arithmetic, GPT-3 is a game-changer for A.I. This week's brilliant guest, Melanie Subbiah, was a lead author of the GPT-3 paper.

GPT-3 is a natural language processing (NLP) model with 175 billion parameters that has demonstrated unprecedented and remarkable "few-shot learning" on the diverse tasks mentioned above (translation between languages, question-answering, performing three-digit arithmetic) as well as on many more (discussed in the episode).

Melanie's paper sent shockwaves through the mainstream media and was recognized with an Outstanding Paper Award from NeurIPS (the most prestigious machine learning conference) in 2020.

Melanie:
• Developed GPT-3 while she worked as an A.I. engineer at OpenAI, one of the world’s leading A.I. research outfits.
• Previously worked as an A.I. engineer at Apple.
• Is now pursuing a PhD at Columbia University in the City of New York specializing in NLP.
• Holds a bachelor's in computer science from Williams College.

In this episode, Melanie details:
• What GPT-3 is.
• Why applications of GPT-3 have transformed not only the field of data science but also the broader world.
• The strengths and weaknesses of GPT-3, and how these weaknesses might be addressed with future research.
• Whether transformer-based deep learning models spell doom for creative writers.
• How to address the climate change and bias issues that cloud discussions of large natural language models.
• The machine learning tools she’s most excited about.

This episode does have technical elements that will appeal primarily to practicing data scientists, but Melanie and I put an effort into explaining concepts and providing context wherever we could so hopefully much of this fun, laugh-filled episode will be engaging and informative to anyone who’s keen to learn about the start of the art in natural language processing and A.I.

The SuperDataScience show's available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

Jon’s Answers to Questions on Machine Learning

The wonderful folks at the Open Data Science Conference (ODSC) recently asked me five great questions on machine learning. I thought you might like to hear the answers too, so here you are!

Their questions were:
1. Why does my educational content focus on deep learning and on the foundational subjects underlying machine learning?
2. Would you consider deep learning to be an “advanced” data science skill, or is it approachable to newcomers/novice data scientists?
3. What open-source deep learning software is most dominant today?
4. What open-source deep learning software are you looking forward to using more?
5. Do you have a case study where you've used deep learning in practice?

The SuperDataScience show's available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

ODSC's blog post of our Q&A is here.

Deep Reinforcement Learning — with Wah Loon Keng

For an intro to Deep Reinforcement Learning or to hear about the latest research and applications in the field (which is responsible for the most cutting-edge "A.I."), today's episode with Wah Loon Keng is for you.

Keng:
• Co-authored the exceptional book "Foundations of Deep Reinforcement Learning" alongside Laura Graesser.
• Co-created SLM-Lab, an open-source deep reinforcement learning framework written with the Python PyTorch library.
• Is a Senior A.I. Engineer at AppLovin, a marketing solutions provider.

In this episode, Keng details:
• What reinforcement learning is.
• A timeline of major breakthroughs in the history of Reinforcement Learning, including when and how Deep RL evolved.
• Modern industrial applications of Deep RL across robotics, logistics, and climate change.
• Limitations of Deep RL and how future research may overcome these limitations.
• The industrial robotics and A.I. applications Deep RL could play a leading role in in the coming decades.
• What it means to be an A.I. engineer and the software tools he uses daily in that role.

The SuperDataScience show's available on all major podcasting platforms, YouTube, and at SuperDataScience.com.

Deep Learning's Neurobiology Origins

Interested in understanding what Deep Learning is? Here's a ten-minute talk I gave last month that visually explains the approach, how it originated in neuroscience research, and how it's now leading the A.I. revolution.

Thanks to Kate Strachnyi and Ravit Jain for organizing the fun "Holiday Book Party" that I gave this talk at. And thanks to Aglae Bassens for the stunning illustrations I used throughout.

Backpropagation

This week's video explains the relationship between Partial-Derivative Calculus and the Backpropagation ("Backprop") approach used widely in training Artificial Neural Networks, including Deep Learning networks.

We publish a new video from my "Calculus for Machine Learning" course to YouTube every Wednesday. Playlist is here.

More detail about my broader "ML Foundations" curriculum and all of the associated open-source code is available in GitHub here.

A.I. Robotics at Home

This week's guest is mad genius Dave Niewinski, who creates A.I.-powered robots for use at home (e.g., cold beer retriever, flame-throwing weed killer) and to teach people A.I. robotics via his popular YouTube channel.

In the episode, Dave covers:
• The specific robotics hardware and open-source software incorporated into his wildest and most well-known robots
• Where machine vision algorithms, particularly deep learning models, are critical for enabling robot functionality
• His tips for folks who’d like to get started in A.I. robotics themselves
• How his interest in robotics led him to founding his Dave's Armoury Ltd. consulting business, which allows him to automate and improve real-world industrial processes with robots
• What excites him most about the societal impact A.I. robotics will have in our lifetimes

Specific robots of Dave's that we detail on the show include ones that:
• Play the sandbag-tossing game "cornhole"
• Defend his machine shop from kids by spraying them with a hose
• Exterminate weeds by throwing flames
• Carve pumpkins for Halloween
• Solve Rubik's cubes
• Use Lego pieces to create 2D artwork
• Race kids at assembling 3D Lego unicorns
• Bring cold beer from the fridge to wherever you are in the house

Thanks to Graham McCormick for introducing me to Dave! I learned so much from him and had such a good time hanging with him "on air".

The SuperDataScience show's available on all major podcasting platforms, YouTube, and at SuperDataScience.com. You can check out the Dave's Armoury YouTube channel here!

Deep Learning Battle: Pytorch vs. Tensorflow

When I teach Deep Learning, the question I get most often is: "Should I be using TensorFlow or PyTorch?" In this recent talk at the DataScienceGO conference, I provide my most thorough and polished response yet.

Thanks to Harpreet Sahota for hosting the session masterfully and leading the audience Q&A at the end.

Intro to Deep Reinforcement Learning at Columbia University

My goodness did I ever miss lecturing in-person! Was finally back in front of a live classroom on Friday, providing an intro to Deep Reinforcement Learning to engineering graduates at Columbia University in the City of New York.

Thank you Chong and Sam for having me. It's always a delight to lecture to your brilliant ELEN E6885 students, especially now that the pandemic is subsiding and I can interact with them meaningfully again.

You can see the slides from here as well as the associated GitHub repository here.

Translations of Deep Learning Illustrated – Now available in Traditional Chinese

My book, Deep Learning Illustrated, recently became available in Traditional Chinese alongside the existing Russian, German, and Korean translations. The new edition instantly became a #1-bestseller in Taiwan.

Thanks to Neville Huang for diligently translating to Traditional Chinese from the original English. Neville also tipped me off to the #1-bestseller status — I've put the screenshot he shared with me on LinkedIn.

Deep Reinforcement Learning for Robotics with Pieter Abbeel

Very special guest this week! Pieter Abbeel is a serial A.I. entrepreneur, host of star-studded The Robot Brains Podcast, and the world's most preeminent researcher of Deep Reinforcement Learning applications.

As a professor of Electrical Engineering and Computer Science at the University of California, Berkeley, Pieter directs the Berkeley Robot Learning Lab and co-directs the Berkeley A.I. Research Lab.

As an entrepreneur, he's been exceptionally successful at applying machine learning for commercial value. Gradescope, a machine learning company in the education technology space that he co-founded, was acquired in 2018. And the A.I. robotics firm Covariant, which he co-founded more recently, has raised $147 million so far, including raising $80 million in a Series C funding round in July.

In this episode, Pieter eloquently discusses:
• His exciting current research in the field of Deep Reinforcement Learning
• Top learning resources and skills for becoming an expert in A.I. robotics
• How academic robotics research is vastly different from R&D for industry
• Productivity tips
• Traits he looks for in data scientists he hires
• Skills to succeed as a data scientist in the coming decades

He also had time to answer thoughtful questions from distinguished SuperDataScience listeners Serg Masís and Hsieh-Yu Li.

Listen or watch here.

AI Recruitment Technology & Deep Learning - Guest Appearance on the Engineered-Mind Podcast

Thanks to Jousef Murad for having me on the popular Engineered-Mind podcast.! Jousef had deeply insightful questions and I enjoyed the experience immensely :)

I spoke with Jousef back in April 2021, where we discussed:

- untapt and how AI powered recruiting works
- My background in neuroscience
- Where to get started when learning ML
- Tips for becoming a deep learning specialist
- What is I’m most excited about in terms of AI
- How I come up with the idea of writing a book

You can listen to the podcast anywhere podcasts are available including Apple Podcasts, Spotify, and Anchor.fm. You can also check out the video directly on YouTube here.