#496 – FFmpeg: The Incredible Technology Behind Video on the Internet

Episode of Lex Fridman Podcast · May 6, 2026 · 4 h 24 min

Jean-Baptiste Kempf and Kieran Kunhya explain FFmpeg, VLC, video codecs, handwritten assembly, open-source maintainer burnout and the Google security drama with Lex Fridman.

Summary

Lex Fridman spoke with Jean-Baptiste Kempf, president of VideoLAN and lead developer of VLC, and Kieran Kunhya, an FFmpeg contributor and codec engineer who runs the FFmpeg account on X, about the software infrastructure behind nearly all internet video. The guests walked through the full playback pipeline — from URL to byte stream, demuxing, entropy decoding, intra-prediction and inverse transforms — and explained the distinction between containers such as MP4, MOV and Matroska and codecs such as H.264, HEVC, AV1 and AV2. Kempf stressed that video compression is lossy by design and tuned to human perception, while Kunhya noted that each codec generation buys roughly 30% better compression at one to two orders of magnitude more compute.

A large part of the conversation concerned open source as a social system. Kempf described the origin of VLC in the Network 2000 student project at École Centrale Paris, his relicensing of libVLC from GPL to LGPL, which required tracking down more than 350 contributors, and his repeated refusal of multimillion-dollar offers to bundle toolbars or advertising into VLC. Kunhya criticised Google's handling of AI-generated security reports against FFmpeg, arguing that the security industry over-escalates trivial issues on obscure codecs while under-funding the volunteers who fix them; both said the resulting public drama increased donations, awareness and, eventually, patches from Google. They also discussed maintainer burnout, the XZ compromise, and death threats Kempf received after dropping PowerPC support.

On engineering, Kunhya and Kempf defended handwritten SIMD assembly over compiler auto-vectorization, citing the dav1d AV1 decoder's roughly 240,000 lines of assembly and speedups of 10x to 62x over C. They praised reverse engineers such as Kostya Shishkov, described FFmpeg's FATE testing matrix, and explained Kunhya's assembly lessons project. Kempf was sceptical of rewriting existing code in Rust while acknowledging its value for new projects, and described his new company Kyber, an AGPL/commercial dual-licensed SDK aiming at four-millisecond glass-to-glass latency for teleoperating robots, drones and vehicles.

The episode closed with the video archiving community's use of FFmpeg and the lossless FFV1 codec, patent pools around HEVC, VLC's refusal of government backdoor requests, the CIA's modified VLC build revealed in Vault 7, and speculation that multimedia frameworks will eventually handle point clouds, haptics and other sensory data.

Chapters

0:00
Introduction and the legend of VLC
Fridman introduces FFmpeg and VLC as invisible infrastructure behind internet video and thanks the volunteer engineers behind them.
10:48
The weirdest files VLC can open
Kempf and Kunhya recount VHS capture, DVD-Audio support, obscure game codecs and a competition to build the most horrible playable file, plus the origin of the traffic cone logo.
16:21
How video playback works
The guests walk through the playback pipeline from URL to demuxing, entropy decoding, intra-prediction, inverse transforms and display, and explain perceptual compression and YUV.
24:33
Containers versus codecs
Kempf and Kunhya distinguish MP4, MOV and Matroska containers from codecs like H.264, and explain why VLC probes file contents rather than trusting extensions.
35:12
What FFmpeg is and how open source licensing works
The guests describe FFmpeg's libraries and command line, then discuss permissive versus copyleft licences and Kempf's relicensing of libVLC from GPL to LGPL.
56:20
Linus Torvalds and code review culture
Kempf discusses Torvalds, Git, and why small maintainer teams demand excellence, and how harsh review tone is shaped by subculture and non-native English.
1:00:59
Turning down millions to keep VLC ad-free
Kempf tells the story of VLC's origins in the Network 2000 student project and explains why he refused toolbar, spyware and advertising deals.
1:15:14
FFmpeg, Google and the security reporting drama
Kunhya criticises Google's AI-generated security reports and the security industry's alarmism, and both describe how public pressure on X produced donations and patches.
1:34:32
Who contributes and why
The guests describe teenage contributors, the motivations of volunteers, and the relationship between FFmpeg, VideoLAN and x264 as a binary star system.
1:45:29
History of FFmpeg and reverse engineering codecs
Kunhya traces FFmpeg's eras from Fabrice Bellard to Michael Niedermayer and Kostya Shishkov, and both explain how binary blobs are reverse engineered to bit exactness.
2:02:14
FATE testing and handwritten assembly
The guests explain FFmpeg's automated testing matrix, then argue for handwritten SIMD assembly over compilers, citing dav1d's 240,000 lines and 62x speedups.
2:24:04
Learning assembly and opinions on Rust
Kunhya describes his assembly lessons project and both guests give mixed assessments of Rust's utility versus rewriting existing codebases.
2:39:56
The Libav fork and maintainer burnout
Kempf explains the 2011 FFmpeg/Libav split and its resolution, then both discuss AI slop, the XZ compromise and death threats over dropped PowerPC support.
2:56:04
x264, psychovisual optimization and anime
Kunhya explains how x264 rejected PSNR in favour of psychovisual rate distortion and adaptive quantization, and Kempf ties the project to the anime fansub community.
3:09:19
Frame types, streaming and the CIA's fake VLC
The guests explain I, P and B frames and adaptive streaming, then cover the Vault 7 CIA VLC build, fake German installers and VLC's sandboxing work.

Key points

What was said, by whom, with the exact words. Timestamps jump to the moment.

Kieran Kunhya
Kunhya said roughly 45% of video files cannot be decoded on the GPU and must fall back to software decoding.
“I think it's up to 45% of files are not GPU decodable”
Listen from 18:06 · Video codecs
Jean-Baptiste Kempf
Kempf argued that video and audio compression is designed around human perception rather than lossless data reduction, which is why YUV is used instead of RGB.
“all the compressions that we do, and that's very important, people forget about that, is to be viewed by humans”
Listen from 20:04 · Video codecs
Kieran Kunhya
Kunhya said each codec generation's roughly 30% compression gain comes at one to two orders of magnitude more CPU cost.
“It's 30% better, but an order of magnitude, perhaps even two orders of magnitude more compression power”
Listen from 22:50 · Video codecs
Jean-Baptiste Kempf
Kempf said VLC deliberately ignores file extensions and probes file contents because it was built to handle broken and untrusted input.
“we discard the file format, right? We look into the file to understand what's in it”
Listen from 27:32 · VLC media player
Kieran Kunhya
Kunhya argued FFmpeg democratized video production, putting individuals and trillion-dollar corporations on the same technology stack.
“I think it's incredible that your home videos, your grandmother's home videos and trillion-dollar corporations effectively are on a level playing field using the same technology stack”
Listen from 36:19 · FFmpeg
Jean-Baptiste Kempf
Kempf said relicensing libVLC from GPL to LGPL required personally tracking down more than 350 individual contributors because copyright remains with each author.
“if you want to properly re-license, you need to find all the contributors. And at that time, I had to contact more than 350 people”
Listen from 53:07 · Open source licensing
Jean-Baptiste Kempf
Kempf said the core maintainer teams are tiny — about five people for VLC and ten to fifteen for FFmpeg — which is why code quality standards must be uncompromising.
“The core community of VLC is five people. The core community of FFmpeg is 10 to 15”
Listen from 57:40 · FFmpeg
Linked moment · 1:02:13Skip to 1:02:13 using a listen link in the info box.
Jean-Baptiste Kempf
Kempf said he turned down tens of millions of dollars in toolbar, spyware and advertising offers because he considered them unethical and a betrayal of contributors.
“Yes, I could be a multimillionaire and be somewhere on the beach. But I did not do it because I thought it was not moral and it was not the right thing to do”
Listen from 1:02:13 · VLC media player
Kieran Kunhya
Kunhya criticised the security industry for marking trivial issues as high severity, saying constant alarmism erodes credibility.
“at some point, the security industry needs to realize you can't keep crying wolf like this”
Listen from 1:18:31 · Open source maintainer burnout
Kieran Kunhya
Kunhya said donations to FFmpeg increased substantially after the public dispute but still do not cover a single full-time developer.
“Donations have increased substantially. They're still not enough to cover even a single full-time developer”
Listen from 1:25:42 · FFmpeg
Jean-Baptiste Kempf
Kempf said VideoLAN's only effective leverage with large platform companies is public pressure on social media, citing a year-long inability to update VLC on the Android Play Store.
“The only way we got someone to answer was to put a very spicy, as you say, tweet saying that we were going to stop distributing VLC for Android”
Listen from 1:26:23 · VideoLAN
Jean-Baptiste Kempf
Kempf advised new open-source contributors to work on something they love, and said contributing to FFmpeg is the best programming education available.
“Work on something you love. I am working on VLC because I love movies”
Listen from 1:35:07 · FFmpeg
Kieran Kunhya
Kunhya said hand-written SIMD in FFmpeg yields 10x to 50x speed improvements over compiled C, not marginal percentage gains.
“when we write SIMD is we have a 10X, not percentage, 10X to 50X speed improvement”
Listen from 2:09:47 · Assembly language and SIMD optimization
Jean-Baptiste Kempf
Kempf said dav1d contains roughly 30,000 lines of C and 240,000 lines of handwritten assembly, more than all of FFmpeg's assembly combined.
“We are talking about 30,000 lines of C, but 240,000 lines of handwritten assembly, right? Handwritten assembly, 240,000 lines”
Listen from 2:12:25 · dav1d
Jean-Baptiste Kempf
Kempf said roughly 30% of Netflix video and 50% of YouTube video is now delivered in AV1, often decoded in software because few devices have hardware decoders.
“30% of the video from Netflix are now in AV1. 50% of YouTube, right?”
Listen from 2:13:53 · AV1
Kieran Kunhya
Kunhya said Rust has an 'Esperanto vibe' and that Rust rewrites must match or exceed existing feature sets and testing rigour before they can replace incumbents.
“It has a very big Esperanto vibe about it. It's like, we're going to solve this and we're doing this in a particular way”
Listen from 2:31:00 · Rust (programming language)
Jean-Baptiste Kempf
Kempf said Rust is excellent for new projects but advised against rewriting existing codebases, because reading code is far harder than writing it.
“It is an order of magnitude easier to write code than read code”
Listen from 2:34:24 · Rust (programming language)
Jean-Baptiste Kempf
Kempf said he received a death threat containing powder after deciding to stop maintaining VLC's PowerPC port around 2009-2010.
“So I received a death threat with some powder in it, right?”
Listen from 2:50:24 · Open source maintainer burnout
Kieran Kunhya
Kunhya said x264's breakthrough was rejecting PSNR-driven optimization in favour of psychovisual rate distortion and adaptive quantization.
“Two big things, psychovisual rate distortion. So using block energy, trying to compensate for human perception when making decisions”
Listen from 2:56:25 · x264
Jean-Baptiste Kempf
Kempf said Kyber currently achieves seven milliseconds glass-to-glass latency between machines and is targeting four milliseconds.
“So far, we achieve seven milliseconds from Windows to Windows or Windows to Mac”
Listen from 3:40:21 · Kyber (low-latency streaming SDK)
Jean-Baptiste Kempf
Kempf said HEVC licensing became unmanageable, with multiple patent pools and uncapped fees potentially costing large streamers hundreds of millions of dollars per year, which motivated the creation of AV1.
“for YouTube or Netflix, we could talk about hundreds of millions of dollars of licensing for patents per year”
Listen from 3:52:09 · Video patents and patent pools
Jean-Baptiste Kempf
Kempf said two intelligence agencies asked for a backdoor in VLC and he refused, adding that VideoLAN would shut the software down rather than compromise it.
“If we had to compromise our software, we would shut it down. This is clear”
Listen from 3:58:22 · VLC media player
Kieran Kunhya
Kunhya said the archiving community funded the lossless FFV1 codec and GPU encoding in FFmpeg because preserving multimedia heritage in a playable form is a long-term concern.
“They funded the development of the FFV1 codec. So that's a lossless codec”
Listen from 4:06:08 · FFV1 and digital video archiving
Jean-Baptiste Kempf
Kempf defined multimedia as a digital representation of streams for the human senses and said FFmpeg and VLC will eventually handle odour, haptics and brainwaves.
“multimedia is a digital representation of several streams for the human senses. And we will do that”
Listen from 4:13:44 · FFmpeg

People

In the conversation

Lex FridmanAmerican podcaster and AI researcher; same host, second diarization label covering the interview segments
Jean-Baptiste KempfFrench software engineer, president of VideoLAN and lead developer of VLC media player
Kieran KunhyaBritish codec engineer, longtime FFmpeg contributor, founder of Open Broadcast Systems and the person behind the FFmpeg account on X

Discussed

Linus TorvaldsCreator of the Linux kernel and Git, cited for his harsh code review style
Fabrice BellardFrench programmer who created FFmpeg
Michael NiedermayerLongtime FFmpeg maintainer credited with exhaustive MPEG-4 Part 2 support in the 2000s
Kostya ShishkovUkrainian developer known for reverse engineering complex proprietary codecs including GoToMeeting
Loren Merrittx264 developer who wrote much of its assembly and the x86inc abstraction layer
Henrik GramnerSwedish assembly expert on x86 SIMD in the VideoLAN/FFmpeg community
Martin StorsjöFinnish developer specializing in ARM NEON assembly optimization
Yuri ReznikMultimedia and signal processing researcher and contributor to H.264 and H.265 standards
Daniel StenbergMaintainer of curl, cited for criticizing AI-generated bug reports
Dave RiceArchivist associated with the FFmpeg-based video preservation community
Andrew KelleyCreator of the Zig programming language and former FFmpeg developer
Alex StrangeFormer FFmpeg developer quoted on security researcher incentives
Jon Lech JohansenNorwegian programmer who broke DVD CSS encryption, referenced for demonstrating DRM reverse engineering
Christophe MassiotEarly VideoLAN developer mentioned as a co-founder of the project
Tim SweeneyEpic Games founder who publicly praised VLC
Andrej KarpathyAI researcher cited as having learned assembly after following the FFmpeg account
TheoSoftware developer and streamer involved in the public FFmpeg security dispute
Pavel DurovTelegram founder, cited as an example of European legal pressure on platform operators

Topics

From the publisher

Jean-Baptiste Kempf is lead developer of VLC and president of VideoLAN. Kieran Kunhya is a longtime FFmpeg contributor, codec engineer, and the person behind the now-infamous FFmpeg account on X. Thank you for listening ❤ Check out our sponsors: https://lexfridman.com/sponsors/ep496-sc See below for timestamps, transcript, and to give feedback, submit questions, contact Lex, etc. Transcript: https://lexfridman.com/ffmpeg-transcript CONTACT LEX: Feedback – give feedback to Lex: https://lexfridman.com/survey AMA – submit questions, videos or call-in: https://lexfridman.com/ama Hiring – join our team: https://lexfridman.com/hiring Other – other ways to get in touch: https://lexfridman.com/contact EPISODE LINKS: FFmpeg on X: https://x.com/FFmpeg FFmpeg: https://ffmpeg.org/ VideoLAN (VLC): https://www.videolan.org/ VideoLAN on X: https://x.com/videolan Jean-Baptiste’s Website: https://jbkempf.com/ Jean-Baptiste’s LinkedIn: https://www.linkedin.com/in/jbkempf/ Jean-Baptiste’s GitHub: https://github.com/jbkempf Kieran’s X: https://x.com/kierank_ Kieran’s LinkedIn: https://bit.ly/3OORhmC Kieran’s GitHub: https://github.com/kierank SPONSORS: To support this podcast, check out our sponsors & get discounts: Larridin: Measure AI adoption in your business. Go to https://larridin.com Blitzy: AI agent for large enterprise codebases. Go to https://blitzy.com/lex BetterHelp: Online therapy and counseling. Go to https://betterhelp.com/lex Fin: AI agent for customer service. Go to https://fin.ai/lex LMNT: Zero-sugar electrolyte drink mix. Go to https://drinkLMNT.com/lex Perplexity: AI-powered answer engine. Go to https://perplexity.ai/ OUTLINE: (00:00) – Introduction (03:00) – Sponsors, Comments, and Reflections (10:48) – Weirdest things VLC opens (15:12) – How video playback works (24:33) – Video codecs and containers (35:20) – FFmpeg explained (56:20) – Linus Torvalds (1:00:59) – Turning down millions to keep VLC ad-free (1:15:17) – FFmpeg & Google drama (1:34:31) – FFmpeg develope…

Summary, chapters and key points were generated from the episode transcript on September 27, 2026. Quotes are checked word for word against the transcript.

Categories: Technology·Entrepreneurship·Science