The memory towers behind the AI boom

1:23“Imagine a restaurant with hundreds of chefs and one narrow ingredient door.”

As checked: A workload can be limited by compute throughput or memory bandwidth depending on how much arithmetic it performs per byte moved.

3:38“HBM4 doubles the interface cited for HBM3E, from 1,024 to 2,048 data connections per stack. Samsung says its fastest HBM4 configuration can reach up to 3.3 terabytes per second for one stack.”

As checked: HBM4 doubles the cited per-stack data interface from 1,024 to 2,048 I/Os relative to HBM3E.

3:38“HBM4 doubles the interface cited for HBM3E, from 1,024 to 2,048 data connections per stack. Samsung says its fastest HBM4 configuration can reach up to 3.3 terabytes per second for one stack.”

As checked: Samsung began HBM4 mass production and commercial shipment in February 2026, offering 12-layer 24-36 GB products and reporting up to 3.3 TB/s per stack.

3:58“And none of this began with generative AI. AMD and SK hynix brought HBM to commercial graphics cards in 2015, when the hungry machine was a gaming GPU.”

As checked: HBM existed before the generative-AI boom; AMD commercialized the first HBM-powered GPU family in 2015 after development with SK hynix.

4:20“But proximity alone is not enough. Thousands of lanes need somewhere to run.”

As checked: TSMC CoWoS integrates logic SoCs and HBM stacks, making advanced packaging part of accelerator capability rather than a cosmetic final step.

5:14“And HBM4 makes the relationships even tighter. Its base die uses advanced logic, so memory design now reaches into the foundry and packaging worlds.”

As checked: HBM4 makes compute-memory co-design more important by using advanced logic base dies and a wider interface.

5:25“SK hynix works with TSMC on its HBM4 base die and CoWoS integration. Samsung describes a more integrated route, combining its memory, foundry, and packaging capabilities around a four-nanometer base die.”

As checked: SK hynix uses TSMC advanced logic for its HBM4 base die and collaborates with TSMC on CoWoS integration.

6:03“Designing that machine is hard.”

As checked: Stacking multiple TSV-connected dies creates distinct yield, test, and reliability challenges.

6:51“The testing cannot happen only at the end.”

As checked: HBM manufacturing relies on screening before stacking and repeated testing after stacking because assembly can introduce connection, stress, and yield failures even when constituent dies passed earlier tests.

7:55“Now make each floor communicate faster.”

As checked: Higher stack density and faster interfaces make heat removal and thermal resistance important HBM design constraints.

9:02“Future stacks may use hybrid bonding, joining copper connections more directly and reducing space occupied by microscopic bumps. But adoption depends on cost, process maturity, and the generation being built.”

As checked: Future HBM stacks may use hybrid bonding to connect copper more directly and reduce space and resistance associated with micro-bumps, but adoption is not yet universal.

9:35“By 2026, the established HBM market was dominated by three companies: SK hynix, Samsung, and Micron.”

As checked: SK hynix, Samsung, and Micron are the three major established HBM suppliers at the 2026 checkpoint.

9:54“SK hynix pairs its memory and stacking expertise with TSMC's advanced logic and packaging ecosystem. The company said it began mass shipments of HBM4 in the second quarter of 2026 and planned a second-half ramp.”

As checked: SK hynix began mass shipments of HBM4 in Q2 2026 and planned to ramp production in H2.

10:27“Micron, the US-headquartered supplier, said HBM4 volume shipments began in the first quarter. By its fiscal third quarter, it said it had shipped more than one billion dollars in HBM4 revenue.”

As checked: Micron reported HBM4 volume shipments beginning in Q1 2026 and more than $1 billion in HBM4 revenue shipped by fiscal Q3 2026.

11:49“Samsung forecast that the broader memory market would remain undersupplied in the second half of 2026 despite production increases. Broader memory - not proof that every Samsung HBM product was unavailable.”

As checked: Samsung expected memory supply constraints to continue in H2 2026 despite production increases.

12:33“But adding new capacity years ahead is nothing like restocking an empty warehouse.”

As checked: New HBM/advanced-packaging capacity takes years to construct; SK hynix's 2026 Indiana groundbreaking targets mass production in H2 2029.

12:48“In August 2026, SK hynix broke ground on an Indiana HBM packaging and research facility.”

As checked: The United States is supporting domestic HBM advanced-packaging capacity through the SK hynix Indiana project.

13:48“The United States controls certain advanced HBM in defined export contexts. It is not a universal HBM ban; classification, destination, customer, and packaging arrangements matter.”

As checked: Advanced HBM is strategically controlled by the United States because it is a key component of advanced AI and supercomputing systems.

14:08“But spending billions to move more data is only half the answer.”

As checked: Software can materially reduce HBM traffic; FlashAttention reorganizes exact attention to reduce reads/writes between HBM and on-chip SRAM.