How AI Chips Fit Compute And HBM Into One Package
AIThis post was created with the assistance of artificial intelligence (AI).

🔍 Read the full analysis: How AI Chips Fit Compute And HBM Into One Package on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on monitors, keyboards and dev gear

  • Business-only prices and quantity discounts
  • Tax-exempt purchasing
  • Multiple users, one account, clear invoices
As an affiliate, we earn on qualifying purchases.

TL;DR

Modern AI accelerators package compute dies beside stacks of high-bandwidth memory, linking them through silicon interposers and placing the assembly under liquid cooling. Nvidia and AMD designs show how these components scale from one package to racks containing dozens of GPUs, while advanced packaging and memory supply remain constraints. Some specifications are vendor claims, and production capacity, package dimensions and costs are not fully disclosed.

A new hardware explainer published November 22 describes how AI accelerators combine compute dies, stacked high-bandwidth memory (HBM) and silicon interposers in one package, then connect dozens of packages into a rack. The architecture matters because performance depends not only on processing power but also on moving data to the processors—and on whether manufacturers can build the memory and advanced packaging at scale.

The explainer describes the package as a layered assembly rather than a single chip. A package substrate carries power and signals to the surrounding module. Above it, a silicon interposer provides dense connections among compute dies and memory stacks. The compute dies perform the main calculations; HBM stacks sit nearby to supply data through many short connections. A cold plate above the assembly transfers heat to liquid-cooling systems.

HBM is made from DRAM chips stacked vertically and connected by through-silicon vias, or TSVs. Those vertical connections let memory sit close to the compute logic, supporting high bandwidth in a compact area. The source cites the JEDEC HBM4 standard, published in April 2025, as allowing stacks of four, eight, 12 or 16 layers and up to 64 gigabytes per stack. That is a standard capability, not confirmation that every accelerator uses a 16-layer stack or reaches the maximum capacity.

Product examples illustrate different approaches. According to Nvidia’s architecture materials cited in the explainer, its Rubin GPU uses two reticle-limited compute dies and has 336 billion transistors. AMD says its MI350 Series combines eight vertically stacked accelerator complex dies with two I/O dies; the MI355X is listed at 185 billion transistors. The source reports up to 19.2 terabytes per second of memory bandwidth for Rubin, compared with 900 gigabytes per second for Nvidia’s V100. These are product specifications from different generations; they do not by themselves establish application performance or a directly comparable benchmark.

At a glance
reportWhen: Explainer published November 22, 2026;…
The developmentA November 22, 2026 explainer details how current AI accelerators combine compute dies and HBM within advanced packages, and how those packages scale into rack systems.

Packaging Sets the Pace for AI

Putting memory close to compute helps address a central hardware constraint: processors can only work quickly when data reaches them quickly. As AI models and workloads grow, memory bandwidth and capacity influence how much work an accelerator can perform, while packaging determines how many dies and memory stacks can be linked together.

The package also has implications beyond engineering. TSMC’s CoWoS technologies are used to connect logic dies and HBM, but producing large interposers and assembling complex packages requires specialized manufacturing capacity. The explainer cites SK hynix’s expectation that HBM demand will exceed supply through 2030. That is the company’s forecast, not a guarantee of a specific shortage or price. If supply remains tight, access to accelerators could depend as much on memory and packaging availability as on demand for compute itself.

At rack scale, the design draws power and cooling into the same capacity question. Nvidia’s GB200 NVL72 configuration contains 72 GPUs and 36 Grace CPUs, along with network switches and power equipment. Nvidia lists 120 kilowatts of cooling capacity for the rack. Such systems are not just collections of chips: they require facilities able to deliver substantial power, remove heat and maintain high-speed connections among components.

Products Worth Considering

DJI Mic 3 Wireless Microphone (2 Transmitters, 1 Receiver, Charging Case)

DJI Mic 3 Wireless Microphone (2 Transmitters, 1 Receiver, Charging Case)

Anker SOLIX C1000 Gen 2 Portable Power Station with 200W Solar Panel

Anker SOLIX C1000 Gen 2 Portable Power Station with 200W Solar Panel

As an affiliate, we earn on qualifying purchases.

From Silicon Dies to Rack Systems

A reticle is the maximum area a chipmaking scanner can print in one exposure. The source says TSMC describes CoWoS-S interposers of up to about 2,700 square millimeters, or 3.3 times reticle size. For larger packages, CoWoS-L connects smaller silicon bridges within a carrier layer. TSMC’s CEO said on October 15, 2026, that the company was in “real world production” of five-reticle-size packaging and working on larger designs. AMD identifies CoWoS-L for its MI450X and MI455X packages.

In the explainer’s account, Nvidia’s GB200 and GB300 NVL72 systems use 18 compute trays, each with four GPUs and two Grace CPUs, plus nine NVLink switch trays. The Open Compute Project listing cited for the GB200 rack gives dimensions of 600 millimeters wide, 1,068 millimeters deep and 2,236 millimeters high. Nvidia’s current Vera Rubin NVL72 listing retains a 72-GPU, 36-CPU configuration, according to the source. Rack configurations and product specifications can change as vendors update systems.

The source’s 3D models are schematic: it says Nvidia and AMD do not publish the millimeter dimensions of the dies, packages or memory stacks, and that the models do not guess those measurements. That distinction matters because product diagrams can show component order and counts without establishing true physical proportions.

““large silicon interposer area to accommodate various functional top dies, including logic chiplets, with high-bandwidth memory (HBM) cubes stacked over it””

— TSMC, as quoted in the explainer

Dimensions, Supply and Performance

Several important details remain unavailable or conditional. Nvidia and AMD do not publish the physical dimensions of the dies, packages or individual HBM stacks in the cited materials, so the source’s models are schematic. The explainer also does not provide independent measurements of manufacturing yields, package output, delivery times or prices, making it difficult to quantify how strongly packaging limits accelerator supply today.

Vendor specifications and forecasts need to be read as such. The reported bandwidth figures do not show how a particular model performs in real workloads, and SK hynix’s supply outlook is a company forecast. The cited 120-kilowatt cooling figure applies to Nvidia’s GB200 NVL72 rack; the explainer does not provide facility-level energy use or comparable cooling figures for every system. It is also unclear from the source how quickly announced next-generation package designs will reach broad commercial availability.

Watch HBM and Packaging Capacity

The next indicators are production and delivery updates from memory makers, TSMC and accelerator vendors. Those announcements can show whether HBM output and advanced packaging capacity are keeping pace with demand, and whether newer interposer designs are moving from production claims into larger shipments. Product availability statements should be distinguished from shipment volumes and actual customer deployments.

For buyers, the relevant developments will include specifications for shipping systems, the number of accelerators suppliers can deliver, and the power and cooling requirements of complete racks. Until manufacturers publish more detail on capacity, yields and delivery schedules, the effect of packaging constraints on cost and access remains difficult to measure.

Key Questions

What is HBM in an AI accelerator?

HBM, or high-bandwidth memory, is DRAM arranged in vertical stacks and linked by through-silicon vias. It sits close to compute dies so data can move between memory and processors over many connections.

What does a silicon interposer do?

A silicon interposer is a wiring layer that connects compute dies and HBM stacks within a package. It supports denser connections than a conventional circuit board, according to the explainer’s description.

How many GPUs are in an Nvidia NVL72 rack?

The cited Nvidia GB200 and GB300 NVL72 configurations contain 72 GPUs. The source says the GB200 rack also has 36 Grace CPUs, along with compute trays, switch trays and power equipment.

Is HBM supply confirmed to remain short through 2030?

No. SK hynix expects demand to exceed supply through 2030, according to the source, but that is a company forecast. The explainer does not provide independently verified future supply figures.

Are the explainer’s chip diagrams drawn to scale?

No. The source says its 3D models show layer order and component counts schematically. Nvidia and AMD do not publish all the physical dimensions needed to represent the dies, packages and memory stacks to scale.

Source: ThorstenMeyerAI.com

You May Also Like

The Metric Behind Microsoft’s Claim About AI And Doctors

Microsoft’s peer-reviewed MAI-DxO study reported high diagnostic scores on selected cases, but its benchmark does not establish performance in clinical care.

AI Is Helping China’s Trade Surplus Reach A Trillion Dollars Again

China’s customs data show a $1.04 trillion goods surplus through October 2026, alongside sharp growth in chip and computer exports.

AI’s Impact On Work May Be More Complicated Than It Seems

A historical account traces how ATMs first reshaped teller tasks, while branch growth helped sustain jobs before mobile banking drove a later decline.

The SSD Squeeze: Why Storage Joined The Party

Enterprise and consumer SSD prices soar as NAND supply tightens due to AI-driven demand and wafer competition, impacting the entire storage market.