In the first two installments of “Specialized RAM”, we explored how memory evolved from conventional DDR/GDDR to new frontiers. Now, our journey dives into the latest in memory innovation – High Bandwidth Memory (HBM) and Storage-Class Memory (SCM) – plus how they stack up against “ordinary” RAM. We’ll see why GPUs and AI accelerators are adopting these exotic memories, and how future technologies like HBM4 and memory pooling (via CXL) promise even more dramatic shifts. Strap in for a deep dive into memory technologies that power supercomputers, data centers, and the AI revolution.
What is HBM (High Bandwidth Memory)?
Imagine a highway where instead of just widening lanes, engineers magically stack lanes vertically – adding more lanes without spreading out. That’s essentially what High Bandwidth Memory (HBM) does. HBM is a 3D-stacked DRAM that sits right next to (or even on top of) a processor die (GPU/CPU) via a silicon interposer. By stacking 4–16 DRAM dies and connecting them with thousands of tiny through-silicon vias (TSVs), HBM achieves an extremely wide data bus (often 1024 bits or more). This allows massive data throughput even at moderate clock rates.
HBM1 (the original spec) debuted with 1 Gb/s per pin and up to eight dies in a stack. But each new generation brought faster speeds and higher capacities. For example, HBM2 doubled the data rate to 2 Gb/s per pin (256 GB/s per stack) and allowed up to 8 GB per stack. HBM2E (an improved HBM2) went even further: up to 2.5 TB/s per stack (307 GB/s) and support for 12-die stacks (24 GB).
By 2022, HBM3 set the bar even higher. It runs at 6.4 Gb/s per pin with 16-die stacks of 32 Gb each (total 64 GB per device), yielding about 819 GB/s per stack. Next came HBM3E (2023), which bumps the speed to 9.6 Gb/s per pin and squeezes out ~1229 GB/s per stack. At these speeds, a single GPU card with multiple HBM3E stacks can reach terabytes per second of memory bandwidth. These stacks also remain surprisingly compact – even a 64 GB HBM3E module is roughly the size of a postage stamp when laid on the interposer!
HBM’s efficiency comes from this width. Even though each HBM pin may run slower than GDDR or DDR, the aggregate bandwidth is enormous. For example, a 4-stack HBM2E memory system can achieve ~410 GB/s, whereas a single DDR5-4800 channel only delivers ~33.6 GB/s. And all that packed bandwidth comes with lower power per bit: HBM3E uses a core voltage of 1.1 V (down from 1.2 V in HBM2E) and low-voltage I/O (0.4 V), reducing energy usage. In short, HBM packs many DRAM “lanes” into a tiny footprint for huge bandwidth with good energy efficiency.
Key HBM generations:
- HBM (Gen1) – ~128 GB/s per stack; up to 4 GB per stack.
- HBM2 – up to ~256 GB/s per stack; up to 8 GB per stack.
- HBM2E – ~307 GB/s per stack; up to 24 GB (12×2 GB) per stack.
- HBM3 – ~819 GB/s per stack; up to 64 GB (16×4 GB) per stack.
- HBM3E – ~1229 GB/s per stack; same 64 GB capacity but faster speed.
Unlike regular DRAM, HBM packages multiple dies on a single silicon base with TSVs, all mounted with sub-millimeter precision. This “2.5D/3D” magic keeps all chips very close, slashing trace lengths and capacitance. The result is lower latency and power per bit – although HBM’s absolute latency is still in the tens of nanoseconds (comparable to DDR) due to the DRAM technology. In practice, HBM’s massively parallel interface (e.g. sixteen 64-bit channels per stack) and wider bus make up for any extra cycle time.
HBM’s complexity (silicon interposers, precision packaging) means it’s used for premium high-performance applications. Nvidia and AMD use HBM on their top-tier GPUs and accelerators. For instance, Nvidia’s H100 GPU packs six or more HBM3E stacks for about 3.35 TB/s of memory bandwidth. (The H100’s 80 GB of HBM3E sits directly on the GPU silicon.) AMD’s MI250X/H golf GPU boasts 128 GB of HBM2E (two 64 GB stacks) for 3.2 TB/s of bandwidth. These memory speeds are crucial for data-heavy tasks like AI training or scientific simulation, where moving petabytes through the memory system fast is the name of the game.
What is SCM (Storage-Class Memory)?
While HBM tackled the speed problem, Storage-Class Memory (SCM) addresses the memory vs. storage gap. SCM (sometimes called persistent memory or NVDIMM) blends the high capacity and persistence of storage with the low-latency, byte-addressability of DRAM. In other words, it’s a “memory” that can hold data even when power is off.
The poster child of SCM is Intel Optane™ DC Persistent Memory (based on 3D XPoint technology). Optane PMem modules plug into standard memory slots (DDR4 DIMM slots) on servers and can operate in two modes: Memory Mode (volatile, expanding system RAM transparently) or App Direct Mode (persistent, byte-addressable storage). A key benefit is sheer capacity: Optane PMem 200 series supports up to 512 GB per DIMM (and 6 TB per CPU socket) – far beyond what DRAM can economically achieve. It delivered “32% more bandwidth on average” over its predecessor, making it useful for large in-memory databases, big data analytics, and caching. In effect, Optane PMem lets a server hold massive datasets (like large tables or graph in-memory) without needing prohibitively expensive DRAM.
On the performance side, SCM is slower than DRAM but much faster than NAND flash. For example, Optane PMem (when accessed in memory-app-direct mode) exhibits read latencies ~350 ns – roughly 10× slower than DDR4 (~30–50 ns), but 30× faster than an SSD (~10,000 ns). Importantly, its random-read performance is very strong compared to flash. That means workloads with big memory footprints (e.g. AI training jobs, large VM images, real-time databases) can benefit: data moves in and out of “memory” almost as fast as DRAM, without the need to swap from disk.
Beyond Intel’s Optane, the SCM landscape is evolving. 3D XPoint successors (like Chinese firm Numemory’s chips) aim to deliver similar byte-addressable NVM. For example, Numemory recently announced 64 Gb SCM chips (with a standard NAND interface) rivaling 2nd-gen Optane. Meanwhile, emerging technologies like MRAM (magnetoresistive RAM) and ReRAM (resistive RAM) are being developed for faster non-volatile memory. MRAM uses tiny magnetic structures (MTJs) to store bits and can replace SRAM or even DRAM in some designs, offering sub-10 ns speed and high endurance. ReRAM stores data via programmable resistance and promises high density; it already serves as embedded non-volatile memory in some chips. These NVMs are still maturing, but they herald a future where persistence and memory speed could coexist on-chip.
In summary, SCM fills the gap between memory and storage: it offers much larger capacities than DRAM at lower cost, with better performance than SSDs. Intel Optane is the most prominent example today, used in data centers for things like large database caching, in-memory analytics, and virtualization (because dozens of VMs can share that one big memory pool). As persistent memory software ecosystems (APIs like DAX or emerging CXL-attached memory) evolve, SCM will become a standard tier in enterprise architectures.
Comparing HBM, GDDR, DDR, and SCM
How do these specialized memories stack up against “ordinary” RAM? Here are the broad strokes:
- Bus Architecture and Bandwidth: HBM uses a very wide, short bus. A single HBM stack provides a 1024-bit interface (multiple 64-bit channels) at moderate speed, giving hundreds of GB/s per stack. In contrast, a DDR channel is only 64 bits wide – even at DDR5-4800 it tops out around 38.4 GB/s. GDDR (graphics DDR) trades some width for high pin speeds: a GDDR6 chip (32-bit) running 16 Gb/s per pin still only achieves ~64 GB/s, and a typical GPU uses many chips (e.g. 10 chips for a 320-bit bus to get ~640 GB/s total). SCM like Optane has a narrower interface (32 GB/s per DDR4 channel or so), so its raw bandwidth per module (~8 GB/s for reads) is much lower. Table below summarizes typical numbers.
- Latency: Conventional DDR and GDDR have latencies on the order of a dozen nanoseconds (tens of nanoseconds of access time). HBM’s intrinsic latency is similar or slightly lower (thanks to shorter wire lengths), so also in the 10–20 ns range. By contrast, SCM (Optane) is in the hundreds of nanoseconds. This means DRAM/GDDR/HBM remain far faster per access, but the tradeoff is their capacity is smaller. SCM compensates with huge size, and is often used in modes that tolerate its higher latency (e.g. caching or persistent loads in big-data apps).
- Power: HBM achieves high throughput with good energy efficiency. For example, an early HBM2 stack (8 GB) consumed about 3.75 W to deliver ~256 GB/s. In practice, HBM stacks typically draw on the order of 10–15 W each (roughly 0.01–0.02 W per GB/s of bandwidth) – lower than equivalently fast GDDR5/X configurations. GDDR6 chips (say 16 Gb, 2 GB devices) consume a few watts each (2–3 W) at top speed, so a full GPU board may draw 20–30 W for memory. DDR5 DIMMs use a few watts per module (often under 5 W when active). Optane DIMMs draw more (10–15 W each), reflecting their complexity and larger capacity – though per-byte they can be efficient because each module can replace multiple DRAM DIMMs. Notably, HBM3E/3 reduced core voltage to 1.1 V and I/O to 0.4 V to keep power in check even at high speed.
- Density: On-chip DRAM density is growing but limited. A modern DDR5 DRAM chip is on the order of 16–32 Gb (2–4 GB), with DIMMs up to 64 GB. GDDR6 chips are typically 16 Gb (2 GB) or 18 Gb, giving maybe 16–32 GB per card. HBM stacks pack up to 64 GB in one package (16 dies × 32 Gb) – far more than a single DDR chip. SCM modules are the champions of capacity today: Intel Optane DC DIMMs come in 128, 256, and 512 GB sizes, which is orders of magnitude larger than any DRAM DIMM.
Putting some numbers side by side:
| Memory Type | Bandwidth | Latency | Power | Max Density |
|---|---|---|---|---|
| HBM3E (stack) | ~1229 GB/s | ~10–20 ns | ~10 W per stack | 64 GB per stack |
| GDDR6 (per chip) | ~64 GB/s | ~12 ns | ~3 W per chip | 2–8 GB per chip |
| DDR5-4800 (per 64b chan) | ~38.4 GB/s | ~15 ns | ~3–5 W per DIMM | 64 GB per DIMM |
| Optane PMem DIMM | ~8 GB/s (per 256B read) | ~350 ns | ~15 W per DIMM | 512 GB per DIMM |
These numbers illustrate the trade-offs. HBM3E stacks crush conventional memory in raw bandwidth, at moderate power and good density. GDDR6 chips offer high peak speed and are cheaper, making them ideal for 90% of GPU tasks (games, graphics, normal ML) where a few hundred GB/s is enough. DDR5/4 serve general-purpose PCs and servers, balancing decent throughput with lower cost and very low latency. Optane PMem (and future SCMs) stake out huge capacity and persistence, at the cost of higher latency and lower throughput.
Use Cases in Real Systems
Graphics and AI GPUs: High-bandwidth demands are most obvious in GPUs and AI accelerators. Consumer and workstation GPUs (e.g. NVIDIA RTX 40 series, AMD Radeon RX 7000 series) typically use GDDR6/GDDR6X memory, which provides up to ~1 TB/s on flagship models (e.g. RTX 4090, Radeon PRO W7900). These satisfy most gaming and visualization workloads. But NVIDIA’s data-center and AI GPUs (A100/H100 series) and AMD’s Instinct accelerators (MI200/MI300 series) go with HBM:
- NVIDIA H100 (Hopper): Uses HBM3 (80 GB, 3.35 TB/s) on its SXM5 package.
- NVIDIA A100 (Ampere): Used HBM2e (up to 80 GB, 2.0 TB/s) in its SXM modules.
- AMD Instinct MI250X: Packs 2×64 GB HBM2e (128 GB) for ~3.2 TB/s.
- AMD Instinct MI210/MI211: 64 GB HBM2e, ~1.6 TB/s.
These GPUs serve AI training, HPC simulations, scientific computing, and high-end graphics, where every byte-per-second boosts performance. For example, the H100’s 3.35 TB/s bandwidth feeds NVIDIA’s tensor cores during massive deep learning jobs, and its NVLink fabric connects multiple HBM-equipped GPUs with minimal bottleneck. AMD’s MI250 series (deployed in the Frontier supercomputer) similarly uses HBM2e to shred memory bottlenecks for exascale workloads. In such systems, the huge bus-width of HBM (often 5120–8192 bits per GPU) means data moves in parallel like a firehose.
By contrast, mainstream GPUs like the RTX 4000 series stick with GDDR6X, which is simpler and cheaper. According to Exxact, 90% of applications don’t even fully saturate a GDDR6 system, so HBM’s extra bandwidth would bring minimal gains there. Only top-tier AI or HPC setups usually invest in HBM. That said, AMD used HBM even in some gaming cards (e.g. Radeon R9 Fury X), but newer gaming-focused GPUs have mostly moved back to GDDR6 to cut cost.
Servers and Data Centers: On the server side, SCM is making waves. Companies like SAP, Oracle, and Microsoft have deployed Intel Optane DC Persistent Memory in their data centers to run huge databases (e.g. SAP HANA) entirely in memory. Cloud providers use Optane to improve virtualization density and speed up storage tiers. For example, a server with 6 TB of Optane DIMMs per socket can hold much more in-memory data than one with DRAM alone, boosting analytics workloads.
Additionally, server GPUs (e.g. NVIDIA A100/H100 on HGX platforms) rely on HBM just as we described. Even in heterogeneous systems, Optane often sits alongside HBM: for instance, a supercomputer node might have CPU DRAM + Optane PMem as “system memory” and multiple HBM-GPUs for acceleration. In this tiered model, big datasets live (nearly) in memory, with SSDs only needed for cold storage.
AI/ML Workloads: Modern machine learning especially loves bandwidth. Training large models (think GPT-scale) often saturates GPU memory channels. HBM-equipped GPUs like H100 and MI300 are ubiquitous in AI datacenters because the training performance scales almost linearly with memory throughput. Even for inference, high-bandwidth memory allows larger batch sizes and lower latency. While GDDR6 GPUs (like RTX series) can do most ML tasks, data-center accelerators with HBM save crucial time in massive parallel training jobs.
Other Applications: HBM also appears in specialized niches beyond graphics. Some FPGAs (e.g. Xilinx/AMD Versal) offer HBM for data acceleration; networking ASICs use HBM for packet buffering; and high-end image processing (like 8K video or real-time 3D rendering) can benefit. SCM/Optane finds use in any system requiring huge linear address space – for example, in-memory databases, genomics (whole-genome analysis), large-scale graph processing, and virtualization (one host RAM for many VMs). Even PC motherboards (some high-end Xeon boards) include Optane slots to handle massive memory-disk caches for big-data workloads.
Technical Comparison Table
To summarize the characteristics discussed, here’s a comparative table. (Values are representative for modern tech.)
| Memory Type | Bandwidth | Latency | Power | Density (max) |
|---|---|---|---|---|
| HBM3E (stack) | ~1229 GB/s | ~10–20 ns | ~10 W per stack (4–Hi) | 64 GB (16×4 GB) |
| GDDR6 (per 2 GB chip) | ~64 GB/s | ~12 ns | ~2–3 W per chip | 2–8 GB |
| DDR5-4800 (64-bit) | ~38.4 GB/s | ~15 ns | ~3–5 W per DIMM | 64 GB (per DIMM) |
| Optane DC PM (per DIMM) | ~8 GB/s (read) | ~350 ns | ~15 W per DIMM | 512 GB |
Key points from the table:
- HBM3E stacks dwarf the others in bandwidth, thanks to their 1024-bit-wide interface.
- GDDR6 chips are much narrower buses (32-bit) but run at very high pin rates.
- DDR5 channels are intermediate in width and speed.
- Optane (SCM) offers enormous capacity per module but at higher latency (hundreds of ns).
The Future of Specialized Memory
What comes next on this horizon of crazy-fast memory? Several exciting trends are on the radar:
- HBM4 and beyond: JEDEC has projected HBM4 by the mid-2020s, potentially doubling speeds or capacities again. The Wikipedia HBM spec table suggests HBM4 around 2026 with ~1638 GB/s per stack. The details are still under wraps, but we can expect even higher data rates (maybe >10 Gb/s per pin) and possibly wider interfaces (e.g. 2048-bit total) or more layers. Such HBM4 would prepare for exascale computing and next-gen AI accelerators (e.g. Nvidia’s Blackwell architecture or AMD’s future GPUs).
- Persistent memory evolution: With Intel discontinuing Optane drives, the SCM space is open. Chinese firms like Numemory are developing Optane-like SCM (as in Tom’s Hardware news), and others are researching new materials (like 3D XPoint successors, PCRAM, MRAM). We may see MRAM-based persistent memory that operates much closer to DRAM speeds, or ReRAM modules with ultra-high endurance for enterprise flash alternatives. Software stacks (file systems, OS) are also evolving to treat NV memory as a standard tier.
- CXL and memory pooling: The Compute Express Link (CXL) interconnect is revolutionizing how memory is accessed across devices. CXL 2.0 introduced memory pooling, and CXL 3.0 expands it drastically. Synopsys notes that CXL-attached memory could scale into petabytes of shared pool. Imagine multiple servers or GPUs all tapping into a giant, coherent memory pool (including DRAM, HBM, Optane, even storage) like one global address space. This disaggregated memory could break traditional limits: for example, a supercomputer might place its entire dataset in one fabric-attached pool of DRAM+PMem, without manual data sharding. Early CXL hardware (Intel, AMD, IBM) is just appearing now, but by 2025+ we expect “memory expanders” – boxes of DRAM or persistent memory – that any CPU/GPU can use. This trend effectively treats memory as a networked resource.
- DDR6 and on-chip memory: Conventional RAM won’t stand still either. DDR6 is on the horizon (higher speeds, new signaling). Also, advanced packaging techniques (like chiplets or substrate stacking) could allow more DRAM to sit right on top of CPUs (beyond current Embedded DRAM techniques). New CPU architectures may incorporate some amount of NVM on-package for instant-on or checkpointing.
In short, we’re moving into an era of hierarchical, heterogenous memory. HBM, GDDR, DDR, SCM (and soon MRAM, ReRAM, and anything-on-CXL) will all coexist, each used where it fits best. For example, an AI system might use DDR5 for the OS, HBM for the GPU’s tensors, and a petabyte CXL pool of DRAM+PMem as ultra-fast scratch space.
From the wide 3D lanes of HBM to the persistent caverns of SCM, specialized memory is reshaping computing. HBM3E stacks deliver unprecedented throughput for HPC and AI, while Intel Optane and its successors give servers mind-boggling memory capacity with persistence. Compared to traditional DDR/GDDR, these memories trade cost and simplicity for higher bandwidth or capacity. The payoff is in real-world gains: faster AI training (Nvidia H100, AMD MI300), larger databases (in-memory analytics), and new system architectures (memory pooling with CXL).
Electronics students and tech enthusiasts should appreciate that “memory” is no longer one-size-fits-all. Just as SSDs brought tiers of storage, HBM and SCM have brought tiers of memory. As we look ahead, expect even bolder innovations – HBM4 paving the way for super-fast accelerators, MRAM or ReRAM sneaking into everyday processors, and fabrics like CXL turning remote memory into a local resource. By 2025 and beyond, understanding these specialized RAM types will be key for designing anything from gaming PCs to exascale supercomputers.