On Design Principles for Efficient Heterogeneous DRAM-PIM-GPU Systems

Abstract

Heterogeneous DRAM-based processing-in-memory (PIM)-GPU systems promise significant efficiency gains for decode-phase large language model (LLM) inference, particularly in long-output generation, yet current design practices overlook critical factors that determine real-world performance. Through systematic evaluation of diverse architectures and workloads (OPT-7B/70B, Mamba2-2.7B/70B), we reveal three fundamental design principles: (i) static power consumption (DRAM leakage, refresh, and GPU idle power) can dominate the efficiency calculus, causing dynamic-only models to overestimate tokens/s/W by up to 3.85$\times$ for realistic deployments (Mamba2-2.7B, batch size 1, 128 input tokens, and 2,048 output tokens); (ii) decoding performance is monotonically non-decreasing with channel count across all evaluated models and workloads, generally plateauing at high channel counts for low-batch workloads; under a fixed-capacity sweep, all models instead share a common near-optimal hierarchy configuration, with substantially larger misconfiguration penalties for attention-based models; (iii) workload mapping strategies provide bounded improvements (up to 14.0%/17.4% kernel-level latency/energy reduction, up to 5.6% end-to-end gain) and are not primary bottlenecks. Significant efficiency gains require system-wide co-optimization. These principles provide design-space guidance for architects designing the next generation of memory-accelerated LLM systems.

I Introduction

Large Language Models push modern hardware against memory-bandwidth bottlenecks: GPUs excel at compute, but attention, recurrent state updates, and other memory-intensive operations stall on bandwidth. Processing In-Memory (PIM), particularly DRAM-based PIM (DRAM-PIM), offers a promising path forward by bringing computation closer to data and reducing costly data movement [6]. Due to the high maturity of DRAM and low adoption barrier of DRAM-PIM, heterogeneous systems that co-locate or integrate DRAM-PIM units with GPUs, such as those depicted in Fig. 1a-c, are emerging as an attractive platform for decode-phase LLM acceleration in long-output generation [4].

Figure 1: (a) DRAM-PIM chip architecture with reconfigurable channels, ranks, bank groups, and banks. (b) Near-Bank Processing Unit (NBPU) connected to pairs of banks. (c) Heterogeneous DRAM-PIM-GPU systems considered. (d) Decoding performance vs. channel count for GPU+PIM (CXL) (OPT-7B, ISL=2048, OSL=2048); GPU baseline ($\times$1 A100 (80 GiB)) varies batch size.

Although recent work highlights the potential of using DRAM-PIM for LLM execution, the broader architectural implications remain insufficiently understood. Key design factors, namely DRAM hierarchy organization, static power, workload mapping, and interconnect choice, interact in non-obvious ways. For instance, adding DRAM channels can yield diminishing returns when workloads lack sufficient memory-level parallelism (Fig. 1d), and static power from idle components can negate PIM throughput gains entirely at low batch sizes. Through comprehensive system-level evaluation, we derive three actionable design principles that provide concrete guidance for designing efficient heterogeneous memory systems:

• We quantify the impact of static power (DRAM leakage, refresh, GPU idle) on system-level efficiency, showing that neglecting it overestimates tokens/s/W by up to 3.85$\times$ and fundamentally shifts optimal design points;

• We demonstrate that decoding performance is monotonically non-decreasing with channel count across all evaluated models and workloads, generally plateauing at high channel counts for low-batch workloads. Under a fixed-capacity sweep, all models instead share a common near-optimal region; the risk is an approximately 2$\times$ larger fractional misconfiguration penalty for attention-based than SSM-based models;

• Finally, we present RowLocalChunk, a locality-aware workload mapping, showing that it reduces kernel-level latency/energy by up to 14.0%/17.4%, while end-to-end (mapping-level) gains remain bounded (up to 5.6%).

II Background and Motivation

DRAM-based PIM extends commodity memory with low-cost compute capability [6], with feasibility demonstrated in manufactured silicon (Samsung, SK Hynix, UPMEM). In heterogeneous systems, GPUs can handle compute-bound kernels while PIM units accelerate bandwidth-limited operations such as KV-cache accesses and state updates [7, 9], aligning naturally with modern LLMs. Achieving system-level efficiency, however, depends jointly on static power (e.g., DRAM leakage/refresh, GPU idle power during PIM execution), DRAM hierarchy organization (e.g., channels, ranks, bank groups, banks), workload partitioning/mapping (fixed vs. dynamic [5]), data layout [12], and GPU–PIM interconnect choice (e.g., integrated vs. Compute Express Link (CXL)/NVLink-attached [4]), each shaping latency, bandwidth, and power differently. Existing evaluation tools address only subsets of these factors: NicePIM [12] and Pimba [7] target workload mapping; PAPI [5] and AttAcc [9] additionally model interconnects; none jointly capture static power alongside hierarchy exploration for LLM decode. We evaluate all of these factors jointly across OPT and Mamba2 decode workloads.

III Experimental Setup

We study heterogeneous GPU–DRAM-PIM systems across hierarchy organizations, workload mappings, and interconnect options. We instantiate a representative Near-Bank Processing Unit (NBPU) design following Pimba [7]/AttAcc [9]: 640 NBPUs operating at 378 MHz per unit, one per two banks (for the default 40-channel/1,280-bank hierarchy), each supporting MX8 dot-product/multiply–accumulate and accumulation per the cited architectures — an architectural capability we adopt from these proposals rather than independently re-verify at the ALU level, since this simulator tracks NBPU activity only as an opaque per-cycle compute-event count. Active NBPU energy is accounted per executed compute event, scaling with operation count, while idle_pim (Fig. 4) is the co-located PIM-HBM’s precharged-standby background power plus its average refresh power (capacity-normalized as above), charged only to CXL/NVLink co-located configurations and never to integrated DRAM-PIM, which shares the GPU’s own already-installed HBM.

Our goal is comparative design-space evaluation rather than cycle-exact reproduction of a commercial GPU. We therefore calibrate static GPU power using NVML idle measurements, DRAM leakage and refresh using prior experimental DRAM characterization [3] and DRAMPower-style parameters [11], and interconnect latency/bandwidth using published NVLink/CXL values. We adapt and extend ramulator2 [8] (memory) plus attacc_simulator [9] and Pimba [7] (system) to model static power (DRAM refresh/leakage), flexible operation mapping, and both integrated and co-located DRAM-PIM–GPU configurations across interconnect types. This captures GPU idle power during PIM execution, transfer/compute contention, and hierarchy-mapping interplay.

Each system pairs one or more NVIDIA A100 GPUs (80 GiB HBM2e, peak memory bandwidth 1,935 GB/s, 5 HBM stacks, 108 SMs [1]) with DRAM-PIM. We consider three integration strategies, as introduced in Section II: integrated DRAM-PIM sharing the GPU memory bus, and co-located DRAM-PIM connected via CXL (PCIe 6.0) or NVLink3. We hold the physical-channel transfer rate at 3,024 MT/s; aggregate DRAM-PIM bandwidth scales with the configured channel count. HBM2-vs-HBM2e differences fall within our $\pm$20% sensitivity envelope (§IV-A). The DRAM is configured with a refresh interval of 3.9 $\mu$s, matching Pimba’s reported HBM $t_{\mathrm{REFI}}$ [7], and a refresh energy of 4.89 nJ per bank per event (0.72–4.89 nJ sensitivity) – our own density/capacity-scaled modeling estimate, derived using the DRAMPower methodology [11] rather than a value either work reports directly. Where per-unit PIM capacity is fixed but distributed across different numbers of physical banks (Fig. 6), we scale refresh energy per bank event linearly with per-bank capacity:

$$E_{\mathrm{ref,bank}}=4.89\,\mathrm{nJ}\times\frac{C_{\mathrm{bank}}}{64\,\mathrm{MiB}},$$

where 64 MiB is the per-bank capacity of the reference 1,280-bank organization. At fixed $t_{\mathrm{REFI}}$, aggregate refresh power therefore tracks the fixed per-unit capacity rather than the number of banks. A separate, physically co-located PIM-HBM unit (CXL/NVLink-attached) additionally draws a flat 8.41 W/unit precharged-standby background power (2.10–8.41 W sensitivity), our own capacity-scaled estimate for an 80 GiB HBM stack derived from the DDR3L background-power characterization in [3] (no HBM-specific value is directly reported); this value is kept separate from the A100 (80 GiB) system-level power measurement, and this background term is not charged for integrated DRAM-PIM, which shares the GPU’s own already-installed HBM. Context-active GPU power is modeled at a nominal 80 W per A100 board, including installed HBM, based on our observed NVIDIA Management Library (NVML) telemetry. We additionally validate, per GPU, that context power plus dynamic energy over time never exceeds the A100 PCIe 300 W board limit [1]. Table I summarizes all static-power and interconnect parameters (main/nominal values; sensitivity ranges are evaluated in Table II, §IV-A).

TABLE I: Static-power and interconnect parameters. Refresh energy $\times$ 1,280 banks / refresh interval $\approx$ 1.60 W/unit nominal refresh power.

Component	Value
A100 context-active (board, incl. HBM)	80 W
PIM-HBM background power (co-located only)	8.41 W/unit [3]
Refresh energy	4.89 nJ/bank/event [11]
Refresh interval	3.9 $\mu$s [7]
NVLink3 BW (directional)	300 GB/s [1]
NVLink3 latency	1.0 $\mu$s [9]
CXL over PCIe 6.0 BW (directional)	120 GB/s

Following common practice [7], prefill operations are executed entirely on the GPU(s), as they are predominantly compute-bound. Our analysis targets decode, which dominates latency in long-output chat/code-generation deployments. Self-attention and state update operations, which involve high-volume, fine-grained data access patterns naturally suited to in-memory computation, are offloaded to the DRAM-PIM chip(s). All remaining operations execute on the GPU(s). DRAM-PIM computation uses the MX8 format [10], storing KV-cache/state at 1.0 byte/value (metadata included) for both performance and capacity accounting. We additionally report an idealized GPU-INT8 baseline (“GPU+Q”) alongside GPU-FP16: KV-cache/state uses 1.0625 bytes/value for capacity accounting (1 byte plus one BF16 scale factor per 32-value group) but the idealized 1.0 byte/value rate for traffic/latency/energy, with no quantization overhead. Capacity is checked independently per physical allocation domain (weights, activations, KV/state); co-located DRAM-PIM adds a separate 80 GiB PIM-HBM pool per unit, while integrated DRAM-PIM shares the GPU’s own HBM; an equal-unit, not iso-area/iso-cost/fixed-power-budget, comparison, for which integrated results provide the shared-capacity reference. The $\leq$6.25% storage-density gap between GPU-INT8 and MX8 is a second-order effect next to this topology-level capacity difference, but can determine feasibility outright for workloads near the 80 GiB boundary (e.g., OPT-7B at B=128, ISL=2048, OSL=128). Transfer latencies ($k$ bytes) over NVLink3 and CXL/PCIe 6.0 are modeled with a canonical fixed-latency-plus-bandwidth link model:

$$t_{s}(k)=L_{s}+k/B_{s},\quad s\in\{\text{NVLink},\text{CXL}\},$$

with NVLink directional bandwidth $B_{\text{NVLink}}\!=\!300$ GB/s and latency $L_{\text{NVLink}}\!=\!1.0\,\mu$s, and CXL directional bandwidth $B_{\text{CXL}}\!=\!120$ GB/s and fixed latency $L_{\text{CXL}}\!=\!4l_{p}\!+\!2l_{e}\!+\!l_{s}\!=\!180$ ns, with port latency $l_{p}\!=\!25$ ns, PCIe latency $l_{e}\!=\!30$ ns, switch latency $l_{s}\!=\!20$ ns. This replaces an earlier AttAcc-derived empirical curve fit for NVLink calibrated to a bidirectional-aggregate 600 GB/s figure that does not correspond to a one-way transfer rate. GPU$\leftrightarrow$GPU tensor-parallel collective communication (used by every 8-GPU/70B result in this paper) is modeled separately from the GPU$\leftrightarrow$PIM link above, via a canonical recursive halving/doubling all-reduce over the same directional NVLink bandwidth: $t_{\text{AR}}=2\log_{2}(P)L_{\text{AR}}+2\frac{P-1}{P}\cdot\frac{k}{B_{\text{NVLink}}}$, with per-round collective latency $L_{\text{AR}}\!=\!6.06\,\mu$s chosen to preserve the empirical intercept of the AttAcc curve fit it replaces, which is retained in our codebase only as a legacy comparison mode and does not produce any result reported in this paper. Our 8-GPU results therefore combine PCIe-A100 board specifications (1,935 GB/s HBM2e, 300 W board limit) with an NVLink3/NVSwitch-style all-to-all collective network.

Algorithm 1 RowLocalChunk mapping.

1:$chunk_{r}\!\leftarrow\!\lfloor\text{atom\_sz}/\text{elem\_w}\rfloor$;  $chunk_{c}\!\leftarrow\!\lfloor\text{page\_sz}/\text{atom\_sz}\rfloor$

2:$N_{r}\!\leftarrow\!\lceil d_{\text{model}}/chunk_{r}\rceil\!\cdot\!B$;  $N_{c}\!\leftarrow\!\lceil L_{\max}/chunk_{c}\rceil$

3:$\text{quota}\leftarrow\text{distribute}(N_{r}N_{c},\,U)$ $\triangleright$ largest-remainder

4:Init $\text{row\_own}[r],\text{col\_own}[c]\leftarrow\varnothing$;  $N_{core}\!\leftarrow\!\max(1,\text{heads}\!\times\!B)$

5:for all $(r,c)$ in row-major order do

6:  $C\leftarrow\{u\in U\mid\text{quota}[u]>0\}$

7:  $chosen\leftarrow\begin{cases}\min(C\cap\text{row\_own}[r]),&\text{if }\neq\emptyset\\ \min(C\cap\text{col\_own}[c]),&\text{else if }\neq\emptyset\\ \arg\min_{u\in C}\text{load}[u],&\text{else}\end{cases}$

8:  Assign $(r,c)\!\to\!chosen$; update quota, ownership

9:  Core: reuse bank’s core, else $idx\!\!\mod N_{core}$ (round-robin)

10:end for

11:return per-unit chunk mapping $\mathcal{M}$

Figure 2: Allocation of 6 tensor chunks ($3$ rows $\times$ $2$ columns) to 4 banks. Heap-based (Pimba) assigns chunks to globally least-filled rows, potentially fragmenting tensor rows across banks. In contrast, RowLocalChunk performs a quota-based triage (row-owner $\to$ col-owner $\to$ least-loaded).

Figure 3: Decoding performance for (a) OPT-7B, (b) OPT-70B ($\times$8), (c) Mamba2-2.7B, (d) Mamba2-70B ($\times$8). DRAM hierarchy: $ch\!=\!40,pseudochannel\!=\!2,bg\!=\!4,bank\!=\!4$.

We evaluate two standard decoder-based transformers with KV caching, representative of GPT-like LLMs, OPT-7B and OPT-70B [13], and two State Space Models: Mamba2-2.7B using the original Mamba2 configuration [2], and a scaled Mamba2-70B model with 192 decoder blocks, hidden dimension 7680, 80 heads, head dimension 384, state dimension 192, and 2-byte data elements. For the smaller networks ($\leq 7B$), a single GPU-DRAM-PIM unit is used. For the larger ones, 8 units are used. Each unit is a tensor-parallel shard pairing one A100 (80 GiB) with a DRAM-PIM chip; weights, activations, and KV cache are sharded uniformly. At B=128 and ISL=OSL=2048, the OPT-style 70B FP16 KV cache exceeds the aggregate 640-GiB GPU capacity, while its one-byte representation alone reaches 640 GiB before accounting for headroom; we therefore mark infeasible baselines as OOM. The GPU-only baseline uses the same number of A100 GPUs as the heterogeneous configuration; total memory capacity is not matched for co-located PIM, while integrated PIM provides the shared-capacity reference. Unless otherwise stated, a fixed DRAM hierarchy of 40 channels, 2 pseudochannels, 4 bank groups, and 4 banks is used to facilitate comparison with related work. For all baselines, the equivalent number of GPUs (to units) is used.

III-A Workload Mapping Strategy

As baseline, we use Pimba’s heap-based allocator [7], which assigns chunks to the least-filled DRAM row via a min-heap. This approach is simple and load-balanced, but it scatters adjacent chunks across distant banks/rows, producing high row-switching and nondeterministic layouts. We introduce RowLocalChunk (Algorithm 1), a simple deterministic locality-aware mapping prioritizing spatial locality while preserving global balance. Per-unit integer quotas are computed via largest-remainder; chunks are traversed in row-major order and assigned via three-stage triage: same-row owner, same-column owner, then least-loaded bank. This mitigates the scattered allocations of heap-based methods (Fig. 2).

Figure 4: Power and latency breakdown for OPT-70B and Mamba2-70B ($\times$8 units, B=128, ISL/OSL=2048, GPU+PIM (CXL)).

IV Results

IV-A Static Power Consumption

Figure 5: Decoding performance vs. channel count [1–64] with fixed pseudochannel/bg/banks (2,4,4) for (a) OPT-7B, (b) OPT-70B ($\times$8), (c) Mamba2-2.7B, (d) Mamba2-70B ($\times$8). GPU(s) vs. GPU+PIM (CXL). All configurations include static power. PIM capacity per unit scales with channel count, from $\sim$2 GiB at 1 channel to $\sim$128 GiB at 64 channels (OPT-70B/Mamba2-70B use 8 units, so total system capacity is 8$\times$ these values).

We evaluate tokens/s/W across batch size (B), Input Sequence Lengths, and Output Sequence Lengths, with and without static power (Fig. 3). Including static power notably reduces relative tokens/s/W, especially at large B and ISL. The 3.85$\times$ headline is relative overestimation (largest where dynamic-only models predict the largest PIM gains, hence at Mamba2-2.7B, B=1), not absolute PIM benefit. At B=1, default-hierarchy PIM matches the GPU baseline; throughput-amortized gains emerge only at B$\geq$32 (a single-channel hierarchy recovers $\leq$4% even at B=1; see §IV-B). Trends are broadly consistent across model scales, though adding more units diminishes per-unit gains as added static power offsets throughput. Integrated and co-located (CXL, NVLink) configurations achieve similar performance, suggesting that interconnect choice is secondary to static power management. Despite the 2.5$\times$ NVLink3-vs-CXL bandwidth gap, transfer volume is small (PIM-offloaded kernels exchange only compact partial sums and KV slices); the comm component remains a small fraction of decode latency and energy across all three configurations, with static power dominating the envelope. For the modeled decode offload strategy, interconnect bandwidth is not the dominant efficiency limiter.

Fig. 4 reports the power/latency breakdown for OPT-70B and Mamba2-70B under GPU+PIM (CXL). Static components (leakage, refresh, GPU idle) account for 66–69% of total system power, dominating the efficiency envelope. The high idle-GPU share arises from context-active board power that persists when no GPU kernels are executing, compounded by frequent synchronization/data-transfer phases of low utilization. Time spent on gemv is substantially larger for transformers than for SSM state updates, reflecting different memory access patterns: attention reads scale with sequence length, while SSM updates operate over fixed-size buffers. Sensitivity analysis (Table II) confirms robustness of the reported DRAM-PIM advantage: independently perturbing DRAM frequency, CXL bandwidth, GPU–GPU all-reduce latency, GPU context-active power, PIM-HBM background power, and PIM refresh energy, each across its full documented sensitivity range, places PIM’s (CXL) tokens/s/W advantage over GPU (OPT-70B, B=28 – the capacity-valid substitute for the submitted B=128 stress point at this ISL/OSL, which OOMs for GPU-FP16 – ISL=OSL=2048, $\times$8 units) between 1.35$\times$ and 2.09$\times$ when static power is included (nominal 1.63$\times$). Excluding static power, the advantage (2.01$\times$, 4.198 tok/s/W nominal) is exactly invariant across every range tested.

TABLE II: Sensitivity analysis: PIM (CXL) tokens/s/W vs. GPU, OPT-70B, B=28, ISL=OSL=2048, $\times$8 units, incl. static power.

Parameter	Range	tok/s/W (vs. GPU)
GPU context-active power	54–105 W	0.863–1.333 (1.35–2.09$\times$)
PIM-HBM background power	2.10–8.41 W	1.044–1.102 (1.63–1.72$\times$)
DRAM frequency	0.8–1.2$\times$ nominal	1.044–1.060 (1.63–1.66$\times$)
CXL bandwidth	32–120 GB/s	1.031–1.044 (1.61–1.63$\times$)
PIM refresh energy/event	0.72–4.89 nJ	1.044–1.056 (1.63–1.65$\times$)
GPU–GPU all-reduce latency	1.0–6.06 $\mu$s	1.041–1.046 (1.63–1.64$\times$)

IV-B DRAM Hierarchy Co-optimization

Given a fixed pseudochannel, bank groups, and banks (2, 4, 4), we vary the number of channels (Fig. 5). Per-unit PIM capacity scales with channel count, from approximately 2 GiB at 1 channel to 128 GiB at 64 channels, matching the canonical 80 GiB only at 40 channels; Fig. 5 therefore represents whole-unit provisioning, where capacity and parallel resources scale together with channel count, rather than an isolated channel-count experiment. Decoding performance is monotonically non-decreasing with channel count for every model and workload we evaluate, generally plateauing at high channel counts for low-batch workloads: more channels provide both more independent memory streams for row-locality-sensitive PIM kernels (gemv, state_update). At B=1, gains from added channels are modest (5–32% depending on model and OSL); performance generally begins to plateau between 32 and 64 channels, depending on the model and sequence length. At B$\geq$32, every capacity-valid configuration for every model peaks at the largest channel count we evaluate (64), with gains of 140–2300% over a single channel.

Figure 6: t-SNE (channels, pseudochannels, ACT4-group count) for the 22 (of 25) configurations in the extended fixed-capacity logical design space; 3 excluded as infeasible (Sec. IV-B). Physical bank count (16–5,120) and NBPU count (8–2,560) vary across the sweep; bank groups/banks fixed at (4,4); PIM capacity fixed at 80 GiB per unit. (a-d) Raw tokens/s/W per model, each embedded independently; (e) normalized tokens/s/W. B=28, ISL/OSL=2048, GPU+PIM (CXL).

Further, in Fig. 6, we fix PIM capacity at 80 GiB per unit and vary its distribution across channels/pseudochannels, holding bank groups/banks fixed at (4,4); the sweep is parametrized as (channels, pseudochannels, num_act4_groups) with their product fixed at 320, where num_act4_groups is the logical count of 4-bank ACT4 activation groups per pseudochannel (distinct from the fixed physical bank-group count of 4) that this sweep varies to hold per-unit capacity fixed while the physical bank count changes. Of 25 mathematically distinct triples, 3 require a refresh-barrier latency exceeding the simulator’s fixed refresh interval for any workload and cannot complete. All four models agree closely on the best configuration – (40,2,4) and (80,1,4) both reach within 0.3% of each model’s own peak tokens/s/W; however, they differ sharply in sensitivity to a poor choice. The worst configuration, (2,2,80), retains only 33.6%/51.6% of peak for OPT-7B/OPT-70B versus 67.3%/76.8% for Mamba2-2.7B/Mamba2-70B.

Figure 7: Comparison of Pimba and RowLocalChunk for OPT-70B (top) and Mamba2-70B (bottom). Columns show PIM-kernel latency, PIM-kernel energy, and end-to-end tokens/s/W. Crosses indicate OOM/invalid configurations.

IV-C Workload Mapping Evaluation

For both Pimba and RowLocalChunk (§III-A, Algorithm 1), DRAM commands are interleaved across banks/pseudochannels/channels, so locality improvements are not masked by controller-level parallelism (Fig. 7). RowLocalChunk reduces kernel latency by 4.4–14.0% and kernel energy by 6.3–17.4% across evaluated OSLs by cutting activate/precharge cycles for adjacent chunks. End-to-end decode gain is modest (0.6–5.6%) because mapping governs only the PIM portion: PIM kernels are a fraction of total decode energy once GPU compute, communication, and static idle power are included (Fig. 4), compressing kernel-level gains.

V Conclusion

Through systematic evaluation of heterogeneous DRAM-PIM-GPU systems across LLM decode workloads, we established three design principles. First, static power can dominate the efficiency calculus and must be included in system-level evaluations. Second, decoding performance is monotonically non-decreasing with channel count across all evaluated models and workloads, generally plateauing at high channel counts for low-batch workloads; under a fixed-capacity sweep, all models instead share a common near-optimal region. Third, workload mapping provides bounded gains and is not the dominant bottleneck; significant efficiency gains require system-wide co-optimization. This motivates architectural reconfigurability.