Systems Research and Architecture SRA
a SRA Theses Hybrid SSD

Exploring the Operating-System Integration of a CXL-like Hybrid Storage Device

image

A memory-semantic SSD [generated with AI].

Context

Conventional SSDs expose a block interface: Linux transfers file data into host DRAM and caches it in the page cache. An SSD may also maintain its own DRAM cache, creating two levels of caching and potentially redundant copies. Memory-semantic interconnects such as Compute Express Link (CXL) offer an alternative: device memory can be mapped into the host's physical address space and accessed through CPU loads and stores.

This thesis considers a hybrid SSD combining directly accessible on-board DRAM with a larger flash-backed storage tier. The proposed device exposes a logical address range covering its complete flash capacity, but only a subset of the data is resident in device DRAM at any time. This is a research device model, not a capability automatically provided by CXL. Its interface must define how nonresident data becomes accessible and how dirty data reaches persistent storage. The following figure visualizes the concept:

image

The conceptual SSD design.

Such a device could allow Linux to retain file data on the device rather than copying it into host DRAM for caching. Linux Direct Access (DAX) provides a related approach by accessing memory-like storage without using the ordinary page cache. However, DAX assumes directly accessible storage; a limited, dynamically remapped DRAM tier is not a drop-in replacement.

Problem

This new class of storage devices raises multiple questions:

1. What should the interface look like?

One option would be to make the operation completely transparent, with the device making all read, write, and remapping decisions based on loads and stores issued through the CXL interconnect. Such a design would make the device easy to use but unpredictable in terms of access latency. To mitigate this issue, a memory-mapped bitmap could be used to communicate to the host which blocks are memory-backed at any given time. Going a step further, writing to the bitmap from the host side could be interpreted as a hint about which blocks are likely to be accessed in the near future. A different approach would be an entirely explicit mechanism, more similar to the current NVMe model. Submission and completion queues could be used to give the host full control over the backing state. Accesses to currently unbacked ranges would be illegal in this design.

2. How should this new device type be integrated into the operating system?

Since the on-device DRAM is directly accessible through native load and store instructions, it is no longer necessary to transfer data back and forth blockwise between host memory and the device. As with DAX, data can bypass the page cache. In contrast, depending on the interface model, the operating system is responsible for tracking and controlling which page frames are accessible at any given time. Compared with current page-cache policies, there is now a strict upper limit on usable capacity: the amount of on-device DRAM. How should the system handle this limitation? Would it be necessary to fall back to the conventional model and use additional host memory?

3. How can the characteristics of an unreleased hardware device be emulated accurately?

As there is currently no commercially available hybrid SSD, the question arises: how can software support and interface designs be evaluated accurately? The most obvious approaches to custom-hardware research have major disadvantages: FPGA: accurate, but it requires specific hardware and software licenses, and the development effort is enormous; full-system simulation: very slow, especially for operating-system-based workloads.

To avoid the problems associated with these extreme approaches, we want to choose a middle ground based on the virtualization features of current CPUs. We want to emulate the device in the guest-physical address space of a QEMU virtual machine. The idea is to use the performance characteristics of existing DRAM, CXL, and NVMe hardware to build a hybrid simulation device. To achieve accurate memory-access timing, we could use multi-NUMA systems or a simple CXL 2.0 expansion card.

Goal

The goal is to build the described emulation setup and quantify possible benefits for different applications.

A possible schedule could follow these steps:

  1. Implement the basic functionality (in terms of quality) as a virtual QEMU device.
  2. Implement a corresponding storage driver for the guest kernel (a simple block-device driver).
  3. Add DAX-like capabilities to the device driver to take advantage of the memory semantics.
  4. Quantify baseline performance using microbenchmarks and/or real-world benchmarks.
  5. Adapt the QEMU device to provide more accurate latency emulation.
  6. Evaluate the impact of different interface designs.

Topics: Virtual Memory, MMU, Nested Paging, QEMU, SSD, CXL

References