Extending Shared Virtual-Memory Objects Across System Boundaries

Typ der Arbeit: Bachelorarbeit
Status der Arbeit: laufend
Projekte: ParPerOS
Betreuer: Alexander Halbuer, Daniel Lohmann
Bearbeiter: Noel Freimuth

Disaggregated memory pools connected with high-bandwidth/low-latency pipes [Generated with AI].

This is an extension of a preceding bachelor's thesis that designed a mechanism for page-granular access control in a multiple-reader/single-writer semantic. The follow-up task is the adaptation and evaluation of this mechanism on a (simulated) disaggregated CXL platform.

Context

The emerging Compute Express Link (CXL) standard extends the border of main memory to a broader circle. In a nutshell, it allows byte-granular memory access via the PCIe interface, enabling devices (e.g., GPUs) to access and cache host memory, as well as hosts to extend memory capacity with extension cards. More recent versions of the CXL standard (2.0/3.0) go even further, allowing for memory disaggregation with centralized memory pools. In combination with coherent access across machines, this allows for efficient communication via shared memory. Unfortunately, due to the high costs of tracking cache line states, we expect only a small fraction of a memory pool to be cache-coherent across machines. For the residual, dominating part of the memory, software mechanisms must be employed to ensure synchronization.

Problem

With Morsels, we introduced a novel memory-management paradigm that shifts from the management of individual pages to larger virtual-memory objects, technically represented as subtrees of the page-table hierarchy. This reduces management overhead and enables very fast transfer between address spaces. With the extension of the memory domain with shared CXL memory pools, we want to extend the Morsel concept in this regard. Shared memory objects should fully reside on the memory pool (including page tables) and multiple hosts should be able to simultaneously interact with this object. To cope with the limited coherency, the idea is to place page tables in coherent memory for synchronization and implement an ownership model for data pages on the software level.

Goal

On the implementation side, this could be achieved with so-called Overlay-Morsels - one per host. Initially, all parts within this overlay are shared read-only with the authoritative truth on the memory pool. The first write access to a page triggers a page fault, which leads to acquiring the ownership of this specific page, meaning that it is mapped writable by the overlay. The remaining parts of the memory object stay unaffected. Additionally, the page fault handler must ensure that other hosts cannot access the page anymore, effectively clearing its present bit in the authoritative truth (and possibly other overlays) and performing a flush. For performing such flushes/invalidations, we expect a mechanism to send interrupts to other attached hosts. This mechanism will also be used to initiate write backs if another machine requests an exclusively owned page.

The main difference between this approach and existing RDMA approaches is that the data always resides on the shared memory pool and accesses are performed on cache-line granularity, not page-wise. Due to the lack of compatible hardware featuring the CXL 3.0 standard, the evaluations will be based on a multi-NUMA server system emulating the performance characteristics of CXL-attached memory.

Current State

The preceding bachelor's thesis already designed a mechanism we call Morsel Views. Each shared memory object can have multiple views. All those views are synchronized in that multiple views can share the same part of the object read-only, but write access is exclusive.

This mechanism works well on a single system but needs to be adapted to work across multiple machines. This includes (but may not be limited to):

Transforming the tracking of existing views from a system-internal mechanism to a shared state.
Designing synchronization protocols to synchronize specific operations (like the removal of pages) across systems.
Establishing a communication channel between all attached systems to trigger specific functions (like a cross-system TLB flush).

Task

With the basic mechanism implemented, we want to use it to solve the initial problem of limited coherence. Our assumptions are:

The original Morsel is located on disaggregated and shared CXL memory.
Each host has a single Overlay-Morsel.
Only page tables are cache-coherent; pages must be explicitly evicted after returning write capability.
There is a mechanism to trigger remote interrupts to perform remote TLB shootdowns and request access to exclusively owned parts.

To simulate the characteristics of remote CXL memory, we want to use a dual-NUMA memory system. The first NUMA node represents the remote memory and the second node the local host memory.

Schedule

The thesis will follow these key steps:

Getting started: Familiarize yourself with kernel development, set up a suitable development environment, and establish a functional test setup.
Dive into the implementation of Morsel Views: Reproduce the results and get used to the implementation.
Implementation of key mechanisms: Design and implement the interface for inter-host communication (e.g., cross-system TLB flush).
Extend the Morsel Views: Use the previously designed interface to make Morsel views ready for cross-system operation.
Creating an evaluation setup: Map the expected key characteristics of disaggregated CXL memory to a multi-NUMA server system.
Evaluation: Evaluate the suitability of Morsels for cross-system operation using synthetic benchmarks and real-world applications.
Possible Extensions: How can extended Morsel features (e.g., copy-on-write) be applied to shared objects?

The given schedule can act as a guideline and may not be strictly followed. Especially, implementation and evaluation typically involve an iterative process.

Topics: CXL, paging, disaggregated memory, Linux kernel

References

Web Links

An Introduction to the Compute Express Link (CXL) Interconnect: General introduction into the CXL concept

Publications

DIMES Workshop Morsels: Explicit Virtual Memory Objects

Alexander Halbuer, Christian Dietrich, Florian Rommel, Daniel LohmannProceedings of the 1st Workshop on Disruptive Memory SystemsAssociation for Computing Machinery2023.
PDF Details Slides 10.1145/3609308.3625267 [BibTex]

Shared Virtual-Memory Objects for Disaggregated Memory with Limited Coherency

CXL will bring shared, disaggregated memory with limited coherency. We will utilize the available coherency to build efficient synchronization for the remaining memory in software.

Typ: Bachelor-/Masterarbeit
Status: laufend
Supervisors: Alexander Halbuer
Daniel Lohmann
Project: ParPerOS
Bearbeiter: Daria Richter