Systems Research and Architecture SRA
a SRA Theses AHA RTOS Benchmark Benchmark

Comparison of exisiting RTOS Benchmark Suites

image

Creating a bench(mark) out of different bench(marks) [Generated with AI]

Context

Real-time systems (RTS) are subject to timing contraints for their execution. This makes their development more complex than general purpose compute systems, since not only the functionality, but also their timing needs to be considered and verified. Often this requires specialization and a high level of integration between hardware and software. Ranging from soft to hard RTS, a diverse set of applications and research topics exist.

Problem

Scientifc research requires measurements and experiments to validate new hypothesis. Since we can not test every indiviudal system, we need to generalize and pick out single systems as represantives for a group of systems. This process collides with the specialized nature of tailored RTS, which are fit to their respective problem and environment.

Nevertheless, previous work established different types of benchmark suits and task set generators to approximate the nature of a wide range of RTS. They allow to quantiy the impact of many variables in a RTS environment: ardware, real-time operating system (RTOS), scheduling algorithms, communication protocols, and more.

But they vary greatly in their scope and detail, ranging from a vague list in the form of "there are 10 tasks that do something every 10ms" to concrete source code implementations for a given RTOS. This is an issue for research, where the main goal is modify some of these variables and verify their results, because even if the same benchmark is used, other implementation details can have an even bigger impact.

Goal

The goal of this thesis is to try out different kinds of exisiting benchmarks and to compare their properties. This means to:

Depending on available time, further hardware platforms can be explored.

Topics: Real-Time Operating Systems, Test automation, C, Python, build systems

This topic can be fitted to either a Bachelors or Masters thesis. TODO

References

Papers

ECRTS Conference
Real-Time System Evaluation Techniques: A Systematic Mapping Study
Tilmann L. Unte, Sebastian Altmeyer37th Euromicro Conference on Real-Time Systems (ECRTS 2025)Schloss Dagstuhl – Leibniz-Zentrum für Informatik2025.
PDF 10.4230/LIPIcs.ECRTS.2025.12 [] 
@inproceedings{unte:2025:ecrts,
location = {Dagstuhl, Germany},
annote = {Keywords: Systematic Mapping Study, Real-Time Systems, Evaluation},
author = {Unte, Tilmann L. and Altmeyer, Sebastian},
booktitle = {37th Euromicro Conference on Real-Time Systems (ECRTS 2025)},
doi = {10.4230/LIPIcs.ECRTS.2025.12},
editor = {Mancuso, Renato},
entrysubtype = {Conference},
isbn = {978-3-95977-377-5},
issn = {1868-8969},
pages = {12:1–12:21},
publisher = {Schloss Dagstuhl – Leibniz-Zentrum f{ü}r Informatik},
series = {Leibniz International Proceedings in Informatics (LIPIcs)},
title = {{Real-Time System Evaluation Techniques: A Systematic Mapping Study}},
url = {https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.ECRTS.2025.12},
urn = {urn:nbn:de:0030-drops-235903},
volume = {335},
year = {2025},
}
OSPERT Workshop B
RTOS-Independent Interaction Analysis in ARA
Gerion Entrup, Jan Neugebauer, Daniel LohmannProceedings of the 16th Annual Workshop on Operating Systems Platforms for Embedded Real-Time Applications (OSPERT '22)2022.
PDF Slides [] 
@inproceedings{entrup:22:ospert,
author = {Gerion Entrup and Jan Neugebauer and Daniel Lohmann},
booktitle = {Proceedings of the 16th Annual Workshop on Operating Systems Platforms for Embedded Real-Time Applications (OSPERT '22)},
entrysubtype = {Workshop},
month = {July},
title = {RTOS-Independent Interaction Analysis in ARA},
venue = {Modena, Italy},
year = {2022},
}

RTSS Conference
Generating Utilization Vectors for the Systematic Evaluation of Schedulability Tests
David Griffin, Iain Bate, Robert I. Davis2020 IEEE Real-Time Systems Symposium (RTSS)2020.
PDF 10.1109/RTSS49844.2020.00018 [] 
@inproceedings{griffin:22:rtss,
author = {Griffin, David and Bate, Iain and Davis, Robert I.},
booktitle = {2020 IEEE Real-Time Systems Symposium (RTSS)},
doi = {10.1109/RTSS49844.2020.00018},
entrysubtype = {Conference},
keywords = {Systematics;Real-time systems;Task analysis;real time;task set generation;UUnifast;RandFixedSum;Dirichlet Rescale;utilization},
number = {},
pages = {76-88},
title = {Generating Utilization Vectors for the Systematic Evaluation of Schedulability Tests},
url = {https://www-users.york.ac.uk/~rd17/papers/DRSRTSS2020.pdf},
volume = {},
year = {2020},
}
WCET Workshop
TACLeBench: A Benchmark Collection to Support Worst-Case Execution Time Research
Heiko Falk, Sebastian Altmeyer, Peter Hellinckx, Björn Lisper, Wolfgang Puffitsch, Christine Rochange, Martin Schoeberl, Rasmus Bo Sørensen, Peter Wägemann, Simon Wegener16th International Workshop on Worst-Case Execution Time Analysis (WCET 2016)Schloss Dagstuhl – Leibniz-Zentrum für Informatik2016.
PDF 10.4230/OASIcs.WCET.2016.2 [] 
@inproceedings{falk:16:wcet,
location = {Dagstuhl, Germany},
annote = {Keywords: Benchmark, WCET analysis, real-time systems},
author = {Falk, Heiko and Altmeyer, Sebastian and Hellinckx, Peter and Lisper, Björn and Puffitsch, Wolfgang and Rochange, Christine and Schoeberl, Martin and S{ø}rensen, Rasmus Bo and Wägemann, Peter and Wegener, Simon},
booktitle = {16th International Workshop on Worst-Case Execution Time Analysis (WCET 2016)},
doi = {10.4230/OASIcs.WCET.2016.2},
editor = {Schoeberl, Martin},
entrysubtype = {Workshop},
isbn = {978-3-95977-025-5},
issn = {2190-6807},
pages = {2:1–2:10},
publisher = {Schloss Dagstuhl – Leibniz-Zentrum f{ü}r Informatik},
series = {Open Access Series in Informatics (OASIcs)},
title = {{TACLeBench: A Benchmark Collection to Support Worst-Case Execution Time Research}},
url = {https://drops.dagstuhl.de/entities/document/10.4230/OASIcs.WCET.2016.2},
urn = {urn:nbn:de:0030-drops-68958},
volume = {55},
year = {2016},
}
RTAS Conference
Taming Non-Blocking Caches to Improve Isolation in Multicore Real-Time Systems
Prathap Kumar Valsan, Heechul Yun, Farzad Farshchi2016 IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS)IEEE2016.
PDF 10.1109/RTAS.2016.7461361 [] 
@inproceedings{valsan:16:rtas,
abstract = {In this paper, we show that cache partitioning does not necessarily ensure predictable cache performance in modern COTS multicore platforms that use non-blocking caches to exploit memory-level-parallelism (MLP).},
author = {Valsan, Prathap Kumar and Yun, Heechul and Farshchi, Farzad},
booktitle = {2016 {IEEE Real-Time} and {Embedded Technology} and {Applications Symposium} ({RTAS})},
date = {2016-04},
doi = {10.1109/RTAS.2016.7461361},
entrysubtype = {Conference},
eventtitle = {2016 {IEEE Real-Time} and {Embedded Technology} and {Applications Symposium} ({RTAS})},
isbn = {978-1-4673-8639-5 978-1-4673-8641-8},
langid = {english},
location = {Vienna, Austria},
pages = {1–12},
publisher = {IEEE},
title = {Taming {Non-Blocking Caches} to {Improve Isolation} in {Multicore Real-Time Systems}},
url = {http://ieeexplore.ieee.org/document/7461361/},
urldate = {2026-03-26},
year = {2016},
}

  1. Trampoline Tests https://github.com/TrampolineRTOS/trampoline/tree/master/tests
  2. Threadmetric Benchmark https://github.com/eclipse-threadx/threadx/tree/master/utility/benchmarks/thread_metric