Zero-copy ML: Doing inference inside the Linux Kernel

While LLMs and deep learning are all the rage nowadays, many problems still make some of these models infeasible. In cases such as network filtering, real-time security monitoring, etc., where one might need to make a decision within a few nanoseconds, there is no time to copy data to a userspace process, let alone to a GPU. This talk will outline work on building ML models to run in Linux kernel space, in particular, the eBPF virtual machine and the restrictions imposed on running models there. We will use a fairly naive dataset from Kaggle for malicious traffic detection and showcase how a model can be trained, compiled, and deployed in a real-life kernel without restarting the machine.

eBPF in and of itself poses a set of interesting constraints:

1. No FPU (no floating point arithmetic that drives many models today)
2. No heap(-ish), keeping weights etc. becomes a bit complicated
3. No stdint/stdlib, normal stdlib features (like memset/malloc) you would expect do not exist, making C-compatibility harder.

What you get in return is a formally-verified program (and model) that is guaranteed to be safe w.r.t., e.g. out-of-bound reads, predictable runtime complexity, and more importantly, a program you can deploy on any Linux machine with a fairly small set of capabilities without having to modify the running kernel.

The talk is aimed at both ML Engineers, developers and a little bit of SecOps, where I will outline the task and dataset with some flashbacks of caveats in traditional feature-engineering, in particular for (network) packet inspection. There might be lessons on how NOT to build datasets and on the importance of domain expertise in both dataset creation and feature engineering. Additionally, Jesper will give an introduction to eBPF and kernel development, and share lessons learned from building code (and models) that is portable across eBPF while still maintaining backwards compatibility with the original embedded-C targets.

In some sense, this is an ode to classical ML and to how traditional feature engineering still has a place, even if it's a small one.

This work is not sponsored by any company; it is open-source and in the public domain.