A Systemd Generator Executes Before Your Logging Is Up. Catch the Write, Not the Run

Every persistence primer on Linux eventually lands on systemd, and most of them stop at .service units and timers because those are the ones you can watch execute. Generators are the awkward cousin nobody wants to write the detection for, and they’re the one you should actually care about. A system generator is a small executable systemd runs at the very start of boot, as root, before it has even parsed the unit tree — and, more to the point, before auditd is running, before your EDR sensor’s userspace agent has loaded, before normal journald, syslog, and audit telemetry is reliably up. The design constraint is right there in the freedesktop docs: generators “may not talk to any other process,” including systemd itself, and cannot use syslog because syslog does not exist yet — the one sanctioned channel is /dev/kmsg, which journald does ingest via the kernel ring buffer, though no attacker is going to volunteer a log line about themselves. (User generators, covered later, run under the per-user systemd --user manager at lower privilege — a smaller surface, but a real one.) That constraint is a feature for systemd and a gift for an attacker. Whatever a planted generator does, it does in a window where your userspace telemetry stack is, functionally, blind.

So the detection can’t live at execution time. It has to live at write time, and at reconciliation time. Get that framing wrong and you’ll spend a sprint building an alert that fires on an event you were never going to capture.

What a generator actually is, and why the timing matters

Generators live in a fixed set of directories. systemd walks them in a defined precedence order and executes everything it finds, in parallel, on every boot and every daemon-reload. The system paths are /etc/systemd/system-generators/, /run/systemd/system-generators/, /usr/local/lib/systemd/system-generators/, and /usr/lib/systemd/system-generators/ (plus /lib/... where it isn’t a symlink to /usr/lib). There’s a parallel user-generators set that runs under each systemd --user manager. Precedence runs admin-over-vendor the way you’d expect: a generator in /etc wins over one of the same name in /usr/lib.

The output side is where it gets pointed. Each generator is invoked with three directory arguments, and one of them — the “early” directory, argv[2] — produces unit files that “take precedence over all configuration, regardless if vendor or user/administrator.” Read that again. A generator can synthesize or override a unit that beats anything an admin wrote in /etc/systemd/system/. So this isn’t only a “run code at boot” primitive. It’s a “run code at boot and rewrite the unit graph before anyone else gets a vote” primitive. That’s why it maps most directly to MITRE T1543.002 (Create or Modify System Process: Systemd Service) when the generator creates or overrides units — with an event-triggered-execution flavor under T1546 as well, since the code fires automatically at boot and on daemon-reload (Elastic’s generator-creation rule tags both) — and why the execution timing matters so much: by the time your EDR agent’s own unit starts, the generator has already run and, if it wanted to, already overridden the definition of the service that’s supposed to be watching it.

Legitimate generators are everywhere, which is the whole problem. On a stock RHEL or Ubuntu box you’ll find systemd-fstab-generator, systemd-gpt-auto-generator, systemd-sysv-generator, the rc-local shim, lvm2-activation-generator, cloud-init-generator on any cloud image, openvpn-generator if openvpn’s installed. All of them ship inside packages and all of them belong in /usr/lib. That’s the seam you exploit for detection: vendor-supplied generators are package-owned, usually live under /usr/lib, and change only during a package transaction. Admin- or locally-built generators can legitimately sit in /etc or /usr/local/lib and won’t be package-owned — but on a well-run host they should be rare, documented, hash-baselined, and change-controlled, which keeps them enumerable rather than a blanket excuse to ignore those paths.

The detection has two halves, and neither is “watch it run”

Half one is the write. Put a file-integrity watch on every generator directory and treat any create, rename, or attribute change as signal. On an auditd host that’s a handful of watch rules:

-w /etc/systemd/system-generators/ -p wa -k systemd_generator
-w /usr/lib/systemd/system-generators/ -p wa -k systemd_generator
-w /usr/local/lib/systemd/system-generators/ -p wa -k systemd_generator
-w /run/systemd/system-generators/ -p wa -k systemd_generator
-w /etc/systemd/user-generators/ -p wa -k systemd_generator
-w /run/systemd/user-generators/ -p wa -k systemd_generator
-w /usr/local/lib/systemd/user-generators/ -p wa -k systemd_generator
-w /usr/lib/systemd/user-generators/ -p wa -k systemd_generator

A note on form before the tuning: the compact -w watch syntax above is shown for readability, and it’s the backward-compatible path — auditd now treats -w watches as legacy in favor of syscall-style rules (-a always,exit -F dir=/etc/systemd/system-generators/ -F perm=wa -k systemd_generator), which are more expressive and lighter-weight at scale. Prefer the syscall form in production where your distro’s auditd baseline supports it; the -w rules here are the legible starting point, not the hardened one.

Do not skip /run. It’s tmpfs, it evaporates on reboot, and a generator planted there runs once and vanishes — which is exactly what you’d want if you were trying to survive a single maintenance window without leaving a disk artifact. But there’s a catch with auditd watches that bites hard here, and the naive version of this advice is genuinely dangerous: the watched path has to exist when the rule loads, and on a minimal image half of these directories won’t be there yet. Do not assume auditd simply skips a missing path and loads the rest — Red Hat documents auditd halting on a rule error and failing to load the rules that follow it (observed across RHEL 6 through 9), so a single missing /run/systemd/user-generators/ can silently drop every rule after it in the file. Handle it deliberately: pre-create the generator directories during hardening so the watches always resolve, or load each watch conditionally only for the paths that exist, and know your loader’s continue-on-error posture (the audit.rules control that tells auditctl to ignore errors and keep going is configuration-dependent, not a default you can assume). Then confirm what actually made it into the kernel with auditctl -l — not journalctl -u auditd, since rule-load warnings don’t reliably reach the journal — because a partially-loaded rule set reads as “deployed” right up until you check it. Better a noisy, fully-loaded rule set than a silent gap you assumed wasn’t there.

In the SIEM, the field you’re keying on depends on your pipeline. With the Elastic/auditbeat path it’s a file event — event.action of creation/rename, file.path matching the generator dirs, and process.executable for the writer. Elastic ships a prebuilt rule for exactly this (“Systemd Generator Created,” EQL, nine directory globs), and it’s a reasonable starting point — but note its coverage seam: the shipped query watches /run/systemd/system-generators/* yet includes no /run/systemd/user-generators/* glob, so a per-user generator planted in tmpfs falls straight through it. Add that path, and don’t trust the shipped exclusion list blindly. If you’re on Splunk with the auditd TA, you’re correlating the SYSCALL record’s key=systemd_generator back to the PATH and CWD records to recover the filename and the writing process, because the watch itself only gives you the key and the syscall. That reassembly is where the Splunk auditd sourcetype earns its reputation for being a pain — the multi-record event model means a dropped PATH line leaves you with a syscall and no filename, and under load, lines do get dropped.

Volume is the good news here. On a stable fleet — one where config management isn’t rewriting generators on every convergence, which is its own problem covered below — legitimate writes to these directories happen only during package installs and upgrades. Baseline is effectively zero events per host per day outside of patch windows. So you are not tuning a numeric threshold — there’s nothing to threshold. You’re building an allow-context: which writes are the package manager, and which aren’t.

Half two is reconciliation, because the write-time detection has a hole you can drive a truck through. If the generator gets planted while the host is offline — a mounted disk image, a golden image poisoned upstream, a restore from a tampered backup — there is no live write event to catch. Nothing fired because nothing was watching the filesystem at the moment of the write. The only way to catch that class is to periodically ask a different question: does every generator on this box belong to a package?

That’s a cheap hunt. Walk each on-disk directory — skip /run, where nothing is ever package-owned and a scheduled sweep arrives long after the tmpfs write is gone — and for every file run it back through the package DB: rpm -qf on RHEL-family, dpkg -S on Debian-family. A file that comes back “not owned by any package” is your lead. It’s not automatically malicious (locally-built software and some config-management drops are legitimately unowned), but the unowned set is small and stable enough to enumerate and adjudicate by hand. Pair it with an AIDE or auditbeat FIM baseline over the same directories so you get hash-level change detection between runs. Belt and suspenders, and on these directories the suspenders are the part that catches the offline plant.

What breaks in week one

The false positives all come from the same place: things that legitimately write generators. Package managers first — dpkg, rpm, dnf, yum — and their transaction temp files, the .dpkg-new and .rpmnew droppings and editor .swp files that land in the directory mid-write. The Elastic rule already excludes those extensions; if you rolled your own auditd-to-SIEM rule you’ll rediscover them the hard way the first time someone patches systemd itself and your generator alert lights up across the entire fleet at once. That fleet-wide simultaneous fire is actually the tell for a benign cause — real persistence doesn’t deploy to 4,000 hosts inside the same fifteen-minute yum window.

Cloud-init is the second one, and it’s worth being precise about, because the lazy version of this warning is simply wrong. Cloud-init runs a generator on first boot — cloud-init-generator, which decides whether cloud-init.target joins the boot goals — but that generator ships inside a package under /usr/lib/systemd/system-generators/ and is not rewritten on each boot, so it should not normally trip a create/rename watch on the generator input directories at all. Where cloud-init actually produces first-boot churn is in the output trees under /run/systemd/generator* — a different set of directories that the Elastic rule and the watches above don’t even cover. So if your FIM scope happens to reach those output dirs, expect provisioning noise there and carve it out by context — first boot, cloud-init as the parent process — never by blanket-ignoring cloud-init, because “ignore cloud-init” written carelessly becomes “ignore anything claiming to be cloud-init.” If your scope is the input directories, cloud-init mostly isn’t your false positive — and a genuinely new generator file appearing there still deserves the alert.

Config management is the third, and the messiest. Puppet, Chef, Ansible, Salt — if any of them manage a generator (some LVM and networking modules do), they’ll rewrite it on every convergence run, which might be every 30 minutes. The instinct is to exclude the config-management agent’s process wholesale. Resist it. That exclusion is a standing invitation: anything that can get code execution as the Puppet agent now has an unmonitored write path straight into a boot-time root primitive. Scope the exclusion to the specific files that tool is known to manage, and alert on everything else it touches. Yeah, it’s more work. It’s also the difference between a detection and a rubber stamp.

Where the environment changes the answer

Immutable and image-based distros shrink the problem in a genuinely useful way. On Fedora CoreOS, RHEL bootc, or Flatcar, /usr is mounted read-only and delivered as an OSTree or OCI image, so /usr/lib/systemd/system-generators/ can’t be tampered with on a running host at all — the whole point of rpm-ostree. (RHEL image mode keeps the root filesystem immutable except for /etc and /var.) That collapses your live-write watch down to /etc and /run, the two writable trees, and it shifts the reconciliation question from “does this file belong to a package” to “does this image match the digest it’s supposed to be” — with the caveat that provenance and digest verification only buy you integrity to the degree your platform actually enforces signature checking, which not every environment does. Different control, same intent.

User generators are the coverage gap most shops never close. /etc/systemd/user-generators/ and the per-user paths run under systemd --user, at a lower privilege, and almost nobody watches them because the persistence primers focus on system-level root. They’re still persistence, they still survive reboot for that user, and on a multi-user jump host or a shared build box that’s a real foothold. Watch them too. The auditd rule costs you nothing.

Containers mostly dodge this — a typical application container doesn’t run systemd as PID 1, so there’s no generator pass. But the systemd-in-container patterns (some OpenShift build scenarios, podman with --systemd=always, sysbox nodes) bring it right back, and there your host-level auditd watch won’t see inside the container’s mount namespace. You need the file watch where the container’s /etc and /run actually live on disk, or FIM inside the image.

Control mapping

These map back to the two-halves model rather than standing on their own — write-time detection on one side, offline reconciliation and integrity on the other:

The honest limitation, stated plainly: none of this observes the execution. If an adversary plants a generator, boots the host once, and removes it, you have exactly two chances to catch it — the write event, if auditd was up and the plant was live-on-disk, and the reconciliation sweep, if it happened to run while the file was present. A /run generator that lives for one boot beats both unless your FIM cadence is tighter than the maintenance window, which it usually isn’t. Close the write path, reconcile against packages on a schedule you can defend, and accept that the residual risk here is real. Boot-time execution below the level your sensors start at is a structural gap, not a tuning problem.

Sources

• Generators (freedesktop.org / systemd wiki)
• systemd.generator(7) manual page (Debian Manpages)
• audit.rules(7) manual page — syscall rules vs. deprecated -w watches (Debian Manpages)
• Auditd stops loading consecutive rules when an error occurs (Red Hat Customer Portal, solution 7001979)
• Boot stages — the cloud-init systemd generator and cloud-init.target (cloud-init docs)
• Systemd Generator Created (Elastic Security prebuilt rules reference)
• Linux Detection Engineering — A primer on persistence mechanisms (Elastic Security Labs)
• Create or Modify System Process: Systemd Service, T1543.002 (MITRE ATT&CK)
• Event Triggered Execution, T1546 (MITRE ATT&CK)
• Hunting for Persistence in Linux (Part 5): Systemd Generators (pberba)