RockyGuard

Blog

The file on disk is clean. The running code isn't.

·Rocky Software ·8 min read

engineering-notessecurity

RockyGuard has always shipped an integrity self-check. It hashes the library and verifies an Ed25519 signature over that hash, so a customer can prove the binary they loaded is the one we built.

It hashes the file on disk.

That is exactly the artifact an in-memory attack leaves untouched. An inline detour hook on the signature-verify path, a 0xCC software breakpoint, a patch written by an attached debugger — none of them touch the file. The bytes on disk stay clean; the bytes executing do not. The check verifies the one copy of the code nobody is running.

The gap was worse for the linkage most people ship

Two facts sharpened the case past “a hook we can’t see.”

The on-disk check needs a separable binary to hash and sign. Static builds don’t have one — the library’s code is linked into the customer’s own executable, and there is nothing standalone to put a .sig next to. So for static linkage the check is skipped entirely. And static is the linkage the customer docs recommend and the one most customers ship. The primary integration path had no integrity coverage at all.

The second fact is a simplification, not a complication. A software breakpoint and a prologue hook are both, physically, a change to the loaded bytes. One mechanism that notices in-memory modification subsumes “detect breakpoints or patches” without a separate anti-debug layer bolted on beside it.

So the thing to build was a check on the running code, that also worked for static builds. Those turn out to be the same requirement.

Why you can’t just sign the expected in-memory bytes

The obvious design is the on-disk one moved into RAM: at build time, compute the hash the code should have once loaded; at runtime, hash the loaded code and compare.

It does not survive contact with the loader. The image in memory is not a copy of the file. The OS applies ASLR base relocations — rewriting absolute addresses baked into the code to wherever the module actually landed this run — and patches the import address table with resolved pointers to other modules. Both differ per run and per machine. There is no single expected hash to precompute, because the loaded bytes are legitimately different every time.

You can undo the relocations before hashing. It is possible, it is per-format, and it is exactly the kind of code that works until a compiler or linker upgrade changes a fixup type and then fails silently. Reintroducing a disassembler’s worth of format-specific fragility to check integrity is a poor trade.

Trust the first run, and no run after

Trust-on-first-use sidesteps the whole problem. Don’t compare against a number computed somewhere else. Compare the process against itself.

The first validation in a process snapshots a SHA-256 baseline over a fixed 64-byte prefix of each monitored function. Every later validation re-hashes the same spans and compares. Within a single process run the load base is already fixed and the relocations are already applied — so the baseline bytes equal the recheck bytes, unless the code was modified in between. Relocations never enter the reasoning, because both sides of the comparison already carry them identically.

The check needs only function addresses. No signature, no file on disk, no PE/ELF parsing. That is precisely why it also covers static builds: it never depended on a separable binary in the first place. It runs alongside the on-disk check — which still earns its place, since it catches the patched-and-saved binary that a within-run comparison, by construction, cannot. Neither replaces the other.

64 bytes, chosen against the threat, not for a round number

The span length is picked against what an attacker actually writes. A rel32 JMP detour is 5 bytes. An absolute mov-then-jmp trampoline is 12 to 14. A software breakpoint is 1. Commercial hooking libraries patch the prologue. 64 bytes covers the prologue and first basic block of every function we monitor, while staying inside a single page and inside the length of the small functions themselves.

A hook placed deeper than 64 bytes into a long function is a gap. It is an accepted, documented gap — the alternative is computing exact function lengths, which needs a disassembler or a symbol-size table the library does not carry, and that reintroduces the toolchain coupling TOFU exists to avoid.

Six functions are monitored: the enforcement chokepoints an attacker would patch to defeat licensing — signature verification, hashing, base64 decode, the clock check, the on-disk integrity checker — and the comparison routine itself. The last one is partial self-protection: an attacker who NOPs the compare loop still has to defeat the span that covers the compare loop.

Two adversaries that never sleep: the OS, and the optimizer

Reading raw code bytes is a good way to fault. A short function at the end of a page, a guard page, a region that isn’t executable — hash straight through any of those and the process dies. Each span is clamped to the executable extent the OS actually reports: VirtualQuery on Windows (rejecting PAGE_GUARD and non-executable protections), /proc/self/maps on Linux. If the extent can’t be read, that span is skipped rather than read blindly. macOS isn’t implemented yet and fails open. No signal or SEH handler is installed to catch a bad read — that path is not async-signal-safe and would clobber a customer’s own handler.

The compiler is the subtler adversary, because it is trying to help. The comparison routine has to survive as a real, addressable function body, so identical-code folding can’t merge it with some other trivial function and collapse the self-check — it carries a unique volatile marker for exactly that. Every monitored address is captured through a volatile pointer so it escapes, which defeats inlining, dead-code elimination, and the --gc-sections stripping the library already turns on. An optimizer that quietly deletes your integrity check is not a hypothetical; it is the default behavior you have to write against.

Off by default, and that is the honest setting

The whole feature is gated on a new ROCKYGUARD_RUNTIME_INTEGRITY CMake option, off by default. A customer who merely rebuilds sees no change.

Default-on was tempting and wrong. The check runs inside the customer’s own process, and plenty of legitimate things modify a running function’s bytes: a function-hooking APM or EDR agent, a profiler, a developer running their own app under a debugger with software breakpoints. Defaulting on would turn a previously-silent skip into a hard integrity failure for a static-linkage customer the moment they upgraded, in environments that were fine the day before. Opt-in keeps the upgrade a no-op until the customer weighs the trade for their own deployment; the remedy string points at the APM exclusion-list fix when they do.

When it is enabled, a detection fails closed — INTEGRITY_RUNTIME_MODIFIED, code 1608, the eighth INTEGRITY_* code and the only one that is a genuinely new detection rather than a finer label on an existing one. It still maps to IntegrityCheckFailed, so a caller switching on status is unaffected. An inability to read any function’s extent fails open to Skipped, mirroring the non-REQUIRE_SIGNATURE path of the on-disk check, so an odd platform is never bricked by a check it can’t run.

The ceiling, stated plainly

This runs on the attacker’s machine. That sentence is the whole limit, and no amount of cleverness moves it.

Tampering that is already present when the baseline is captured is recorded as the baseline and reads clean forever after; capture fires as early as possible to shrink that window, not to close it. A binary patched on disk and reloaded is self-consistent to a within-run check — which is the on-disk .sig check’s job, which is why the two run together. And a determined attacker can NOP the compare, flip the result, or delete the enforcing branch. Self-monitoring the compare raises that bar; it does not make it a wall.

The honest goal is narrower than “unbreakable” and more useful than it sounds: raise the cost of an in-memory patch, and give static builds the only integrity coverage the linkage can physically have. The way past the ceiling is on the roadmap and it is not more checking — it is entanglement: feed the live checksum into a value the crypto path actually consumes, so a patched function produces wrong answers instead of a flag an attacker can find and flip. A check you can locate is a check you can remove. A computation that only comes out right when the code is intact is a harder thing to defeat. That is the next step, and naming it here is the point: we would rather publish the limit than imply it isn’t there.


The mechanism and its limits are in src/runtime_integrity.h; the customer-side build decision is Customer Documentation §12.1, and §9.3 covers how the runtime check differs from the on-disk one. If you want to enable it, it’s one CMake flag — the demo bundle is on the download page.