THREE DAYS. the bug was a missing semicolon in a yaml file.
yaml does not need semicolons. that is the entire point of yaml. but there was a string field that contained the value true; rm -rf data. because someone had typed the literal text into a config and our parser was passing it to a shell.
so the bug is technically: we shell out to handle config strings. the bug is also: i typed true; rm -rf data into a config. but the bug ALSO is: the parser does not warn when a string contains semicolons. WHY DOES THE PARSER NOT WARN.
i added a warning. i replaced the shell-out with a real parser. i wrote a postmortem. i went outside. there is grass.
stay safe out there.