Digital Forensics & IR / Malware Engineering

Malware Analysis & Reverse Engineering

Malware analysis answers a narrow operational question: what does this sample do, what does it talk to, and how would the estate know if it ran somewhere else. Each successive technique costs more analyst time than the last, so the discipline is mostly about stopping at the stage that answers the question actually being asked. The output that matters is not the analysis write-up but the detection content derived from it — byte patterns as YARA rules, behavior as Sigma rules — because that is what protects the hosts nobody has looked at yet.

Static inspection ↔ isolated or hosted detonation ↔ decompilation and memory unpacking ↔ YARA and Sigma rule authoring

Before detonating anything

Running live malware is the one activity on this page with consequences outside the analyst’s machine. Three prerequisites apply before the first execution:

• Authorization: written approval to possess, retain, and execute the sample, from whoever owns the incident. A sample recovered from a customer environment may carry contractual or regulatory handling constraints that have nothing to do with the malware itself.
• A network the sample cannot escape: an analysis segment with no route to production or to the internet, with DNS and HTTP answered by simulated services (INetSim, FakeNet-NG) so the sample reveals its intended destinations without reaching them.
• A disposal rule: where samples live, who can retrieve them, and when they are destroyed. Malware accumulating on a shared fileshare is its own incident waiting to happen.

The four stages of analysis

The four-stage split below follows the taxonomy in Sikorski and Honig’s Practical Malware Analysis, which is the vocabulary most analysts use. Each stage costs more time than the last, so the useful question is which stage answers the question at hand rather than how far down the list one can get.

Self-hosted versus hosted detonation

Stage 2 has two operating models that are frequently treated as one. A self-hosted lab runs on hardware or a hypervisor under local control, on an isolated segment with simulated services. A hosted sandbox is an internet service that runs the sample on someone else’s infrastructure.

A hosted sandbox is by construction not air-gapped, and on free tiers the analysis is normally public — ANY.RUN’s plans list “unlimited public analyses” for the Community tier and reserve private analyses for paid tiers. For a sample pulled from a live incident that matters twice over: the submission can expose victim-identifying artifacts such as internal hostnames, share paths, and targeted user names, and an adversary monitoring public sandbox feeds for their own tooling learns that they have been detected. Incident samples belong in the self-hosted lab; hosted services are for commodity samples where neither concern applies.

What a clean dynamic run does not prove

Sandbox output is evidence of what the sample did in that environment, during that window. Malware routinely declines to act when it detects one:

• Environment checks — hypervisor artifacts, small disk or memory, no user-created documents, no domain membership, a hostname matching a known sandbox pattern.
• Time-based evasion — long sleeps or API-call padding that outlast the analysis window.
• Missing preconditions — a payload that only decrypts when it reaches a specific domain, receives a C2 response, or finds a particular file.

A sample that does nothing in a sandbox is therefore not established as benign; it is unresolved, and the question moves to Stage 3, where capability is visible in the code whether or not it executed.

Packers, crypters, and unpacking from memory

Packers and crypters (UPX, Themida, commodity XOR string encryption) exist to defeat static inspection: the analyst sees a small stub and a high-entropy blob rather than the real code.

1. How packing works at runtime: the packed executable contains a stub and an encrypted payload buffer. On execution the stub allocates writable-executable memory (VirtualAlloc with PAGE_EXECUTE_READWRITE), decrypts the payload into it, and transfers control to the original entry point (OEP) of the unpacked program.
2. How to recover the payload: set breakpoints on memory allocation APIs and on the tail jump into the freshly written region, let the stub finish decrypting, then dump the unpacked image from memory to disk. The dumped binary is what Stage 1 and Stage 3 techniques should be run against — analyzing the packed file yields the stub’s behavior, not the malware’s.

A dumped image is not a valid on-disk executable without import reconstruction, so expect to repair the import table before a decompiler produces useful output.

Turning findings into detection content

Two rule formats carry different halves of the finding. YARA matches bytes and strings in files and memory. Sigma matches behavior in logs. A sample usually justifies one of each: the byte pattern catches the exact build, the behavioral rule catches the technique after the build changes.

The two rules below are illustrative examples written for this journal, not validated detections. The YARA strings and byte pattern are invented to show rule structure; they have not been tested against real samples and would need validation against both malicious and known-good binaries before deployment.

rule Example_Packed_Ransomware_Structure
{
    meta:
        description = "Illustrative YARA rule structure - not a validated signature"
        author      = "Field Journal (illustrative example)"
        date        = "2026-08-15"
        reference   = "https://attack.mitre.org/software/S1202/"
    strings:
        $s1 = "LockBit 3.0" ascii wide nocase
        $s2 = ".README.txt" ascii wide
        $hex_pattern = { 8B 45 ?? 33 C9 89 4D ?? 8B 4D ?? 8B 11 }
    condition:
        uint16(0) == 0x5A4D and (2 of ($s*) or $hex_pattern)
}

uint16(0) == 0x5A4D reads the first two bytes little-endian, so it matches the MZ header and restricts the rule to PE files. That also means it will not match a memory dump or a minidump artifact such as lsass.dmp, which begins with MDMP — scanning dumps needs a separate rule without the header check.

The behavioral counterpart, trimmed to its essential fields:

title: Shadow Copy Deletion via Vssadmin
status: experimental
description: Detects deletion of volume shadow copies, a common pre-encryption step
logsource:
    category: process_creation
    product: windows
detection:
    selection:
        Image|endswith: '\vssadmin.exe'
        CommandLine|contains|all:
            - 'delete'
            - 'shadows'
    condition: selection
falsepositives:
    - Backup software pruning shadow copies on a schedule
level: high

Rule authoring as a pipeline — testing, transpilation to native SIEM queries, rule status lifecycle, and the metrics that tell you whether a rule is earning its place — is covered in Detection Engineering & SIEM Analytics. Note also that YARA-X, the maintained Rust rewrite from VirusTotal, is the direction of travel for new deployments; rule syntax is largely compatible, but validate before migrating an existing ruleset.

What a signature proves

A YARA hit means a byte pattern matched. It is not attribution: shared code, a common packer, and reused open-source components all produce matches across unrelated families. A YARA miss on a repacked or recompiled sample is the expected outcome rather than a failure, which is why file hashes and byte signatures sit at the bottom of the Pyramid of Pain described in Cyber Threat Intelligence & Threat Actor Profiling — they are cheap to deploy and cheap for an adversary to invalidate. The behavioral rule is what survives a rebuild.

Malware analysis capability review checklist

The checklist below is a journal working model, not a published audit standard. When setting up or auditing a malware analysis capability, evaluate these six criteria:

Primary references

• YARA documentation — verified rule syntax, the uint16 header condition, and string modifier semantics used in the example above.
• YARA-X — verified that it is VirusTotal’s maintained Rust rewrite and the recommended direction for new deployments.
• Microsoft VirtualAlloc documentation — verified the allocation and PAGE_EXECUTE_READWRITE protection behavior described in the unpacking mechanics.
• Sigma specification — verified the rule fields and logsource/detection structure used in the behavioral example.
• MITRE ATT&CK: LockBit 3.0 (S1202) — the ATT&CK entry referenced by the illustrative YARA rule.