Malware Analysis: Runtime & Dynamic Techniques for IOC Extraction

Why This Research?

Signature-based AV and static analysis alone can't keep up anymore. Malware authors use packing, obfuscation, and runtime API resolution to hide malicious intent until execution. This research asked one question: can you combine runtime analysis with dynamic analysis in a sandbox to pull IOCs that static methods miss?

The short answer: yes. The longer answer is what this paper documented across five distinct malware types.

Research Methodology

Here's how we ran it, four stages:

1. Set up the sandbox: isolated VM, INetSim for fake internet, CAPE Sandbox for API hooking
2. Collect the samples: five malware types, not variants, each a different threat category
3. Run them: execute each sample, capture runtime API calls and dynamic behavior
4. Extract IOCs: hashes, network indicators, registry changes, dropped files, mutexes, evasion artifacts

Lab Environment

The guest VM ran on an isolated VLAN with no route to the internet. INetSim provided fake DNS, HTTP, SMTP, and other common services so the malware believed it had network access. This is critical, many samples terminate or self-delete if they detect no connectivity, since their C2 communication is essential to their payload.

The Five Samples

We tested five malware types, not variants of one family, but distinct categories with fundamentally different behaviors and objectives:

Runtime + Dynamic: Why Both Matter

Runtime analysis hooks API calls. CAPE Sandbox watched every CreateFile, RegSetValue, WinExec, NtAllocateVirtualMemory, CryptEncrypt: function names, parameters, return values. This is what the malware asked the OS to do.

Dynamic analysis captures the behavioral aftermath, file system changes, registry modifications, network connections, process trees, and memory artifacts. Volatility's psscan, malfind, and netscan plugins provided a forensic snapshot at each execution stage.

Static analysis alone would have failed on every sample. The binaries were packed (UPX, custom cryptors), used runtime API resolution via LoadLibrary + GetProcAddress, and contained zero meaningful static strings. The only way to understand them was to let them run, safely.

IOC Categories Extracted

├─ File IOCs
│   ├─ MD5, SHA1, SHA256 hashes of payload and dropped files
│   └─ File paths, names, and extensions of dropped artifacts
│
├─ Network IOCs
│   ├─ C2 IP addresses and domain names
│   ├─ Non-standard destination ports
│   └─ DNS query patterns (TXT record abuse, DGA domains)
│
├─ Registry IOCs
│   ├─ Persistence keys: Run, RunOnce, Winlogon\Shell
│   └─ Modified security policy and firewall rules
│
├─ Process & Memory IOCs
│   ├─ Process injection targets (explorer.exe, svchost.exe)
│   ├─ Named mutexes for single-instance enforcement
│   └─ Injected code regions (malfind output)
│
├─ Dropped Files
│   └─ Secondary payloads, configuration files, scripts
│
└─ Evasion Techniques
    ├─ VM/Sandbox detection (registry checks, timing)
    └─ Debugger detection (IsDebuggerPresent, NtQueryInformationProcess)

These six categories formed the basis for actionable detection rules, the practical bridge from academic research to operational security.

Results: What Static Analysis Missed

Every sample employed at least one technique that static analysis could not detect:

• Trojan: API calls were resolved dynamically. Static import table showed only kernel32.dll, the actual malicious functions were loaded at runtime.
• Ransomware: Encryption routine existed only as encrypted shellcode inside the packed binary. CAPE caught the CryptEncrypt calls and identified the AES key handle in memory via Volatility.
• Spyware: Browser database access (SQLite files for Chrome, Firefox) was only visible as file handle opens during execution. Static analysis showed a generic "system utility" binary.
• Worm: SMB scanning and admin share writes were entirely runtime behaviors, the static binary contained no hardcoded IPs or share names.
• Botnet Agent: C2 domain was generated algorithmically (DGA) at runtime. Static strings contained only the seed algorithm, not the actual domains.

Scenario 1: The Ransomware Nobody Detected

A medium-sized company with no EDR, no SIEM, and no FIM on their file server. The ransomware executed on a Tuesday at 2 AM via a phishing attachment opened by an accounting clerk. By 3 AM it had encrypted 40,000+ files across the shared finance drive and HR server, including the payroll database. The malware deleted shadow copies on execution. Offline backups were two weeks old. They paid. Recovery still took eight days.

The IOC that would have caught it: vssadmin.exe called with delete shadows arguments, a Sigma rule with 5-minute alerting would have given the SOC a 25-minute window before the encryption sweep completed.

Scenario 2: The Worm That Spread in 12 Minutes

A worm landed on a single workstation. It exploited EternalBlue (MS17-010), a patch from 2017 that was never deployed. Within 12 minutes, 47 endpoints were compromised across three VLANs, including two domain controllers. The worm carried a second-stage ransomware payload. By the time users reported missing files, the entire accounting VLAN was encrypted.

The IOC that would have caught it: Port 445 scanning from a workstation to other internal IPs. Workstations don't normally scan SMB ports across subnets. A simple network traffic baseline + anomaly alert would have flagged this in under 60 seconds.

Scenario 3: The Botnet Agent That Mined for Three Months

A botnet agent ran silently on a finance department workstation for three months. High CPU usage was dismissed as "old hardware." Out-of-band management ports were never monitored. The malware mined cryptocurrency during business hours and joined DDoS campaigns at night. The cloud compute bill spiked by 400%, attributed to "someone left a test server running." The real cause was 12 infected machines participating in coordinated attacks.

The IOC that would have caught it: DNS TXT queries to newly registered domains, combined with sustained CPU usage over 80%. Both are trivial to monitor with basic SIEM correlation rules.

Beyond the Paper: Operationalizing the IOCs

Registry Persistence Monitoring (Wazuh FIM Rule)

<rule id="100200" level="12">
  <if_sid>550</if_sid>
  <match>\\Software\\Microsoft\\Windows\\CurrentVersion\\Run</match>
  <description>Persistence: Registry Run key modified</description>
  <mitre>
    <id>T1547.001</id>
    <id>T1112</id>
  </mitre>
</rule>

Shadow Copy Deletion Detection (Sigma Rule)

title: Shadow Copy Deletion via vssadmin.exe
id: c0e9b4c8-3c1a-4e5f-9f0a-1d2e3f4a5b6c
status: stable
description: Detects vssadmin delete shadows — strong ransomware indicator
logsource:
  product: windows
  category: process_creation
detection:
  selection:
    Image|endswith: '\vssadmin.exe'
    CommandLine|contains|all:
      - 'delete'
      - 'shadows'
  condition: selection
level: critical
tags:
  - attack.defense_evasion
  - attack.T1490

DNS Anomaly Monitoring (from Botnet C2 Findings)

The botnet agent used DNS TXT queries for data exfiltration. Normal office environments produce almost zero TXT queries. A spike in TXT record lookups to newly registered domains, especially from a single internal host, is a high-confidence C2 indicator. Monitor this with Zeek DNS logs fed into your SIEM.

Memory Forensics for Incident Response

If you suspect active malware: acquire a memory dump before powering off the machine. Volatility's malfind reveals injected code regions. psscan shows terminated processes. netscan shows network connections even after the process exits. Combined with CAPE Sandbox's behavioral report, you build a forensic timeline without enterprise forensic tools.

Closing

This research ran on a 16 GB RAM laptop with a consumer SSD. No enterprise hardware. No six-figure lab. Open-source tools, CAPE Sandbox, Volatility, QEMU/KVM, INetSim, handled everything. The methodology scales. The IOCs are indistinguishable from what a commercial sandbox would produce.

The gap between academic research and operational security is narrower than people think. This paper proved that a student with open-source tools and methodical documentation can produce publishable, actionable threat intelligence.

References