Mastering PE Malware Classification and Evasion Attacks with MLSploit: A Hands-On Guide for CS6264

Introduction: Why PE Malware Classification Matters in 2026

With over 1 million new malware samples discovered daily, security teams rely on machine learning (ML) and deep learning (DL) to classify Windows portable executable (PE) files into known families. This tutorial guides you through the CS6264 lab, where you train LSTM, CNN, and RNN models on API call sequences, attack them with a mimicry evasion technique via MLSploit, and then build stronger defenses. Understanding this workflow is crucial for cybersecurity roles in antivirus companies, threat intelligence, and red teaming.

Setting Up Your Environment for MLSploit

Before diving into model training, ensure you have access to MLSploit through Georgia Tech's VPN. Create an account at mlsploit.org with a strong password. Read the 'PE Module Tutorial' and 'MalwareLab Module Tutorial' attachments. The PE module (except the PE transformer) is open-sourced on GitHub. Use the pefile library to parse PE headers without executing malware. Remember: the malware provided (rbot family) is real—never execute it.

Task 1: Training Deep Learning Models on API Call Sequences

You will train three DL models: LSTM, CNN, and RNN. The dataset contains benign PE files and malicious PE files from multiple families. Extract API call sequences using dynamic analysis or simulated traces. For example, an LSTM can capture temporal dependencies in API calls like CreateFile, WriteFile, and RegSetValue. Similar to how streaming services predict your next watch, LSTM predicts the next API call to classify malware families.

# Example: Training an LSTM with Keras
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense, Embedding

model = Sequential()
model.add(Embedding(input_dim=vocab_size, output_dim=128, input_length=max_seq_len))
model.add(LSTM(128, dropout=0.2))
model.add(Dense(num_families, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

Save your model as pe.model.zip and predictions as prediction.zip. Include log files.

Task 2: Attacking DL Models with Mimicry Attack

Using MLSploit's mimicry attack, you will craft adversarial PE samples that evade your trained models. The attack adds benign-like API calls to malicious samples, fooling the classifier—similar to how a deepfake video mimics a real person. Configure the attack via MLSploit's interface and generate attack-exe.zip, attack-feature.zip, attack-prediction.zip, and attack.cfg.zip. Log all outputs.

Task 3: Detecting the Attack with Static Features

Train new models (e.g., Random Forest, XGBoost) on static features like PE header fields, section entropy, and imports to distinguish attacked samples from benign. For instance, attacked samples may show unusual import table sizes. Write Python scripts (detection1.py, etc.) and save trained models (model1.zip, etc.). This step mirrors how antivirus updates signatures after a new evasion technique emerges.

# Example: Random Forest detection
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report

clf = RandomForestClassifier(n_estimators=100)
clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)
print(classification_report(y_test, y_pred))

Task 4: Training Classic ML Models on Enhanced Features

Extend features to include API existence (binary), frequency (counts), and arguments (e.g., registry key paths). Train models like SVM, Logistic Regression, and Gradient Boosting. These models are lighter than DL and often used in production due to interpretability. Think of it as a credit score model: it uses multiple factors (API usage patterns) to decide 'malicious' or 'benign'. Save models and predictions.

Task 5: Attack Transfer to ML Models

Evaluate how well the mimicry attack samples from Task 2 evade your classic ML models. This tests transferability—a key concept in adversarial ML. If your SVM misclassifies the attacked sample, it indicates the attack generalizes. Log predictions and analyze false negatives. This is analogous to a phishing email that bypasses one spam filter but gets caught by another.

Bonus Task: Evading Malware Detection with Reinforcement Learning

Use two open-source tools: a static malware detector and a reinforcement learning (RL) agent that modifies PE files to evade detection. The RL agent learns to add benign sections or modify entry points—similar to how a game AI learns to beat levels. This task is worth 40 points and requires deep understanding of PE structure and RL algorithms like DQN.

Written Report Tips

Your report should include diagrams (e.g., model architectures, confusion matrices) and citations (e.g., the mimicry attack paper). Use Times New Roman 12pt. Include an appendix with MLSploit screenshots. Explain why certain models performed better: for example, CNNs may capture local API patterns better than RNNs for short sequences.

Conclusion

By completing this lab, you gain hands-on experience in adversarial ML for cybersecurity—a skill in high demand as AI-powered attacks grow. Whether you're defending against ransomware or building next-gen antivirus, these techniques are foundational. Submit your deliverables as a .tar.gz file and ensure all logs are included.