AI-Enhanced VPN Traffic Classifier

An encrypted-traffic ML pipeline that separates VPN from non-VPN flows on the ISCX VPN-nonVPN dataset with 98.63% test accuracy — without ever inspecting payloads.

Solo ML engineer — research, pipeline, evaluation

Test accuracy

98.63%

ROC-AUC (VPN class)

0.99

Features used

Metadata only

Inference target

Real-time gateway

Pythonscikit-learnXGBoostPandasNumPySMOTEPCAMatplotlibJupyter

Problem

Modern enterprises need to distinguish VPN-tunnelled traffic from regular traffic for QoS, policy enforcement, and security analytics — but payload inspection is impossible (encrypted) and often illegal. Existing port- and IP-based heuristics break the moment a VPN uses a common port (443) or a rotating endpoint.

▸Classify flows as VPN vs non-VPN using only metadata observable at a gateway.
▸Operate in near real-time so it can sit inline with traffic without bottlenecking it.
▸Preserve user privacy — never read packet contents.

Approach

How I built it.

Data & preprocessing

Built on the ISCX VPN-nonVPN dataset. Flows were cleaned, deduplicated, and reduced to metadata-only features: packet size statistics, inter-arrival timing, direction ratios, and flow duration. Class imbalance was handled with SMOTE on the training split only, and PCA explored dimensionality reduction for the latency-sensitive deployment story.

▸StandardScaler normalization fit on train, applied to val/test.
▸SMOTE oversampling restricted to the training fold to avoid leakage.
▸Stratified train/val/test split (70/15/15) preserving class ratios.

Modeling

A Random Forest baseline established a strong reference point, then XGBoost was tuned for the final classifier. Both models were chosen for their robustness to mixed-scale tabular features and their interpretability via feature importance — important for justifying decisions to a security team.

▸Random Forest baseline with grid-searched depth and estimators.
▸XGBoost with early stopping on a held-out validation set.
▸Feature importance reviewed to confirm the model leaned on timing/size patterns, not artifacts.

Evaluation

Evaluated on accuracy, precision/recall per class, F1, ROC-AUC, and confusion matrices — plus a latency budget measurement for inline use. The metadata-only constraint was treated as a hard requirement, not a nice-to-have, so any feature that hinted at payload content was rejected.

Results

The tuned XGBoost classifier reached 98.63% accuracy on the held-out test set with balanced precision and recall across both classes, and a ROC-AUC of 0.99 for the VPN class. Feature importance confirmed the model relied on flow timing, packet size distribution, and directional ratios — the exact signals available at a real gateway.

▸98.63% test accuracy, balanced across VPN and non-VPN classes.
▸Per-flow inference fast enough to sit inline with traffic.
▸Privacy preserved — zero payload bytes used in features or training.

Learnings

What I'd carry forward.

→Imbalance handling belongs strictly inside the training fold; leaking SMOTE into validation inflates metrics dramatically.
→Tabular boosting still beats deeper architectures on small/medium structured-traffic datasets when latency matters.
→Framing constraints (metadata-only, real-time) up-front made model selection much easier than chasing the highest possible accuracy.

More case studies

Browse other deep-dives or jump back to the full project list.

LearnTube — AI YouTube Learning Assistant →SmartHire AI — Job Portal →All projects →