Comparative Evaluation of OWASP ZAP, Burp Suite Community Edition, and Acunetix for Detecting Web Application Vulnerabilities

Abstract

Dynamic Application Security Testing (DAST) is often evaluated by simply counting scanner alerts, even though alert volume conflates crawler reachability, reporting redundancy, vulnerability judgment quality, and false positives. This Stage 1 registered report presents a controlled, reproducible experiment comparing ZAP (formerly OWASP ZAP), Burp Suite Community Edition, and an academic or trial deployment of Acunetix, across the DVWA and OWASP Juice Shop environments. The design evaluates SQL injection, stored and reflected cross-site scripting, command injection, cross-site request forgery, authentication and access control failures, security misconfiguration, sensitive data exposure, path traversal, local and remote file inclusion, weak cookies, missing security headers, open redirects, and server information disclosure; coverage is further extended to map onto the OWASP Top 10:2021. Positive test units and matching negative controls are established before scanner reports are opened.

Two detection modes separate end-to-end crawl effectiveness from reachability-conditioned detection. Ten randomized iterations per scanner, per application, per mode provide timing and resource observations, while double-blind adjudication determines true and false findings. Pre-registered metrics include detection rate, precision, recall, false positive rate, false discovery rate, overall accuracy, F1, OWASP coverage, execution time, CPU and memory consumption, CVSS distribution, severity agreement, remediation utility, report quality, setup effort, usability, learning burden, automation, CI/CD integration, and export support. Paired tests, non-parametric alternatives, confidence intervals, multiple-comparison correction, and effect sizes are also pre-specified.

The capability gap is treated explicitly as a structural difference: Community Edition includes neither Burp Scanner nor automated crawling, so its self-reported DAST results are structurally inapplicable, not zero. Burp Community is therefore evaluated on a separate track explicitly labeled as analyst-assisted and educational in nature. This protocol contributes a falsifiable benchmark, a closed ground-truth model, a weighted, profile-sensitive selection framework, and a complete reproduction package, designed to prevent artificial or incomparable scanner rankings and to support an evidence-based Stage 2 article.