Integrating E2E and Application Security Testing: HOWTo with NightwatchJS and OWASP ZAP

The Problem

Ever since I started my journey in DevSecOps and Application Security Automation, one of the key areas of my work has been “Parameterized Scanning”. “Parameterized Scanning” started off when we were attempting to automate an application security test for one of our largest clients, a World’s Top 10 Travel Portal. Before Parameterized Scanning, the pen tester had to rely either on manually crawling through the application to identify parameters, consequently testing them to identify security vulnerabilities, or use the “spider” feature of the Web Application Scanner. The spider feature of most Web Application Vulnerability Scanners has become a relic that we should be associating with the dark ages. Why? Simply because traditional spidering simply does not work. Traditional spidering works on the principle of crawling a particular HTML page, discovering new links, forms, etc on the HTML page and attempting to identify more such pages with the same technique, recursively. This largely has been rendered obsolete today.

  • Powerful front-end JavaScript frameworks like Angular/React/Vue/Mojo2, etc that are often Single Page Apps, and generate links and other content on the fly, based on events and user actions. These frameworks have made it very easy for a user to interact with a web application, making them highly responsive and user-friendly, but have made it notoriously difficult for spiders to identify content, forms and parameters, thereby reducing the spider’s result to a “nothingburger”.

This creates a significant problem for DAST Scanners. Some of them provide “Recorded Authentication” settings. They figure that since “Authenticated Scans” (scans where you need to be authenticated as the user) are the only thing that’s important, they allow a user to record an authentication sequence in their scan tool and replay this scan sequence to recreate the authentication and subsequently, the scanner still resorts to spidering the application. This still doesn’t solve the problem, because spiders dont find anything anymore.

Which is why => Parameterized Scanning

The Solution — Parameterized Scanning

Parameterized Scanning is the practice of creating/leveraging software test automation to use as an input for Application Security Testing. Let’s assume that your organization has a QA department. Let’s assume that this QA department creates test automation (End to End Tests, e2e test) in Selenium or similar framework. The idea is to leverage that existing test automation to serve as an input for the DAST Scanner. Essentially:

  1. The Test runs against the application, with the DAST scanner as the proxy.

As you can see, this model has some non-trivial benefits (only a few are captured here):

  • The entire test has valid parameters, urls and modules that are being used by the e2e test. Therefore, this can be used against even complex front-end heavy apps, web services, etc. This makes the “coverage” aspect of a security test more specific, with all the right parameters and inputs. This makes the test more effective, and the possibility of finding more security vulnerabilities, more probable.

We at we45, have created a host of solutions/tools that facilitate Parameterized Scanning, right from our product Orchestron Client to RoboZap, (Github link) etc that can be used by our customers and the community at large to perform Parameterized, Automated Scanning, leveraging existing or new e2e tests. However, thus far, my examples and work had largely been with Python based libraries like Python-Selenium, Python-requests among others.

I have written this blogpost to showcase an experiment in a JavaScript End-to-End Testing Framework. I have been working extensively with NodeJS recently, so I thought that I would give this a shot and write about some of my experiences for everyone’s benefit.

For this example, I have used the following:

  • NightwatchJS (an E2E Testing Framework for NodeJS that is meant to facilitate “Browser Automation Testing”. It uses Selenium under the hood =>

The Good

  • OWASP ZAP is probably one of the best tools that you can use for integration into an automated pipeline. Its API is extremely powerful and allows the user to control even the smallest operational aspect of ZAP. Highly recommended for this reason. ZAP also has a host of other benefits including some really powerful Add-ons etc

The Bad:

  • Nightwatch is NodeJS, which is JavaScript, which is Asynchronous. There were several times that this led me to nearly tear my hair out in frustration. Async events tend to be a little alien to Python devs like myself. Also, with Testing, you like things to run in sequence. Node and JS make you work for it. You’d see evidence of my frustration in the before and after hooks, where I have used “hacky” solutions to stitch things together.

Steps in the Example

The first thing we want to do is set up Nightwatch to proxy the browser’s traffic through our ZAP’s proxy port. In my example, this port is 8090 (ZAP). Here’s a gist of the Proxy Settings in nightwatch.conf.js. I have also setup a bunch of other settings in this file, including the ZAP executable path, the report path, etc.

Chrome Proxy Settings in nightwatch.conf.js
Other Settings that capture ZAP Settings

The next thing I did, was setup a “before” test hook. True to the name, this is a test hook that is fired, BEFORE the test commences. What I like to do is to start the ZAP executable in this hook and get it ready to receive requests from the E2E test. I have executed ZAP in the GUI mode, but adding a headless flag when you are invoking the executable runs it without the GUI (needed in CI-style environments)

‘before’ hook with ZAP Start

Next, the E2E test executes and runs all the tests. Here, I have two very simple tests setup. This is your actual test.

NightwatchJS Tests — Authenticate and Browse another page

Once the test executes, we want the browser to close. Subsequently, I signal to ZAP (through the ZAP JSON RPC service) to commence an “Active Scan” against the application. Once the Active Scan is triggered, its status (in percentage) is queried every 10 seconds. Be advised that with large apps, scans can take the proverbial “several hours”. Once the scan completes (100%), the result is written to the the report path (from nightwatch.conf.js) as a JSON document. Once all this is done, ZAP shuts down and all is well with the world 🙂 All of this magic happens in the “after” test hook.

‘after’ test hook — Start ZAP Scan, Print status, Export Report and Shutdown ZAP

Tips and Tricks

  • I make this recommendation to all my clients. And I maintain it. Never write massive tests. They result in scans running for hours. Smaller tests have smaller scan runtimes, hence more focused results.

This entire example can be downloaded from this repo:

Follow Abhay on Twitter: @abhaybhargav

Abhay Bhargav is the CTO of we45, a focused Application Security company. Abhay is the author of two international publications. “Secure Java for Web Application Development” and “PCI Compliance: A Definitive Guide”. Abhay is a builder and breaker of applications, and has authored multiple applications in Django and NodeJS. He is the Chief Architect of “Orchestron”, a leading Application Vulnerability Correlation and Orchestration Framework. He is a passionate Pythonista and loves the idea of automation in security. This passion prompted him to author the world’s first hands-on Security in DevOps training that has been delivered in multiple locations, and recently as a highly successful training programs at the OWASP AppSecUSA 2016, OWASP AppSec EU and USA 2017. Abhay recently delivered a workshop on SecDevOps at DEFCON 25.



CTO of we45 (An AppSec Company), DevSecOps Greasemonkey, Passionate Security Technologist and Creator

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Abhay Bhargav

CTO of we45 (An AppSec Company), DevSecOps Greasemonkey, Passionate Security Technologist and Creator