Integrating E2E and Application Security Testing: HOWTo with NightwatchJS and OWASP ZAP

The Problem

Ever since I started my journey in DevSecOps and Application Security Automation, one of the key areas of my work has been “Parameterized Scanning”. “Parameterized Scanning” started off when we were attempting to automate an application security test for one of our largest clients, a World’s Top 10 Travel Portal. Before Parameterized Scanning, the pen tester had to rely either on manually crawling through the application to identify parameters, consequently testing them to identify security vulnerabilities, or use the “spider” feature of the Web Application Scanner. The spider feature of most Web Application Vulnerability Scanners has become a relic that we should be associating with the dark ages. Why? Simply because traditional spidering simply does not work. Traditional spidering works on the principle of crawling a particular HTML page, discovering new links, forms, etc on the HTML page and attempting to identify more such pages with the same technique, recursively. This largely has been rendered obsolete today.

• Powerful front-end JavaScript frameworks like Angular/React/Vue/Mojo2, etc that are often Single Page Apps, and generate links and other content on the fly, based on events and user actions. These frameworks have made it very easy for a user to interact with a web application, making them highly responsive and user-friendly, but have made it notoriously difficult for spiders to identify content, forms and parameters, thereby reducing the spider’s result to a “nothingburger”.
• A lot of the apps we test are web services or micro-services today. With the rise of mobile apps and multiple client requirements, developers seldom create “monolithic” web applications any more, but rather prefer a modular approach to creating a web service that has the ability to work with multiple clients, and other applications as API. Clearly, spiders are completely useless here as there’s no content to discover in the first place.

This creates a significant problem for DAST Scanners. Some of them provide “Recorded Authentication” settings. They figure that since “Authenticated Scans” (scans where you need to be authenticated as the user) are the only thing that’s important, they allow a user to record an authentication sequence in their scan tool and replay this scan sequence to recreate the authentication and subsequently, the scanner still resorts to spidering the application. This still doesn’t solve the problem, because spiders dont find anything anymore.

Which is why => Parameterized Scanning

The Solution — Parameterized Scanning

Parameterized Scanning is the practice of creating/leveraging software test automation to use as an input for Application Security Testing. Let’s assume that your organization has a QA department. Let’s assume that this QA department creates test automation (End to End Tests, e2e test) in Selenium or similar framework. The idea is to leverage that existing test automation to serve as an input for the DAST Scanner. Essentially:

1. The Test runs against the application, with the DAST scanner as the proxy.
2. Once the test completes the scan process, the DAST Scan starts scanning for security vulnerabilities
3. Once the scan is done, the DAST scanner generates/publishes reports/artifacts that can be used by engineering teams to triage and fix the identified vulnerabilities.

As you can see, this model has some non-trivial benefits (only a few are captured here):

• The entire test has valid parameters, urls and modules that are being used by the e2e test. Therefore, this can be used against even complex front-end heavy apps, web services, etc. This makes the “coverage” aspect of a security test more specific, with all the right parameters and inputs. This makes the test more effective, and the possibility of finding more security vulnerabilities, more probable.
• The entire test can be run in an Automated manner, thereby making it a great candidate for DevOps integration. Once this E2E test is available, and ready to use, it becomes a matter of running and re-running the security test against the application
• Security Testing becomes more modular — You can choose to test only specific modules of the app that are captured by the E2E test. Let’s say you want to test the Patient Management Functionality of the application, you can do so, without having to configure the scanner’s crawler from identifying all kinds of URLs that may even be out of scope.

We at we45, have created a host of solutions/tools that facilitate Parameterized Scanning, right from our product Orchestron Client to RoboZap, (Github link) etc that can be used by our customers and the community at large to perform Parameterized, Automated Scanning, leveraging existing or new e2e tests. However, thus far, my examples and work had largely been with Python based libraries like Python-Selenium, Python-requests among others.

I have written this blogpost to showcase an experiment in a JavaScript End-to-End Testing Framework. I have been working extensively with NodeJS recently, so I thought that I would give this a shot and write about some of my experiences for everyone’s benefit.

For this example, I have used the following:

• NightwatchJS (an E2E Testing Framework for NodeJS that is meant to facilitate “Browser Automation Testing”. It uses Selenium under the hood => http://nightwatchjs.org/
• Selenium Server
• ChromeDriver => I felt that Chrome was far faster and more reliable than Firefox for Selenium Tests recently.
• Export Report => Its a ZAP Add-on that I use to generate my JSON report
• An intentionally vulnerable web application called “we care” that was developed by us at we45. Its available as a Docker container => https://hub.docker.com/r/nithinwe45/wecare/
• ZAP JSON-RPC Service. A Small JSON-RPC service that I created that you can use to interact with ZAP’s API. => https://github.com/we45/OWASP-ZAP-JSON-RPC-Service

The Good

• OWASP ZAP is probably one of the best tools that you can use for integration into an automated pipeline. Its API is extremely powerful and allows the user to control even the smallest operational aspect of ZAP. Highly recommended for this reason. ZAP also has a host of other benefits including some really powerful Add-ons etc
• Writing the End-to-End test in NightwatchJS was a breeze. But there were several issues with the other things (see “The Bad”). Nightwatch also provides Test Hooks. These are events that can be fired before the test, after the test and before and after each test. I have used test hooks (before and after) extensively in this example
• (Shameless Plug) I am glad I wrote my minimalistic ZAP JSON RPC service. While ZAP does have a REST API, its painful to use REST endpoints over more intuitive “functions” that a JSON-RPC service allows you to use.

The Bad:

• Nightwatch is NodeJS, which is JavaScript, which is Asynchronous. There were several times that this led me to nearly tear my hair out in frustration. Async events tend to be a little alien to Python devs like myself. Also, with Testing, you like things to run in sequence. Node and JS make you work for it. You’d see evidence of my frustration in the before and after hooks, where I have used “hacky” solutions to stitch things together.
• OWASP ZAP Javascript API is not very usable. I had to make up for it with the JSON-RPC service. It all worked out in the end 🙂
• Firefox and Geckodriver. I had initially envisioned using Firefox and Geckodriver for the test. But the proxy experience was so terrible and Geckodriver was so slow, I decided to switch the Chrome. It was far far more consistent, and wayyy faster.

Steps in the Example

The first thing we want to do is set up Nightwatch to proxy the browser’s traffic through our ZAP’s proxy port. In my example, this port is 8090 (ZAP). Here’s a gist of the Proxy Settings in nightwatch.conf.js. I have also setup a bunch of other settings in this file, including the ZAP executable path, the report path, etc.

Chrome Proxy Settings in nightwatch.conf.js

Other Settings that capture ZAP Settings

The next thing I did, was setup a “before” test hook. True to the name, this is a test hook that is fired, BEFORE the test commences. What I like to do is to start the ZAP executable in this hook and get it ready to receive requests from the E2E test. I have executed ZAP in the GUI mode, but adding a headless flag when you are invoking the executable runs it without the GUI (needed in CI-style environments)

‘before’ hook with ZAP Start

Next, the E2E test executes and runs all the tests. Here, I have two very simple tests setup. This is your actual test.

NightwatchJS Tests — Authenticate and Browse another page

Once the test executes, we want the browser to close. Subsequently, I signal to ZAP (through the ZAP JSON RPC service) to commence an “Active Scan” against the application. Once the Active Scan is triggered, its status (in percentage) is queried every 10 seconds. Be advised that with large apps, scans can take the proverbial “several hours”. Once the scan completes (100%), the result is written to the the report path (from nightwatch.conf.js) as a JSON document. Once all this is done, ZAP shuts down and all is well with the world 🙂 All of this magic happens in the “after” test hook.

‘after’ test hook — Start ZAP Scan, Print status, Export Report and Shutdown ZAP

Tips and Tricks

• I make this recommendation to all my clients. And I maintain it. Never write massive tests. They result in scans running for hours. Smaller tests have smaller scan runtimes, hence more focused results.
• Optimize ZAP’s Scan Policy to suit your application. I highly recommend that you don’t run the default, especially with plugins like Active Scan ++, etc enabled. I have used a scan policy called “Light” (in this example) which runs a low intensity set of checks for only a few vulnerabilities. You should tailor your scan policy to your application. For instance an Active Scan check for “Trace axd Information Disclosure” would probably be useless against a NodeJS app as its only meant to capture flaws with a ASP.NET application. Or similarly, checking for SQL Injection with SQLite doesnt make any sense when your DB is MySQL.
• If you are testing HTTPS sites, you should consider installing ZAP’s certificate into the browser that you are planning on using.

NightWatchJS with OWASP ZAP

This entire example can be downloaded from this repo: https://github.com/we45/Nightwatch-ZAP

Follow Abhay on Twitter: @abhaybhargav

Abhay Bhargav is the CTO of we45, a focused Application Security company. Abhay is the author of two international publications. “Secure Java for Web Application Development” and “PCI Compliance: A Definitive Guide”. Abhay is a builder and breaker of applications, and has authored multiple applications in Django and NodeJS. He is the Chief Architect of “Orchestron”, a leading Application Vulnerability Correlation and Orchestration Framework. He is a passionate Pythonista and loves the idea of automation in security. This passion prompted him to author the world’s first hands-on Security in DevOps training that has been delivered in multiple locations, and recently as a highly successful training programs at the OWASP AppSecUSA 2016, OWASP AppSec EU and USA 2017. Abhay recently delivered a workshop on SecDevOps at DEFCON 25.