Progressive scanning is a feature within Qualys Web Application Scanning (WAS) that is now available to all customers. The intent and goal of progressive scanning is to add a mechanism to effectively scan very large web applications over the course of several scans. This is sometimes needed due to restrictions that may prevent you from scanning an entire application at one time. Examples of restricting factors include small scan windows, slow-responding servers, and huge or complex applications with many pages/forms/cookies that need to be tested during the scan.
Progressive scanning works by allowing WAS to run an initial scan and then store the state of the scan. The saved state includes information such as links crawled, vulnerability tests run on those links, and vulnerabilities identified. When the scan is then re-launched, the scanning "picks up where it left off" and finishes the remaining tests as well as begins to crawl and test new links. In this way, WAS can complete a scan of a very large application over the course of several scans.
When is Progressive Scanning Useful?
As you may be aware, there are two key limitations within WAS as far as the maximum size of a scan is concerned. WAS can test up to a maximum of 8000 links or run for 24 hours in a single scan, whichever comes first. If your web application is large and/or slow such that it doesn't finish in a single scan, it may be a candidate for progressive scanning. However, be aware that depending on the nature of your scanning practices and the size of the application, it may take months to obtain a full scan. Let's say you have an application with 20,000 links for example. It would take 3 scans to test the application entirely (i.e., the first two scans test 8000 links and the third scan tests 4000 links). If you scan on a daily basis, it will take only 3 days to have a fully tested application. However, if you scan the application once a month, only partial results will be available on this app for two full months. Therefore, progressive scanning is most useful for organizations that are able to do frequent scanning (or at least have scan windows readily available) versus those organizations who scan relatively infrequently.
How Progressive Scanning Works
Continuing vs. Non-continuing
This is one of the most important concepts with progressive scanning. WAS will behave differently on a "continuing" scan versus a "non-continuing" scan. Knowing how to begin each type of scan is important for the success of progressive scanning within your organization.
A continuing scan, as the name implies, is a continuation of a previous scan. In this mode, progressive scanning uses the stored data from the previous scan and will prioritize unfinished tasks - that is, it will not re-test content or vulnerabilities that were discovered in an earlier iteration. A continuing scan is created by using "Scan Again" functionality on the Scans tab or by using a scheduled scan with progressive scanning enabled. The first scan in a schedule is non-continuing, but each subsequent scan within the same schedule is "continuing" by default and prioritize crawling and testing new links.
A non-continuing scan does not take into account any previous scans. This is a traditional, stand-alone scan and occurs when it's the first time an application is scanned with progressive scanning enabled. It also occurs, of course, when a scan is run with progressive scanning turned off.
How to Enable Progressive Scanning
If you don't see the Progressive Scanning options in WAS, it probably needs to be enabled for your WAS subscription. Simply send a request to email@example.com (or ask your Qualys Technical Account Manager) to have progressive scanning enabled on your subscription. Once enabled, you'll see three locations where the progressive scanning option appears.
Web Application Profile
The first location where Progressive Scanning can be configured is within the web application profile. Here you'll see a checkbox that enables you to turn on or off progressive scanning at the web app level. You can override this when you run a scan, but it's recommended to set the progressive scanning checkbox appropriately within the web app profile. This default value is "off".
Launching a New Scan
When you choose to launch a scan, you'll also be presented with a progressive scanning checkbox. This will override the web app configuration setting. Remember, when launching a scan this way, you'll be creating a non-continuing scan, unless you've launched the scan by selecting "Scan Again" option from the Scans tab (in which case you'll be creating a continuing scan).
Scheduling a Scan
Finally, when scheduling a scan you'll also be presented with progressive scanning options. In this case, you can choose to enable or disable progressive scanning, or to honor the setting in the web app profile. If it's enabled, this will result in the first scan of the schedule being non-continuing, with every subsequent scan being a continuing scan.
Important Related Features
Keep in mind that other features within WAS can and should be used to help you scan very large web applications. Configuring blacklists and defining redundant links in the web app profile are two common best practices in this regard. Blacklists and redundant links can help eliminate unnecessary scanning, reduce scan times, and increase the overall effectiveness of your testing program. For example, a shopping site or product catalog may have dynamically-generated URLs for each and every product. Although the links are unique for each product in this scenario, the code behind the link is often exactly the same. It is therefore unnecessary to crawl & test each link for every single product in the catalog. Another common example is a web application where the developers have appended a time stamp to URLs (a "t" parameter for instance) to prevent browser caching. WAS only needs to scan these types of URLs once, not once for every time stamp.
The option profile settings within WAS will also impact your scans and should be considered as well. The scan intensity and the search criteria are the main factors when it comes to scanning large applications. Longer scan times will result when a low scan intensity is used versus a higher intensity. Likewise, using a "complete" search criteria results in longer scan times compared to testing for only certain vulnerabilities/QIDs.