Exporting the Vulnerability KnowledgeBase to an external Database

Document created by Eric Perraudeau Employee on Oct 5, 2011Last modified by Parag Baxi on Jul 22, 2013
Version 20Show Document
  • View in full screen mode

Note: This is not officially supported code.

Update July 11th 2013

Maximun CSV field size limited to 32000 charaters to accomodate Excel maximun field size. Note that LibreOffice does not have this issue and can open CSV file with field size greater than 32kB. Not that as defined in RFC 4180 (http://tools.ietf.org/html/rfc4180), CSV field does not have a size limit.

 

Update May 30th 2013

Fixed an issue with xsltproc (A potential infinite template recursion) by using the option "--maxdepth 5000". Attached zip file as been updated accordingly.

Update Dec. 21st 2011

Added support for new XML elements release in QualysGuard 6.23. The script now create a new CSV file (or table) that contains the list of vendor and software for each QID.

 

More information about the change in the knowledgebase in the QualysGuard 6.23 API Notification. See the section marked "Knowledge Base APIV2: Vendor and Product Info Included in XML Output".

Introduction

 

There are many situations where you want a local copy of the QualysGuard KnowledgeBase, including:

  • to create custom reports,
  • to correlate KnowledgeBase data against a third-party solution, or
  • to feed an external ticketing system.

 

The above cases all require manipulation of the KnowledgeBase data, and it is more efficient to manipulate a local copy than it is to make individual queries into QualysGuard.

 

The reason for that is pretty simple, a detected vulnerability (or a detection) is an instance of a QID for an asset. Most likely, this detection will be found multiple time for different assets in your network. And all the description of this QID, such as the threat, impact, solution, don't need to be replicated for each and every detection. So the idea is to reference another table that contains this information, the QID will be used as a unique identifier.

 

This document explains how you can leverage the KnowledgeBase API v2 in order to get a local copy of the vulnerability knowledge base.

 

WARNING: Before trying to use this script, on need to make sure that the optional "Knowledgebase Download" module is enabled for your subscription. If you are getting an error like:

<TEXT>You are not allowed to download the KnowledgeBase, 
please contact your sales representative for more information.</TEXT>

it means that the module is not enabled. Contact me or QualysGuard support for more information.

 

How it works

The attached script uses the program "curl" to download a compete copy of the QualysGuard KnowledgeBase in XML. Then it uses the program "xsltproc" with 8 different XSLT files to create 8 CSV files. The main CSV file is kb_v2-vuln.csv. All the other files use the QID as a foreign key to this file.

 

More information about XSLT at: http://en.wikipedia.org/wiki/Xslt

 

Installation and execution

  1. This script is built to run on a linux/unix/mac system. The script requires curl and xsltproc, which come by default with most of the distributions. On Windows, curl can be used (note: make sure to use double quotes for the command line parameters, as it doesn't support single quotes like on Linux), but the XSLT transformations should be performed with msxsl.exe, and the script "go.sh" should be modified.
  2. Download the attached zip archive on your machine and uncompress it to a folder.
  3. Edit the file "go.sh" and set your QualysGuard API user login and password.
  4. run go.sh
  5. Et voila! You're done.

 

At this point, this script produces output in CSV format. Use your favorite data import tool to import the CSV file into your database.

 

Notes and Disclaimer

This script is delivered as is. It should not be run in a production environment without a review and a good knowledge of its limitations.

It should be considered as proof of concept / prototype and should be customized to serve your needs.

 

Concerning the security of the script, the QualysGuard login and password are stored in clear text in the file "go.sh" and will be displayed in the list of the running processes when the script is launched. I recommend using the file "~/.netrc" to secure this file properly, in conjunction with the curl option "-n" in order to protect the login/password.

 

If you want to maintain a copy of the KB in your local database, I recommend running a differential update on a regular basis using the following request that gives you all the QIDs that change after the date YYYY-MM-DD:

> https://qualysapi.qualys.com/api/2.0/fo/knowledge_base/vuln/?action=list&details=All&last_modified_after=YYYY-MM-DD

 


Attachments

Outcomes