Qualys Cloud Platform VM "detection" API examples and use cases

Document created by Eric Perraudeau Employee on Feb 1, 2011Last modified by Jeff Leggett on Jun 11, 2016
Version 20Show Document
  • View in full screen mode

Updates

2012-11-02: added explanation for pagination logic

2012-08-30: get the list of port and services added

2012-08-27: "Differential download" paragraph added

2016-6-11: Made link to github example code


Introduction

The host list "detection" API (/api/2.0/fo/asset/host/vm/detection/ API endpoint) gives API users the ability to obtain “automatic” vulnerability detection data that can be easily imported into a third party solution. Using the host detection API users can request a list of hosts with the hosts latest vulnerability data, based on the “automatic” data available in the user’s account. The GET or POST access method may be used to make a host list detection API request.

 

Automatic data brings a lot of value to customers because they provide the latest complete vulnerability status for the hosts (NEW, ACTIVE, FIXED, REOPENED) and history information. Automatic data are completely independent of scan results and option profiles. QualysGuard normalizes the vulnerability scan results into the database using a complex and sophisticated process. This mechanism generates what is called the vulnerability “automatic" data.

 

The new host list detection API is recommended as a replacement for other QualysGuard APIs when the API user wants to manage “automatic” data and integrate this with third party applications. For many third party applications this API is a replacement for the following existing QualysGuard APIs:

  • asset_range_info.php
  • asset_data_report.php
  • asset_search.php
  • get_host_info.php

 

Use Cases and example

1. Create Custom Technical Reports with vulnerability details

 

Technical reports need additional information for each vulnerability such as description, solution, threat or impact. The detection API provides the QID for each vulnerability found for an asset. The QID is a unique ID that references a vulnerability within the QualysGuard KnowledgeBase.

 

Use the following workflow to create custom technical reports:

  • Step 1. Use the host list detection API to return “automatic” vulnerability data for hosts in your account, as described in these release notes.
  • Step 2. Use the KnowlegeBase Download API (knowledgebase_download.php) to obtain vulnerability data, such as the vulnerability description, threat and impact. It’s possible to make a request for all vulnerabilities (QIDs) in the KnowledgeBase or just a specific vulnerability.

 

For example, to make a request for QID 90082 use the following URL:

https://qualysapi.qualys.com/msp/knowledgebase_download.php?vuln_id=90082

 

Example XML output for QID 90082 from the KnowledgeBase Download API:

...

<VULNS>

<VULN>

<QID>90082</QID>

<VULN_TYPE>Vulnerability</VULN_TYPE>

<SEVERITY_LEVEL>3</SEVERITY_LEVEL>

<TITLE><![CDATA[Microsoft Windows DHCP Server Configured To Evade Rogue Detection]]></TITLE>

<CATEGORY>Windows</CATEGORY>

<LAST_UPDATE><![CDATA[2009-05-12T22:47:20Z]]></LAST_UPDATE>

<DIAGNOSIS>

<![CDATA[A rogue DHCP server is any DHCP server not authorized to serve IP addresses on the network. The presence of any such server can cause severe IP address collisions, and in the presence of Dynamic DNS updates can propagate to DNS servers across the domain. <P> Microsoft Windows DHCP servers have a mechanism to protect against roguedetection. Every Windows DHCP server periodically checks with the Active Directory if it is authorized to function in the (sub)domain. However, this security check can be disabled with a registry tweak. <P> The scanner found this registry modification on the target Windows host. If this modification is not authorized, it's most likely a rogue DHCP server maliciously evading detection.]]>

</DIAGNOSIS>

<CONSEQUENCE>

<![CDATA[The rogue DHCP server can potentially cause severe IP collisions and corrupt DNS databases across the (sub)domain.]]>

</CONSEQUENCE>

<SOLUTION>

<![CDATA[If the registry key modification was not authorized, remove the following value from the key: <P> DisableRogueDetection under HKLMSYSTEM\CurrentControlSet\Services\DHCPServer\Parameters (with the REG_DWORD value of 1)]]>

</SOLUTION>

</VULN>

...

 

  • Step 3. Correlate the vulnerability information in the third party application using the QID number provided in the <QID> XML output which is returned by the host detection API (Step 1) and the KnowledgeBase Download API (Step 2).

 

A typical integration would be to create tables in a database for the XML output from both QualysGuard API functions and use QID as a key for a join. This way it would be possible to create queries that will provide all the vulnerabilities for a given set of hosts (according to custom search criteria) and their description.

 

2. Get All PCI Vulnerabilities

  • Step 1. First you need to create a dynamic search list titled “PCI Vulns” using the QualysGuard user interface. When creating the dynamic search list, select the PCI option next to Compliance Type as shown below.

Screen shot 2011-02-01 at 1.11.26 PM.png

  • Step 2. Create an asset group titled “PCI Hosts” containing the hosts which are in scope for PCI compliance.
  • Step 3. Make the following host list detection API request using the asset group title “PCI Hosts” and the search list title “PCI Vulns”:

https://qualysapi.qualys.com/api/2.0/fo/asset/host/vm/detection/?action=l ist&ag_titles=PCI+Hosts&include_search_list_titles=PCI+Vulns'

 

3. Asset Search Portal Capability

The host detection API endpoint allows users to do the same kind of searches as the Asset Search Portal user interface using these search parameters:

  • Asset Groups
  • IPs/Ranges
  • Operating System
  • QID

The host detection API offers users more search parameters and capability than the Asset Search Portal user interface. Using the API users can get the details of vulnerability detections. Also, the API allows users to do more advanced searches on QIDs using all the power of search lists.

Screen shot 2011-02-01 at 1.14.42 PM.png

 

4. Download the list of port and services

Similar to what you can get in the UI in the "host -> port and services" page, the detection API can be used to download all the port and services by using specific QIDs, like in the example below:

 

curl -u 'LOGIN:PASSWORD' -H 'X-Requested-With: curl' 'https://qualysapi.qualys.com/api/2.0/fo/asset/host/vm/detection/?action=list&show_igs=1&qids=82023,82004' > open_ports.xml

 

Extract of the XML output that provide the list of port and services in the <RESULT> section:

 

 

    <HOST_LIST>
      <HOST>
        <ID>1876448</ID>
        <IP>10.10.10.26</IP>
        <TRACKING_METHOD>NETBIOS</TRACKING_METHOD>
        <OS><![CDATA[Windows XP Service Pack 0-1]]></OS>
        <DNS><![CDATA[winxp-dhchnage-aan]]></DNS>
        <NETBIOS><![CDATA[WINXP-DHCHNAGE-]]></NETBIOS>
        <LAST_SCAN_DATETIME>2010-04-21T21:53:24Z</LAST_SCAN_DATETIME>
        <DETECTION_LIST>
          <DETECTION>
            <QID>82004</QID>
            <TYPE>Info</TYPE>
            <RESULTS><![CDATA[Port      IANA Assigned Ports/Services    Description     Service Detected
123     ntp     Network Time Protocol   ntp
135     msrpc-epmap     epmap DCE endpoint resolution   msrpc udp
137     netbios-ns      NETBIOS Name Service    netbios ns
138     netbios-dgm     NETBIOS Datagram Service        unknown
445     microsoft-ds    Microsoft-DS    unknown
500     isakmp  isakmp  unknown
1026    unknown unknown unknown
1027    unknown unknown unknown
1900    unknown unknown unknown]]></RESULTS>
          </DETECTION>
          <DETECTION>
            <QID>82023</QID>
            <TYPE>Info</TYPE>
            <RESULTS><![CDATA[Port      IANA Assigned Ports/Services    Description     Service Detected        OS On Redirected Port
135     msrpc-epmap     epmap DCE endpoint resolution   DCERPC Endpoint Mapper   
139     netbios-ssn     NETBIOS Session Service netbios ssn      
445     microsoft-ds    Microsoft-DS    microsoft-ds     
1025    blackjack       network blackjack       msrpc    
3389    ms-wbt-server   MS WBT Server   win remote desktop       
5000    Socket23        backdoor commplex-main  ssdp]]></RESULTS>
          </DETECTION>
        </DETECTION_LIST>
      </HOST>
      <HOST>


 

Download the list of "FIXED" vulnerabilities

By default, the ouput of the detection API contains all the vulnerabilities that are current, including New, Active and Re-Opened Vulenerabilities. In order to doanload the vulenrabilties that have been fixed, the detection API needs to be called with the optional parameter "status={value}" that will show only hosts with one or more of these status values: New, Active, Re-Opened, Fixed. Multiple status values can be entered as a comma-separated list like in the following example:

 

curl -u "LOGIN:PASSWORD" -H "X-Requested-With:curl" "https://qualysapi.qualys.com/api/2.0/fo/asset/host/vm/detection/?action=list&status=New,Active,Fixed,Re-Opened"

 

Differential download

The detection API can be used to only download the detections of hosts that have been scanned since a specific date by using the parameter "vm_scan_since=date". This can be used to implement a differential mechnism by using the detection API on a regular basis with this parameter sets to the date of the previous call (you can add a small overlap to make sure you don't miss anything). In this case you probably need to to an initial full dowload for the first call.

 

Here is an example of such a call:

 

curl -n -H "X-Requested-With:curl" "https://qualysapi.qualys.com/api/2.0/fo/asset/host/vm/detection/?action=list&vm_scan_since=2012-01-01"

 

Paginated output

The output of this API is paginated. By default, a maximum of 1,000 host records are returned per request. If the requested list identifies more than 1,000 hosts, then the XML output includes the <WARNING> element and instructions for making another request to download the nest page of host records.

 

You can customize the page size (i.e. the number of host records) by using the parameter "truncation_limit=10000" for instance. In this case the results will be return with pages of 10,000 host records.

 

When using "truncation_limit=0" it means that the output is not paginated and all the records are returned in a single output. WARNING: this can generate very large output and processing large XML files can consume a lot of resources on the client side. In this case it is recommended to use the pagination logic and parallel processing. The previous page can be processed while the next page is being downloaded.

 

Sample Script for Pagination Logic

The following script illustrates how it’s possible for a third party application to retrieve paginated results using the host list detection API.

 

#!/bin/sh

BASE_URL='https://qualysapi.qualys.com'
OUTPUT_FILENAME='detection'
USERNAME='username'
PASS='password'
N='01'
## Specify filtering parameters here (Remember to URL encode os_pattern, if specified):
#PARAMETER="&show_igs=1&include_search_list_titles=QID+45038&os_pattern=%5Cd%2B&output_format=CSV&suppress_duplicated_data_from_csv=0"

echo "${BASE_URL}/api/2.0/fo/asset/host/vm/detection/?action=list${PARAMETER}"
curl -H 'X-Requested-With: curl' -u "${USERNAME}:${PASS}" "${BASE_URL}/api/2.0/fo/asset/host/vm/detection/?action=list${PARAMETER}" > "${OUTPUT_FILENAME}-${N}.xml"
while grep '<URL><!\[CDATA\[https:\/\/.*qualys\.com' "${OUTPUT_FILENAME}-${N}.xml"
do
          NEXT_URL=`grep "<URL>" "${OUTPUT_FILENAME}-${N}.xml" | sed s/'^.*<URL><!\[CDATA\['//g | sed s/'\]\]\>\<\/URL\>'//g`
          N=`expr ${N} + 1`
          if [[ ${N} -lt 10 ]] ; then N=`echo "0"${N}` ; fi
          if [[ ${N} -gt 99 ]] ; then echo "bailing out after fetching the first 99 batches of data"; exit 0; fi
          echo "next url = ${NEXT_URL}"
          curl -H 'X-Requested-With: curl example QWEB 6.17' -k -u "${USERNAME}:${PASS}" "${NEXT_URL}" > "${OUTPUT_FILENAME}- ${N}.xml"
done

 

Other example code for the host detection API can be found here: community/hostdetection at master · Qualys/community · GitHub

1 person found this helpful

Attachments

    Outcomes