Qualys App for ServiceNow CMDB Documentation (aka CMDB Sync)

Document created by Jeff Leggett Employee on Mar 18, 2016Last modified by Jeff Leggett Employee on Jul 18, 2017
Version 14Show Document
  • View in full screen mode

About the Qualys App for ServiceNow CMDB

The Qualys App for ServiceNow CMDB synchronizes Qualys IT asset discovery and classification with the ServiceNow Configuration Management Database (CMDB) system. The App automatically updates the ServiceNow CMDB with any assets discovered by Qualys and with up-to-date information on existing assets, giving ServiceNow users full visibility of their global IT assets on a continuous basis. Conversely, if an asset is added to the ServiceNow CMDB, the App will add it to the Qualys asset inventory. For assets that exist in both asset repositories, selected metadata can be synchronized.

 

Update 9/16/16: Version 1.1.0 published to the ServiceNow store now supports Fuji, Geneva and Helsinki. 

 

Prerequisites

Make sure you have a valid Qualys Account Subscription with API Access.

 

Visit the ServiceNow Store, search for this app, and click Contact Seller. Your TAM will be in touch regarding pricing, and then ServiceNow will provision the app into an instance of your choice.  After that, the app will start appearing in the "Downloads" list in your instance. Then you need to click the "Install" button there to start using the app. After you are done, you will have a new module in your ServiceNow instance that looks like this:

 

Overview1.png

 

Setup

After installation, add API source(s).

  1. Go to Qualys App for ServiceNow CMDB > Configuration > API Sources, and click “New” button.
  2. Enter required details in the form and click “Submit”.

setup1.png

 

Name is anything you would like to call it, and Username and Password are valid Qualys Cloud Platform credentials, with API access enabled.

 

setup2.png

 

After you configured your API source, and saved, Choose the connection you just built, and "Test Connection":

 

setup3.png

 

One you have a successful connection you are ready to move on to Schedules.

 

Schedules

You will need to setup at least 1 schedule.  You may eventually want many more.

 

A note about the Service Now user's Timezone setting

In the schedule scripts we use ServiceNow's new GlideDateTime().getDisplayValueInternal(); function to update the schedule last_run_timestamp. When this object is instantiated directly and used (e.g. in scoped application background script), it returns time in GMT, irrespective of the timezone set for user under whom this script runs. That's how it is designed. Also, scoped applications are not allowed to set the timezone. (otherwise we could have queried the timezone, and set it for the script execution). BUT, the time value you see on the UI is shown in the user set timezone - even if you set GMT date-time in this column.When the schedule runs next time, it gets value in GMT, and not the one you see on UI. That may lead to confusion, and log entries show time in GMT, for this reason we recommend that the Service Now user set his or her time to GMT.

 

Limitation of Service Now in open API calls

Service Now has a 10 minute limit for leaving a connection open, so any schedule you wish to run that may result in LARGE data return sets should be set to a schedule of every 15 minutes or so to run.

 

schedule1.png

 

Qualys to ServiceNow Scheduling

 

schedule2.png

 

You will give this configuration a Name, Choose the API Source you setup int he previous step, and a Qualys Asset Tag you want synced over.  We do not recommend leaving this blank.  Also, choose if you would like us to sync Ports, Software and Hardware information.  The more detailed a scan you have done with Qualys Cloud Platform, the more detail you will have here.  Cloud Agent will have the most detail of an asset, while Authenticated Scans will have the next most detail, with Un-authenticated scans having the least.

 

ServiceNow to Qualys Scheduling

 

schedule3.png

 

The "Qualys Asset Tag" box will assign that tag in Qualys Cloud Platform to any assets synced from ServiceNow. We also highly recommend you add filter conditions (at minimum IP Address) to assets to be synced.  Finally make sure you enable VM (Vulnerability Management) and/or PC (Policy Compliance) checkbox(es) to be able to scan these assets you sync.

 

Properties

 

properties1.png

 

You may define application specific properties on this page.

  1. Select the Qualys Import API call truncation limit . This property defines how many host assets to include in a single Qualys API response.  
    For hostasset APIs, default truncation limit is 100 - i.e. if you do not provide that in preferences, it will return 100 records. However, you can provide any value between 1-1000. If you provide truncation limit which is greater than 1000, it results in INVALID_REQUEST error.
    In our SN app, we have set the default value to 100. If SN is killing the import queue processing jobs, then user can lower that value so that XML processing time fits in job execution time limits. 
    We have provision to up that truncation limit up to 1000, in case customer knows their assets do not have much data (ultimately resulting in smaller XML size) and if they want to keep number of API calls made as low as possible. 
    For example, you can set higher truncation limit if you aren't pulling any hardware/software information. In such a case, each host asset record will not have huge information associated. 
    One should use that only if they KNOW that information in each record will be smaller.
  2. Size of Import batch. This property defines the batch size for import queue. Import queue processor will pick up only these many records from queue at a time.
  3. Select Size of Export batch. This property defines the batch size for export queue. Export queue processor will pick up only these many records from queue at a time.

 

Sync

 

Import Queue

This shows the list of jobs run from Qualys TO ServiceNow Assets and their status.  The XML that was transferred is also available here (usually attached as response.xml):

 

importqueue.png

 

Approve Qualys Assets

Assets imported from Qualys to Service Now will be here for approval to be added to your ServiceNow CMDB.  You will need to approve each individually or a screen at a time.  It will overwrite data in your CMDB if you approve the asset.

 

approve.png

 

If fields gathered aren't showing in your list, do the following:
1. Referring the same screen shot, click on the gear icon that's to the upper left of main pane.
2. In the pop up that opens, you see two lists - Available and Selected. 
3. Find and double-click "MAC Address" from the Available list. It should end up in Selected list. 
4. Click OK. Now your view refreshes, and you should start seeing the MAC address column. 
We set values when that tag is present in XML. So, if for some interfaces MAC address is not available (XML does not contain it OR its empty), the value in SN table column would be empty. 
It's the same reason why you don't see Hostname for all the network interfaces in the screen shot. 

 

Export Queue

This is the list of assets synced from ServiceNow to Qualys Cloud Platform.  If an IP Address exists in Qualys Cloud Platform we do not overwrite, we skip it and move on.

 

exportqueue.png

 

Advanced

 

App Scheduled Jobs

All of the Apps schedules Jobs are listed from here.  An important one to be aware of is the "Qualys Asset Tags fetching job" which runs daily by default.  This syncs all of the Asset Tags in Qualys Cloud Platform for use within the App.  You may wish to run this more than once a day if you generate tags in Qualys Cloud Platform on a more regular basis.

 

approve.png

 

Transform Maps

A transform map is a set of field maps that determine the relationships between fields in an import set and fields in an existing ServiceNow table, such as Incidents [incident] or Users [sys_user]. After creating a transform map, you can reuse it to map data from another import set to the same ServiceNow table.The Transform Maps module enables an administrator to define destinations for imported data on any ServiceNow tables. Transform mapping can be as simple as a drag and drop operation to specify linking between source fields on an import set table and destination fields on any ServiceNow table. Use transform mapping to map source and destination fields dynamically.The Transform Maps the Qualys App for ServiceNow CMDB uses are now all listed in a handy location here.  FOr more information on Transform Maps see http://wiki.servicenow.com/index.php?title=Creating_New_Transform_Maps#gsc.tab=0

 

transform.png

 

Reports

We give you a few canned reports as an example of the kind of data visualization you can do with ServiceNow and the Qualys App for ServiceNow data.

 

Qualys Assets Tags by Source

 

report1.png

 

Assets Tag Distribution

 

report2.png

 

OS Distribution

 

report3.png

 

Support

  • Hours of Operation: 8am - 5pm PST
  • Days of Operation: Monday - Friday (except national holidays, or as defined by law)
  • Promised Call Response Time: Within 12 hours of received support request
  • Promised Call Resolution Time: Within 5-10 business days of response
  • Contact Method: Website
  • Contact Details
  • Online Documentation

Debugging and Troubleshooting

 

How to debug

  1. Application writes log entries at appropriate places, and after each important step.
  2. Also, whenever application finishes important activity, it logs “ <activity> Completed” entries.
  3. In case of problems, one should search the Application Logs module to find all the entries related to this application.
    1. See what all messages are logged by application, related to problem area.
  4. If application’s log entries are not sufficient enough, and if you have access to script includes, you may add your own log statements.

 

Observed Issues, how to troubleshoot them and work-arounds

  1. In case of huge data returned by Qualys API, the Import Queue Processor may timeout and terminate.
    1. In such a case, go to Properties page and lower the Import API call truncation limit.
  2. Issue with ServiceNow GlideSysAttachment.getContent():
    1. It is observed that, if attachment size is more than 5 mb, the getContent() method returns empty string (“”), even though attachment in Import Queue record shows correct and complete XML.
    2. In such a case, application puts that import queue entry in “Error” state, and updates the “processing_notes” column with “Cannot process the attachment. File size maybe too large.”
    3. If you encounter such a situation, you are advised to lower the “x_qual5_cmdb_sync.import_truncation_limit” property value to such a number, where response size will be under 5 mb.

 

Anticipated Issues

  1. No connection to API server.
    1. Such a case should get handled in Qualys Assets Sync script include, leading to graceful exit with proper log entries.
  2. Import Queue Processor timeout during processing a particular response.
    1. This may leave the corresponding Import Queue entry in “Processing” state for quite a long time.
    2. In such a case, user should manually change the status back to
      1. “Queued”, if he wants to process that response again.
        1. If you reprocess any response, it will not lead to duplicate data, as application checks whether the record already exists in staging tables before inserting.
      2. “Error”, if he does not want to process it again.

 

List of expected failure modes

  1. Qualys API server down.
  2. Qualys subscription expired.
  3. User credentials used are incorrect.
  4. User credentials are correct, but they do not have API access.

 

Frequently Asked Questions

 

Qualys to ServiceNow Sync

  • Do you currently or do you plan to support the IndentifyAndReconcile API for CMDB CRUD actions? https://docs.servicenow.com/product/configuration_management/concept/c_CMDBIdentifyandReconcile.html  Goal of this API is to maintain the integrity of the database, and to correctly identify Cis so that new records are created only if CI is truly new to CMDB. The current version does not support this API. And, as of now, there is no plan to use it. However, we use transform maps and coalesce feature to update the matching record, if found. (matched on IP address only) If no matching record found, only then it creates a new one.
  • Is the comparison delta derived from just a few tables or the base CMDB_CI table? The records are primarily compared and updated/created on cmdb_ci_computer table. However, user wants to use any other table, they can easily update the transform map to work with some other table of their choice.
  • Do you re-class the CI record if your IP endpoint device changes?  Do you have a list of classes you have mapped for CI record creation?  We do not alter the class of CI record.
  • When you create/update a CI record do you record a datetime and identifier somewhere other then the description field for proper sorting/filtering? Whenever the record in cmdb_ci_computer table  is updated/newly created, we set “discovery_source” column to “Qualys”. If you search with “Discovery source contains Qualys”, you should get all these records.
  • What fields in SN do ports, software and hardware write to if checked?  Since we didn’t find tables serving our purpose to store this information, we have added new tables in the application scope. Except network adapters and volumes, rest of the information (open ports, installed software, processors) go into these tables in app scope. Network adapters information goes into cmdb_ci_network_adapter table and volumes information goes into cmdb_ci_file_system table.

 

ServiceNow to Qualys Sync

  • Is it possible to sync back more then one table?  Yes, you need to create one schedule per such table.
  • Do you handle syncing back the CMDB_CI or server base tables?  The table field is user-selectable. You can select any table as long as it have column named “ip_address”, containing valid IP address.

Attachments

    Outcomes