About the Qualys App for ServiceNow CMDB
The Qualys App for ServiceNow CMDB synchronizes Qualys IT asset discovery and classification with the ServiceNow Configuration Management Database (CMDB) system. The App automatically updates the ServiceNow CMDB with any assets discovered by Qualys and with up-to-date information on existing assets, giving ServiceNow users full visibility of their global IT assets on a continuous basis. Conversely, if an asset is added to the ServiceNow CMDB, the App will add it to the Qualys asset inventory. For assets that exist in both asset repositories, selected metadata can be synchronized.
Update 9/16/16: Version 1.1.0 published to the ServiceNow store now supports Fuji, Geneva and Helsinki.
Make sure you have a valid Qualys Account Subscription with API Access.
Visit the ServiceNow Store, search for this app, and click Contact Seller. Your TAM will be in touch regarding pricing, and then ServiceNow will provision the app into an instance of your choice. After that, the app will start appearing in the "Downloads" list in your instance. Then you need to click the "Install" button there to start using the app. After you are done, you will have a new module in your ServiceNow instance that looks like this:
After installation, add API source(s).
- Go to Qualys App for ServiceNow CMDB > Configuration > API Sources, and click “New” button.
- Enter required details in the form and click “Submit”.
Name is anything you would like to call it, and Username and Password are valid Qualys Cloud Platform credentials, with API access enabled.
After you configured your API source, and saved, Choose the connection you just built, and "Test Connection":
One you have a successful connection you are ready to move on to Schedules.
You will need to setup at least 1 schedule. You may eventually want many more.
A note about the Service Now user's Timezone setting
In the schedule scripts we use ServiceNow's new GlideDateTime().getDisplayValueInternal(); function to update the schedule last_run_timestamp. When this object is instantiated directly and used (e.g. in scoped application background script), it returns time in GMT, irrespective of the timezone set for user under whom this script runs. That's how it is designed. Also, scoped applications are not allowed to set the timezone. (otherwise we could have queried the timezone, and set it for the script execution). BUT, the time value you see on the UI is shown in the user set timezone - even if you set GMT date-time in this column.When the schedule runs next time, it gets value in GMT, and not the one you see on UI. That may lead to confusion, and log entries show time in GMT, for this reason we recommend that the Service Now user set his or her time to GMT.
Limitation of Service Now in open API calls
Service Now has a 10 minute limit for leaving a connection open, so any schedule you wish to run that may result in LARGE data return sets should be set to a schedule of every 15 minutes or so to run.
Qualys to ServiceNow Scheduling
You will give this configuration a Name, Choose the API Source you setup int he previous step, and a Qualys Asset Tag you want synced over. We do not recommend leaving this blank. Also, choose if you would like us to sync Ports, Software and Hardware information. The more detailed a scan you have done with Qualys Cloud Platform, the more detail you will have here. Cloud Agent will have the most detail of an asset, while Authenticated Scans will have the next most detail, with Un-authenticated scans having the least.
ServiceNow to Qualys Scheduling
The "Qualys Asset Tag" box will assign that tag in Qualys Cloud Platform to any assets synced from ServiceNow. We also highly recommend you add filter conditions (at minimum IP Address) to assets to be synced. Finally make sure you enable VM (Vulnerability Management) and/or PC (Policy Compliance) checkbox(es) to be able to scan these assets you sync.
You may define application specific properties on this page.
- Select the Qualys Import API call truncation limit . This property defines how many host assets to include in a single Qualys API response. For hostasset APIs, default truncation limit is 100 - i.e. if you do not provide that in preferences, it will return 100 records. However, you can provide any value between 1-1000. If you provide truncation limit which is greater than 1000, it results in INVALID_REQUEST error.In our SN app, we have set the default value to 100. If SN is killing the import queue processing jobs, then user can lower that value so that XML processing time fits in job execution time limits.We have provision to up that truncation limit up to 1000, in case customer knows their assets do not have much data (ultimately resulting in smaller XML size) and if they want to keep number of API calls made as low as possible.For example, you can set higher truncation limit if you aren't pulling any hardware/software information. In such a case, each host asset record will not have huge information associated.One should use that only if they KNOW that information in each record will be smaller.
- Size of Import batch. This property defines the batch size for import queue. Import queue processor will pick up only these many records from queue at a time.
- Select Size of Export batch. This property defines the batch size for export queue. Export queue processor will pick up only these many records from queue at a time.
This shows the list of jobs run from Qualys TO ServiceNow Assets and their status. The XML that was transferred is also available here (usually attached as response.xml):
Approve Qualys Assets
Assets imported from Qualys to Service Now will be here for approval to be added to your ServiceNow CMDB. You will need to approve each individually or a screen at a time. It will overwrite data in your CMDB if you approve the asset.
This is the list of assets synced from ServiceNow to Qualys Cloud Platform. If an IP Address exists in Qualys Cloud Platform we do not overwrite, we skip it and move on.
App Scheduled Jobs
All of the Apps schedules Jobs are listed from here. An important one to be aware of is the "Qualys Asset Tags fetching job" which runs daily by default. This syncs all of the Asset Tags in Qualys Cloud Platform for use within the App. You may wish to run this more than once a day if you generate tags in Qualys Cloud Platform on a more regular basis.
A transform map is a set of field maps that determine the relationships between fields in an import set and fields in an existing ServiceNow table, such as Incidents [incident] or Users [sys_user]. After creating a transform map, you can reuse it to map data from another import set to the same ServiceNow table.The Transform Maps module enables an administrator to define destinations for imported data on any ServiceNow tables. Transform mapping can be as simple as a drag and drop operation to specify linking between source fields on an import set table and destination fields on any ServiceNow table. Use transform mapping to map source and destination fields dynamically.The Transform Maps the Qualys App for ServiceNow CMDB uses are now all listed in a handy location here. FOr more information on Transform Maps see http://wiki.servicenow.com/index.php?title=Creating_New_Transform_Maps#gsc.tab=0
We give you a few canned reports as an example of the kind of data visualization you can do with ServiceNow and the Qualys App for ServiceNow data.
Qualys Assets Tags by Source
Assets Tag Distribution
- Hours of Operation: 8am - 5pm PST
- Days of Operation: Monday - Friday (except national holidays, or as defined by law)
- Promised Call Response Time: Within 12 hours of received support request
- Promised Call Resolution Time: Within 5-10 business days of response
- Contact Method: Website
- Contact Details
- Online Documentation
Debugging and Troubleshooting
How to debug
- Application writes log entries at appropriate places, and after each important step.
- Also, whenever application finishes important activity, it logs “ <activity> Completed” entries.
- In case of problems, one should search the Application Logs module to find all the entries related to this application.
- See what all messages are logged by application, related to problem area.
- If application’s log entries are not sufficient enough, and if you have access to script includes, you may add your own log statements.
Observed Issues, how to troubleshoot them and work-arounds
- In case of huge data returned by Qualys API, the Import Queue Processor may timeout and terminate.
- In such a case, go to Properties page and lower the Import API call truncation limit.
- Issue with ServiceNow GlideSysAttachment.getContent():
- It is observed that, if attachment size is more than 5 mb, the getContent() method returns empty string (“”), even though attachment in Import Queue record shows correct and complete XML.
- In such a case, application puts that import queue entry in “Error” state, and updates the “processing_notes” column with “Cannot process the attachment. File size maybe too large.”
- If you encounter such a situation, you are advised to lower the “x_qual5_cmdb_sync.import_truncation_limit” property value to such a number, where response size will be under 5 mb.
- No connection to API server.
- Such a case should get handled in Qualys Assets Sync script include, leading to graceful exit with proper log entries.
- Import Queue Processor timeout during processing a particular response.
- This may leave the corresponding Import Queue entry in “Processing” state for quite a long time.
- In such a case, user should manually change the status back to
- “Queued”, if he wants to process that response again.
- If you reprocess any response, it will not lead to duplicate data, as application checks whether the record already exists in staging tables before inserting.
- “Error”, if he does not want to process it again.
- “Queued”, if he wants to process that response again.
List of expected failure modes
- Qualys API server down.
- Qualys subscription expired.
- User credentials used are incorrect.
- User credentials are correct, but they do not have API access.
Frequently Asked Questions
Qualys to ServiceNow Sync
- Do you currently or do you plan to support the IndentifyAndReconcile API for CMDB CRUD actions? https://docs.servicenow.com/product/configuration_management/concept/c_CMDBIdentifyandReconcile.html Goal of this API is to maintain the integrity of the database, and to correctly identify Cis so that new records are created only if CI is truly new to CMDB. The current version does not support this API. And, as of now, there is no plan to use it. However, we use transform maps and coalesce feature to update the matching record, if found. (matched on IP address only) If no matching record found, only then it creates a new one.
- Is the comparison delta derived from just a few tables or the base CMDB_CI table? The records are primarily compared and updated/created on cmdb_ci_computer table. However, user wants to use any other table, they can easily update the transform map to work with some other table of their choice.
- Do you re-class the CI record if your IP endpoint device changes? Do you have a list of classes you have mapped for CI record creation? We do not alter the class of CI record.
- When you create/update a CI record do you record a datetime and identifier somewhere other then the description field for proper sorting/filtering? Whenever the record in cmdb_ci_computer table is updated/newly created, we set “discovery_source” column to “Qualys”. If you search with “Discovery source contains Qualys”, you should get all these records.
- What fields in SN do ports, software and hardware write to if checked? Since we didn’t find tables serving our purpose to store this information, we have added new tables in the application scope. Except network adapters and volumes, rest of the information (open ports, installed software, processors) go into these tables in app scope. Network adapters information goes into cmdb_ci_network_adapter table and volumes information goes into cmdb_ci_file_system table.
ServiceNow to Qualys Sync
- Is it possible to sync back more then one table? Yes, you need to create one schedule per such table.
- Do you handle syncing back the CMDB_CI or server base tables? The table field is user-selectable. You can select any table as long as it have column named “ip_address”, containing valid IP address.