API v2 with Powershell 3.0 Invoke-RestMethod

Document created by gowen on Jan 13, 2016
Version 1Show Document
  • View in full screen mode

API v2 is a RESTful API, and starting with Powershell 3.0 InvokeRestMethod provides a smooth native method for making REST calls.  Here I will present a script which retrieves the list of reports from Qualys, selects the newest, and downloads that for manipulation within Powershell.  This will demonstrate all of the following:

 

  • RESTful API calls with InvokeRestMethod
  • Session authentication with cookies in Powershell
  • Powershell manipulation of XML returned by the API
  • Powershell manipulation of CSV returned by the API

 

 

As a disclaimer, this is my first Powershell script.  But in the process of writing it, I've tried to learn and use its strengths, and not just write a batch file using Powershell.  Don't hesitate to call out fails (e.g., my handling of messy CSV input is inelegant; feel free to point out a better way)!

 

Code Walkthrough

 

Because this is a development script, I hard-wired the parameters - the credentials and the name of the report to be looking for - at the top of the script.  Obviously, for production use, you'd replace these with either command line arguments or interactive prompts.  But for our learning purposes, this is acceptable.  The "target" is the name of the Report as seen in the Reports repository.

 

# Tweak these
$username = 'me_user'
$password = 'me_password'
$target = "Daily Whatsis Roundup"








 

Now we will set up the first API call.  The $headers and $base variables will be re-used throughout the script, but $body will only be applicable to this first call.  Also note that a new variable called "sess" is going to be created and set as a side effect of the Invoke-RestMethod call; we'll use it later.

 

# Not these
$hdrs = @{"X-Requested-With"="powershell"}
$base = "https://qualysapi.qualys.com/api/2.0/fo"
$body = "action=login&username=$username&password=$password"
Invoke-RestMethod -Headers $hdrs -Uri "$base/session/" -Method Post -Body $body -SessionVariable sess








 

This has the effect of performing step 1 of the Session Based Authentication example on page 19 of the API v2 User Guide.  The next Invoke-RestMethod call(s) in this script will use the cookie set and stored in $sess for authentication.  It's useful to note that this is true both in the script and interactively at the CLI - if you are using Powershell interactively, you can type in these lines to get a session, and then any trial-and-error REST calls you want to play with can take advantage of that session.  That's a really powerful tool for interactive development and debugging.

 

The next thing we'll do is download a list of reports found in the repository, then filter it so that we only have a list of the reports with the name that we looked for.  This is the REST call documented on pages 113-115 of the User Guide.

 

# Get the list of reports and select the most recent
$rpts = (Invoke-RestMethod -Headers $hdrs -Uri "$base/report?action=list" -WebSession $sess).SelectNodes("//REPORT[contains(TITLE, '$target')]")








 

It's important to understand what's happening here.  The Invoke-RestMethod call is returning an XML object, because Qualys returns an XML document.  By putting the Invoke-RestMethod call inside () we can manipulate that returned object with the SelectNodes call, which uses an XPath query to only return REPORT elements.  Without the SelectNodes, the $rpts object would be the full XML tree - filtering it at acquisition time simplifies the data we need to work with.

 

Now we want to pull out the specific pieces of data I care about to move forward:

 

$objs = @()
foreach ($rpt in $rpts) {
  $obj = New-Object PSObject
  Add-Member -InputObject $obj -MemberType NoteProperty -Name ID -Value $rpt.ID
  Add-Member -InputObject $obj -MemberType NoteProperty -Name Title -Value $rpt.TITLE."#cdata-section"
  Add-Member -InputObject $obj -MemberType NoteProperty -Name Timestamp -Value $rpt.LAUNCH_DATETIME
  $objs += $obj
}








 

We create an empty array called $objs, then for each REPORT element in the XML we create an object to contain the ID, Title, and Timestamp (LAUNCH_DATETIME).  We then append that object to the array, so that when this loop is done we have an array describing the things we need to know about these reports.  By shifting the XML data into objects, we make it possible to do things like sort on individual properties:

 

$goget = $objs | Sort -Descending -Property Timestamp | Select -First 1
Write-Host "Will download ID" $goget.ID "(" $goget.Title ") dated" $goget.Timestamp "`r`n"








 

Here I use Sort to determine which report is the most recent; that's the one I want to download.  I can also print out the list of older reports I'm not going to download in case that's useful information for the user:

 

Write-Host "Ignoring these older reports:"
$objs | Sort -Descending -Property Timestamp | Select -Skip 1 | Format-Table








 

Now it's time for our next API call to fetch the report, found on page 142-144 of the User Guide.

 

# Retrieve the most recent report
$csvhead = "IP","DNS","NetBIOS","Tracking Method","OS","IP Status","QID","Title","Vuln Status","Type","Severity","Port","Protocol","FQDN","SSL","First Detected","Last Detected","Times Detected","CVE ID","Vendor Reference","Bugtraq ID","Threat","Impact","Solution","Results","PCI Vuln","Ticket State","Instance","Category","Associated Tags"
$rpt = (Invoke-RestMethod -Headers $hdrs -Uri "$base/report?action=fetch&id=$($goget.ID)" -WebSession $sess | ConvertFrom-CSV -Header $csvhead






 

When we download the report, we are receiving CSV data.  That's not automatically turned into an object the way that XML data was, so we'll use ConvertFrom-CSV to create an object.  Unfortunately, ConvertFrom-CSV can't be told that there's are several lines of report header before the CSV data actually begins.  That's why we're specifying the headers in $cvshead; if the first row of the file contained the column labels, Powershell would handle that automagically.  Instead  the data we get from Qualys looks like this, where the column labels show up on line 5:

 

"Daily Whatsis Roundup","01/12/2016 at 22:04:01 (GMT)"
"Acme, Inc","123 Somewhere Street",,"Boston","Massachusetts","United States of America","02120"
"Me User","me_user","Manager"

"IP","DNS","NetBIOS", ...








 

so the resulting CSV object is going to have some crap rows.  I couldn't find an elegant way to avoid that, so I brute forced the problem by creating an array with all the rows that I knew contained valid data.  In this case, the first column - IP - will always contain a 10.* IP address on valid lines:

 

# Unfortunately, I can't figure out how to skip the header lines in the report before parsing the CSV,
# So we have to do it afterwards by filtering on valid IP values
$rows = @()
foreach ($row in $rpt) {
  if ($($row.IP).StartsWith("10.")) {
  $rows += $row
  }
}








 

Once I've got a clean copy of the data, I can start manipulating it... for demonstration purposes, let's just print out some discrete columns:

 

# Printing it out...  Obviously, would be better to iterate and generate reports...
Write-Host "Some discrete data for your consideration:"
$rows | Select IP,DNS,QID,Title | Format-Table






 

And the last thing we'll do is log out of our session, also page 19 of the User Guide:

 

# Logout
Invoke-RestMethod -Headers $hdrs -Uri "$base/session/" -Method Post -Body "action=logout" -WebSession $sess








 

 

Results

 

Here's the output when this script is run:

 

PS C:\tmp> powershell -ExecutionPolicy ByPass -File psQualysREST.ps1

 

 

xml                                                         SIMPLE_RETURN

---                                                         -------------

version="1.0" encoding="UTF-8"                              SIMPLE_RETURN

Will download ID 10954910 ( Daily Whatsis Roundup ) dated 2016-01-11T22:02:22Z

 

 


Ignoring these older reports:

 

 

 

 

 

 


ID                                      Title                                   Timestamp

--                                      -----                                   ---------

10943305                                Daily Whatsis Roundup                   2016-01-10T22:02:20Z

10937750                                Daily Whatsis Roundup                   2016-01-09T22:01:24Z

10933955                                Daily Whatsis Roundup                   2016-01-08T22:01:21Z

 

 

 

 

 

 


Some discrete data for your consideration:


IP                      DNS                          QID                     Title

--                      ---                          ---                     -----

10.1.28.19              enterprise.dev.examp...      86476                   Web Server Stopped R...

10.1.28.19              enterprise.dev.examp...      86476                   Web Server Stopped R...

10.1.27.34              voyager.dev.example....      123844                  VLC Media Player "m3...

10.1.15.80              agamemnon.b5.example...      124423                  Wireshark Multiple D..

 

 

 

 

 

 

 

xml                                                         SIMPLE_RETURN

---                                                         -------------

version="1.0" encoding="UTF-8"                              SIMPLE_RETURN

 

 

 

 

PS C:\tmp>

 

Caveats & Comments

  1. The real power of Powershell, here, is the way it allows you to fluidly turn input into objects, and then to manipulate those objects in a granular way.
  2. If Powershell is about objectifying data, REST is about making it easy to say "Hey, I want data," and Invoke-RestMethod is Powershell's way of saying "Okay, let's GO there!"
  3. You will need Powershell 3+ for this.  This work was done on Windows 7; it's not hard to download Powershell 3 for Windows 7 even though it's not native to it.
  4. There's got to be a better way to handle messy CSV input.  I was in a hurry.  It beat the alternatives.  My bus was late.  Etc. etc.
  5. My coworkers and I all had trouble with the Powershell Authentication example posted here, and using session cookies was actually quite elegant, so we did that instead.
  6. There is no error handling; if the login call fails, the rest of the script will bang ahead anyway.  This script makes it work; but making it work right is left to the reader.  Caveat lector.
  7. I shortened some of the variable names (e.g., myWebSession -> sess) to make example code fit on a line better for this post.  I would never use short and cryptic variable names otherwiseHonest.

Attachments

Outcomes