The Case of Invoke-RestMethod vs. The Bad JSON Feed

I work with some incredibly smart and talented people on a daily basis at work who build some pretty cool systems that integrate with the services my team runs for our campus.  One of those services is an API that is run by our Identity Management team that gives us an interface to work with all of the identity data for campus.  This REST API allows us to query not only information about accounts in our environment, but feed data back in for things like provisioning email aliases or just notifying them that we’ve given someone a mailbox.  Like I said, pretty cool stuff!

We use the API for both interactively querying account information and for working with accounts in automation.  Several of the members of my team started noticing that some of our PowerShell functions weren’t returning data periodically.  The bells and whistles started going off when we started receiving failures on several pieces of PowerShell automation due to null data being received when pulled data on an individual account.  So, let’s look one piece of code we’re using :

This PowerShell function is a fairly basic example of how to work with a REST API using the Invoke-RestMethod cmdlet.  This particular function lets me run

on myself and get back everything that is stored about my account when Invoke-RestMethod hits the API.  In this case, I’m getting back null data.  That’s not right…

Invoke-RestMethod is a pretty cool cmdlet for working with a REST API.  It’s smart enough to recognize that it is receiving back data from my API in JSON format and take that data and turn it into objects that I can work with in PowerShell.  The small gotcha is that Invoke-RestMethod really depends on this being valid JSON.  See where I’m going here???

So, it looks like we’re getting back null data, but that shouldn’t be happening.  What do we do next?  The first thing I checked is the data that we’re actually pulling.  I have the ability to pull back “all” in the requested_attributes.  Starting there, if I decrease the scope of data I’m requesting and limit it to something like “primaryemailaddress”, all of a sudden everything is happy.  Strange…  Pulling “all” again… nope… null…

The API is returning a ton of data, but we typically only care about the result data and not all the other messages, logs, and other general fluff that comes back from the API… until now.  In my function, you’ll see that we have a line with

where we’re getting out results back from the API. In this case, I want to see all the messages back from the API to see if we’re receiving any errors or other useful information.  To do that, we’re going to change that line to

so we get back everything. That will give us the full response from the REST API so we can see what we’re getting back and play around with all the data in the response.

After re-loading the function, I’m going to re-run my command and see what I get back. (Sparing you the ugly output, there was nothing useful is the messages that were returned in the huge glob of JSON I got back.) So, let’s dump it everything in a variable so we can play with it a bit:

Next, let’s try piping that through ConvertFrom-Json and then we can parse through the data to see what’s going on:

Yeah buddy!  An error!  Now this is something we can actually work with!  “Cannot convert the JSON string because a dictionary that was converted from the string contains the duplicated keys ‘persondirectoryid’ and ‘PersonDirectoryId’.”  BINGO!  It seems our REST API JSON feed is giving us duplicate data in the form of one attribute in lower case and another in camel case.

As a short term fix, we can use “.Replace” to replace the bad data that we’re getting so things work properly:

In this case, we notified the team that owns the API application and they were able to correct the issue with the duplicate attribute.  Though, this does bring up an interesting shortcoming of Invoke-RestMethod:  in this case, it had no tolerance whatsoever for the invalid data.  Both entries were the same except that one was all lower case and one was camel case.  I guess in a perfect world, it would be nice if there was a -CaseSensitive parameter to allow different cases of entries or some other way I could -IgnoreErrors or –DropErrors.  But, bad data is bad data and fixing the data fixed my problem in this case.

Leave a Reply