Whoops! Cleaning up Mistakes via API

Posting again after kind of a lengthy break. It's summertime, and for lots of disparate reasons I've queued up topics but haven't had the ambition or taken the time to write them all out. So today we get a tale of automation mistakes and the subsequent cleanup.

For some background on what I'm writing about, we have an inventory application that can be managed via API, and a few (less than five) times per year we 'bulk' order equipment. For reasons I won't get into here, we pre-load all of these devices into our inventory application, but until they are ordered or go to fulfillment their inventory records are incomplete. Populating the basic data is pretty manual (transcription of device serial numbers, though sometimes by barcode scan depending on vendor), and since it happens 1-3 times per year I've not yet automated much of that initial process. And so, the serial numbers are obtained in whatever way convenient and relayed to me via some spreadsheet or dataset.

The Improvement

This winter I built a process to use the inventory application's API to handle the two most laborious processes: creating new items and, to a lesser degree, updating certain data points on existing items. The process of adding new items via the app's interface is pretty wild and requires a lot of steps. For adding more than just a few, it takes forever, and is ripe with opportunity for data entry errors, etc. So I wrote a thing in PHP to basically taken an array of new object details and create (or update, if necessary) objects:

$newResources[] = array(
    'barcode'=>'1234567890',
    'serialNumber'=>'566aabb',
    'deviceName'=>'prefixmodel-566aabb',
    'resourceType'=>$computerResource,
    'manufacturer'=>'Dell',
    'budgetNumber'=>'1234-56789-01011',
    'purchaseOrder'=>'abcd123',
    'vendor'=>'Dell',
    'assignedUsername'=>'username',
    'assignedDeptname'=>'deptname'
);

This is then fed to the mechanism that parses them out into the proper series of API calls and creates/updates the record. As a bonus (not covered here, but perhaps a future topic), this other mechanism I made queries either our Jamf management (for Apple devices) or the Dell API directly for warranty and other information, and adds that information to the records automatically. Works super swell, and has saved a load of time.

The example above works great for a small number or creates/updates, but this summer I had a batch of about 150 devices to add based on a single bulk order (where many of the details are identical such as purchase order, manufacturer, etc.). Further, at the time of bulk load, we don't have barcode numbers or the assigned user/dept since these are still new in box. So several of these items will remain unused until the time of fulfillment.

An important side note: In the example above, we have a resourceType (of $computerResource for a desktop computer). In our environment, there are also distinct other types such as $laptopResource and $dockResource to use accordingly.

The New Issue

Since all the details save for the serialNumber (and resulting deviceName), this example doesn't scale well for larger sets. So I wrote a simple mechanism to auto-create the proper structure:

$desktops = array('1234567','2345678', ... ,'9990123');
foreach($desktops as $svctag) {
  $newResources[] = array(
    'barcode'=>'',
    'serialNumber'=>$svctag,
    'deviceName'=>'prefixmodel-'.$svctag,
    'resourceType'=>$computerResource,
    'manufacturer'=>'Dell',
    'budgetNumber'=>'1234-56789-01011',
    'purchaseOrder'=>'abcd123',
    'vendor'=>'Dell',
    'assignedUsername'=>'',
    'assignedDeptname'=>''
  );  
}

Then I was able to use my Generate IN Statement script (well, a form of it since its mechanism is useful in myriad ways) to transform the list I received in a spreadsheet to the resulting $desktops array.

Running this bit of code then populates the larger array with details based just on the input I gave it (for serial numbers), and the upstream mechanism to do the adds takes just a minute. PERFECT!

Haste Makes Waste, Right?

As I noted above, resourceType ($computerResource, $laptopResource, and $dockResource) is important to distingush when creating the record. They can be changed after the fact, but it's not trivial in the UI. Also, records can never truly and fully be deleted from the system once created (they can be "deleted" in a sense that makes them hidden from normal use). Since my test worked great and it was the end of the day, I plunged ahead...

Three arrays ($desktops, $laptops, and $docks) with serial numbers were fed to the system...EXCEPT the system all made them $dockResources and gave them all the wrong prefixmodel because of my hasty copy/paste error.

As such, I ended up with about 100 records of both the wrong type AND the wrong naming scheme. Well, shit.

The Fix Was Trivial

Since I'd built this underlying API interface/mechanism to address both creates and updates a nicely-formatted 35 lines of code was an easy fix to implement. In particular, the most important bits are below (truncated for clarity):

$desktops = array('1234567','2345678', ... ,'9990123');
foreach($desktops as $svctag) {
  $newResources[] = array(
    'serialNumber'=>$svctag,
    'deviceName'=>'prefixmodel-'.$svctag,
    'oldName'=>'badprefix-'.$svctag,
    'resourceType'=>$computerResource
  );  
}
foreach ($newResources as $newResource) {
  $serialNumber = $newResource['serialNumber'];
  $deviceName = $newResource['deviceName'];
  $oldName = $newResource['oldName'];

  $queryArgs = array("name"=>$oldName);
  $resourceID = getResourceOIDByQuery($resourceQuery, $queryArgs);
  $updateFallback = array(
    "name"=>$deviceName,
    "serialNumber"=>$serialNumber,
    "resourceType"=>$resourceType,
  );
  $update = updateResource($resourceID, $updateFallback);
}

This simply looks up the resource by "bad" name and corrects the name and the type. Since I had the the "bad" information at hand in the existing arrays it was pretty simple to implement, even in one of those "Oh shit!" crisis moments.

Leftovers

Moving forward I've put in some basic controls to prevent this from happening. In a weird way, the "best" part of this hasty mistake is that the "fix" really only took about fifteen minutes to conceptualize and implement, and mere seconds to run and fix. If the fixes had to have been done manually (or this set of data manually added as it was in the Before Times), fifteen minutes would've been a mere fraction of the time it'd have taken to use the application's UI. Ultimately the mistake has created a new resource available for us in the future, though hopefully never to be necessary!