Nutanix : Get Software Version for each Clusters

Background

When doing upgrade campaign, you need to go to each cluster to verify who's got what software version and then check the progress. Sometimes, it could be painful to collect that information, especially when you have a lot of clusters. I was wondering how this could automated, some obvious details can easy be obtained with "classic" API calls, but not everything. Challenge accepted!

Hidden API ?

The best place to get software versions using the UI is the LCM module. This is where you upgrade everything these days, not only hardware firmware, but easy Nutanix software pieces.

It would be awesome if I can get those details from a simple API call. I tried to dig into all API endpoint versions I know but, I cannot get them all. It is easy to get the AOS version and NCC but others are not findable anywhere.

Guess what ? There is a hidden API called LCM API. Nutanix only introduced it recently. It is not visible by default in some AOS versions.

To determine if the LCM API is enable on your cluster, your can try to reach it from a browser : 

https://<cluster_vIP>:9440/lcm/api_explorer/index.html#

It should open the Swagger explorer with various LCM API endpoints. If not, you can try to enable the functionality from any of the CVMs by typing this command :

$ ./configure_lcm --enable_api

Next, you need to restart genesis.

$ genesis restart

Next, try again to reach the LCM API explorer :

The endpoints we need to retrieve the software versions are :

/lcm/v1.r0.b1/resources/entities/list

and :

/lcm/v1.r0.b1/resources/status

Beware that the overall structure of the LCM API will change in a near future, indeed, this is going to be more widely used and some architecture changes will be made.

Anyway, I was in need for a quick solution, so I decided to use the current version. If needed I will certainly update my scripts later.

What you can do with /entities/list (a POST request) is to pass some filtering to get what you need.

Exemples

To retrieve all software details from a cluster, use this payload :

{

  "filter": "entity_type==software"

}

The cURL syntax will be this one : 

curl -X POST "https://x.x.x.x:9440/lcm/v1.r0.b1/resources/entities/list" -H "accept: application/json" -H "Authorization: Basic Zmxob2VzdDIxxxxxxxxxxGFuZ2VNZSE=" -H "Content-Type: application/json" -d "{\"filter\":\"entity_type==software\"}"

And it will return something like this : 

{

  "data": {

    "entities": [

      {

        "entity_class": "PC Core Cluster",

        "entity_model": "AOS",

        "entity_type": "software",

        "id": "prism-hht5",

        "location_id": "pc:7e51208f-4697-xxxx-b787-2e1374941fcc",

        "request_version": "5.20.1.1",

        "uuid": "66979a7e-6cad-xxxx-b032-b64f9c9c2274",

        "version": "5.20.1.1"

      },

      {

        "available_version_list": [

          {

            "disable_reason": "",

            "is_enabled": true,

            "order": "15",

            "single_group_uuid": "0d1d50aa-b5ef-4f15-xxxx-9d6ef1f74f98",

            "status": "available",

            "uuid": "36xxxxxx9-00fb-422f-92ab-e22d40108206",

            "version": "2.3.0"

          }

        ],

[...]

]      {

        "available_version_list": [

          {

            "child_entities": [],

            "order": "1",

            "release_notes": "https://portal.nutanix.com/page/documents/details?targetId=Release-Notes-Cluster-Maintenance-Utilities:Release-Notes-Cluster-Maintenance-Utilities",

            "single_group_uuid": "xxxxxxxx-c0e0-457c-8710-a76df67de3ea",

            "status": "available",

            "uuid": "9febe4c8-xxxx-48ae-95ae-5052a0e66179",

            "version": "2.0.3"

          }

        ],

        "entity_class": "Cluster Service",

        "entity_model": "Cluster Maintenance Utilities",

        "entity_type": "software",

        "location_id": "node:aaacb1f5-xxxx-4c7a-8fc0-f022c51f017c",

        "request_version": "1.0.0",

        "single_group_uuid": "965d323a-xxxx-457c-8710-a76df67de3ea",

        "uuid": "fb096687-567d-46cc-xxxx-e627350021a0",

        "version": "1.0.0"

      }

    ],

    "metadata": {

      "filter": "entity_type==software",

      "length": 25,

      "offset": 0,

      "total_matches": 25

    }

  },

  "metadata": null

}

I have truncated the above list on purpose, this is very long, but I believe you got the idea. So, the list is long, it is the reason why we can have more powerful filtering capabilities to get exactly what we are looking for.

Let's focus on the NCC version for now, and use this filtering payload : 

{

  "filter": "entity_model==NCC"

}

Here is the response  : 

{

  "data": {

    "entities": [

      {

        "entity_class": "Core Cluster",

        "entity_model": "NCC",

        "entity_type": "software",

        "location_id": "cluster:7e51208f-xxxx-478d-xxxx-2e1374941fcc",

        "request_version": "4.2.0.2",

        "uuid": "b1e8a3e6-fb60-4ec3-b072-73324a86e328",

        "version": "4.2.0.2"

      }

    ],

    "metadata": {

      "filter": "entity_model==NCC",

      "length": 1,

      "offset": 0,

      "total_matches": 1

    }

  },

  "metadata": null

}

You just need to navigate this JSON-style string with the language of your choice ...

In Php, it will be $data->entities[0]->version. Entities is returned as an array since you may have different software release on each node. You will see this later. In this example, I'm querying a Prism Central instance, so there is only "one node".

Other useful filters

"filter": "entity_type==software"

"filter": "entity_class==Core Cluster"

"filter": "entity_class==Hypervisor"

"filter": "entity_class==LICENSING SERVICE"

"filter": "entity_model==AOS"

"filter": "entity_model==NCC"

"filter": "entity_model==Foundation"

"filter": "entity_model==Foundation Platforms"

"filter": "entity_model==Licensing"

"filter": "entity_model==File Server"

"filter": "entity_model==PC"

"filter": "entity_type==firmware"

I'm sure you can guess what is the expected result ;)

You can even combine two or more filters if required : 

"filter": "entity_type==firmware;entity_class==BIOS"

What about LCM version ?

Yes, indeed, how can we get the LCM version number ? It requires another endpoint.

/lcm/v1.r0.b1/resources/status

This request is a GET, here is the cURL version : 

curl -X GET "https://x.x.x.x:9440/lcm/v1.r0.b1/resources/status" -H "accept: application/json" -H "Authorization: Basic Zmxob2xxxxxxxxxOb3RDaGFuZ2VNZSE="

And the response is : 

{

  "data": {

    "cancel_intent": {

      "is_set": false

    },

    "framework_version": {

      "local_version": "2.4.3.1.26954",

      "remote_version": "2.4.3.1.26954",

      "update_needed": false

    },

    "in_progress_operation": {

      "type": null,

      "uuid": null

    },

    "upload_in_progress": {}

  },

  "metadata": null

}

local_version is the currently installed version on the cluster,

remote_version is the next available version on the support portal (ready for download)

Where is the script ?!?

Ok, ok ... Yes, there is a script that gather all the software version at once on any number of clusters from 1 to .... any size.

Here is a sample output :

The Hypervisor column is special. You can see the hypervisor version, either vmware or AHV or both in the case of a mixed cluster. But as well, for Prism Central, I have decided to display the PC version there. Not sure this is wise, but at least this is how I see it now. It might change later.

When a software package is not deployed (because not needed), you see "n/a" in the relevant column.

You can get the script on my GitHub Repo. Do not forget that the updated version of my Php Framework is also required.

There are 2 new useful functions added in my Framework :

function nxGetSWVersions($clusterConnect,$filter)

function nxGetLCMVersion($clusterConnect)

The first one needs a filter to be passed to it. This is the filter in the table above. There is no error checking nor syntax checking on the filter, so be sure to send the correct one. For example, you can call it this way : 

// Get Foundation version

$data=nxGetSWVersions($clusterConnect,"\"entity_model==Foundation Platforms\"");

$data=$data->data->entities;

It will return something similar to this, depending of the number of nodes in your cluster : 

  [0]=>

  object(stdClass)#13 (9) {

    ["entity_class"]=>

    string(15) "Cluster Service"

    ["entity_model"]=>

    string(20) "Foundation Platforms"

    ["entity_type"]=>

    string(8) "software"

    ["last_updated_time"]=>

    string(20) "2021-10-01T12:28:31Z"

    ["location_id"]=>

    string(41) "node:79ce82b4-xxxx-4cc4-9efa-53664ff55165"

    ["request_version"]=>

    string(3) "2.7"

    ["single_group_uuid"]=>

    string(36) "fff46f40-4bc3-xxxx-b595-dbb53b5777f7"

    ["uuid"]=>

    string(36) "cca552b0-xxxx-4180-afe6-156632e5dfdf"

    ["version"]=>

    string(3) "2.7"

  }

  [1]=>

  object(stdClass)#12 (9) {

    ["entity_class"]=>

    string(15) "Cluster Service"

    ["entity_model"]=>

    string(20) "Foundation Platforms"

    ["entity_type"]=>

    string(8) "software"

    ["last_updated_time"]=>

    string(20) "2021-10-01T12:27:20Z"

    ["location_id"]=>

    string(41) "node:0fa5a075-xxxx-4e95-a172-2048beea45aa"

    ["request_version"]=>

    string(3) "2.7"

    ["single_group_uuid"]=>

    string(36) "fff46f40-xxxx-479b-b595-dbb53b5777f7"

    ["uuid"]=>

    string(36) "3cd01858-xxxx-4134-b8ee-7a89b191390d"

    ["version"]=>

    string(3) "2.7"

  }

  [2]=>

  object(stdClass)#16 (9) {

    ["entity_class"]=>

    string(15) "Cluster Service"

    ["entity_model"]=>

    string(20) "Foundation Platforms"

    ["entity_type"]=>

    string(8) "software"

    ["last_updated_time"]=>

    string(20) "2021-10-01T12:29:41Z"

    ["location_id"]=>

    string(41) "node:0d8125c6-xxxx-4e40-bbfd-760653382d39"

    ["request_version"]=>

    string(3) "2.7"

    ["single_group_uuid"]=>

    string(36) "fff46f40-xxxx-479b-b595-dbb53b5777f7"

    ["uuid"]=>

    string(36) "51caf6c4-xxxx-45b8-8f8f-790e5fed4de0"

    ["version"]=>

    string(3) "2.7"

  }

}

The reason why you get an array of 3 elements is because this cluster got 3 nodes. Indeed, there is most of the time a software package for each nodes - and the version could be different (while upgrading for example).

The interesting field is of course located in $data[$i]->version. 

What is done to simplify display is to list only the different version using this trick :

    // Get unique Foundation version 

    $tmp=array();

    for($h=0;$h<count($ver[$i]["Foundation"]);$h++)

    {

     $tmp[$h]=$ver[$i]["Foundation"][$h]->version;

    }

    $tmp2=array_unique($tmp);

    $ver[$i]["Foundation"]=array_values($tmp2);

It is returning this : 

    array(1) {

      [0]=>

      string(3) "2.7"

    }

The next function is more straightforward

    // Get LCM version

    $lcm=nxGetLCMVersion($clusterConnect);

And the version is returned as a string :

    string(13) "2.4.3.1.26954"

Conclusion

As a conclusion, I would say LCM API is providing a lot of powerful methods to get access to various LCM component. You can also initiate an inventory if you want, you can even start an upgrade. The way I use LCM might not be efficient. Indeed, instead of requesting for all information at once, I decided to request details one by one, which result in slowness. I initially retrieved all the software version at once but I was struggling to dive into the right information. This is the reason why I switched to the current version, adding delay for each calls/retrievals.

At the end of the day, the script work with the current LCM API implementation. And this is all I have on target now. I could certainly revisit the script when the API stack will be changed.

Now, for the next massive upgrade campaign, it will be easier to follow what's been done and what's remaining, without browsing in each and every clusters manually. That's automation after all ! :)

I hope it helps! ;)