Rubrik : Bug on AHV Export




Background

When we had an issue on an AHV VM, the first reflex was to restore it from last night backup. It was very quick and we were all happy ... unless we realized that no networking was connected. We opened a case with Nutanix to troubleshoot the issue and ....

We have found a bug !

After a couple of investigation and forensic, we have discovered the below phenomenon.

Here is a sample of the faulty VM with the original VLAN order :

Here is the exported VM VLAN order :


We can see the difference, the VLAN ID order is inverted. It was not very obvious at first, but when looking at the ARP table on the hypervisor managing that VM, we have seen that MAC address were inverted.
Now, we need to understand what's wrong and how to solve it. Indeed, what's wrong ? The export itself ? or the backups ? If this is the backups it means all our backups are corrupt since a long time and this is serious.

Other option, it can be an issue when the newly exported VM is created. In that case, a hotfix can definitely help.

At this stage we can be sure we have found a bug and not a small one. It could affect other customers with different VLANs created the same way as we did.

We have involved Rubrik support and it has been escalated to Engineering department. After some times reviewing the code they confirmed there is indeed a "sort" instruction on the export module that should be removed. They managed to correct the code, do the Q/A and we have been delivered with a hotfix. The internal issue number is CDM-271956.

The hotfix has been installed today (2nd Nov 2021) and the exact CDM version number is 6.0.1-12877_HP798644

The hotfix can only be applied by Rubrik support and it will be incorporated into any CDM release starting from 6.0.2.

I can confirm the hotfix is indeed solving our issue on restoring AHV VMs with multiple vNIC. The AHV version does not matter, this is not an issue on the Nutanix side.

I hope this is also helping other customers.




Comments

Post a Comment

Thank you for your message, it has been sent to the moderator for review...

What's hot ?

ShredOS : HDD degaussing with style

Image
Background Have you ever had the need to properly erase HDD or SSD before disposing them ? Or selling them ? You do not want to have someone else playing around with your data right ? What we usually do is using a bootable Linux Live CD and use dd style command to write random data on the disk right ? But what if you have 10 disks in your machine ? What if you need a report to proof the world that wiping is a success ? I came across this challenge recently.
Read more

Nutanix : CVM stuck into Phoenix

Image
Background Last night, I was in my monthly upgrades process for my clusters. I was about to catch up with all the versions. I was about to start the last run of patches, so I ran the LCM inventory for the last time. Of course I was doing something else in parallel - LCM is so time consuming, you'd better parallel tasking. When I came back to the LCM inventory screen, I saw 6 pre-checks errors : Cassandra down, network down, connection timeout, .... Most scary : node removed from Metadata Ring.
Read more

ThingSpeak : Create some useful formulas

Image
Background In a previous post related to the thingspeak integration for my solar panel production monitoring , I created some basic dashboard showing the production throughout the day. I wanted to see what was the production since the beginning of the day. Since ThingSpeak is part of MathWorks (MATLAB), I have a good feeling that some strong formulas were available for our creativity. 
Read more