Nutanix : CVMs are running out of space (sic!)


Background

Trying to upgrade to the latest AOS, NCC, Hypervisor .... but you are running out of space on your cluster and pre-checks are failing ? There is a fix !

Let's do some cleanup

[Edit Oct 25th, 2021] : Nutanix recently issued a script that is dealing with this exact issue. This is not by default installed in the CVM and it is fully explained in KB1540 available here. It only works on AHV. Use the below with caution !

Logs are taking a lot of space, same for temp files and old binaries. Those simple commands can be ran on each of the CVMs to save space :

nutanix@NTNX-15SM15xxxxxx-A-CVM:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       9.8G  5.2G  4.1G  56% /
devtmpfs         14G     0   14G   0% /dev
tmpfs            14G  4.0K   14G   1% /dev/shm
tmpfs            14G  696K   14G   1% /run
tmpfs            14G     0   14G   0% /sys/fs/cgroup
/dev/sda3        40G   23G   17G  59% /home
tmpfs           2.8G     0  2.8G   0% /run/user/1000
/dev/sdd1       917G  238G  670G  27% /home/nutanix/data/stargate-storage/disks/9XG843J5
/dev/sdc1       917G  248G  660G  28% /home/nutanix/data/stargate-storage/disks/9XG86131
/dev/sdb1       917G  249G  660G  28% /home/nutanix/data/stargate-storage/disks/9XG8619W
/dev/sde1       917G  248G  660G  28% /home/nutanix/data/stargate-storage/disks/9XG8618V
/dev/sda4       308G  148G  157G  49% /home/nutanix/data/stargate-storage/disks/BTTV451306A6400HGN


In the above example, /home has 17 Gb of available space. Let initiate the cleanup.


 nutanix@NTNX-15SM15xxxxxx-A-CVM:~$ allssh rm -rf /home/nutanix/data/cores/*
================== 192.168.8.183 =================
================== 192.168.8.184 =================
================== 192.168.8.185 =================
nutanix@NTNX-15SM15xxxxxx-A-CVM:~$ allssh rm -rf /home/nutanix/data/binary_logs/*
================== 192.168.8.183 =================
================== 192.168.8.184 =================
================== 192.168.8.185 =================
nutanix@NTNX-15SM15xxxxxx-A-CVM:~$ allssh rm -rf /home/nutanix/data/ncc/installer/*
================== 192.168.8.183 =================
================== 192.168.8.184 =================
================== 192.168.8.185 =================
nutanix@NTNX-15SM15xxxxxx-A-CVM:~$ allssh rm -rf /home/nutanix/data/log_collector/*
================== 192.168.8.183 =================
================== 192.168.8.184 =================

================== 192.168.8.185 =================


Note : this tip and trick has been provided directly by the Nutanix Support !

After the cleanup, here is the new available space :

nutanix@NTNX-15SM15xxxxxx-A-CVM:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       9.8G  5.2G  4.1G  56% /
devtmpfs         14G     0   14G   0% /dev
tmpfs            14G  4.0K   14G   1% /dev/shm
tmpfs            14G  696K   14G   1% /run
tmpfs            14G     0   14G   0% /sys/fs/cgroup
/dev/sda3        40G   22G   18G  55% /home
tmpfs           2.8G     0  2.8G   0% /run/user/1000
/dev/sdd1       917G  238G  670G  27% /home/nutanix/data/stargate-storage/disks/9XG843J5
/dev/sdc1       917G  248G  660G  28% /home/nutanix/data/stargate-storage/disks/9XG86131
/dev/sdb1       917G  249G  660G  28% /home/nutanix/data/stargate-storage/disks/9XG8619W
/dev/sde1       917G  248G  660G  28% /home/nutanix/data/stargate-storage/disks/9XG8618V
/dev/sda4       308G  148G  157G  49% /home/nutanix/data/stargate-storage/disks/BTTV451306A6400HGN

We have a gain of 1 Gb. Hoora ! ;)

Time to deploy the latest firmware !

If ever this is not sufficient to install the new firmware, you'd better have to contact the Nutanix support directly, you may have missed some gigantic files in the way that could be deleted or moved elsewhere. Always contact the Nutanix support at first, especially on the commercial product. This trick also apply to the Community Edition.

Comments

  1. vivMonday, December 3, 2018 at 10:56:00 AM GMT+1

    Thank you! Your posts concerning nutanix CE are quite useful.

    ReplyDelete
    Replies
    1. Let's Talk About Tech!Friday, June 7, 2019 at 1:36:00 PM GMT+2

      My pleasure, I just realized that I have a lot of comments on almost every articles. I should read the "moderate section" a little bit more often ;)

      Delete

Post a Comment

Thank you for your message, it has been sent to the moderator for review...

What's hot ?

ShredOS : HDD degaussing with style

Image
Background Have you ever had the need to properly erase HDD or SSD before disposing them ? Or selling them ? You do not want to have someone else playing around with your data right ? What we usually do is using a bootable Linux Live CD and use dd style command to write random data on the disk right ? But what if you have 10 disks in your machine ? What if you need a report to proof the world that wiping is a success ? I came across this challenge recently.
Read more

ThingSpeak : Create some useful formulas

Image
Background In a previous post related to the thingspeak integration for my solar panel production monitoring , I created some basic dashboard showing the production throughout the day. I wanted to see what was the production since the beginning of the day. Since ThingSpeak is part of MathWorks (MATLAB), I have a good feeling that some strong formulas were available for our creativity. 
Read more