Description

This article introduces the procedure used to gracefully shut down a node when installing the Nutanix Acropolis Hypervisor (AHV) and Controller VM (CVM) in a Nutanix cluster. Before shutting down the AHV, you need to shut down the CVM. When you shut down the CVM, you have to put the node into maintenance mode.

Applicable Systems

ThinkAgile HX

Solution

Caution: Verify the data resiliency status of your cluster:

1. Log in to the Prism Element.
   In Health -> select Run NCC Checks from the Actions drop-down menu.
   


2. In the Run Checks pop-up window, select All Checks and click Run to begin the health checks.
   If there are any errors or failures, contact Nutanix Support.
   

   You can also run health checks with the ncc health_checks run_all command using SSH to access any CVM. If there are any errors or fails, send the /home/nutanix/data/logs/ncc-output-latest.log file to Nutanix Support.

3. Make sure the cluster's Data Resiliency status is displayed as OK in Prism.
   
   ​Before you begin, shut down any guest VMs running on the node or migrate them to other nodes in the cluster.

Caution: Verify the data resiliency status of your cluster:

4. Use the management IP address to log in to the CVM with SSH.
   Connect to one of the CVM IP addresses with the user name that has administrator privileges. For more information, refer to CONTROLLER VM ACCESS.
   After you are connected, you will see the SSH prompt.
   
    

5. Run the acli host.list command to list all the hosts in the cluster and check if the Acropolis hosts are in maintenance mode.
   Note the Hypervisor address of the host that will be put into maintenance mode.
   
   
   In the Schedulable column:
   • TRUE means the host is not in maintenance mode.
   • FALSE means the Acropolis host is in maintenance mode.


6. Use the following command to check whether it’s possible to change the Nutanix AHV host maintenance mode of the target host:

   nutanix@cvm$ acli host.enter_maintenance_mode_check AHVHypervisorAddress

   The value of the AHVHypervisorAddress will be either the IP address of the AHV host or the hostname.
   
    

7. Run the following command to put the node into maintenance mode:

   nutanix@cvm$ acli host.enter_maintenance_mode AHVHypervisorAddress [wait="{ true | false }" ]

   Specify wait=true to wait for the host evacuation attempt to finish.
   
    

8. Shut down the CVM.

   nutanix@cvm$ cvm_shutdown -P now

   
   
   The CVM can also be shut down through SSH access to the AHV host.
   1. Log in to the AHV host with SSH.
   2. Run virsh list --all to list all the CVMs. Note the name of CVM that will be shut down.
   3. Run the virsh shutdown CVM_Name command to shut down the CVM.
   4. Replace CVM_name with the name of the CVM that you found from the preceding command.


9. Log in to the AHV host with SSH.
   
    

10. Run the shutdown -h now command to shut down the host.
    
     
11. You can now carry out hardware maintenance for the node.


12. After a shutdown, turn on the node either by pressing the power button or by logging in to the IMM to perform the power-on action.
    It may take several minutes to start all the services including AHV host and CVM.


13. Log in to any CVM in the cluster with SSH, and run the cluster status command to verify that all services are up on all the CVMs.
    – If the cluster is running properly:
     The output for each node in the cluster will be similar to the output displayed below.
    

    – If the CVM is not automatically started with the AHV host, perform the following steps:

    1. Log in to the AHV host with SSH, and run the virsh list --all | grep CVM command to find the name of the CVM. Make a note of the CVM name in the second column.
       • If the CVM is off, a line similar to the following example should be returned:

         NTNX-12AM2K470031-D-CVM shut off

       • If the CVM is on, a line similar to the following example should be returned:

         NTNX-12AM2K470031-D-CVM running

    2. If the CVM state is shut off, start the CVM by executing the following command:

       root@ahv# virsh start CVM_name

       * Replace CVM_name with the name of the CVM that you found from the preceding command.
       – If the node is in maintenance mode, log in to any CVM with SSH. Run the following command to take the node out of maintenance mode:

       nutanix@cvm$ acli host.exit_maintenance_mode AHVHypervisorAddress
       nutanix@cvm$ cluster start

       Then run the cluster start command to start the cluster.

​Additional Information

Nutanix Default Redundancy Factor: 2 (data resiliency status)

• By default, Nutanix clusters have Redundancy Factor - 2, which means they can tolerate the failure of a single node or drive. 
• If the cluster only has Replication Factor 2 (RF2), you can only shut down one node for each cluster.
• If more than one node in an RF2 cluster has to be shut down, shut down the entire cluster.   

Nutanix Tips

• Controller VM Access
• Shutting Down a Node in a Cluster (AHV)
• Starting a Node in a Cluster (AHV)
• Redundancy Factor vs Replication Factor
• Node cannot be removed: Cluster needs at least 5 usable nodes

Related Articles

• Lenovo DCG product support site for Converged HX5510 Appliance - Type 8695
• Lenovo Converged HX3500, HX5500, and HX7500 Nutanix Appliances Product Guide
• Lenovo ThinkAgile HX Series knowledge base article landing page
• ThinkAgile HX Series User Guide