Skip to content

docs: add full cloud shutdown and startup procedure #2432

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bbezak merged 1 commit into from
Jul 20, 2026
Merged

docs: add full cloud shutdown and startup procedure #2432

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
235 changes: 24 additions & 211 deletions doc/source/operations/control-plane-operation.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,162 +84,53 @@ the monitoring service rather than the host being monitored).
Control Plane Shutdown Procedure
================================

Overview
--------
For a shutdown of the whole cloud, see :doc:`shutdown-and-startup`.

* Verify integrity of clustered components (RabbitMQ, Galera, Keepalived). They
should all report a healthy status.
* Put node into maintenance mode in bifrost to prevent it from automatically
powering back on
* Shutdown down nodes one at a time gracefully using systemctl poweroff
Single-node maintenance
-----------------------

Controllers
-----------

If you are restarting the controllers, it is best to do this one controller at
a time to avoid the clustered components losing quorum.
Before shutting down a compute node, migrate its instances to another node.
See :doc:`migrating-vm`.

Checking Galera state
+++++++++++++++++++++
Before shutting down a controller, repeat the cluster checks from
:ref:`shutdown-pre-flight`. Shut down only one controller at a time.

On each controller perform the following:
For a Bifrost-managed compute, controller, network or monitoring node, run on
the seed VM:

.. code-block:: console

[stack@controller0 ~]$ docker exec -i mariadb mysql -u root -p -e "SHOW STATUS LIKE 'wsrep_local_state_comment'"
Variable_name Value
wsrep_local_state_comment Synced
docker exec bifrost_deploy baremetal --os-cloud bifrost \
node maintenance set --reason maintenance <node>
docker exec bifrost_deploy baremetal --os-cloud bifrost \
node power off --soft <node>

The password can be found using:
Wait for the node to reach ``power off``. To start it again:

.. code-block:: console

ansible-vault view $KAYOBE_CONFIG_PATH/kolla/passwords.yml \
--vault-password-file <Vault password file path> | grep ^database
docker exec bifrost_deploy baremetal --os-cloud bifrost \
node power on <node>

Checking RabbitMQ
+++++++++++++++++

RabbitMQ health is determined using the command ``rabbitmqctl cluster_status``:
After the node is healthy, remove maintenance mode:

.. code-block:: console

[stack@controller0 ~]$ docker exec rabbitmq rabbitmqctl cluster_status

Cluster status of node rabbit@controller0 ...
[{nodes,[{disc,['rabbit@controller0','rabbit@controller1',
'rabbit@controller2']}]},
{running_nodes,['rabbit@controller1','rabbit@controller2',
'rabbit@controller0']},
{cluster_name,<<"rabbit@controller2">>},
{partitions,[]},
{alarms,[{'rabbit@controller1',[]},
{'rabbit@controller2',[]},
{'rabbit@controller0',[]}]}]

Checking Keepalived
+++++++++++++++++++
docker exec bifrost_deploy baremetal --os-cloud bifrost \
node maintenance unset <node>

On (for example) three controllers:
For a seed VM hosted on the seed hypervisor, run on the seed hypervisor to stop
it:

.. code-block:: console

[stack@controller0 ~]$ docker logs keepalived
virsh shutdown <seed VM>

Two instances should show:
To start it again:

.. code-block:: console

VRRP_Instance(kolla_internal_vip_51) Entering BACKUP STATE

and the other:

.. code-block:: console

VRRP_Instance(kolla_internal_vip_51) Entering MASTER STATE

Ansible Control Host
--------------------

The Ansible control host is not enrolled in bifrost. This node may run services
such as the seed virtual machine which will need to be gracefully powered down.

Compute
-------

If you are shutting down a single hypervisor, to avoid down time to tenants it
is advisable to migrate all of the instances to another machine. See
:ref:`evacuating-all-instances`.

Ceph
----

#. Check that the cluster is healthy (i.e. ``ceph -s``). Where possible, solve
or isolate any issues before the shutdown e.g. by marking unhealthy OSDs as
'out' in the cluster.

#. Stop all clients. This includes

* **All** OpenStack VMs (if their storage is RBD-backed).

* CephFS mounts.

* Ceph-backed OpenStack services such as Glance, Cinder, Manila, and RGW/S3/Swift.

#. Set the ``noout`` flag, so that the cluster does not attempt to redistribute
data when OSDs go down. Use the following command on a MON node:

.. code-block:: console

sudo cephadm shell -- ceph osd set noout

#. Shut down all the nodes, with those holding MON services last.

Note that if it is not desired for Ceph services to automatically start later
with the operating system, extra steps need to be taken and are not described
here.

Shutting down the seed VM
-------------------------

.. code-block:: console

virsh shutdown <Seed hostname>

.. _full-shutdown:

Full shutdown
-------------

In case a full shutdown of the system is required, we advise to use the
following order:

* Perform a graceful shutdown of all virtual machine instances
* Shut down compute nodes
* Shut down monitoring node (if separate from controllers)
* Shut down network nodes (if separate from controllers)
* Shut down controllers
* Shut down Ceph nodes (if applicable)
* Shut down seed VM
* Shut down Ansible control host

Full startup
-------------

If the entire control plane is powered down, it is best to bring the nodes up
in the reverse order of shutdown:

* Power on Ansible control host
* Power on seed VM (and other service VMs)
* Power on Ceph nodes (if applicable)
* Where possible, start the nodes running MON services first.
* Make sure that all OSD services are back up and running. At this point
it is safe to unset the ``noout`` cluster flag.
* Power on controllers
* Power on network nodes (if separate from controllers)
* Power on monitoring node (if separate from controllers)
* Power on compute nodes
* Power on virtual machines
virsh start <seed VM>

Rebooting a node
----------------
Expand All @@ -251,84 +142,6 @@ Example: Reboot all compute hosts apart from compute0:

kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/maintenance/reboot.yml --limit 'compute:!compute0'

References
----------

* https://galeracluster.com/library/training/tutorials/restarting-cluster.html

Control Plane Power on Procedure
================================

Overview
--------

* Remove the node from maintenance mode in bifrost
* Bifrost should automatically power on the node via IPMI
* Check that all docker containers are running
* Check OpenSearch Dashboards for any messages with log level ERROR or equivalent

Controllers
-----------

If all of the servers were shut down at the same time, it is necessary to run a
script to recover the database once they have all started up. This can be done
with the following command:

.. code-block:: console

kayobe overcloud database recover

Ansible Control Host
--------------------

The Ansible control host is not enrolled in Bifrost and will have to be powered
on manually.

Seed VM
-------

The seed VM (and any other service VM) should start automatically when the seed
hypervisor is powered on. If it does not, it can be started with:

.. code-block:: console

virsh start <Seed hostname>

Full power on
-------------

Follow the order in :ref:`full-shutdown`, but in reverse order.

Shutting Down / Restarting Monitoring Services
----------------------------------------------

Shutting down
+++++++++++++

Log into the monitoring host(s):

.. code-block:: console

ssh stack@monitoring0

Stop all Docker containers:

.. code-block:: console

monitoring0# for i in `docker ps -a`; do systemctl stop kolla-$i-container; done

Shut down the node:

.. code-block:: console

monitoring0# sudo shutdown -h

Starting up
+++++++++++

The monitoring services containers will automatically start when the monitoring
node is powered back on.

Software Updates
================

Expand Down
1 change: 1 addition & 0 deletions doc/source/operations/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ This guide is for operators of the StackHPC Kayobe configuration project.

ceph-management
control-plane-operation
shutdown-and-startup
customising-horizon
database-backups
gpu-in-openstack
Expand Down
Loading
Loading