Know it's healthy.
Prove you can recover.

Keep this exercise small and reversible

Start with one running code-managed Ubuntu 24.04 lab after its Nginx Ansible lesson. Use Ubuntu/WSL on Windows or your Linux control laptop. Keep the existing verified SSH key and one-host inventory. Do not practice on production or shared infrastructure.

A health check is checking that the toy still works. A backup is a spare piece kept somewhere else. A recovery test is actually fitting the spare piece—not just admiring its box.

1. Create a separate local directory, ~/contoso-operations.
2. Copy your chosen lab's inventory.ini into it. Check the current server IP, username and private-key path. Keep only one host in group lab.
3. Download the playbooks below into that folder. Store all real backups outside the website/source tree.
4. Unlock the key in your SSH agent and activate an existing Ansible virtual environment, or prepare one using this block.

mkdir -p ~/contoso-operations
cd ~/contoso-operations
python3 -m venv ~/.venvs/contoso-operations
source ~/.venvs/contoso-operations/bin/activate
python -m pip install ansible-core
ansible lab -i inventory.ini -m ansible.builtin.ping

The virtual-environment command needs the python3-venv package from the server setup lesson. Do not keep your infrastructure running longer than needed just to practice; VM, disk and address costs still apply.

1. Three different kinds of healthy

This playbook reads service/capacity information and checks HTTP on the server's loopback address. It does not install packages, restart anything, create paid monitoring resources or test an external firewall.

ansible-playbook -i inventory.ini health.yml --syntax-check
ansible-playbook -i inventory.ini health.yml

Then test from your laptop with curl --max-time 5 -I http://YOUR-SERVER-IP/, or http://127.0.0.1:8080/ if you completed the SSH-tunnel lesson. A 200 status proves a response, not that every application feature is correct.

2. Learn from a short service failure

sudo nginx -t
sudo journalctl -u nginx -n 30 --no-pager
sudo tail -n 20 /var/log/nginx/access.log
sudo tail -n 20 /var/log/nginx/error.log

Access logs can reveal addresses and URL parameters. System logs may contain names or configuration details. Do not publish raw logs or send them to an external service without reviewing the data and access policy.

Optional lab-only interruption

First confirm the exact disposable server and that nginx -t succeeds. Keep your SSH session open. Run these lines one at a time; do not walk away between stop and start.

sudo systemctl stop nginx
# Expected: connection fails while the service is stopped.
curl --max-time 5 -I http://127.0.0.1/
sudo systemctl start nginx
curl --max-time 5 -I http://127.0.0.1/

While stopped, the cloud VM may still look healthy. The health playbook should fail its service check. Once started, run it again and expect success. If startup fails, read the journal/config test; do not reboot or widen firewall rules as your first reaction.

This drill changes only Nginx state. It does not delete files, alter SSH access or simulate a whole-region outage.

3. Design one notification before enabling more telemetry

A useful alert answers: what happened, which resource, how long, who should act and which runbook to follow. A budget alert answers a money question; a service-health alert answers a different question.

Optional: price and configure one CPU alert in your chosen cloud. A sample learning threshold is average CPU above 80% for 10 minutes, not a universal production recommendation. Check whether the UI expresses utilization as a percent or fraction. Do not generate load simply to force an alert.

1. Verify resource scope, metric units, evaluation window and threshold.
2. Choose an explicit missing-data behavior. Stopping/deleting a lab can remove metrics; do not treat no data as guaranteed healthy.
3. Use notification-only actions. Do not add automatic shutdown, restart or scaling actions in this exercise.
4. Test the channel through the provider's available test flow where supported. Channel delivery is not proof the metric condition works.
5. Record every alarm, action group/topic/channel you create for later cleanup.

Guest RAM and filesystem fullness are not automatically included in every provider's basic VM metrics. Adding an agent can require workload permissions and separate billing. A CPU alert will not reliably detect a stopped Nginx service; choose an application-health signal when designing real monitoring.

AWS CPU alarmCloudWatch pricingAzure VM alertsAzure Monitor pricingGoogle alertingGoogle monitoring pricing

4. Save the page somewhere outside the server

This backup copies only the nonsensitive public /var/www/html/index.html to your control laptop. It is not a full VM, disk, database or secret backup.

Its default label is baseline. The playbook refuses an existing destination rather than overwriting it. Use a new lowercase label to save another copy. The output directory is protected locally and excluded from routine commits.

ansible-playbook -i inventory.ini backup.yml --syntax-check
ansible-playbook -i inventory.ini backup.yml -e backup_label=baseline
# Replace YOUR-SERVER-IP with the exact host label in inventory.ini.
sha256sum backups/baseline/YOUR-SERVER-IP/index.html
less backups/baseline/YOUR-SERVER-IP/index.html

The fetch module validates its transfer checksum. Inspect the saved content too: a perfectly copied wrong file is still the wrong backup. This lesson uses actual backup runs, not check mode, to create the local file.

Ansible fetch and checksum validation

5. Restore it and verify the bytes

In your original server configuration folder, change page_message in server.yml and apply that playbook. Verify that the page now differs from the baseline. Do not run the backup again under the same label.

Return to the operations folder. Confirm the inventory still points at the same lab and the backup is the intended file. Restoration overwrites the current demo page. Save a separate before-restore backup first if you want to preserve the changed version.

ansible-playbook -i inventory.ini restore.yml --syntax-check
# Simulation: inspect the proposed page replacement; no checksum proof yet.
ansible-playbook -i inventory.ini restore.yml \
  -e backup_label=baseline -e confirm_restore=true --check --diff
# Actual restoration: run only after confirming this is the disposable lab.
ansible-playbook -i inventory.ini restore.yml \
  -e backup_label=baseline -e confirm_restore=true
ansible-playbook -i inventory.ini health.yml

The real run compares local and restored SHA-256 checksums and requires local HTTP 200. Check mode skips those post-write checks because it did not actually write the file. View the page through your intended access path and confirm the baseline text returned.

The script does not restart Nginx: replacing static HTML should not need it. If the service is stopped, the HTTP check fails and you must investigate/start it deliberately. If the VM IP changed, the backup folder's host label and new inventory may no longer match; verify ownership and choose the intended saved copy rather than blindly renaming backups.

Ansible SHA-256 statAnsible copy

6. What a real recovery plan adds

Disk snapshots can preserve a disk without being application-consistent. Databases often need their own backup procedures. A laptop-only backup can be lost with the laptop; important data needs an approved off-host/off-device design, encryption, restricted access and tested retention.

Terraform/Ansible can rebuild infrastructure and configuration, but they do not magically recreate unsaved user data. Keep sensitive backups outside public buckets and websites. Price snapshots, vaults and replicated copies before creating them.

7. Stop paying for finished exercises

1. Close any SSH tunnel. It does not stop the server.
2. Use the original cloud guide's reviewed Terraform destroy/manual cleanup process.
3. If you opted into alerts, delete the exact lab alert policies and any dedicated notification resources you created. Do not delete shared topics/action groups/channels.
4. Inspect disks, snapshots, reserved IPs, monitoring destinations and retained logs. Removing the VM alone does not remove every billable service.
5. Keep or delete your local backups deliberately; they are not in the website package. Log out of temporary cloud sessions and check cost reports after reporting catches up.