Comprehensive Guide for Proxmox Cluster Status Monitoring in Linux

Maintaining a Proxmox cluster’s stability and performance is crucial for IT professionals and managed service providers (MSPs). A Proxmox cluster offers numerous benefits, including high availability, efficient resource management, and seamless virtualization. However, monitoring its status is essential to ensure smooth operation.

This blog post delves into a Bash script designed to get the Proxmox Cluster Status and save it to a custom field. We will explore the script’s functionality, potential use cases, comparisons, FAQs, implications, and best practices.

Background

Proxmox Virtual Environment (VE) is an open-source server virtualization management solution that allows IT professionals to deploy and manage virtual machines and containers. Clustering in Proxmox is a fundamental feature enabling multiple Proxmox VE nodes to work together, providing high availability and load balancing. Monitoring the cluster’s status is vital for maintaining the integrity and performance of the virtual environment.

This script is tailored for IT professionals who need an efficient way to monitor Proxmox clusters. It fetches the cluster status and saves the information into either a multiline custom field or a WYSIWYG custom field, making it easy to manage and review the cluster’s health.

The Script

#!/usr/bin/env bash

# Description: Get the Proxmox Cluster Status and save it to a multiline and/or WYSIWYG custom field
#
# Release Notes: Fixed 10% width bug.
# By using this script, you indicate your acceptance of the following legal terms as well as our Terms of Use at https://www.ninjaone.com/terms-of-use.
# Ownership Rights: NinjaOne owns and will continue to own all right, title, and interest in and to the script (including the copyright). NinjaOne is giving you a limited license to use the script in accordance with these legal terms. 
# Use Limitation: You may only use the script for your legitimate personal or internal business purposes, and you may not share the script with another party. 
# Republication Prohibition: Under no circumstances are you permitted to re-publish the script in any script library or website belonging to or under the control of any other software provider. 
# Warranty Disclaimer: The script is provided “as is” and “as available”, without warranty of any kind. NinjaOne makes no promise or guarantee that the script will be free from defects or that it will meet your specific needs or expectations. 
# Assumption of Risk: Your use of the script is at your own risk. You acknowledge that there are certain inherent risks in using the script, and you understand and assume each of those risks. 
# Waiver and Release: You will not hold NinjaOne responsible for any adverse or unintended consequences resulting from your use of the script, and you waive any legal or equitable rights or remedies you may have against NinjaOne relating to your use of the script. 
# EULA: If you are a NinjaOne customer, your use of the script is subject to the End User License Agreement applicable to you (EULA).

# Command line arguments, swap the numbers if you want the multiline custom field to be the second argument
multiline_custom_field=$1 # First argument is the multiline custom field name
wysiwyg_custom_field=$2   # Second argument is the WYSIWYG custom field name

if [[ -n "${multilineCustomField}" && "${multilineCustomField}" != "null" ]]; then
    multiline_custom_field=$multilineCustomField
fi
if [[ -n "${wysiwygCustomField}" && "${wysiwygCustomField}" != "null" ]]; then
    wysiwyg_custom_field=$wysiwygCustomField
fi

if [[ -n "${multiline_custom_field}" && "${multiline_custom_field}" == "${wysiwyg_custom_field}" ]]; then
    echo "[Error] multilineCustomField and wysiwygCustomField cannot be the same custom field."
    exit 1
fi

if [[ -z "${multiline_custom_field}" ]]; then
    echo "[Info] multilineCustomField is not set."
fi
if [[ -z "${wysiwyg_custom_field}" ]]; then
    echo "[Info] wysiwygCustomField is not set."
fi

# Check that we have the required tools
if ! command -v pvecm &>/dev/null; then
    echo "[Error] The Proxmox VE API tool 'pvecm' is required."
    exit 1
fi

# Check that we are running as root
if [[ $EUID -ne 0 ]]; then
    echo "[Error] This script must be run as root."
    exit 1
fi

# Check if ninjarmm-cli command exists
ninjarmm_cli="/opt/NinjaRMMAgent/programdata/ninjarmm-cli"
if [[ -z $ninjarmm_cli ]]; then
    echo "[Error] The ninjarmm-cli command does not exist in the default path. Please ensure the NinjaRMM agent is installed before running this script."
    exit 1
else
    # ninjarmm-cli command exists in the default path
    echo -n
fi

# Run the pvecm command to get the status information
if ! pvecm_status_output=$(pvecm status); then
    echo "[Error] Failed to get the Proxmox Cluster Status."
    echo "$pvecm_status_output"
    exit 1
fi
# Example Output:
# Cluster information
# -------------------
# Name:             cluster1
# Config Version:   4
# Transport:        knet
# Secure auth:      on
#
# Quorum information
# ------------------
# Date:             Mon Apr  8 10:33:16 2024
# Quorum provider:  corosync_votequorum
# Nodes:            4
# Node ID:          0x00000004
# Ring ID:          1.631
# Quorate:          Yes
#
# Votequorum information
# ----------------------
# Expected votes:   4
# Highest expected: 4
# Total votes:      4
# Quorum:           3
# Flags:            Quorate
#
# Membership information
# ----------------------
#     Nodeid      Votes Name
# 0x00000001          1 10.10.10.17
# 0x00000002          1 10.10.10.18
# 0x00000003          1 10.10.10.19
# 0x00000004          1 10.10.10.20 (local)

# Cluster Table
# Get the cluster name
cluster_name=$(echo "$pvecm_status_output" | grep -oP 'Name:\s+\K\w+' | head -n 1)
# Get the Config Version
config_version=$(echo "$pvecm_status_output" | grep -oP 'Config Version:\s+\K\d+' | head -n 1)
# Get the Transport
transport=$(echo "$pvecm_status_output" | grep -oP 'Transport:\s+\K\w+' | head -n 1)
# Get the Secure auth
secure_auth=$(echo "$pvecm_status_output" | grep -oP 'Secure auth:\s+\K\w+' | head -n 1)

# Create Cluster Status label
cluster_table="<h2>Cluster Status</h2>"
# Create the Cluster Status table
cluster_table+="<table style='white-space:nowrap;'><tr><th>Cluster Name</th><th>Config Version</th><th>Transport</th><th>Secure Auth</th></tr>"
cluster_table+="<tr><td>$cluster_name</td><td>$config_version</td><td>$transport</td><td>$secure_auth</td></tr></table>"

# Quorum Table
# Get the Quorum Date
quorum_date=$(echo "$pvecm_status_output" | grep -oP 'Date:\s+\K.*' | head -n 1)
# Get the Quorum provider
quorum_provider=$(echo "$pvecm_status_output" | grep -oP 'Quorum provider:\s+\K\w+' | head -n 1)
# Get the Nodes
nodes=$(echo "$pvecm_status_output" | grep -oP 'Nodes:\s+\K\d+' | head -n 1)
# Get the Node ID
node_id=$(echo "$pvecm_status_output" | grep -oP 'Node ID:\s+\K\w+' | head -n 1)
# Get the Ring ID
ring_id=$(echo "$pvecm_status_output" | grep -oP 'Ring ID:\s+\K[\d.]+')
# Get the Quorate
quorate=$(echo "$pvecm_status_output" | grep -oP 'Quorate:\s+\K\w+')

# Create Quorum Status label
quorum_table="<h2>Quorum Status</h2>"
# Create the Quorum Status table
quorum_table+="<table style='white-space:nowrap;'><tr><th>Quorum Date</th><th>Quorum Provider</th><th>Nodes</th><th>Node ID</th><th>Ring ID</th><th>Quorate</th></tr>"
quorum_table+="<tr><td>$quorum_date</td><td>$quorum_provider</td><td>$nodes</td><td>$node_id</td><td>$ring_id</td><td>$quorate</td></tr></table>"

# Votequorum Table
# Get the Expected votes
expected_votes=$(echo "$pvecm_status_output" | grep -oP 'Expected votes:\s+\K\d+')
# Get the Highest expected
highest_expected=$(echo "$pvecm_status_output" | grep -oP 'Highest expected:\s+\K\d+')
# Get the Total votes
total_votes=$(echo "$pvecm_status_output" | grep -oP 'Total votes:\s+\K\d+')
# Get the Quorum
quorum=$(echo "$pvecm_status_output" | grep -oP 'Quorum:\s+\K\d+')
# Get the Flags
flags=$(echo "$pvecm_status_output" | grep -oP 'Flags:\s+\K\w+')

# Create Votequorum Status label
votequorum_table="<h2>Votequorum Status</h2>"
# Create the Votequorum Status table
votequorum_table+="<table style='white-space:nowrap;'><tr><th>Expected Votes</th><th>Highest Expected</th><th>Total Votes</th><th>Quorum</th><th>Flags</th></tr>"
votequorum_table+="<tr><td>$expected_votes</td><td>$highest_expected</td><td>$total_votes</td><td>$quorum</td><td>$flags</td></tr></table>"

# Get the Membership information table
memberships=$(echo "$pvecm_status_output" | grep -oP '0x000000\d+\s+\d+\s+\d+\.\d+\.\d+\.\d+')
# Split memberships into an array
OLDIFS=$IFS
IFS=$'\n' read -r -d '' -a membership_array <<<"$memberships"
IFS=$OLDIFS

# Membership Table
# Create Membership Status label
membership_table="<h2>Membership Status</h2>"
# Create the Membership Status table
membership_table+="<table style='white-space:nowrap;'><tr><th>Node ID</th><th>Votes</th><th>Name</th></tr>"
for membership in "${membership_array[@]}"; do
    node_id=$(echo "$membership" | grep -oP '0x000000\d+')
    votes=$(echo "$membership" | grep -oP '\d+\s+(?=\d+\.\d+\.\d+\.\d+)')
    name=$(echo "$membership" | grep -oP '\d+\.\d+\.\d+\.\d+')
    membership_table+="<tr><td>$node_id</td><td>$votes</td><td>$name</td></tr>"
done
membership_table+="</table>"

# Combine all tables into one
result_table="$cluster_table</br>$quorum_table</br>$votequorum_table</br>$membership_table"

# Save the result to the custom field
_exit_code=0
if [[ -n "$multiline_custom_field" ]]; then
    if [[ -x "$ninjarmm_cli" ]]; then
        if hideOutput=$("$ninjarmm_cli" set "$multiline_custom_field" "$pvecm_status_output" 2>&1); then
            echo "[Info] Successfully set custom field: $multiline_custom_field"
        else
            echo "[Error] Failed to set custom field: $multiline_custom_field. Custom Field does not exit or does not have write permissions."
            _exit_code=1
        fi
    else
        echo "[Error] NinjaRMM CLI not found or not executable"
        _exit_code=1
    fi
fi

if [[ -n "$wysiwyg_custom_field" ]]; then
    if [[ -x "$ninjarmm_cli" ]]; then
        if hideOutput=$("$ninjarmm_cli" set "$wysiwyg_custom_field" "$result_table" 2>&1); then
            echo "[Info] Successfully set custom field: $wysiwyg_custom_field"
        else
            echo "[Error] Failed to set custom field: $wysiwyg_custom_field. Custom Field does not exit or does not have write permissions."
            _exit_code=1
        fi
    else
        echo "[Error] NinjaRMM CLI not found or not executable"
        _exit_code=1
    fi
fi

# Error out after checking both custom fields
if [[ $_exit_code -ne 0 ]]; then
    exit 1
fi

# Output the result if no custom fields are set
if [[ -z "${wysiwyg_custom_field}" ]] && [[ -z "${multiline_custom_field}" ]]; then
    echo "${pvecm_status_output}"
fi

 

Detailed Breakdown

Let’s break down the script step-by-step to understand its operation and the logic behind it.

1. Initialization and Command Line Arguments: The script starts by taking two command-line arguments: the names of the multiline and WYSIWYG custom fields.
2. Validation and Configuration: It validates and assigns values to the custom fields, ensuring they are not null or identical.
3. Tool Availability and Permissions Check: The script checks if the required tools, such as pvecm and ninjarmm-cli, are available and ensures the script runs with root privileges.
4. Fetching and Parsing Cluster Status: It executes the pvecm status command to retrieve the cluster status and stores the output in a variable.
5. Extracting Relevant Information: The script uses grep and regular expressions to parse and extract relevant information, such as the cluster name, config version, transport method, secure auth status, quorum information, and membership details.
6. Creating HTML Tables: The script formats the extracted data into HTML tables for better readability and organization.
7. Saving Results to Custom Fields: Depending on the availability of the custom fields, the script saves the raw and formatted results to the specified custom fields using the ninjarmm-cli command.

Potential Use Cases

Consider an MSP managing multiple Proxmox clusters for various clients. Regularly checking and recording the cluster status can be time-consuming. This script automates the process, ensuring up-to-date status information is always available in a structured format. For example, an IT professional can use this script to:

• Automatically collect and save Proxmox cluster status every morning.

• Generate status reports for clients, highlighting cluster health and any potential issues.

• Integrate with monitoring tools to trigger alerts if specific cluster metrics indicate problems.

Comparisons

While there are other methods to monitor Proxmox clusters, such as manual checks or using third-party monitoring tools, this script offers several advantages:

• Automation: Unlike manual checks, this script automates the process, reducing the chances of human error and saving time.

• Customization: The ability to save status information to custom fields allows for easy integration with existing monitoring and reporting systems.

• Cost-Effective: This solution leverages built-in tools and scripts, avoiding the need for expensive third-party solutions.

FAQs

Q1: Do I need to be an expert in Bash scripting to use this script?

No, you only need basic knowledge of how to run a script and understand its parameters.

Q2: Can I customize the script for other types of custom fields?

Yes, you can modify the script to handle additional custom fields by adjusting the relevant sections.

Q3: What happens if the script encounters an error?

The script includes error handling to inform you of missing tools, permissions issues, or command failures.

Implications

Regularly monitoring Proxmox cluster status helps in maintaining the system’s reliability and performance. By using this script, IT professionals can proactively manage cluster health, prevent downtime, and ensure optimal resource utilization. This contributes to overall IT security and operational efficiency.

Recommendations

• Run Regularly: Schedule the script to run at regular intervals using cron jobs to ensure continuous monitoring.

• Review Outputs: Regularly review the saved outputs to identify any anomalies or issues early.

• Maintain Backups: Keep backups of your cluster configurations and regularly update your scripts to handle new Proxmox versions.

Final Thoughts

Efficient monitoring is crucial for managing Proxmox clusters, and this Bash script provides a streamlined solution for IT professionals and MSPs. By automating the status collection and formatting the results into custom fields, it enhances the ability to maintain cluster health and performance. Tools like NinjaOne complement this script by providing a comprehensive platform for IT management, ensuring all your monitoring needs are met efficiently.