How to Monitor Proxmox Node Status with an Automated Bash Script

Efficient management of IT infrastructure is crucial for maintaining the stability and performance of an organization’s digital environment. One of the essential tasks for IT professionals and Managed Service Providers (MSPs) is monitoring the status of nodes within a Proxmox Virtual Environment (PVE) cluster. The provided script offers a robust solution to automate this process, ensuring that node status information is readily available and up-to-date.

Background

Proxmox VE is an open-source server virtualization environment used by many IT professionals to manage virtual machines, containers, and storage. Monitoring the health and status of nodes within a Proxmox cluster is critical for proactive maintenance and troubleshooting.

This script simplifies the process by retrieving node status information and saving it to custom fields for easy access and analysis. This automation is particularly valuable for MSPs managing multiple client environments, as it reduces manual effort and minimizes the risk of oversight.

The Script

#!/usr/bin/env bash

# Description: Get the Proxmox Node Status and save it to a multiline and/or WYSIWYG custom field
#
# Release Notes: Fixed 10% width bug.
# By using this script, you indicate your acceptance of the following legal terms as well as our Terms of Use at https://www.ninjaone.com/terms-of-use.
# Ownership Rights: NinjaOne owns and will continue to own all right, title, and interest in and to the script (including the copyright). NinjaOne is giving you a limited license to use the script in accordance with these legal terms. 
# Use Limitation: You may only use the script for your legitimate personal or internal business purposes, and you may not share the script with another party. 
# Republication Prohibition: Under no circumstances are you permitted to re-publish the script in any script library or website belonging to or under the control of any other software provider. 
# Warranty Disclaimer: The script is provided “as is” and “as available”, without warranty of any kind. NinjaOne makes no promise or guarantee that the script will be free from defects or that it will meet your specific needs or expectations. 
# Assumption of Risk: Your use of the script is at your own risk. You acknowledge that there are certain inherent risks in using the script, and you understand and assume each of those risks. 
# Waiver and Release: You will not hold NinjaOne responsible for any adverse or unintended consequences resulting from your use of the script, and you waive any legal or equitable rights or remedies you may have against NinjaOne relating to your use of the script. 
# EULA: If you are a NinjaOne customer, your use of the script is subject to the End User License Agreement applicable to you (EULA).

# Command line arguments, swap the numbers if you want the multiline custom field to be the second argument
multiline_custom_field=$1 # First argument is the multiline custom field name
wysiwyg_custom_field=$2   # Second argument is the WYSIWYG custom field name

# Check if the custom fields are set to null
if [[ -n "${multilineCustomField}" && "${multilineCustomField}" != "null" ]]; then
    multiline_custom_field=$multilineCustomField
fi
if [[ -n "${wysiwygCustomField}" && "${wysiwygCustomField}" != "null" ]]; then
    wysiwyg_custom_field=$wysiwygCustomField
fi

# Check if the custom fields are the same
if [[ -n "${multiline_custom_field}" && "${multiline_custom_field}" == "${wysiwyg_custom_field}" ]]; then
    echo "[Error] multilineCustomField and wysiwygCustomField cannot be the same custom field."
    exit 1
fi

# Check if the custom fields are not set
if [[ -z "${multiline_custom_field}" ]]; then
    echo "[Info] multilineCustomField is not set."
fi
if [[ -z "${wysiwyg_custom_field}" ]]; then
    echo "[Info] wysiwygCustomField is not set."
fi

# Check that we have the required tools
if ! command -v pvesh &>/dev/null; then
    echo "[Error] The Proxmox VE API tool 'pvesh' is required."
    exit 1
fi

# Check that we are running as root
if [[ $EUID -ne 0 ]]; then
    echo "[Error] This script must be run as root."
    exit 1
fi

# Get the version of proxmox-ve
_version=$(pveversion --verbose | grep "proxmox-ve" | awk '{print $2}')

# Check if the version
if [[ "$(echo "${_version}" | awk -F. '{print $1}')" -eq 7 ]]; then
    echo "[Info] Proxmox VE $_version is greater than or equal to 8."
else
    echo "[Warn] Proxmox VE $_version is less than 8. Some data may not be formatted as expected. See: https://pve.proxmox.com/pve-docs/chapter-pve-faq.html#faq-support-table"
fi

# Check if ninjarmm-cli command exists
ninjarmm_cli="/opt/NinjaRMMAgent/programdata/ninjarmm-cli"
if [[ -z $ninjarmm_cli ]]; then
    echo "[Error] The ninjarmm-cli command does not exist in the default path. Please ensure the NinjaRMM agent is installed before running this script."
    exit 1
else
    # ninjarmm-cli command exists in the default path
    echo -n
fi

# Run the pvesh command to get the status information
if ! pvesh_status_output=$(pvesh get /cluster/status --noborder); then
    echo "[Error] Failed to get the Proxmox Node Status."
    echo "$pvesh_status_output"
    exit 1
fi
# Example Output from: pvesh get /cluster/status --noborder
# id        name     type    ip            level local nodeid nodes online quorate version
# cluster   cluster1 cluster                                      4        1       4
# node/pve1 pve1     node    192.168.1.10  c     0          1       1
# node/pve2 pve2     node    192.168.1.20  c     0          2       1
# node/pve3 pve3     node    192.168.1.30  c     0          3       1
# node/pve4 pve4     node    192.168.1.40  c     1          4       1

# Exclude the cluster information then skip the first line
node_status=$(echo "$pvesh_status_output" | grep -v "cluster" | tail -n +2)

# Create a table with the node status information with only the columns named id, name, ip, and online
if [[ "$(echo "${_version}" | awk -F. '{print $1}')" -ge 8 ]]; then
    data_table=$(echo "$node_status" | awk '{print $7, $2, $4, $8}' | column -t)
else
    data_table=$(echo "$node_status" | awk '{print $7, $2, $4, $7}' | column -t)
fi

# Convert the table to an HTML table with headers
result_table=$(echo "$data_table" | awk 'BEGIN {print "<table style=\"white-space:nowrap;\"><tr><th>Node ID</th><th>Node Name</th><th>IP Address</th><th>Online Status</th><th>Votes</th></tr>"} {print "<tr>"; for(i=1;i<=NF;i++) print "<td>" $i "</td>"; print "<td>1</td></tr>"} END {print "</table>"}')

# Save the result to the WYSIWYG custom field
if [[ -n "$wysiwyg_custom_field" ]]; then
    # Check if the NinjaRMM CLI exists and is executable
    if [[ -x "$ninjarmm_cli" ]]; then
        # Save the result to the custom field
        if hideOutput=$("$ninjarmm_cli" set "$wysiwyg_custom_field" "$result_table" 2>&1); then
            echo "[Info] Successfully set custom field: $wysiwyg_custom_field"
        else
            echo "[Error] Failed to set custom field: $wysiwyg_custom_field. Custom Field does not exit or does not have write permissions."
            _exit_code=1
        fi
    else
        echo "[Error] NinjaRMM CLI not found or not executable"
        _exit_code=1
    fi
fi

# Format the output for the multiline custom field
pvesh_status_output=$(
    # Exclude the cluster information then skip the first line
    echo "$data_table" | awk '{if (NR == 1) print "--------"; print "Node ID: " $1 "\nNode Name: " $2 "\nIP Address: " $3 "\nOnline Status: " $4 "\nVotes: 1\n"; if (NR != NF) print "--------"}'
)

# Save the result to the multiline custom field
_exit_code=0
if [[ -n "$multiline_custom_field" ]]; then
    if [[ -x "$ninjarmm_cli" ]]; then
        if hideOutput=$("$ninjarmm_cli" set "$multiline_custom_field" "$pvesh_status_output" 2>&1); then
            echo "[Info] Successfully set custom field: $multiline_custom_field"
        else
            echo "[Error] Failed to set custom field: $multiline_custom_field. Custom Field does not exit or does not have write permissions."
            _exit_code=1
        fi
    else
        echo "[Error] NinjaRMM CLI not found or not executable"
        _exit_code=1
    fi
fi

# Output the result
echo "${pvesh_status_output}"

exit $_exit_code

 

Detailed Breakdown

The script is designed to fetch the Proxmox node status and save the information to either a multiline custom field or a WYSIWYG custom field. Let’s break down its functionality step by step:

1. Command Line Arguments

• The script accepts two command-line arguments: the names of the multiline custom field and the WYSIWYG custom field.

2. Custom Field Validation

• It checks if the custom fields are set and ensures they are not null or the same.

3. Tool and Permissions Check

• The script verifies the presence of necessary tools (pvesh and ninjarmm-cli) and checks if it is running with root privileges.

4. Fetch Node Status

• It runs the pvesh get /cluster/status –noborder command to retrieve the status of the nodes in the Proxmox cluster.

5. Data Processing

• The output is processed to extract relevant information, such as node ID, name, IP address, and online status. This information is formatted into a table.

6. Save to Custom Fields

• The formatted data is saved to the specified custom fields using the ninjarmm-cli command. If the WYSIWYG field is specified, the data is converted into an HTML table.

7. Output

• If no custom fields are specified, the script outputs the node status directly to the console.

Potential Use Cases

Case Study: An IT Professional Managing a Proxmox Cluster

Imagine an IT professional named Alex, who manages a Proxmox cluster for a mid-sized company. Alex needs to regularly monitor the status of the nodes to ensure they are online and functioning correctly. By using this script, Alex can automate the retrieval and logging of node statuses into custom fields within their monitoring system. This automation saves Alex significant time and ensures that any issues are promptly detected and addressed.

Comparisons

Manual Monitoring vs. Automated Script

Traditionally, monitoring node statuses involves manually running commands and recording the results, a time-consuming and error-prone process. This script automates the entire workflow, ensuring consistent and accurate data collection. Compared to other methods, such as using third-party monitoring tools, this script offers a lightweight and customizable solution tailored to Proxmox environments.

FAQs

1. What is Proxmox VE? Proxmox VE is an open-source server virtualization environment used to manage virtual machines, containers, and storage.
2. Why do I need to run the script as root? The script requires root privileges to access Proxmox node status information and perform necessary system checks.
3. What happens if the pvesh command fails? If the pvesh command fails, the script will output an error message and terminate.
4. Can I use this script without the ninjarmm-cli tool? No, the script relies on ninjarmm-cli to save the node status information to custom fields.

Implications

Using this script ensures that node status information is consistently collected and logged, which is crucial for maintaining IT infrastructure health. By automating this process, IT professionals can focus on more strategic tasks, improving overall efficiency and reducing the risk of unnoticed issues.

Recommendations

• Run Regularly: Schedule the script to run at regular intervals to ensure up-to-date node status information.

• Monitor Outputs: Regularly check the custom fields or output logs for any anomalies.

• Ensure Tools are Installed: Verify that both pvesh and ninjarmm-cli are installed and accessible on the system.

Final Thoughts

Automating node status monitoring with this script enhances the efficiency and reliability of managing Proxmox clusters. By integrating this script into your workflow, you can ensure timely and accurate status updates, ultimately improving the stability and performance of your IT environment.

NinjaOne’s suite of tools, including NinjaRMM, provides additional capabilities for comprehensive IT management, complementing the functionality of this script.