How to Download Files from URLs with a Bash Script

In today’s dynamic IT environment, automation, and scripting are essential tools for IT professionals and Managed Service Providers (MSPs). Whether it’s for deploying applications, managing infrastructure, or troubleshooting, scripts can save countless hours of manual labor, increase efficiency, and reduce human error. One common task that professionals often need to automate is file downloading from a URL using a shell script, specifically a bash script.

Let’s take a closer look at a bash script that not only downloads files from URLs but also verifies the file content with a provided MD5 sum and creates necessary directory structures if they don’t already exist.

Download Files from URLs Bash Script

#!/usr/bin/env bash
# Description: Downloads a file from a URL to a specified path, and can verify the file content with a provided md5 sum.
#
# Release Notes: Initial Release
#
# Usage: <url> <download file path> [expected md5 sum]
# <> are required
# [] are optional
# Example: https://www.nirsoft.net/utils/advancedrun.zip /tmp/advancedrun.zip
#  Downloads advancedrun.zip
# Example: https://www.nirsoft.net/utils/advancedrun.zip /tmp/advancedrun.zip 1f0913135878bb6cd30c1f3f6cf4b882
#  Downloads advancedrun.zip, verify's the provided md5 summed hash
#
# Notes: If the path doesn't exist this script will create the folders needed to place it there.
#  If you used /tmp/MyFiles/advancedrun.zip and the MyFiles folder didn't exist then it would create it.
#  The same for /tmp/MyFiles/Tools/advancedrun.zip, it would create MyFiles and Tools.
# By using this script, you indicate your acceptance of the following legal terms as well as our Terms of Use at https://www.ninjaone.com/terms-of-use.
# Ownership Rights: NinjaOne owns and will continue to own all right, title, and interest in and to the script (including the copyright). NinjaOne is giving you a limited license to use the script in accordance with these legal terms. 
# Use Limitation: You may only use the script for your legitimate personal or internal business purposes, and you may not share the script with another party. 
# Republication Prohibition: Under no circumstances are you permitted to re-publish the script in any script library or website belonging to or under the control of any other software provider. 
# Warranty Disclaimer: The script is provided “as is” and “as available”, without warranty of any kind. NinjaOne makes no promise or guarantee that the script will be free from defects or that it will meet your specific needs or expectations. 
# Assumption of Risk: Your use of the script is at your own risk. You acknowledge that there are certain inherent risks in using the script, and you understand and assume each of those risks. 
# Waiver and Release: You will not hold NinjaOne responsible for any adverse or unintended consequences resulting from your use of the script, and you waive any legal or equitable rights or remedies you may have against NinjaOne relating to your use of the script. 
# EULA: If you are a NinjaOne customer, your use of the script is subject to the End User License Agreement applicable to you (EULA).

# Parameters
URL=$1
SAVE_PATH=$2
SUM=$3

URL_CHECK_REGEX='(https?|ftp|file)://[-[:alnum:]+&@#/%?=~_|!:,.;]*[-[:alnum:]+&@#/%=~_|]'
if [[ "${URL}" =~ ${URL_CHECK_REGEX} ]]; then
    echo "${URL} is a valid URL."
else
    echo "${URL} is an invalid URL."
    exit 1
fi

# Verify that the path provided is in a valid format, and create the folder structure if needed.
PATH_CHECK_REGEX='^(/[^/ ]*)+/?$'
if [[ "${SAVE_PATH}" =~ ${PATH_CHECK_REGEX} ]]; then
    echo "${SAVE_PATH} is a valid path."
    FOLDER=$(dirname "${SAVE_PATH}")
    if [[ -d "${FOLDER}" ]]; then
        echo "Folder ${FOLDER} exists"
    else
        echo "Folder ${FOLDER} does not exist, creating."
        mkdir -p -v "${FOLDER}"
        if [ -f "${FOLDER}" ]; then
            echo "Created ${FOLDER}"
        else
            echo "Failed to created ${FOLDER}"
            exit 1
        fi
    fi
else
    echo "${SAVE_PATH} is an invalid path."
    exit 1
fi

function private_download() {
    # $1 = URL
    # $2 = File Path
    if [ "$(command -v wget)" ]; then
        echo "Downloading using wget"
        wget -O "$2" "$1"
    elif [ "$(command -v curl)" ]; then
        echo "Downloading using curl"
        curl "$1" --output "$2"
    else
        echo "Failed to find wget or curl."
        exit 1
    fi
}

function private_gethash() {
    FILE=$1
    HASH=$2
    WAS_ERROR=0
    if ! command -v md5 &>/dev/null; then
        # This should never happen
        echo "md5 could not be found"
        WAS_ERROR=1
    fi
    if [ "$(command -v md5)" ]; then
        CURRENT_HASH=$(md5 "${FILE}")
        # "##* " in "${CURRENT_HASH##* }" gets the last word in a string
        if [ "${HASH}" = "${CURRENT_HASH##* }" ]; then
            echo "File matches md5sum hash"
            WAS_ERROR=0
        else
            echo "File does not match md5sum hash"
            echo "Expected: ${CURRENT_HASH}"
            echo "File: ${HASH}"
            WAS_ERROR=1
        fi
    fi
    if [ ${WAS_ERROR} == 1 ]; then
        echo "Error verifying hash sum."
        exit 1
    fi
}

function private_validate() {
    if [ -f "$1" ]; then
        echo "File Downloaded."
        private_gethash "$1" "$2"
    else
        echo "Failed to download file."
        exit 1
    fi
}

# Download file
private_download "${URL}" "${SAVE_PATH}"
# If SUM is not empty
if [[ -n "${SUM}" ]]; then
    # Verify that the file was download
    # Compare hash from parameter to the file's calculated md5 sum
    private_validate "${SAVE_PATH}" "${SUM}"
fi

 

Script Overview

This script primarily takes three parameters: the URL from where the file is to be downloaded, the path where the file should be saved, and an optional MD5 hash for content verification. If the path doesn’t exist, the script will create the necessary folders, offering a significant advantage when dealing with complex directory structures.

The script begins by validating the URL and path parameters. If either of these parameters is invalid, it will exit immediately and throw an error. The script uses regular expressions to validate the URL and the path.

Once it has verified the parameters, it proceeds to download the file. The script is designed to use either ‘wget’ or ‘curl’ based on what is available in the environment.

After successfully downloading the file, if an MD5 hash was provided, it verifies the downloaded file’s content against the provided hash. This feature is particularly useful when ensuring the file integrity, a critical aspect in many IT operations.

Script Commands

The script primarily uses ‘wget’ or ‘curl’ commands to download files, which are common tools in Unix-based systems. ‘wget’ and ‘curl’ are both free utilities for non-interactive download of files from the web, supporting HTTP, HTTPS, and FTP protocols.

The script also uses ‘md5’, a command-line utility that computes and checks MD5 message digest. The MD5 hash (128-bit) value is typically rendered as a sequence of 32 hexadecimal digits. It’s used to ensure the data integrity of files by producing a unique hash value for the contents of a file. If two files have the same MD5 hash value, it’s highly likely that the files are the same.

The ‘mkdir -p’ command is used to create the directory path if it does not exist. The ‘-p’ flag ensures that no error is reported if a directory given as an operand already exists.

The script uses ‘command -v’ to check the availability of ‘wget’, ‘curl’, and ‘md5’ commands in the environment. It also uses regular expressions and conditional statements ‘[[‘ and ‘]]’ to validate the URL and the path.

Next, let’s delve deeper into the script and explore its potential applications.

Potential Applications

This bash script can be utilized in a variety of scenarios:

1. Software Updates: The script can be used to automate the download of software updates from a vendor’s website. Once downloaded, another script could be used to apply these updates across multiple systems in a network.
2. Security Patches: Security patches are often released as downloadable files on the internet. This script can automate the process of downloading these patches, ensuring that all systems in a network remain up-to-date and secure.
3. Data Files: In data-heavy environments, it’s common to have data files hosted on web servers. This script can be used to download such files for further data analysis or processing.
4. Distributed Systems: In a distributed system, there might be a need to download and verify a specific file across multiple systems. The script can be used to ensure the same version of a file is used across all systems.
5. Backup and Recovery: The script can be used to download backup files from a remote server for disaster recovery purposes.

By understanding the mechanics of the script, IT professionals can modify and expand it to suit their specific needs, further enhancing their automation capabilities.

Benefits for IT Professionals and MSPs

This script offers numerous benefits for IT professionals and MSPs:

• Timesaving: Instead of manually downloading files, professionals can use this script to automate the process, thus saving valuable time.
• Error reduction: With automated download and MD5 verification, the chances of human error are significantly reduced.
• Flexibility: The script can use either ‘wget’ or ‘curl’, making it adaptable to different environments.
• File integrity: By comparing MD5 hashes, the script ensures the downloaded file’s integrity, which is crucial in security-sensitive operations.

Final Thoughts

This Bash script simplifies a common task, improves efficiency, and enhances reliability and security. Whether you’re an experienced sysadmin or just starting out in the IT world, this script is a valuable addition to your toolkit.

NinjaOne’s highly rated remote monitoring and management platform can allow the deployment and execution of this bash script across multiple systems in a seamless manner. Whether you’re looking to download software updates, security patches, or data files, integrating this script with NinjaOne can help automate and streamline these IT processes. It can ensure consistency, reduce manual errors, and save time by allowing mass deployments. Thus, NinjaOne’s robust capabilities, when used in conjunction with this script, can greatly enhance operational efficiency and productivity.