Mitigating BorgBackup Client Compromise

Greg Kapoustin

BorgBackup (or Borg for short) is a popular, highly-rated open-source backup solution that efficiently compresses, encrypts, and de-duplicates data. Borg’s attack model assumes that the client environment is secure. If the client is compromised, an attacker can not only silently (undetectably) alter future backups but may also be able to modify older archives. Such time-travel/time-rewind/DeLorean attacks enable undetectable exploits and corruption of archives made before system compromise. This article explains the Borg security model, discusses the implications of client compromise, considers several solutions, and demonstrates simple mitigation using secure timestamps backed by blockchain-based commitments (stamps). We use BorgBackup as an illustration – the general problem applies to all backup systems with a similar architecture.

The Borg Security Model

Both Backup is used by system administrators, developers, scientists, and organizations for general backups, disaster recovery, long-term archiving, and regulatory compliance. Borg’s security model is built around the assumption that the client is trusted and the repository is not:

The attacker can always impose a denial of service – block connections to the repository or delete it entirely.
The attacker cannot modify the data of any archive without the client detecting the change. The attacker also cannot rename, remove, or add an archive without the client detecting the change.

If the client (the system or environment within which the command borg create is run) is compromised, such silent (undetected by the client) changes become possible. Depending on the repository configuration, in addition to the denial of service, the attacker may not be able to do the following:

Modify the data of any archive without the client detecting the change.
Rename, remove, or add an archive or its contents without the client detecting the change.

Note that these attacks go beyond the denial of service of the base security model and are far more severe: Under denial of service, a client is not able to access an archive but can determine whether an archive is valid. When client security is compromised, there is no longer a way to determine if an archive is valid.

The Consequence of a Compromised Client

One consequence of a compromised client is that an attacker can re-write the past. Specifically, the attacker may recreate historical backups with changes. For example, an attacker could remove certain critical emails from a backup or compliance archive, alter financial records, or tamper with security logs.

Borg does record archive timestamps and IDs. The latter is constructed using cryptographic hashes of the archive contents. Nevertheless, this data is stored in the client database and can be changed by the attacker. Consequently, an attack at time t makes all archives, including those backed up before t, suspect. Thus, within the base security model, “Who controls the present controls the past.”

Such silent compromise of historical Borg archives enables a variety of attacks:

A file can be silently removed from an old email archive, as illustrated below. If your archive1, backed up by a secure client, silently became archive2, how would you know?

borg_list_archive_1.txt borg_list_archive_2.txt
/home/greg/tmp/data                            /home/greg/tmp/data
/home/greg/tmp/data/email_from_adam.txt        /home/greg/tmp/data/email_from_adam.txt
/home/greg/tmp/data/email_from_beth.txt     <
/home/greg/tmp/data/email_from_charlie.txt     /home/greg/tmp/data/email_from_charlie.txt

The record of intrusions can be removed from system logs. All log history, even entries prior to compromise time t, is suspect:

auth.log
Sep 24 10:13:34 ubuntu sshd[12345]: Accepted password for nice_user from 192.168.1.100 port 4523
Sep 24 11:00:12 ubuntu sshd[23456]: Accepted password for naughty_user from 203.0.113.42 port 53   <
Sep 24 11:01:04 ubuntu sshd[23456]: pam_unix(sshd:session): session opened for user naughty_user   <
Sep 24 12:10:47 ubuntu sshd[23456]: pam_unix(sshd:session): session closed for user naughty_user   <
Sep 24 12:12:02 ubuntu sshd[34567]: Accepted publickey for nice_user from 192.168.1.100 port 452

Financial transactions, their contents, timestamps, and thus ordering, can be changed. Data may otherwise remain consistent with other sources, but records become reordered:

Trades
Ticker  Timestamp               Action  Quantity                Ticker  Timestamp               Action  Quantity
SPY     2024-09-24 09:00:00     Buy     100                     SPY     2024-09-24 09:00:00     Buy     100
SPY     2024-09-24 11:30:00     Sell    100                     SPY     2024-09-24 11:30:00     Sell    100
SPY     2024-09-25 10:15:00     Buy     150                   | SPY     2024-09-25 10:15:00     Sell    200
SPY     2024-09-25 13:45:00     Sell    200                   | SPY     2024-09-25 13:45:00     Buy     150

In all of the above examples, the compromised client does not have the information necessary to distinguish data that has been silently changed from data that remains unchanged since the original backup. Below is the comparison (diff) of the original (valid) repository on the left and a compromised (invalid) repository on the right. How could one distinguish between these two and determine which, if any, is valid?

borg_list_repo_1.json borg_list_repo_2.json

{                                                               {
    "archives": [                                                   "archives": [
        {                                                               {
            "archive": "backup",                                            "archive": "backup",
            "barchive": "backup",                                           "barchive": "backup",
            "id": "284a377d5aff135e4ff92137ab0cd8db33bbdefb04 |             "id": "aae662e970aa1fb6503e51c5ea25cfb9bd5450c253
            "name": "backup",                                               "name": "backup",
            "start": "2024-09-23T00:24:18.000000",                          "start": "2024-09-23T00:24:18.000000",
            "time": "2024-09-23T00:24:18.000000"                            "time": "2024-09-23T00:24:18.000000"
        }                                                               }
    ],                                                              ],
    "encryption": {                                                 "encryption": {
        "mode": "none"                                                  "mode": "none"
    },                                                              },
    "repository": {                                                 "repository": {
        "id": "d492cf09410d14d120c99db9b1e5a56d93445c7f5545d2           "id": "d492cf09410d14d120c99db9b1e5a56d93445c7f5545d2
        "last_modified": "2024-09-23T00:24:18.000000",                  "last_modified": "2024-09-23T00:24:18.000000",
        "location": "/home/greg/borg/borg_demo                          "location": "/home/greg/borg/borg_demo
    }                                                               }
}

Legacy Approaches to Mitigating Borg Client Compromise

Borg client compromise can be mitigated using several standard security approaches, all of which rely on some higher and more secure source of authority to validate the client database:

Additional backups, supervisors, or monitors of the client security database within the overall client or enterprise network: This and related family of approaches involve using a more secure environment (a reference, monitor, or kernel) to supervise and protect a less secure one. Such supervisors are a fine approach, but the complexity and security challenges of constructing and maintaining the required reference system are daunting. They are even more daunting in our example, where one secure system has already been compromised. Nor do they prove which of the conflicting copies of a client database is valid.
Physical hardware that provides Write Once, Read Many (WORM) guarantees or services that promise WORM, versioning, and immutability and emulate these in software: These approaches introduce additional hardware and vendor dependencies, require custom tooling to secure Borg client data, and depend on sound configuration to protect the added services and interfaces. It is worth noting that all major public cloud providers (AWS, Google Cloud, and Microsoft Azure) have accidentally deleted customer data due to errors or misconfiguration.
Trusted timestamping, such as RFC3161: This is an important and complex mechanism for timestamping and verifying. Its severe drawbacks merit their separate discussion.

Mitigating Borg Client Compromise Using Public Blockchain Infrastructure

Thanks to the recent advances in blockchain infrastructure, we can now leverage the best features of secure backup, supervisors, immutable storage, and trusted timestamping discussed above and eliminate the need for proprietary services, hardware, and custom tooling.

Using public blockchain as the service for secure backup/supervisor/monitor functionality offers a number of advantages over legacy solutions:

Because a blockchain is a tamper-proof append-only ledger, it is impossible to re-write history and tamper with historical archive IDs, even if the client is compromised. Control of the present no longer gives control of the past. An entire peer-to-peer network must be compromised to impair security.
Archives IDs, timestamps, and other client data can be secured and verified independently of any single entity, company, or service provider and rely on the public internet.
A rich and rapidly growing ecosystem of open-source tools and public infrastructure can be used independently of or in addition to commercial services. Archives can be verified using several tools, such as Dune Analytics or HTTP requests to network nodes.
You don’t need to roll your idiosyncratic tools, deploy monitors, stress over operational security, and manage additional cloud infra and services.

validityBase has developed lightweight tools that serve this very purpose. The vBase command line interface (CLI) can receive a Borg archive ID and publish it to a public blockchain, thus assigning the archive ID a permanent, immutable, and globally verifiable timestamp backed by a large peer-to-peer network that secures $billions. Since the archive ID derives from the hash of the underlying archive data, it serves as a secure fingerprint of the archive itself.

By comparing the ID of the archive you are looking to restore or verify to the timestamped archive ID recorded on the blockchain, it is trivial to tell whether the archive you’re seeing today is the same one you backed up. An additional benefit is that blockchain data can serve as a secure source of truth for two Borg clients to synchronize state.

Securing Borg Client with the vBase CLI

The vBase CLI provides a convenient abstraction to blockchain operations. The tool makes it trivial to create a secure stamp and, thus, a timestamp or the archive ID with a single command. We colloquially refer to a timestamped cryptographic commitment as a stamp:

borg list --json $REPO_NAME | \
jq -r '.archives[].id’ | \
vbase commitment-service add-object --object-cid-stdin
{'objectCid': '0x23629be451287238b8f3ccfc9e7280db8ae5d8d29461602e3f64185afe50555c',
 'timestamp': '2024-09-24 05:51:52+00:00',
 'transactionHash': HexBytes('0x344b410ce6e131236cd9e6500ff4d551d8373bf316c84273c5368a8cbb1445c9'),
 'user': '0x8626f6940E2eb28930eFb4CeF49B2d1F2C9C1199'}

A client can verify the archive just as easily:

borg list --json $REPO_NAME | \
jq -r '"\(.repository.id) \(.archives[0].time)"' | \
vbase commitment-service verify-object --object-cid-stdin --timestamp-stdin

To get started, check out the CLI or other integration and verification options on GitHub. This open-source tool relies on public infrastructure and does not require permission or registration. Nevertheless, you may find if convenient to register for a free vBase account and use our optional managed services that abstract and simplify access to the public blockchains.

Conclusion

Borg’s security model assumes the client is always secure.
If a client is compromised today, all Borg archives made in the past may become suspect and could be silently modified.
The legacy mitigations of Borg client compromise require complex configuration, setup, integration, and the development of custom tooling.
Cryptographic stamps secured by public blockchains are a simple way to prevent compromised clients from silently modifying archives.
vBase tools and services offer a simple interface that abstracts public blockchains and makes mitigation simple.

vBase Blog