Scroll to navigation

GB-STORAGE-TOOL(1) User Manual GB-STORAGE-TOOL(1)

NAME

gb-storage-tool - Manage guerillabackup backup data storages

SYNOPSIS

gb-storage-tool --Config file --DryRun

DESCRIPTION

This is the manual page for the gb-storage-tool command. The tool is used to perform operations on backup file storage locations as used by gb-backup-generator or gb-transfer-service to store backup data.

Currently the tool supports checking storage file naming to identify incomplete backups due to aborts during backup generation or transfer e.g. by reboots or crashes. To ignore files for a reason, e.g. notes, add entries to the status file, e.g.

For all files defining valid backup data elements, configurable policies are applied. See POLICIES section below for supported policies.

OPTIONS

--Config file

This optional parameter specifies an alternative configuration file instead of /etc/guerillabackup/storage-tool-config.json.

--DryRun

This optional parameter will make gb-storage-tool perform policy checks only but will not modify the storage, e.g. by deleting files flagged for deletion by a retention policy.

POLICIES

gb-storage-tool can apply multiple policies to each backup data source but it is only possible to have one policy of a given type (see policy types below). Which policies to apply is defined by the gb-storage-tool configuration file "Policies" parameter. A regular expression is used to select which sources policies should be applied to with the first matching expression taking precedence. For each regular expression a list of polices with parameters is defined. See /data/etc/storage-tool-config.json.template for examples.

To ease policy selection in large setups, policy inheritance can be used. A included configuration (see "Include" configuration parameter) may also define policies, which can extend or override the policies from the parent configuration(s) but also policies defined just earlier in the same configuration. The overriding policy definition has to have a higher priority, otherwise it will be ignored. To disable policy inheritance a subconfiguration may set the "Inherit" configuration parameter to false (default is true). This will also prevent any policies defined earlier in the very same configuration to be ignored. Thus to disable inheritance for all sources in a configuration, the first entry in the policy list should match all sources (.*) and disable inheritance.

Each policy defined in the gb-storage-tool configuration file may also keep policy status information in the status file. The status data is usually updated as the policy is applied unless there is a significant policy violation. That will require the user either to fix the root cause of the violation (e.g. backup data was found to be missing) or the user may update the status to ignore the violation. The later cannot be done interactively via gb-storage-tool yet, one has to adjust the storage status configuration manually. Therefore the user has to create or update the status configuration with the the backup element name (the filename relative to the data directory without any suffix) as key and the status information for the policy as described below (and sometimes given as hint on the console too).

gb-storage-tool supports following policies:

Interval:

Verify that all sources generate backups at expected rates and all backups were transferred successfully. Thus this policy eases spotting of system failures in the backup system. An example policy configuration is: 

...


  "Policies": [


    {


      "Sources": "^(.*/)?root$",


      "Inherit": false,


      "List": [


        {


          "Name": "Interval",


          "Priority": 100,


          "FullMin": "6d20H",


          "FullMax": "6d28H",


          "IncMin": "20H",


          "IncMax": "28H"


        }, {
...

This configuration specifies that to all backups from source with name "root" (the default backup created by the gb-backup-generator) an Interval policy shall be applied. The policy will expect full backups every 7 days +- 4 hours and incremental backups each day +- 4 hours.

When policy validation fails for a given source, the policy configuration may be adjusted but also the violation may be ignored by updating the check status. Thus the validation error will not be reported any more in the next run. The status data in that case may look like:

...


  "20200102000000-root-full": {


    "Interval": {


      "Ignore": "both"


    }


  },
...

This status indicates, that the both interval checks for the interval from the previous full and incremental backup to the backup named above should be disabled. Do disable only one type of checks, the "full" or "inc" type keyword is used instead of "both".

While above is fine to ignore singular policy violations, also the policy itself may be adjusted. This is useful when e.g. the backup generation intervals where changed at the source. The status data in that case could look like:

...


  "20200102000000-root-full": {


    "Interval": {


      "Config": {


        "FullMax": "29d28H",


        "FullMin": "29d20H",


        "IncMax": "6d28H",


        "IncMin": "6d20H"


      }


    }


  },
...

LevelRetention:

This defines a retention policy defined by retention levels, e.g. on first level keep each backup for 30 days, next level keep 12 weekly backups, on the next level keep 12 monthly backups, then 12 every three month and from that on only yearly ones. 

...


  "Policies": [


    {


      "Sources": "^(.*/)?root$",


      "List": [


        {


          "Name": "LevelRetention",


          "Levels": [
# Keep weekly backups for 30 days, including incremental backups.


            {


              "KeepCount": 30,


              "Interval": "day",


              "TimeRef": "latest",


              "KeepInc": true


            },
# Keep weekly backups for 3 month, approx. 13 backups.


            {


              "KeepCount": 13,


              "Interval": "day",


              "AlignModulus": 7


            },
...


            {


              "KeepCount": 12,


              "Interval": "month",


              "AlignModulus": 3,


              "AlignValue": 1


            },
...

This configuration defines, that on the finest level, backups for 30 days should be kept counting from the most recent on ("TimeRef": "latest"), including incremental backups ("KeepInc": true). Thus for machines not producing backups any more, the most recent ones are kept unconditionally.

On the next level, 13 weekly backups are kept, while may overlap with backups already kept due to the first level configuration from above. But here only full backups will be kept, that were generated after every 7th day due to "AlignModulus", preferring the one generated on day 0.

At another level, only one backup is kept every three month, preferring the one from the month numbered 1, 4, 7, 10 due to "AlignModulus" and "AlignValue". Hence the first backup in January, April, ... should be kept.

Size:

This policy checks that backup data sizes are as expected as size changes may indicate problems, e.g. a size increase due to archives, database dumps, local file backups ... forgotten by the administrator (thus wasting backup space but sometimes also causing security issues due to lack of as strict access permissions on those files compared to their source), size increase due to rampant processes filling up database tables or log files in retry loops (also monitoring should catch that), core dumps accumulating, ...

A "Size" policy can be defined for both full and incremental backups. For each backup type, the accepted size range can be defined by absolute or relative values. Without providing an expected size, the size of the first backup of that type seen is used. Therefore for servers without accumulating data, following policy could be defined:

...


  "Policies": [


    {


      "Sources": "^(.*/)?root$",


      "List": [


        {


          "Name": "Size",


          "Priority": 0,


          "FullSizeMinRel": 0.9,


          "FullSizeMaxRel": 1.1,


          "IncSizeMin": 100000,


          "IncSizeMaxRel": 10.0


        }, {
...

This configuration will check sizes of "root" backups using the first full and incremental size as reference. Full backups may vary in size between 90% and 110% while incremental backups have to be at least 100kb large but may vary 10-fold in size. All supported policy parameters are:

FullSizeExpect [size (int)]

Specify the expected full backup size. When missing the size of first full backup seen is used as default.

FullSizeMax [size (int)]

Specify the absolute maximum backup size. You cannot use "FullSizeMaxRel" at the same time.

FullSizeMin [size (int)]

Specify the absolute minimum backup size. You cannot use "FullSizeMinRel" at the same time.

IncSizeExpect, IncSizeMax, IncSizeMin [size (int)]

Same as "Full..." parameters just for incremental backups. See above.

FullSizeMaxRel [ratio (float)]

Specify the maximum backup size in relation to the expected size (see "FullSizeExpect"). You cannot use "FullSizeMax" at the same time.

FullSizeMinRel [ratio (float)]

Specify the minimum backup size in relation to the expected size (see "FullSizeExpect"). You cannot use "FullSizeMin" at the same time.

IncSizeExpectRel [ratio (float)]

Specify the expected incremental backup size in relation to the expected full backup size (see "FullSizeExpect"). You cannot use "IncSizeExpect" at the same time.

IncSizeMaxRel, IncSizeMinRel [ratio (float)]

Same as "Full..." parameters just for incremental backups. See above.

When policy validation fails for a given source, the policy configuration may be adjusted but also the violation may be ignored by updating the check status. Thus the validation error will not be reported any more in the next run. The status data in that case may look like:

...


  "20200102000000-root-full": {


    "Size": {


      "Ignore": true


    }


  },
...

While above is fine to ignore singular policy violations, also the policy itself may be adjusted. This is useful when e.g. the size of backups changed due to installing of new software or services. The updated policy configuration can then be attached to the first element it should apply to:

...


  "20200102000000-root-full": {


    "Size": {


      "Config": {


        "FullSizeExpect": 234567890,


        "FullSizeMinRel": 0.9,


        "FullSizeMaxRel": 1.1,


        "IncSizeMin": 100000,


        "IncSizeMaxRel": 10.0


      }


    }


  },
...

FILES

/etc/guerillabackup/storage-tool-config.json

The default configuration file for gb-storage-tool tool. Use storage-tool-config.json.template to create it. The template also contains the documentation for each available parameter. The most relevant parameters for gb-storage-tool are DataDir, Include and Status.

/var/lib/guerillabackup/state/storage-tool-status.json

This is the recommended location for the toplevel gb-storage-tool status file. The file has to contain valid JSON data but also comment lines starting with #. See POLICIES section above for description of policy specific status data.

BUGS

For guerillabackup setups installed from packages, e.g. .deb or .rpm files usually installed via package management software, e.g. apt-get, aptitude, rpm, yast, please report bugs to the package maintainer.

For setups from unpackaged software trunk, please report at https://github.com/halfdog/guerillabackup/issues.

SEE ALSO

gb-transfer-service(1)

AUTHOR

halfdog <me@halfdog.net>

Wrote this manual page.

COPYRIGHT

Copyright © 2022-2023 halfdog

This manual page was written for guerillabackup system on Linux systems, e.g. Debian.

Permission is granted to copy, distribute and/or modify this document under the terms of the Lesser GNU General Public License, Version 3.

On Debian systems, the complete text of the Lesser GNU General Public License can be found in /usr/share/common-licenses/LGPL-3.

09/16/2023 guerillabackup