Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Periodic failures #71

Open
rgaudin opened this issue Jul 23, 2021 · 0 comments
Open

Periodic failures #71

rgaudin opened this issue Jul 23, 2021 · 0 comments
Labels
bug Something isn't working

Comments

@rgaudin
Copy link
Member

rgaudin commented Jul 23, 2021

On our large download backup, the backup stops working every 2 months and requires to be restarted. Borg base alerts informs us.

Logs mentions a connection issue but given the number of layers there are, logs are frequently useless.

Configuration:

BORGBASE_NAME=download
BITWARDEN_PASSWORD=*******
BITWARDEN_EMAIL=******
BACKUP_HOUR=2
BACKUP_MINUTE=0
KEEP_WITHIN=0H
KEEP_DAILY=7
KEEP_WEEKLY=5
KEEP_MONTHLY=12
KEEP_YEARLY=1
QUOTA=0
REGION=eu
ALERT=1
PERIODICITY=1d
BACKUP_DAY=1
DATABASES=
MAX_BORGMATIC_RETRY=10
WAIT_BEFORE_BORGMATIC_RETRY=5

Here's the tail of the log

M /storage/openzim/release/libzim/feed.xml
M /storage/openzim/release/feed.xml
Remote: Storage quota: 879.21 GB out of 3.96 TB used.
Remote: Storage quota: 879.21 GB out of 3.96 TB used.
[email protected]:repo: Running consistency checks
Remote: Starting repository check
Remote: packet_write_wait: Connection to 65.21.33.81 port 22: Broken pipe
Connection closed by remote host
[email protected]:repo: Error running actions for repository
Command 'borg check --prefix {hostname}- --info [email protected]:repo' returned non-zero exit status 2.
/root/.config/borgmatic/config.yaml: Error running configuration file

summary:
/root/.config/borgmatic/config.yaml: Error running configuration file
[email protected]:repo: Error running actions for repository
Remote: Starting repository check
Remote: packet_write_wait: Connection to 65.21.33.81 port 22: Broken pipe
Connection closed by remote host
Command 'borg check --prefix {hostname}- --info [email protected]:repo' returned non-zero exit status 2.

Need some help? https://torsion.org/borgmatic/#issues
[email protected]:repo: Pruning archives
Keeping archive: download__backup__2021-07-20T02:00:40 Tue, 2021-07-20 02:00:42 [05e109f47cdabf316764a0b364a3adead675200a8ee482119b85dc63d16e38d9]
Keeping archive: download__backup__2021-07-19T02:00:13 Mon, 2021-07-19 02:00:15 [1949cc1def2dff8d7a82544a3bfb53a1774e1f0f3581174c4805add9e5ee3360]
Keeping archive: download__backup__2021-07-18T02:00:33 Sun, 2021-07-18 02:00:35 [b55f59819bf147a0c6b09a11de9e0e20483754eed30e73072f3564b5e17be8b8]
Keeping archive: download__backup__2021-07-17T02:00:16 Sat, 2021-07-17 02:00:18 [022630cb7db2a35a71afde80d23e62a842db7654d6d353e388c3d384457e220c]
Keeping archive: download__backup__2021-07-16T02:00:17 Fri, 2021-07-16 02:00:19 [9f8675681dd627010dbe72e1d8a822c3a6a7daa299b5893518838a716e6bb901]
Keeping archive: download__backup__2021-07-15T02:00:30 Thu, 2021-07-15 02:00:32 [cf314a498013c72c14166cc10b0e8e19dec5a45f455c2e02f62147b6e21065c6]
Keeping archive: download__backup__2021-07-14T02:00:38 Wed, 2021-07-14 02:00:40 [2eb19a6f256abf34f838b62d0221b78029d2339e055b5381c630cf8d1c1d6019]
Pruning archive: download__backup__2021-07-13T02:00:29 Tue, 2021-07-13 02:00:32 [25418a95bc2fb698119d2fd5084c877d778244a30da703fad7ece94c3b4b0585] (1/1)
Remote: Storage quota: 879.21 GB out of 3.96 TB used.
Keeping archive: download__backup__2021-07-11T02:00:10 Sun, 2021-07-11 02:00:11 [e07405e3f376e2e36097a37573897ad99afd192c0c623bacddc446cc9956f3b0]
Keeping archive: download__backup__2021-07-04T02:00:14 Sun, 2021-07-04 02:00:16 [12dcaaf12fc33eb79fecc3dec2d925eb41e614bea69e4a5e4d087ad42cf04bd4]
Keeping archive: download__backup__2021-06-30T02:00:31 Wed, 2021-06-30 02:00:33 [513c6714e5d7834e989fae4dbca9844b873d04c6f920298394dfdeebb2cf47d8]
Keeping archive: download__backup__2021-06-27T02:00:28 Sun, 2021-06-27 02:00:29 [718e41af436a75f3050fdbdcf5e889520933a1142b7c1149bee6fdfcd1481d5d]
Keeping archive: download__backup__2021-06-20T02:00:15 Sun, 2021-06-20 02:00:16 [47d59aa9cfc9f539214d7e674dc9c658af30f2bae9c1ba2184566fb2e438da06]
Keeping archive: download__backup__2021-06-13T02:00:17 Sun, 2021-06-13 02:00:18 [de186b99a4f1cc4908315d97b081fcbb8027e65b4a27bb890081ff854693693f]
Keeping archive: download__backup__2021-05-31T02:00:19 Mon, 2021-05-31 02:00:20 [8ccbccd160fc7aec5bb98c6ec1bfd3be46b7a434bc828eb9981715a4f62713b9]
Keeping archive: download__backup__2021-04-30T02:00:31 Fri, 2021-04-30 02:00:35 [6269745493856f09aec0503d9998573f015379db6ca6d5bfd8d17cb657590f4b]
Keeping archive: download__backup__2021-03-31T02:00:14 Wed, 2021-03-31 02:00:15 [f7d129c53127055953b848cafc84dfcb7bbf57aaf8ec6ae512844af19e017944]
Keeping archive: download__backup__2021-02-28T02:00:40 Sun, 2021-02-28 02:00:48 [516ad18252ddbffa126adce29de587e79d481ac8f88000d76763bd050c10615d]
Keeping archive: download__backup__2021-01-31T02:00:21 Sun, 2021-01-31 02:00:23 [d792c3fcb404d68fcd9beed1894e515f9212a0d70b1c29d50b3e5d4a019cca17]
Keeping archive: download__backup__2020-12-31T02:03:27 Thu, 2020-12-31 02:04:08 [709e198f3a9f53baed68fb3dcd82f85a6de520f24923ddfd63ed1a5be6d95029]
Remote: Storage quota: 879.21 GB out of 3.96 TB used.
[email protected]:repo: Creating archive
Creating archive at "[email protected]:repo::download__backup__{now}"
M /storage/kiwix/archive/.htaccess
M /storage/kiwix/other/.htaccess
M /storage/kiwix/dev/.htaccess
M /storage/kiwix/release/.htaccess
M /storage/kiwix/release/kiwix-android/2.5.0/feed.xml
M /storage/kiwix/release/kiwix-android/2.5.1/feed.xml
M /storage/kiwix/release/kiwix-android/2.5.2/feed.xml
M /storage/kiwix/release/kiwix-android/2.5.3/feed.xml
M /storage/kiwix/release/kiwix-android/2.3/feed.xml
M /storage/kiwix/release/kiwix-android/feed.xml
M /storage/kiwix/release/browsers/chrome/feed.xml
A /storage/kiwix/release/browsers/firefox/feed.xml
M /storage/kiwix/release/browsers/feed.xml
M /storage/kiwix/release/browsers/edge/feed.xml
M /storage/kiwix/release/kiwix-js-windows/feed.xml
M /storage/kiwix/release/feed.xml
M /storage/kiwix/release/kiwix-desktop-macos/feed.xml
M /storage/kiwix/release/kiwix-desktop/feed.xml
M /storage/kiwix/release/kiwix-hotspot/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v1.0.0/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.0.4/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.0.6/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.0.7/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.0.5/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.4.0/feed.xml
M /storage/kiwix/release/kiwix-hotspot/master/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.4.1/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.4.2/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.0.8/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.1.0/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.1.1/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.0.3/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.0.0/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.0.1/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.0.2/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.3.7/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.3.8/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.3.0/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.2.0/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.2.1/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.3.2/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.3.1/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.3.3/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.3.5/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.3.6/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.4.3/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.4.4/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.4.5/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.4.6/feed.xml
M /storage/kiwix/release/kiwix-hotspot/v2.3.4/feed.xml
M /storage/kiwix/release/libkiwix/feed.xml
M /storage/kiwix/release/kiwix-tools/feed.xml
M /storage/openzim/release/zimwriterfs/feed.xml
M /storage/openzim/release/zim-tools/feed.xml
M /storage/openzim/release/libzim/feed.xml
M /storage/openzim/release/feed.xml
Remote: Storage quota: 879.21 GB out of 3.96 TB used.
Remote: Storage quota: 879.21 GB out of 3.96 TB used.
[email protected]:repo: Running consistency checks
Remote: Starting repository check
Remote: packet_write_wait: Connection to 65.21.33.81 port 22: Broken pipe
Connection closed by remote host
[email protected]:repo: Error running actions for repository
Command 'borg check --prefix {hostname}- --info [email protected]:repo' returned non-zero exit status 2.
/root/.config/borgmatic/config.yaml: Error running configuration file

summary:
/root/.config/borgmatic/config.yaml: Error running configuration file
[email protected]:repo: Error running actions for repository
Remote: Starting repository check
Remote: packet_write_wait: Connection to 65.21.33.81 port 22: Broken pipe
Connection closed by remote host
Command 'borg check --prefix {hostname}- --info [email protected]:repo' returned non-zero exit status 2.

Need some help? https://torsion.org/borgmatic/#issues
[email protected]:repo: Pruning archives
Remote: write: Connection reset by peer
Connection closed by remote host. Is borg working on the server?
[email protected]:repo: Error running actions for repository
Command 'borg prune --keep-daily 7 --keep-weekly 5 --keep-monthly 12 --keep-yearly 1 --prefix download__backup__ --info --list [email protected]:repo' returned non-zero exit status 2.
/root/.config/borgmatic/config.yaml: Error running configuration file

summary:
/root/.config/borgmatic/config.yaml: Error running configuration file
[email protected]:repo: Error running actions for repository
Remote: write: Connection reset by peer
Connection closed by remote host. Is borg working on the server?
Command 'borg prune --keep-daily 7 --keep-weekly 5 --keep-monthly 12 --keep-yearly 1 --prefix download__backup__ --info --list [email protected]:repo' returned non-zero exit status 2.

Need some help? https://torsion.org/borgmatic/#issues
[email protected]:repo: Pruning archives
Remote: ssh_exchange_identification: read: Connection reset by peer
Connection closed by remote host. Is borg working on the server?
[email protected]:repo: Error running actions for repository
Command 'borg prune --keep-daily 7 --keep-weekly 5 --keep-monthly 12 --keep-yearly 1 --prefix download__backup__ --info --list [email protected]:repo' returned non-zero exit status 2.
/root/.config/borgmatic/config.yaml: Error running configuration file

summary:
/root/.config/borgmatic/config.yaml: Error running configuration file
[email protected]:repo: Error running actions for repository
Remote: ssh_exchange_identification: read: Connection reset by peer
Connection closed by remote host. Is borg working on the server?
Command 'borg prune --keep-daily 7 --keep-weekly 5 --keep-monthly 12 --keep-yearly 1 --prefix download__backup__ --info --list [email protected]:repo' returned non-zero exit status 2.

Need some help? https://torsion.org/borgmatic/#issues
@rgaudin rgaudin added the bug Something isn't working label Jul 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@rgaudin