meteohub.de

the system keeps on bringing me the below messages - different restores from different MB on RPi backups
now started forced rebuild on reboot - will take some time as the DB size is 10 GB.

Any ideas ?

System
Platform: Meteobridge x86 (VM)
RAM: 238 MB total, 120 MB free (49% used)
Storage: /dev/root, 121.9 GB total, 91.7 GB free (25% used)
SW Version: Meteobridge 6.0 (Aug 20 2024, build 1209), FW 1.0

setup (24.08.2024 16:41:26): use UTC stamp 2024-08-24 14:41:26 (1 tries)
dbase (24.08.2024 16:41:31): process started.
dbase (24.08.2024 16:41:31): Content of RAM database restored.
wnetd (24.08.2024 16:41:33): process started.
system (24.08.2024 16:41:33): start of logging process initiated.
logger (24.08.2024 16:41:33): Free trial period will end in 1 days. When you want to use this device beyond that period, you are welcome to use online licensing feature on "License" tab to purchase a license.
logger (24.08.2024 16:41:33): databases connected. Update interval 3600 seconds.
logger (24.08.2024 16:41:33): data logger started.
wnetd (24.08.2024 16:41:34): send data thread started (pid:2879 tid:2882)
logger (24.08.2024 16:41:34): connect station 0 (GW1000 via TCP/IP).
logger (24.08.2024 16:41:34): connect station 9 (System Data via Plug-in).
logger (24.08.2024 16:41:37): waiting for sensors (wind0, rain0, th0, thb0) to provide data.
logger (24.08.2024 16:41:37): services paused.
dbase (24.08.2024 16:41:39): failed with step 12 on 'lgt0dist_hour' (after 0 retries in 1ms): database disk image is malformed (11)
dbase (24.08.2024 16:41:39): rollback on sync forced (hour): lgt0dist
logger (24.08.2024 16:41:39): station 0 (GW1000), Radio Frequency: 868 MHz
logger (24.08.2024 16:41:45): station 0 (GW1000), FW version: GW2000A_V3.1.5
logger (24.08.2024 16:41:51): sensors are now providing data.
logger (24.08.2024 16:41:51): services started.
dbase (24.08.2024 16:42:41): failed with step 12 on 'lgt0dist_hour' (after 0 retries in 1ms): database disk image is malformed (11)
dbase (24.08.2024 16:42:41): rollback on insert forced (2) (hour): lgt0dist
dbase (24.08.2024 16:42:41): failed with step 12 on 'lgt0dist_hour' (after 0 retries in 1ms): database disk image is malformed (11)
dbase (24.08.2024 16:42:41): rollback on insert forced (2) (hour): lgt0dist
dbase (24.08.2024 16:42:41): failed with step 12 on 'lgt0dist_hour' (after 0 retries in 1ms): database disk image is malformed (11)
dbase (24.08.2024 16:42:41): rollback on insert forced (2) (hour): lgt0dist
dbase (24.08.2024 16:42:41): failed with step 12 on 'lgt0dist_hour' (after 0 retries in 1ms): database disk image is malformed (11)
dbase (24.08.2024 16:42:41): rollback on insert forced (2) (hour): lgt0dist
dbase (24.08.2024 16:42:41): failed with step 12 on 'lgt0dist_hour' (after 0 retries in 1ms): database disk image is malformed (11)
dbase (24.08.2024 16:42:41): rollback on insert forced (2) (hour): lgt0dist
dbase (24.08.2024 16:42:41): failed with step 12 on 'lgt0dist_hour' (after 0 retries in 1ms): database disk image is malformed (11)
dbase (24.08.2024 16:42:41): rollback on insert forced (2) (hour): lgt0dist
dbase (24.08.2024 16:42:41): failed with step 12 on 'lgt0dist_hour' (after 0 retries in 1ms): database disk image is malformed (11)
dbase (24.08.2024 16:42:41): rollback on insert forced (2) (hour): lgt0dist

after an 8 hour database rebuild the error messages have disappeared (and 8 hours data are missing now)
why can't MB VM not properly import a database backup created on a MB on RPi without needing to rebuild the database ?
I say so, because I have tried it about 10 times - always with the same result (database malformed).
Or how is a database migration supposed to happen ?
Via export on the RPi and import on the VM ? This will even take much much longer (the import) and actual data will also be missing for that period ...
data occurring during the import are not saved in the WAL file and later on written to the database.
At least I cannot see that - for the time of the import the minute data show gaps.
To avoid or to minimize that I had already on the RPi cut the import into hourly chunks - but with a 5-year data containing database this will take forever .... 24 x 365 x 5 = 43,800 !! Each import 5 minutes .. it will take 1/2 year day and night ...
I'm wondering why the upcoming data are not stored in the WAL file and then added after the rebuild ...
during a backup the database is also locked and occurring data are saved in the WAL file ...

Maybe going down the rebuild road and importing the missing data (as my MB on RPi keeps on running) in small chunks is the least time consuming and data missing/losing approach ... - but in case of a needed migration without a backup system, this would be a fatal data loss

after reading another thread with seemingly the same issue I conclude that the SQL export/import might work - but again will take a lot of time and create actual data loss ... - may be nice for small database but not for big ones ...
plus I cannot export my database as SQL as I have kept my RPis on 5.8 (resp. went back) as the new Service Event handling broke my whole 20 events - and I didn't have the fun time yet to edit everything again from scratch

Regarding your problems to migrate to version 6.0 you can send me your config file, so that I can test and fix the conversion bug (if there is any).

Regarding database backups. This is a binary copy of the database file. No one can guarantee that this binary copy will work on other platforms. Doing the SQL export is for sure portable, but when you have millions of datasets it will take time to import. I tried to have this done in the background but sqlite3 is blocking tables during write so restoring data and storing new data from sensors collides and doe snot work. I tested this.

the SQL export appears to be an all-or-nothing affair, meaning the whole database is exported.
Is there an option (maybe availabe soon) to limit the SQL export by a from - to window (at least by the day - the already existing entries will anyway be discarded) ?

You can easily help yourself by reducing to the records within an selected time frame by running this on the generated full database dump (here sql-202408301447.gz). The example selects month July 2024 which has epoch times 1719784800 to 1722463200. Simply run this on the Meteobridge shell inside the /tmp/mnt/data/backup directory.

admin wrote: ↑Fri Aug 30, 2024 3:48 pm You can easily help yourself by reducing to the records within an selected time frame by running this on the generated full database dump (here sql-202408301447.gz). The example selects month July 2024 which has epoch times 1719784800 to 1722463200. Simply run this on the Meteobridge shell inside the /tmp/mnt/data/backup directory.

Code: Select all

zcat sql-202408301447.gz | awk ' BEGIN{FS="[0-9S][,(]"} { if ($2 > 0) {if ($2 >= 1719784800 && $2 < 1722463200) {print $0} } else {print $0} }' | gzip >sql-202407.gz