MB on RPi - scripts, export, http uploads progressively delayed - severe issues ** symptomatically solved **

[Gyvate]

my scenario:
MB build 3007 on RPi4B-2GB - posting (amongst other targets) to two Meteotemplate instances on two different webhosts.
Until recently everything went fine, but suddenly Meteotemplate stopped automatically archiving data - manually adding records to the MT DB (mySQL) worked. I had already noticed earlier that the display of current data was lagging behind the real time by 20-30 minutes.
Server time checked - OK, MB RPi time checked OK.

The data received on Meteotemplate showed the delayed EPOCH timestamp. It appeared that once the delay between real server time and the MB posted data timestamp became bigger than 30 minutes, Meteotemplate stopped archiving. The programmer of the Meteotemplate (@Jachym) is supposed to look into this, but Meteobridge should post the data properly in the first place and not so strongly delayed.
When I moved the http post from the MB RPi to a MB Pro, there was no delay in posting. That shows to me that something is wrong with the MB RPi implementation.

Both MB Pro and MB Rpi are connected via Ethernet to the same switch => local network issue can be excluded.
Both have the same gateway and DNS entries.

After rebooting the MB Rpi, the posted time stamp and the remote server time are +/- in synch.
However, as time goes by, the MB RPi posts lag more and more behind - which can also be seen in in the Upload Status
Posting interval: 15 seconds
e.g. after reboot
HTTP Upload: 2023-09-08 16:36:11 Sent: 2023-09-08 16:36:08 success
after 17 minutes, already 3 minutes delay - and it keeps on growing
HTTP Upload: 2023-09-08 16:53:11 Sent: 2023-09-08 16:50:28 Success
HTTP Upload: 2023-09-08 17:03:02 Sent: 2023-09-08 16:58:44 success

that's not the only http upload on my MB PRi (nor on my MB Pro) and the Upload status monitor doesn't say which post it is referring to, but at the the remote server end the delay is also clearly visible.

I assume the question why it works with the MB Pro (no delay) and not with the MB RPi (increasing delay) probably only @admin can answer ...

It was running "well" (i.e. without major issues noticed at the Meteotemplate end, the delay itself was there already for a long time but didn't create major issues - therefore tolerated) for years, but now the delay crossed the 30 minute tolerance line at the Meteotemplate end and it became noticeable.
And it's obviously a MB issue, not a MT issue. Data is supposed to be actual for the current postings.

All the above observations together point to the MB on RPi implementation, which is the latest update.
The conditions in LAN and WAN are the same for both devices.
It's not a DB issue or remote server performance issue either - plenty of space, plenty of processing power.
Only what it could be remains in the dark for me ...
a queing issue ? A cache issue ? Housekeeping issue ? ....

I don't want to reboot my MB on RPi every second day ...

any thoughts ?

[Gyvate]

just to add:
I was running well for years up to build 2996 - it only occurred after the upgrade to 3005.
Maybe something was added then which is responsible for the delayed posting ...

[Gyvate]

I started a radical test:
- copied a new image on a SD card
- started with a new database from scratch
the issues remain ... - something seems to be terribly wrong
the posting gets delayed, even the start of scripts (.sh) - I have two scripts - one is an hourly backup - even that gets delayed

MB-20-uploads_20230912_0634.JPG (61.28 KiB) Viewed 1216 times

pressing the "test" button in the events page doesn't show an effect ...
even initiating a backup from the admin page ... - no reaction
Only a reboot helps - for a few hours ....
Somehow this system seems to confuse the time ....
It's starts on time 20:02 and gets more and more delayed.
what in the end looks like three minutes too early is in fact 57 minutes late ...
2nd last entry was a manual backup after reboot, last entry event triggerred script like 11 hours before at 20:00

MB-20-backups_20230912_0701.JPG (62.56 KiB) Viewed 1216 times

history:
I ran suddenly in a larger number of database issues when MB tried to create records for already existing entries.
Then MB complained of a malformed database imiage.
Doing a sqlite ".recover" and rebuilding the database externally didn't help.
Internally it wouldn't even start the forced rebuilt claiming not enough storage available - which wasn't the case.
The same with a restore. Even with the option to delete the database before restore brought the message of not enough storage.
The databse had a size of 6GB, 8 GB were still free - after the delete 14 GB would have been free.
But the delete wasn't executed ...
That's when I created a new image and startd the database from scratch.
It's hard to believe that all my bakups are faulty. And I went back three weeks.
I can go back 2 years ... will do this in parallel when time allows on a separate MB/RPi install, maybe also going back in MB version ....
Let me see what the new database does, but already the timings are delayed, seemingly ignored, maybe stuck in some queue ...

as you can see from the pictures below, MB doesn't tell the truth - neither was the current database deleted (ls -l) nor was there insufficient space (df -h) ....

MB-20-logfile-b2998_20230912_0754.jpg (168.63 KiB) Viewed 1211 times

MB-20-df-h-ls-l_20230912_0754.jpg (108.57 KiB) Viewed 1211 times

[admin]

sorry, it is hard to follow your line of thought. When you want to inspect the upload characteristic of a certain HTTP upload, I would recommend to pause all the others while inspecting. That way you can easily see the schedules Meteobridge is trying to apply to the one service you are keen about.

When it works ok, it might be some overdosing with the 19 other events you defined for various uploads. Some with repeat rates of 15 seconds pointing to private servers with unknown http/s response times.

Does the Buffer indicator on "System - Info" tab show any signs for a queue building up?

[Gyvate]

sorry, it is hard to follow your line of thought. When you want to inspect the upload characteristic of a certain HTTP upload, I would recommend to pause all the others while inspecting. That way you can easily see the schedules Meteobridge is trying to apply to the one service you are keen about.

When it works ok, it might be some overdosing with the 19 other events you defined for various uploads. Some with repeat rates of 15 seconds pointing to private servers with unknown http/s response times.

Does the Buffer indicator on "System - Info" tab show any signs for a queue building up?

I think I have finally after a long search and many tests indentified the "culprit":
the one and only FTP upload event I have ran into a timeout after the FTP server was upgraded and set a new default option (FTP with SSL/TLS (explicit) ) that MB somehow couldn't handle. (I won't exclude that the reason for this is at the FTP server end).

What comes now is circumstancial evidence.
It looks as if because of long timeout waiting time of the FTP post all other (http) posts got delayed (what in my opinion shouldn't be the case - something for @admin to consider to have a look into).
It all ended up with my database getting inconsistent - and when the restore wouldn't work, I got alarmed (see other post - viewtopic.php?t=17487).

Finally I had the FTP server option untagged, a 2nd RPi set up with a new image, started data logging from scratch - and everything works fine again. No delays, and all my 19 events aren't an issue for neither the RPi4B nor for my remote VPS (that the "private server" hosting my domains) - and has never been before.

The original RPi4B MB installation was in a hopeless mess. I had downgraded it to build 2998 to exclude newly introduced bugs - but it wouldn't go back to build 3007 anymore. So I had to "flatten" it too, install a new image and started from scratch again.

As a lesson learnt (for me) I will now run my MB on two RPi4B in parallel - in hot stand-by - one doing the logging and posting, the other doing logging only - and in case one fails, I only have to activate the postings on the 2nd and I have an uninterrupted service. Both have hourly external backups.

Now comes the work finding the last not corrupted database backup file and then filling the gap to the newly started data collection by importing the daily CSV exports again. Luckily this can now be done on the 2nd RPi4B and once everything is done, I can restore the database backup into the 1st RPi. Uninterrupted data collection safeguarded.

The whole original issue became visible, when Meteotemplate refused to accept the too much belated data posts from Meteobridge.
The MB Pro I was using to do the postings instead of the misbehaving RPi/MB didn't have a FTP event - so all went smoothly.