Raintotal failed spike during import meteohub data. **solved**

[jpouyaud2]

Hi,

I'm experiencing some failed spike with rain0total when Importing old meteohub data (2015) on my meteobridge

Platform: Raspberry Pi 3 Model B
SW Version: Meteobridge 5.6 (Mar 11 2023, build 2709), FW 1.3
per-minute data are used

when importing the meteohub RAW file for a month, the import is processing well.
The issue is when I'm inspecting the data imported:

I see some wrong spike for raintotal

here is an example for 5 Jan 2015 (the spike is at 14:00) => 12,6 mm are reported (it was not such raining at this time)

Capture d’écran 2023-03-15 à 20.36.58.png (163.87 KiB) Viewed 1536 times

Capture d’écran 2023-03-15 à 20.37.11.png (215.13 KiB) Viewed 1536 times

Capture d’écran 2023-03-15 à 20.37.20.png (250.13 KiB) Viewed 1536 times

I was initially thinking of a meteohub corrupted Raw file (bad sequencing for rain0total value): it is not. the files is fine

it's look like their is a "stack" counter which is flushing at a time in the code

I've this behavior a lot of time for a year period imported, which is ruining the precipitation data

I can share the meteohub raw file

Regards

[jpouyaud2]

Hi

the raw file can be downloaded here

https://paloux.meteo-triadou.fr/upload/raw.txt

cheers

[Gyvate]

obviously the same amount of rain was already there earlier, first registration of 12.6 mm was at 20150105103040.
There it should show.
Assuming that the time stamp is UTC, the increase from 12.4 to 12.6 shows correctly in the 11:00 hourly window.
The 12.4 mm had been taken over from 20150504 - where they shouldn't have been in the first place if this was a daily count.
I don't know what rain0 x y z means. If the z stands for monthly count, the increase was taken over correctly. If it was daily, it should have been reset to 0 at midnight.
Anyhow, it looks like it was wrongly interpreted by the import program - there is not visible reason in the data import file for me why to add 12.6 mm at 14:00 ...
Possibly a bug ...

[jpouyaud2]

Hi Gyvate,

yes, I confirm the 12,4mm had been taken over wrongly by the import. I see this behavior in a lot of occurence during the import

for your reference.
rain0 x y z : z is the cumulative value for rain total. supposed to be a Yearly one (not daily neither monthly)

what is strange is this import function was working without issue when I used it some years ago (old version)

cheers

[WETRHE]

I found out the same issue with wswin import:
viewtopic.php?t=17222

[jpouyaud2]

hi Wetrhe,
did you find some ways to workaround this bug with import data ?
regards

[jpouyaud2]

Hi,

would be great to just have an ack/confirmation from the dev that something is wrong with the import code

thanks

[admin]

When you send me files that import incorrectly, I can test: info(at)meteobridge.com

[Gyvate]

When you send me files that import incorrectly, I can test: info(at)meteobridge.com

the OP has provided a link for a raw data upload in the 2nd post, something to begin with, and the 1st post refers to that data - but he should have more data available onsite.
https://paloux.meteo-triadou.fr/upload/raw.txt

with this data and the screenshots it can be seen where the error occurs.
I looked that up myself and could see that the import doesn't seem to work properly for the rain data (hence my response).
Looks like the rain counter is not reset when it should => rain events with data where there shouldn't be any.

[admin]

When you import rain data into a database that already has rain data for the same period things will go crazy. That is to be expected. Does the error occur even when rain data has been erased before (you can do on "edit hourly history" tab)?

[jpouyaud2]

Hello,

I've just re-uploaded the raw.txt file if you want to reproduce the bug

cheers

https://paloux.meteo-triadou.fr/upload/raw.txt

[jpouyaud2]

I confirm there is no rain data for the same period.

yes, the error occur when rain data have been erased before

cheers

[admin]

I was able to identify the error. It has to do with Meteobridge importing the file in chunks and rain totals as a cumulative value is not handled correctly when starting with the next chunk. Just released version should have that fixed, please also read here (April 22, 2023): https://meteobridge.com/wiki/index.php/Forum

[jpouyaud2]

Excellent !

this is a really good news Borris

I will proceed again for import mostly 5 year on one of my Meteobridge

thanks a lot