Meteohub recomputation errors

Discussion of the Meteohub software package

Moderator: Mattk

Post Reply
Unconnected
Junior Boarder
Junior Boarder
Posts: 34
Joined: Wed May 07, 2008 2:58 pm
Location: The Netherlands

Meteohub recomputation errors

Post by Unconnected » Mon Nov 30, 2015 1:09 pm

I'm having an issue with 2 of my graphs, wind and rain. I get graphs like this:

Image
(hope you can see it, the graph is transparant :?)

more problems over here: http://hetweerinbleiswijk.nl/jaaroverzi ... zicht-2014 and here http://hetweerinbleiswijk.nl/jaaroverzi ... zicht-2015
(in the 'neerslag' aka rain section)

some values are waaay to large. now when i check the raw file of that month (2014-10)

root@meteohub:/data/weather/201410$ tail -n 20 raw | grep rain0
20141031235839 rain0 0 0 49599
20141031235926 rain0 0 0 49599
root@meteohub:/data/weather/201410$ head -n 20 raw | grep rain0
20141001000014 rain0 0 0 48768
20141001000101 rain0 0 0 48768

the total of the rain0 looks to me like it should be 49599 - 48768 = 831mm. now when I check the monthly total:

root@meteohub:/data/weather/201410$ cat ../rain0-month1 | grep 2014
...
20140801000000 0.8 0.0 90.9 132.90 22
20140901000000 0.0 0.0 1.8 12.40 3
20141001000000 0.5 0.0 588.8 4976.40 7
20141101000000 9.9 0.0 180.8 9867.10 4
20141201000000 0.1 0.0 128.8 114.50 11
root@meteohub:/data/weather/201410$

the value is almonst 5000mm? where does that value come from? how does meteohub do the total recomputation? hope you can shed some light on this. if you want to, i can send you the raw file for inspection :)

btw: been using your program on a NSLU2 since 2008. working ever since! well, i do have to change the usb stick once in a while, but ey, it 'just works'! good job! :D

Unconnected
Junior Boarder
Junior Boarder
Posts: 34
Joined: Wed May 07, 2008 2:58 pm
Location: The Netherlands

Re: Meteohub recomputation errors

Post by Unconnected » Tue Feb 16, 2016 8:43 am

no idea how to fix this? :(

User avatar
YJB
Platinum Boarder
Platinum Boarder
Posts: 387
Joined: Thu Feb 19, 2009 5:53 pm
Location: Venhuizen, Netherlands
Contact:

Re: Meteohub recomputation errors

Post by YJB » Tue Feb 16, 2016 11:17 am

Hi,

I can't tell you where these values are coming from. What I can tell you is that you will need to:
- Remove the offending values from the raw data files
- Followed by a recomputation, as this will clear the incorrect aggregates

You can edit the raw data by going to the "Inspect data" menu, or edit the raw files directly. I would chose the second option only if you understand what you are doing. In that case a backup upfront might be a good idea.

Ysbrand

Unconnected
Junior Boarder
Junior Boarder
Posts: 34
Joined: Wed May 07, 2008 2:58 pm
Location: The Netherlands

Re: Meteohub recomputation errors

Post by Unconnected » Tue Feb 16, 2016 12:02 pm

thanks for your reply! the problem is that i don't know where the values come from either :/ there aren't any wrong or ridiculous high values in any of the raw files, only in the cummulation files. changing them won't do anything since they will be overwritten when a recalculation starts again :(

User avatar
YJB
Platinum Boarder
Platinum Boarder
Posts: 387
Joined: Thu Feb 19, 2009 5:53 pm
Location: Venhuizen, Netherlands
Contact:

Re: Meteohub recomputation errors

Post by YJB » Tue Feb 16, 2016 12:25 pm

Yes, I agree it looks funny, but I would not be surprised if there is an odd (read: corrupt) value somewhere in the middle of the month.

Currently you are listing the start and end of the month values, but I would recommend to look at the entire month. For analysis pruposes, I would grep for rain0 redirect it to a file, import the file in xls and sort on the total. Once you've indentified an issue you can then go back to the original file and fix it.

Unconnected
Junior Boarder
Junior Boarder
Posts: 34
Joined: Wed May 07, 2008 2:58 pm
Location: The Netherlands

Re: Meteohub recomputation errors

Post by Unconnected » Tue Feb 16, 2016 12:34 pm

YJB wrote:For analysis pruposes, I would grep for rain0 redirect it to a file, import the file in xls and sort on the total. Once you've indentified an issue you can then go back to the original file and fix it.
ah! that's the idea i was looking for! i can do an easy sort in excel. will give that a shot today. perfect :D

Unconnected
Junior Boarder
Junior Boarder
Posts: 34
Joined: Wed May 07, 2008 2:58 pm
Location: The Netherlands

Re: Meteohub recomputation errors

Post by Unconnected » Tue Feb 16, 2016 3:39 pm

hmm. when importing and sorting the file from 10-2014 in excel, i do get some high values as shown in the attachment.

if only i knew how to map these values onto the values of the bold line below :?

20140901000000 0.0 0.0 1.8 12.40 3
20141001000000 0.5 0.0 588.8 4976.40 7
20141101000000 9.9 0.0 180.8 9867.10 4
Attachments
values_201410-rain0.PNG
values_201410-rain0.PNG (20.89 KiB) Viewed 3731 times

User avatar
YJB
Platinum Boarder
Platinum Boarder
Posts: 387
Joined: Thu Feb 19, 2009 5:53 pm
Location: Venhuizen, Netherlands
Contact:

Re: Meteohub recomputation errors

Post by YJB » Tue Feb 16, 2016 4:08 pm

20141001000000 0.5 0.0 588.8 4976.40 7 is referring to the entire month.

Keep in mind that the file rain0-month1 will just contain a single line for each and every month. So this is your statiscal information that has bene collected for that month.

Most likely are the 0 values and 20808 values confusing the aggregate functionality. It appears to me that those entries are not correct and should be removed from the raw file.

Unconnected
Junior Boarder
Junior Boarder
Posts: 34
Joined: Wed May 07, 2008 2:58 pm
Location: The Netherlands

Re: Meteohub recomputation errors

Post by Unconnected » Wed Feb 17, 2016 3:49 pm

YJB wrote:Most likely are the 0 values and 20808 values confusing the aggregate functionality. It appears to me that those entries are not correct and should be removed from the raw file.
I analysed all raw files in excel and removed all the lines containing ridiculous values. Have to wait a few days for the results though :| (the system is running on a NSLU2 with weather data from 2008 up until now. recomputation takes around 5 days :P)

Code: Select all

Process
Weather Data Recomputation:	in progress (■ ■ □ □ □ □ □ □ □ □ □ )
Weather Data Logging:	running (last data: 6 sec)

Post Reply