Page 1 of 1
Logging protocol problem...
Posted: Sun Aug 31, 2008 8:25 pm
by skyewright
Only noticed this today, so I don't know if it has been ever since install of 2.7a, but I don't believe it was there before...
A logging request with a datetime is producing a good response, but a "most recent record" request (the type that WD uses most of all) is now giving :
Error (internal): Invalid formed record (0 4117
) in file "/data/weather/200808/raw".
:(
Any ideas...
Re:Logging protocol problem...
Posted: Sun Aug 31, 2008 8:37 pm
by skyewright
skyewright wrote:
Error (internal): Invalid formed record (0 4117
) in file "/data/weather/200808/raw".[/quote]
Further to this. Looking in raw, I see that that the rain1 total was 4117 between
20080829201756 rain1 03 0 4117
and
20080831005414 rain1 0 0 4117
I'm studying that range for possible corruptions...
Re:Logging protocol problem...
Posted: Sun Aug 31, 2008 8:43 pm
by skyewright
[quote="skyewright"]skyewright wrote:
I'm studying that range for possible corruptions.../quote]
Nothing obvious...
Ideas welcome (end of month is a very critical time for WD, and is only hours away :( ).
Re:Logging protocol problem...
Posted: Sun Aug 31, 2008 8:48 pm
by admin
you could stop data logging, and make a "cleanup" with "data inspection" page for month 2008-08.
This should purge the bad data, that could be caused by a hard system shutdown... or other reasons.
Re:Logging protocol problem...
Posted: Sun Aug 31, 2008 9:10 pm
by sevenless
On the topic of cleanups, is there any automated way to purge bad data from the raw files?
I recently suffered a faulty USB stick and there are at least a few bad records in my files according to the meteohub logs, but unfortunately the log data doesn't pinpoint the dates/times/sensors that have the problem, and who knows where "line 892374" of the file /200808/raw is in relation to the exact date/time!
Re:Logging protocol problem...
Posted: Sun Aug 31, 2008 10:38 pm
by skyewright
admin wrote:you could stop data logging, and make a "cleanup" with "data inspection" page for month 2008-08.
This should purge the bad data,
Good idea, but as the month is about to change I'll settle for letting that happen - which will presumably make the request problem go away - then tidy up after WD has handled its 'change of month'.
WD is managing okay because if its "on the minute" extra request that does include a date/time - the only thing being missed is rain because to 'solve' an problem that meant that
WD occasionally double counted rain, Brian decided to ignore rain records in the 'on the minute' request. This has resulted in WD not seeing 3mm so far - not a big deal here on Skye! :laugh:
or other reasons.
Meteohub log reports two recent 'illegal data records'
Code: Select all
wmr928eval (29.08.2008 17:47:13): illegal data record in line 833364 while reading from file /data/weather/200808/raw: 20080829164712 wmr928eval (29.08.2008 17:47:14): illegal data record in line 833365 while reading from file /data/weather/200808/raw: rain1 0 0 4096
wmr928eval (30.08.2008 11:47:18): illegal data record in line 855307 while reading from file /data/weather/200808/raw: 20080830wmr928eval (30.08.2008 11:47:19): illegal data record in line 855308 while reading from file /data/weather/200808/raw: 104718 t8 179
Both are noticeably earlier than the '4117' rain1 mentioned earlier (and before installing 2.7a).
Does that give any clue for what I might be looking for?
If I look in the raw file using a text editor that can show all characters (i.e. including line feeds, etc.) I can see no anomalies at or around those positions...
I'll try a clean up tomorrow.
Re:Logging protocol problem...
Posted: Sun Aug 31, 2008 10:45 pm
by skyewright
sevenless wrote:and who knows where "line 892374" of the file /200808/raw is in relation to the exact date/time!
Most text editors allow you to 'Goto' a specific line number, and in the error messages of that type that I've (just!) seen there was also a date & time, and right at the very far right of the error message a snippet from the raw file.
However I've just tried to make use of those hints to caste any light on the problems in my own 'raw' and I couldn't see anything wrong at or around the indicated point, so maybe it is more subtle?
Re:Logging protocol problem...
Posted: Sun Aug 31, 2008 11:27 pm
by skyewright
admin wrote:or other reasons.
Like maybe I should be thinking about a new stick?
Current one has been in use for a few days under 6 months. In that whole time there have been 4 'illegal data records', two in May, one 29th August, one 30th August.
I'll try to keep an eye on the log...
Re:Logging protocol problem...
Posted: Sun Aug 31, 2008 11:49 pm
by admin
You can search for these two entries in the raw file "..../200808/raw" with your favorite text editor:
1) There are two defective lines (833364/5) you should find by searching for "rain1 0 0 4096"
2) There are two defective lines (855307/8) you should find by searching for "104718 t8 179"
Re:Logging protocol problem...
Posted: Mon Sep 01, 2008 10:29 am
by skyewright
admin wrote:You can search for these two entries in the raw file "..../200808/raw" with your favorite text editor:
Thnak you.
I'd already found the lines using a combination of line number and search pattern. What I can't see is anything wrong with them.
(833364/5) you should find by searching for "rain1 0 0 4096"
[pre]20080829164702 t7 152
20080829164704 wind1 248 25 25 0149
20080829164712 rain1 0 0 4096
20080829164714 wind0 227 24 24 0149
20080829164715 t8 156[/pre]
(855307/8) you should find by searching for "104718 t8 179"
[pre]20080830104710 wind1 202 24 20 0163
20080830104716 th5 161 98 0158
20080830104718 t8 179
20080830104721 wind0 221 16 20 0163
20080830104721 th3 174 78 0135[/pre]
That's exactly as the lines are in the file - there are no extra characters in there disturbing the usual patterns.
Before I do the clean up I'll take an extra copy. Then I'll diff the two files afterwards.
PS. On restart of WD this morning the "latest data record" request is again working fine as hoped (i.e. since it is now looking at the new 2008-09 raw). :)
PPS. Sorry for being in a bit of a panic yesterday evening. I get nervous about things that might disturb the relatively stable state I have achieved with WD (and change of month is an especially busy time internally for WD).
PPPS. Since (I imagine) most of us don't look in meteohub log very often, might it be an idea to consider a "Send e-mail Notification" option for when an "illegal data record" occurs?
Re:Logging protocol problem...
Posted: Mon Sep 01, 2008 4:02 pm
by skyewright
skyewright wrote:Meteohub log reports two recent 'illegal data records'
Code: Select all
wmr928eval (29.08.2008 17:47:13): illegal data record in line 833364 while reading from file /data/weather/200808/raw: 20080829164712 wmr928eval (29.08.2008 17:47:14): illegal data record in line 833365 while reading from file /data/weather/200808/raw: rain1 0 0 4096
wmr928eval (30.08.2008 11:47:18): illegal data record in line 855307 while reading from file /data/weather/200808/raw: 20080830wmr928eval (30.08.2008 11:47:19): illegal data record in line 855308 while reading from file /data/weather/200808/raw: 104718 t8 179
<SNIP>
I'll try a clean up tomorrow.
I've done a clean up. Then a 'diff'. The before and after files are identical.
The error messages seem to suggest breaks in those two records - but there is no trace of any such break in the actual files (viewed using a good prorgrammer's text editor that I trust to show me what's really there).
You'd know better than I at what point in processing the error message might have fired. Is there any chance that it was a false alarm?