Page 1 of 2
data corruption in raw file
Posted: Sat Jul 25, 2015 6:46 pm
by goombah
I've been having some problems, which I think are related to the SDHC card I was using. System has been running fine for months, then a few weeks ago I lost power. System wouldn't reboot off the SDHC card. Plugged in a backup and it came up fine, so its not the Pi. FSCK attempts against the SDHC card didn't help - still unbootable. Ended up reformatting it and loading the latest Meteohub software after making a backup of the /data partition. After reconfiguration, all looked clean.
Came back last night and notice that WUnderground was reporting the instantaneous data correctly, but was not reporting accumulated rainfall. Poked around a bit and discovered a lot of boot errors. Somewhere along the line, the /dev/ttyUSB0 device ceased to exist as well. Reboots wouldn't find it. Again plugged in one of my backup cards, and all was fine. Decided to just run off it for awhile and see if it runs clean.
Unsure if my Pi is having issues, or the SDHC card is.
Also checked the meteohub.log file and noticed a few dozen corrupt data records in the raw file. Manually fixed most of those, removed the undecipherable one. (It was stuff like "2015" being "2u15" - looked like single bit errors. They were batched around certain times, and not consistent, so am not sure what to make of that.)
Any words of wisdom?
FYI: Changing the SDHC card apparently changed my system ID. Took a few to figure out how to change my registration code.
Re: data corruption in raw file
Posted: Sun Jul 26, 2015 9:32 am
by admin
typical signs of a worn out card. You can cleanup the raw data with the corresponding function
on "inspect data" page. It removes ill formatted rows from raw data.
Re: data corruption in raw file
Posted: Sun Jul 26, 2015 6:17 pm
by goombah
Thanks for the inspect data command. That beats the heck out of me going into vi and manually cleaning it up. So far, of the dozens of lines of bad data I'm finding, its always single character issues, and always clumped together around one time period.
I am surprised, the 16gb card I bought was only used for Meteohub, was a high end, high speed one, and put in service in May.
Re: data corruption in raw file
Posted: Sat Aug 15, 2015 6:24 pm
by Manhart
I observed the same problem since several weeks with my Raspberry 2. Every 2-3 weeks i found such corrupt data records in a raw file (meteohub.log). The month of the raw file was random.
I made nearly every day a complete backup of the public folder to reduce the loss of data. If i found such entries in the log, i replaced this raw file with the last good copy of the same file and made a restart of the system.
After power outage the Raspberry 2 failed to boot from the SDCARD nearly every time. I had to build the SDCARD new with the image and to restore the data from the last backup. I use a SDCARD from Kingston (32GB) and tests always went without a failure.
Today i changed my Meteohub system. I use now an USB memory stick for the data partition (public folder). I hope, that the problem with these corrupt data records is now solved. I made a short description about the change of the system:
http://www.beathis.ch/cgi/Wordpress/?p=1790 (only in german).
Regards
Matthias
Re: data corruption in raw file
Posted: Sat Aug 15, 2015 6:27 pm
by goombah
Please let us know how that works. I love the idea of moving the /data partition to a USB drive - although I have burned a few of those up over the years as well (mostly when using them as linux boot drives).
Re: data corruption in raw file
Posted: Sat Aug 15, 2015 6:36 pm
by Manhart
I used Google Translate to translate my description:
https://translate.google.com/translate? ... edit-text=
I hope the translation is usable.
Regards
Matthias
Re: data corruption in raw file
Posted: Fri Sep 18, 2015 1:40 pm
by Nimbus43
Mathias,
danke für deinen Bericht.
Ich habe Zugriff auf den Raspberry, aber wenn ich fdisk -1 eingebe, sagt er mir "invalid option".
4GB USB Stick eingesteckt, sogar mit ext4 formatiert, Meteohub läuft ...
Der USB Stick wird also nicht erkannt.
Für einen Hinweis wäre ich dir sehr dankbar.
Gruß,
Mario
Re: data corruption in raw file
Posted: Fri Sep 18, 2015 3:07 pm
by Manhart
Hallo Mario,
der Befehl lautet "fdisk -l" (kleines L).Dort muss /dev/sda auftauchen, sonst ist entweder die USB-Schnittstelle tot oder der Stick.
Gruss
Matthias
Re: data corruption in raw file
Posted: Fri Sep 18, 2015 7:18 pm
by Nimbus43
Mathias,
danke, mit -l ging es ...
Nach Neustart des Raspberry habe ich die auf meiner Festplatte gespeicherten Daten auf Meteohub-rpi zurückkopiert. Aber es scheint ein Problem mit der Anbindung der Wetterstation zu geben. Wenn ich in Meteohub auf Sensoren gehe, erscheint die Fehlermeldung: "error connect socket 127.0.0.1:5555 :Connection refused
Warning: Data logging has been halted."
Auch kann ich unter "Inspect Data" (verwende Englisch) keine alten Daten sehen.
Der USB Stick enthält Daten. In System Info, Data, werden 55 MB von 3698 MB angezeigt.
Kannst du mir helfen?
Re: data corruption in raw file
Posted: Fri Sep 18, 2015 7:28 pm
by Manhart
Hallo Mario,
ich habe es genau nach meiner Anleitung gemacht. Nach Schritt 11 (Daten zurückkopieren) folgt Schritt 12 (Datensicherung wiederherstellen). Schritt 12 ist wichtig. Versuche das mal.
Gruss
Matthias
Re: data corruption in raw file
Posted: Fri Sep 18, 2015 7:36 pm
by Nimbus43
Matthias,
Datensicherung wieder herstellen habe ich gemacht, Meteohub Maintenance zeigt mir auch an, dass er 22,6 MB mit Datum von heute importiert hat ...
Ich habe ausserdem auch keine Log Einträge ... grübel, grübel
Re: data corruption in raw file
Posted: Fri Sep 18, 2015 7:43 pm
by Manhart
Im letzten Schritt (nicht im Blog notiert) habe ich die Firmware auf 5.0j aktualisiert. Vielleicht hilft das erneute Installieren dieser Firmware.
Gruss
Matthias
Re: data corruption in raw file
Posted: Fri Sep 18, 2015 7:48 pm
by Nimbus43
5.0j hatte ich schon ... werde es aber noch einmal installieren.
Melde mich morgen wieder, danke jedenfalls für deine Hilfe ...
Re: data corruption in raw file
Posted: Mon Sep 21, 2015 7:26 am
by Nimbus43
Hallo Mathias,
ich habe den Rasp mit Meteohub neu aufgesetzt und den USB-Stick nach deiner Anleitung implementiert. Funktioniert wunderbar, danke!
Jetzt bleibt noch abzuwarten, wie sich diese Konfiguration im Langzeitbetrieb verhält ...
Beste Grüße,
Mario
P. S.: Meteohub Website incl. Forum war gestern nicht zu erreichen, so dass ich erst heute auf 5.0j updaten kann.
Re: data corruption in raw file
Posted: Mon Sep 21, 2015 5:10 pm
by Manhart
Hallo Mario,
Danke für Deine Info. Gut, dass es nun geklappt hat. Seit der Umstellung meines Systems auf den externen USB-Stick, konnte ich keinerlei Fehler mehr in den Rohdaten im Log entdecken. Letzte Woche hatte ich einen kurzen Stromausfall und das führte prompt wieder zu einem Problem auf der SD-Karte. Die Daten auf dem USB-Stick aber blieben unbeschädigt.
Gruss
Matthias