meteohub.de

Last week, my graphs stopped mid-stream, at about 10:30am Wednesday, for an unknown reason under 2.3, and I've since upgraded to 2.6.

The unit was offline for about 24 hours until I rebooted it following the upgrade, but now when I try to display a graph, even one encompassing only the last 2 minutes, I get:

plot '< /srv/www/cgi-bin/goto.cgi thb0-min5 20080713000000' using 1:8 axis x1y1 title "Indoor Humidity" with lines linecolor rgb "#0000ff", '< /srv/www/cgi-bin/goto.cgi th0-min5 20080713000000' using 1:8 axis x1y1 title "Outdoor Humidity" with lines linecolor rgb "#00ccff", '< /srv/www/cgi-bin/goto.cgi thb0-min5 20080713000000' using 1:($2*9/5)+32 axis x1y2 title "Indoor Temperature" with lines linecolor rgb "#ff0000", '< /srv/www/cgi-bin/goto.cgi th0-min5 20080713000000' using 1:($2*9/5)+32 axis x1y2 title "Outdoor Temperature" with lines linecolor rgb "#00ff00" ... etc ...

... on Internet Explorer.

Did the data logging process get stopped during the upgrade?

I also noticed this at the top of the manage-graphs page when I go to display a graph:

Error file open: /data/graphs/__test.mg: Read-only file system

So perhaps that's why there's no data being stored locally.

Perhaps it failed fsck? I'll try another reboot.

The reboot via the web page didn't help. I can't seem to log in to the unit via SSH to check on the filesystem mount status of /data.

Any suggestions?

It looks like the crux of the problem is the /data filesystem is mounted read-only. This may also be the cause of my wind gust on WU being stuck at 8.0mph.

http://www.wunderground.com/weatherstat ... KNHMERRI11

I think if I can get in via SSH, I should be able to fix it, but the web console password, and meteohub/meteohub don't work.

Can someone point me to docs on how to get a shell on the Meteohub?

Okay, I got the shell - root/meteohub, if anyone was wondering.

I found /data mounted "ro" as expected:

/dev/sda3 on /data type ext3 (ro,noatime,data=ordered) []

However, when I try to switch it over to rw, it reports that the /dev/sda3 device is write-protected:

root@weatherhub:/data$ mount -o remount,rw /data
mount: block device /dev/sda3 is write-protected, mounting read-only
root@weatherhub:/data$

I made no change beyond upgrading from 2.3 to 2.6, so I'm not quite clear on how the device itself could have gotten write-protected. I didn't even touch the device during the upgrade process.

Looking at /var/log/messages shows more interesting stuff:

Jul 19 11:43:32 (none) user.info kernel: sd 0:0:0:0: SCSI error: return code = 0x08000002
Jul 19 11:43:32 (none) user.info kernel: sda: Current: sense key=0x7
Jul 19 11:43:32 (none) user.info kernel: ASC=0x27 ASCQ=0x0
Jul 19 11:43:32 (none) user.warn kernel: end_request: I/O error, dev sda, sector 1968984
Jul 19 11:43:32 (none) user.err kernel: Buffer I/O error on device sda3, logical block 106859
Jul 19 11:43:32 (none) user.warn kernel: lost page write due to I/O error on sda3
Jul 19 11:43:32 (none) user.err kernel: Aborting journal on device sda3.
Jul 19 11:43:40 (none) user.crit kernel: ext3_abort called.
Jul 19 11:43:40 (none) user.crit kernel: EXT3-fs error (device sda3): ext3_journal_start_sb: Detected aborted journal
Jul 19 11:43:40 (none) user.crit kernel: Remounting filesystem read-only
Jul 19 11:43:54 (none) user.info kernel: sd 0:0:0:0: Device not ready: <6>: Current: sense key=0x2
Jul 19 11:43:54 (none) user.info kernel: ASC=0x3a ASCQ=0x0
Jul 19 11:43:54 (none) user.warn kernel: end_request: I/O error, dev sda, sector 1900576
Jul 19 11:43:54 (none) user.err kernel: Buffer I/O error on device sda3, logical block 98308
Jul 19 11:43:54 (none) user.warn kernel: lost page write due to I/O error on sda3

I'm going to try shutting down enough stuff to unmount /data and run an FSCK on it. The problem is I don't think I can back up my historical data since the backup operation wants to write to /data.

Can't seem to get /data unmounted, even though I've stopped pretty much everything on the unit that might possibly be making it busy.

Going to try commenting it out of /etc/fstab and rebooting... okay, that worked, I'm running badblocks on the filesystem now.

Okay, the fsck -c finished, and there were a good number of filesystem errors corrected:

Checking for bad blocks (read-only test): done 936
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Entry 'weather' in / (2) has deleted/unused inode 31297. Clear<y>? yes

Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Inode 2 ref count is 10, should be 9. Fix<y>? yes

Pass 5: Checking group summary information
Block bitmap differences: -(66027--66432) -(66434--66596) -(67584--67600) -69632 -(69634--69708) -(69710--69923) -(71681--73241) -(73243--79662) -(79723--79735) -(79872--80239) -(80242--80756) -(80759--83061) -(83063--83085) -(83087--83231) -(83971--83972) -(86016--86408) -(86410--86722) -(86724--87698) -(88064--88065) -92160 -(94208--94214) -(102400--106859)
Fix<y>? yes

Free blocks count wrong for group #0 (23582, counted=23583).
Fix<y>? yes

Free blocks count wrong for group #2 (18544, counted=32277).
Fix<y>? yes

Free blocks count wrong for group #3 (27815, counted=32275).
Fix<y>? yes

Free blocks count wrong (311403, counted=329597).
Fix<y>? yes

Inode bitmap differences: -(31297--31351) -(46945--46948)
Fix<y>? yes

Free inodes count wrong for group #2 (15593, counted=15648).
Fix<y>? yes

Directories count wrong for group #2 (4, counted=0).
Fix<y>? yes

Free inodes count wrong for group #3 (15644, counted=15648).
Fix<y>? yes

Directories count wrong for group #3 (2, counted=0).
Fix<y>? yes

Free inodes count wrong (172007, counted=172066).
Fix<y>? yes


/dev/sda3: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sda3: 62/172128 files (0.0% non-contiguous), 14339/343936 blocks

After that, I was able to mount the /data filesystem rw, but it remains to be seen whether the data files will be corrupted at all. I'm running an fsck -cc (write/read bad-block check) now, and everything should come back up upon reboot.

... so far:

Checking for bad blocks (non-destructive read-write test)
Testing with random pattern: badblocks: Input/output error during test data write, block 45760
...

I'm rather surprised that a USB memory stick would develop bad-blocks given that there's no moving parts, but then what do I know...

Okay, after a rather odd wedge perhaps caused by trying to log in while badblocks was running, the system is back up and running with a writeable /data and looks okay now.

My feeling is that perhaps a lot of file system errors or data errors which are reported in the last posts are caused by UNRELIABLE USB sticks. You know there was a big discussion in c't about this subject, especially with stick sizes bigger 1 or 2 Gb.
NEVER TRUST a stick before you checked it several times with the hwtool in download section.
I already had sticks showing errors on different adresses at each test run! Sometimes only one error, sometimes more (always the same stick!)
Only after we are 100% sure that the stick is OK we should blame a problem to the meteohub software...
BR
Peter

you are both right, and respect to "mvpel" your linux knowledge is impressive!

I would run the following test to check if this was just a hick-up of the system or if your stick has a problem. "cp /dev/zero /data/zero" this will ru some time and will flood your /dev/sda3 partition until completely filled. When operation stops with an error, that there is no more room on the device, then delete the /data/zero file and your /dev/sda3 partition looks ok.

Do the same this the /dev/sda1 partition: "cp /dev/zero /zero". again wait what is happening and delete the file afterwards.

If your file system swaps to read-only mode again (and reports read/write errors in the /data/log/messages file), then your stick has gone and cries for a replacement.

I'm using the KingMax stick that was supplied with the unit by Ambient Weather. They're tiny, but also pretty cheap.

I'll try the /dev/zero suggestion, thanks - I think I'm in good shape with the /data filesystem now since I ran an fsck -cc (read/write bad-block scan), so what it found is marked unusable. The /dev/zero idea is a good way to check for bad blocks on active filesystems.

However, I'm not sure what I'd be able to do if the root filesystem were switched to read-only, I'd probably need to build a Linux CD-only image for my laptop and scan the ext3fs filesystem on the stick using that.

Turns out that one of the corrupted inodes was the one for the /data/weather directory, so it appears that I've lost about five and a half months of locally-stored weather and indoor climate data. (Everything is still up on WU at KNHMERRI11.)

I looked around for my firmware-upgrade backup of the data, since I figured a two-month gap in the data is better than losing it all, but it looks like I deleted it or left it on my work machine that I shipped back to California for recycling a month ago.

Oh well.

Is there a particularly good brand of stick that you recommend when I decide to upgrade to 4GB? Or maybe I'll wait for the new hardware platform...

Did you try to sync your weather data with rsync to your server? Meteohub has a rsync daemon setup for this and if you have a linux machine running somewhere this is a snap to setup from the requesting side. It also runs with Windows in the cygwin environment. The manual gives some details on this in appendix F.

Nope, didn't realize this was available, or necessary - never thought of the possibility of a problem with the memory stick, so I never did an off-system backup except for the firmware upgrade. I'll definitely look into it now, though.