Read-only /data filesystem problem in 2.6

[mvpel]

Last week, my graphs stopped mid-stream, at about 10:30am Wednesday, for an unknown reason under 2.3, and I've since upgraded to 2.6.

The unit was offline for about 24 hours until I rebooted it following the upgrade, but now when I try to display a graph, even one encompassing only the last 2 minutes, I get:

plot '< /srv/www/cgi-bin/goto.cgi thb0-min5 20080713000000' using 1:8 axis x1y1 title "Indoor Humidity" with lines linecolor rgb "#0000ff", '< /srv/www/cgi-bin/goto.cgi th0-min5 20080713000000' using 1:8 axis x1y1 title "Outdoor Humidity" with lines linecolor rgb "#00ccff", '< /srv/www/cgi-bin/goto.cgi thb0-min5 20080713000000' using 1:($2*9/5)+32 axis x1y2 title "Indoor Temperature" with lines linecolor rgb "#ff0000", '< /srv/www/cgi-bin/goto.cgi th0-min5 20080713000000' using 1:($2*9/5)+32 axis x1y2 title "Outdoor Temperature" with lines linecolor rgb "#00ff00" ... etc ...

... on Internet Explorer.

Did the data logging process get stopped during the upgrade?

I also noticed this at the top of the manage-graphs page when I go to display a graph:

Error file open: /data/graphs/__test.mg: Read-only file system

So perhaps that's why there's no data being stored locally.

Perhaps it failed fsck? I'll try another reboot.

[mvpel]

The reboot via the web page didn't help. I can't seem to log in to the unit via SSH to check on the filesystem mount status of /data.

Any suggestions?

[mvpel]

It looks like the crux of the problem is the /data filesystem is mounted read-only. This may also be the cause of my wind gust on WU being stuck at 8.0mph.

http://www.wunderground.com/weatherstat ... KNHMERRI11

I think if I can get in via SSH, I should be able to fix it, but the web console password, and meteohub/meteohub don't work.

Can someone point me to docs on how to get a shell on the Meteohub?

[mvpel]

Okay, I got the shell - root/meteohub, if anyone was wondering.

I found /data mounted "ro" as expected:

/dev/sda3 on /data type ext3 (ro,noatime,data=ordered) []

However, when I try to switch it over to rw, it reports that the /dev/sda3 device is write-protected:

root@weatherhub:/data$ mount -o remount,rw /data
mount: block device /dev/sda3 is write-protected, mounting read-only
root@weatherhub:/data$

I made no change beyond upgrading from 2.3 to 2.6, so I'm not quite clear on how the device itself could have gotten write-protected. I didn't even touch the device during the upgrade process.

Looking at /var/log/messages shows more interesting stuff:

Jul 19 11:43:32 (none) user.info kernel: sd 0:0:0:0: SCSI error: return code = 0x08000002
Jul 19 11:43:32 (none) user.info kernel: sda: Current: sense key=0x7
Jul 19 11:43:32 (none) user.info kernel: ASC=0x27 ASCQ=0x0
Jul 19 11:43:32 (none) user.warn kernel: end_request: I/O error, dev sda, sector 1968984
Jul 19 11:43:32 (none) user.err kernel: Buffer I/O error on device sda3, logical block 106859
Jul 19 11:43:32 (none) user.warn kernel: lost page write due to I/O error on sda3
Jul 19 11:43:32 (none) user.err kernel: Aborting journal on device sda3.
Jul 19 11:43:40 (none) user.crit kernel: ext3_abort called.
Jul 19 11:43:40 (none) user.crit kernel: EXT3-fs error (device sda3): ext3_journal_start_sb: Detected aborted journal
Jul 19 11:43:40 (none) user.crit kernel: Remounting filesystem read-only
Jul 19 11:43:54 (none) user.info kernel: sd 0:0:0:0: Device not ready: <6>: Current: sense key=0x2
Jul 19 11:43:54 (none) user.info kernel: ASC=0x3a ASCQ=0x0
Jul 19 11:43:54 (none) user.warn kernel: end_request: I/O error, dev sda, sector 1900576
Jul 19 11:43:54 (none) user.err kernel: Buffer I/O error on device sda3, logical block 98308
Jul 19 11:43:54 (none) user.warn kernel: lost page write due to I/O error on sda3

I'm going to try shutting down enough stuff to unmount /data and run an FSCK on it. The problem is I don't think I can back up my historical data since the backup operation wants to write to /data.

[mvpel]

Can't seem to get /data unmounted, even though I've stopped pretty much everything on the unit that might possibly be making it busy.

Going to try commenting it out of /etc/fstab and rebooting... okay, that worked, I'm running badblocks on the filesystem now.

[mvpel]

Okay, the fsck -c finished, and there were a good number of filesystem errors corrected:

Checking for bad blocks (read-only test): done 936
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Entry 'weather' in / (2) has deleted/unused inode 31297. Clear<y>? yes

Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Inode 2 ref count is 10, should be 9. Fix<y>? yes

Pass 5: Checking group summary information
Block bitmap differences: -(66027--66432) -(66434--66596) -(67584--67600) -69632 -(69634--69708) -(69710--69923) -(71681--73241) -(73243--79662) -(79723--79735) -(79872--80239) -(80242--80756) -(80759--83061) -(83063--83085) -(83087--83231) -(83971--83972) -(86016--86408) -(86410--86722) -(86724--87698) -(88064--88065) -92160 -(94208--94214) -(102400--106859)
Fix<y>? yes

Free blocks count wrong for group #0 (23582, counted=23583).
Fix<y>? yes

Free blocks count wrong for group #2 (18544, counted=32277).
Fix<y>? yes

Free blocks count wrong for group #3 (27815, counted=32275).
Fix<y>? yes

Free blocks count wrong (311403, counted=329597).
Fix<y>? yes

Inode bitmap differences: -(31297--31351) -(46945--46948)
Fix<y>? yes

Free inodes count wrong for group #2 (15593, counted=15648).
Fix<y>? yes

Directories count wrong for group #2 (4, counted=0).
Fix<y>? yes

Free inodes count wrong for group #3 (15644, counted=15648).
Fix<y>? yes

Directories count wrong for group #3 (2, counted=0).
Fix<y>? yes

Free inodes count wrong (172007, counted=172066).
Fix<y>? yes


/dev/sda3: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sda3: 62/172128 files (0.0% non-contiguous), 14339/343936 blocks

After that, I was able to mount the /data filesystem rw, but it remains to be seen whether the data files will be corrupted at all. I'm running an fsck -cc (write/read bad-block check) now, and everything should come back up upon reboot.

... so far:

Checking for bad blocks (non-destructive read-write test)
Testing with random pattern: badblocks: Input/output error during test data write, block 45760
...

I'm rather surprised that a USB memory stick would develop bad-blocks given that there's no moving parts, but then what do I know...

[mvpel]

Okay, after a rather odd wedge perhaps caused by trying to log in while badblocks was running, the system is back up and running with a writeable /data and looks okay now.

[peteroi]

My feeling is that perhaps a lot of file system errors or data errors which are reported in the last posts are caused by UNRELIABLE USB sticks. You know there was a big discussion in c't about this subject, especially with stick sizes bigger 1 or 2 Gb.
NEVER TRUST a stick before you checked it several times with the hwtool in download section.
I already had sticks showing errors on different adresses at each test run! Sometimes only one error, sometimes more (always the same stick!)
Only after we are 100% sure that the stick is OK we should blame a problem to the meteohub software...
BR
Peter

[admin]

you are both right, and respect to "mvpel" your linux knowledge is impressive!

I would run the following test to check if this was just a hick-up of the system or if your stick has a problem. "cp /dev/zero /data/zero" this will ru some time and will flood your /dev/sda3 partition until completely filled. When operation stops with an error, that there is no more room on the device, then delete the /data/zero file and your /dev/sda3 partition looks ok.

Do the same this the /dev/sda1 partition: "cp /dev/zero /zero". again wait what is happening and delete the file afterwards.

If your file system swaps to read-only mode again (and reports read/write errors in the /data/log/messages file), then your stick has gone and cries for a replacement.

[mvpel]

I'm using the KingMax stick that was supplied with the unit by Ambient Weather. They're tiny, but also pretty cheap.

I'll try the /dev/zero suggestion, thanks - I think I'm in good shape with the /data filesystem now since I ran an fsck -cc (read/write bad-block scan), so what it found is marked unusable. The /dev/zero idea is a good way to check for bad blocks on active filesystems.

However, I'm not sure what I'd be able to do if the root filesystem were switched to read-only, I'd probably need to build a Linux CD-only image for my laptop and scan the ext3fs filesystem on the stick using that.

Turns out that one of the corrupted inodes was the one for the /data/weather directory, so it appears that I've lost about five and a half months of locally-stored weather and indoor climate data. (Everything is still up on WU at KNHMERRI11.)

I looked around for my firmware-upgrade backup of the data, since I figured a two-month gap in the data is better than losing it all, but it looks like I deleted it or left it on my work machine that I shipped back to California for recycling a month ago.

Oh well.

Is there a particularly good brand of stick that you recommend when I decide to upgrade to 4GB? Or maybe I'll wait for the new hardware platform...

[admin]

Did you try to sync your weather data with rsync to your server? Meteohub has a rsync daemon setup for this and if you have a linux machine running somewhere this is a snap to setup from the requesting side. It also runs with Windows in the cygwin environment. The manual gives some details on this in appendix F.

[mvpel]

Nope, didn't realize this was available, or necessary - never thought of the possibility of a problem with the memory stick, so I never did an off-system backup except for the firmware upgrade. I'll definitely look into it now, though.