meteohub.de • Removing data

Page 1 of 2

Removing data

Posted: Sun Nov 21, 2010 3:03 pm

by HeinrichH

One of my Meteohub system the data file is all most running out of space, system info tells me that 2712 MB of the 2903 MB is used and each month is about 95 MB of data so I have about two months to go.
Can I just move the raw data files of each month of 2009 to another location as backup so the data file has more space available?
Or do I have to remove the data with the data inspect option? (just move the data is much easier)

Re: Removing data

Posted: Sun Nov 21, 2010 7:13 pm

by YJB

My experience is that you can just remove those files. Best to stop logging before doing so, and afterwards force a full recomputation of the data.

You can also think about a bigger memorycard. I'm running an Alix board with 8Gb:

Code: Select all

meteohub:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda1             756M  512M  207M  72% /
tmpfs                 122M     0  122M   0% /lib/init/rw
udev                   10M   40K   10M   1% /dev
tmpfs                 122M     0  122M   0% /dev/shm
/dev/hda3             6.1G  1.1G  4.8G  19% /data

Re: Removing data

Posted: Sun Nov 21, 2010 8:46 pm

by HeinrichH

My system is running under VMware, is there a method to expand the size of the image and the size of the datafile? Disk space enough!

Re: Removing data

Posted: Sun Nov 21, 2010 11:11 pm

by YJB

Yes, it should be possible to expand the size of the data filesystem; I've done it myself on my Alix.

The embarrassing thing is that:
- I did make notes, since I was sure that somebody would ask for this one of these days
- I misplaced the notes in a miserable way, and can't find them.

The key is that you will need physical diskspace directly beyond the current data filesystem, or in your case, you might have just a bigger spot on your virtual VM disk. In that case it would be just a matter of creating a new filesystem, move all the data over and mount the new filesystem at the /data mountpoint.

I will try to look a bit further and see if I can retriev my notes somewhere. In that case I will get back....

Re: Removing data

Posted: Sun Nov 21, 2010 11:40 pm

by HeinrichH

I hope you can find your notes, please let me know if so.
I found how to expand the pre allocated disk size for VMware, it's 10GB now but the message I've got was that I have to repartition the guest operating system.
I hope your notes can tell me how to do that

Re: Removing data

Posted: Tue Nov 23, 2010 9:54 pm

by HeinrichH

Any body a idea how to expand the data file of the image under VMware??

Re: Removing data

Posted: Tue Nov 23, 2010 10:46 pm

by YJB

Ok, I've miserably misplaced my notes with all the screen output, so I guess we have to do this in an interactive way. (I've got a shell history, so that helps a bit.

Let's 1st gather some information:

2 notes upfront:
- At the moment we are just gathering information so no danger, later on you might loose data, so a backup is recommended. For VMware that will be easy, just make a copy of all the vmdk files and other that make up your environment.
- If one has no clue about Linux don't even start trying to do this, I won't be responsible anyway.

Get info about the filesystems:
# df
# df -h

we're going to expand the /data filesystem, so make a note of the /dev/hda device

# fdisk -s /dev/hda????

The size of the slice reported by fdisk should be the same as the size reported by the df command (hopefully)

Last step gathering information, print out the current partition table:
# fdisk -l

Code: Select all

Disk /dev/hda: 7633 MB, 7633575936 bytes
255 heads, 63 sectors/track, 928 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hda1               1          98      787153+  83  Linux
/dev/hda2              99         123      200812+  82  Linux swap / Solaris
/dev/hda3             124         928     6466162+  83  Linux

The above is my current table (already expanded) if you look closely you see:
255 heads, 63 sectors/track, 928 cylinders -> 928 cylinders is the total size

In this case all of my current filesystems are using the entire disk (1-98, 99-123, 124-928).

I expect that in your case there will be either a spare partition or some unallocated space directly beyond the partition that holds the /data filesystem.

Can you share some output, and then we continue from there.

Re: Removing data

Posted: Tue Nov 23, 2010 11:53 pm

by HeinrichH

Thanks for the info but it looks like if the fdisk command is not implemented in VMware image.
All the fdisk commands are replied with -bash fdisk: command not found

Re: Removing data

Posted: Tue Nov 23, 2010 11:57 pm

by YJB

Are you logged in as root?

Normally the root prompt is #, while a regular user is identified with a $ prompt.

type "id" to find out

Re: Removing data

Posted: Wed Nov 24, 2010 12:12 am

by HeinrichH

OK, logged in as root, this is what I get

Re: Removing data

Posted: Wed Nov 24, 2010 9:37 am

by YJB

Ok, looks much better.

In the mean time I've decided to get a VMware machine up and running and document the process (hopefully tonight), so if you have some patience you will get the entire procedure, otherwise pm.

Re: Removing data

Posted: Wed Nov 24, 2010 10:35 am

by YJB

0) Make sure that you first read the entire procedure before even starting
I've tried to make this as descriptive as possible, but your specific setup might be different
You will need to be able to use some basic *NIX commands
Don't blame me if something goes wrong, but do report back to me if you feel that something is missing
Make sure that you verify the device id's and partition table entries, your setup might vary slightly, which means that the commands will be slightly different as well

00 ) Physical setup
--- For VMWare, simply grow the virtual disk in the vmware configuration:

: VMware-Resize1.jpg (67.33 KiB) Viewed 12486 times

: VMware-Resize2.jpg (68.97 KiB) Viewed 12486 times

-- For other systems, just get a bigger memory card/stick, configure meteohub on the smaller stick, and then grow

Start of the procedure
1) Disable meteohub auto startup

Code: Select all

mv /etc/rc2.d/S20loggerd /etc/rc2.d/s20loggerd
mv /etc/rc2.d/S20meteonetd /etc/rc2.d/s20meteonetd
mv /etc/rc2.d/S20thttpd /etc/rc2.d/s20thttpd
mv /etc/rc2.d/S89cron /etc/rc2.d/s89cron
mv /etc/rc2.d/S10sysklogd /etc/rc2.d/s10sysklogd
mv /etc/rc2.d/S23ntp /etc/rc2.d/s23ntp
mv /etc/rc2.d/S20samba /etc/rc2.d/s20samba

2) reboot meteohub

Code: Select all

init 6

3) Check if /data is not used anymore:

Code: Select all

cd /
fuser -c /data

result -> no process id listed

4) Verify where /data is located:

Code: Select all

mount |grep data
/dev/hda3 on /data type ext3 (rw,noatime)

5) Assuming there are no process id returned, you can unmount /data

Code: Select all

umount /data

6)Verify data is unmounted

Code: Select all

mount |grep data

result -> /data is NOT displayed

7) Check the current partition table:

Code: Select all

meteohub:/# fdisk -l

Disk /dev/hda: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hda1               1         122      979933+  83  Linux
/dev/hda2             123         146      192780   82  Linux swap / Solaris
/dev/hda3             147         522     3020220   83  Linux

As you can see the total disksize is 1044 cylinders (approx 8GB) and we are only using 1 - 522
Also, in step 4 we've learned that /data is using /dev/hda3
Since /dev/hda3 ends on cylinder 522 we can grow it to cylinder 1044

8 ) Let's do a sanity check on the /data filesystem

Code: Select all

meteohub:/# fsck -n /dev/hda3
fsck 1.40-WIP (14-Nov-2006)
e2fsck 1.40-WIP (14-Nov-2006)
data: clean, 40/377856 files, 30066/755055 blocks

9) As far as I know we can't grow an ext3 filesystem, so we need to convert to ext2 (and later convert back)
This command effectively removes the journal and makes it a ext2 fileystem

Code: Select all

meteohub:/# tune2fs -O ^has_journal /dev/hda3
tune2fs 1.40-WIP (14-Nov-2006)

10) Make sure you have a backup, until now nothing has really changed - point of no return

11) 1st we are going to change the partition table and grow the partition that /data lives in to the max
11a) start fdisk in interactive mode

Code: Select all

meteohub:/# fdisk /dev/hda

The number of cylinders for this disk is set to 1044.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

11b) Print the current partition table

Code: Select all

Command (m for help): p

Disk /dev/hda: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hda1               1         122      979933+  83  Linux
/dev/hda2             123         146      192780   82  Linux swap / Solaris
/dev/hda3             147         522     3020220   83  Linux

11c) Delete the /dev/hda3 (see step 4) partition entry (this does NOT delete your data)

Code: Select all

Command (m for help): d
Partition number (1-4): 3

11d) Verify that it is gone

Code: Select all

Command (m for help): p

Disk /dev/hda: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hda1               1         122      979933+  83  Linux
/dev/hda2             123         146      192780   82  Linux swap / Solaris

11e) Now add the parttion again with "n"

Code: Select all

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)

11f) This is a primary partition, select "p"

Code: Select all

11g) And we were using 3, so let's choose that again "3'

Code: Select all

Partition number (1-4): 3

11h) Accept the defaults, fdisk will automatically choose the 1st available Cylinder and grow to the last

Code: Select all

First cylinder (147-1044, default 147):
Using default value 147
Last cylinder or +size or +sizeM or +sizeK (147-1044, default 1044):
Using default value 1044

11i) And verify our work

Code: Select all

Command (m for help): p

Disk /dev/hda: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hda1               1         122      979933+  83  Linux
/dev/hda2             123         146      192780   82  Linux swap / Solaris
/dev/hda3             147        1044     7213185   83  Linux

Command (m for help):

11j) All looks good, and our new partition is now 7213185 blocks (~7GB), so let's write the partition table "w"

Code: Select all

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.

11k) The kernel is still using the old partition table, this is normal, you can't just change stuff on the fly

12) Make sure that /data is not mounted during the reboot:
I use vi, but you can use another editor if you want:

Code: Select all

vi /etc/fstab

and change:
/dev/hda3 /data ext3 noatime 0 2
into
# /dev/hda3 /data ext3 noatime 0 2

13) Reboot Meteohub and get the new partition table activated

Code: Select all

meteohub:~# init 6

14) Check that /data is NOT mounted

Code: Select all

meteohub:~# mount |grep data

Result -> /data is NOT listed

15) Verify our new partition table

Code: Select all

meteohub:~# fdisk -l /dev/hda

Disk /dev/hda: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hda1               1         122      979933+  83  Linux
/dev/hda2             123         146      192780   82  Linux swap / Solaris
/dev/hda3             147        1044     7213185   83  Linux

16) Conduct a forced filesystem check on our old (small) /data filesystem before we grow it

Code: Select all

meteohub:~# e2fsck -f /dev/hda3
e2fsck 1.40-WIP (14-Nov-2006)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
data: 40/377856 files (2.5% non-contiguous), 13664/755055 blocks

17) Now we are ready to resize, by default linux will use the entire parttion, which will make our command very straightforward:

Code: Select all

meteohub:~# resize2fs /dev/hda3
resize2fs 1.40-WIP (14-Nov-2006)
Resizing the filesystem on /dev/hda3 to 1803296 (4k) blocks.
The filesystem on /dev/hda3 is now 1803296 blocks long.

Don't panic on the size reported, yes it is 4 time smaller when compared with the fdisk output, but remember that fdisk reports 1k and a filesystem is using 4k blocks

18) Let's do a sanity check on our resized filesystem

Code: Select all

meteohub:~# fsck -n /dev/hda3
fsck 1.40-WIP (14-Nov-2006)
e2fsck 1.40-WIP (14-Nov-2006)
data: clean, 40/881664 files, 30030/1803296 blocks

19) And convert it in a ext3 filesystem by adding the journal

Code: Select all

meteohub:~# tune2fs -j /dev/hda3
tune2fs 1.40-WIP (14-Nov-2006)
Creating journal inode: done
This filesystem will be automatically checked every 36 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

20) Check if we can mount the filesystem

Code: Select all

meteohub:~# mount /dev/hda3 /data

21) And check the new size

Code: Select all

meteohub:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda1             942M  509M  386M  57% /
tmpfs                 126M     0  126M   0% /lib/init/rw
udev                   10M   52K   10M   1% /dev
tmpfs                 126M     0  126M   0% /dev/shm
tmpfs                 8.0M     0  8.0M   0% /var/lock
tmpfs                  16M   36K   16M   1% /var/run
tmpfs                  16M   16K   16M   1% /var/log
/dev/hda3             6.8G  138M  6.4G   3% /data

22) Since everything looks well, let's reverse our action step 12 and enable mounting of /data during bootup
I use vi, but you can use another editor if you want:

Code: Select all

vi /etc/fstab

and change:
# /dev/hda3 /data ext3 noatime 0 2
into
/dev/hda3 /data ext3 noatime 0 2

23) Also re-enable the automatic startup of all system/meteohub processes

Code: Select all

mv /etc/rc2.d/s20loggerd /etc/rc2.d/S20loggerd
mv /etc/rc2.d/s20meteonetd /etc/rc2.d/S20meteonetd
mv /etc/rc2.d/s20thttpd /etc/rc2.d/S20thttpd
mv /etc/rc2.d/s89cron /etc/rc2.d/S89cron
mv /etc/rc2.d/s10sysklogd /etc/rc2.d/S10sysklogd
mv /etc/rc2.d/s23ntp /etc/rc2.d/S23ntp
mv /etc/rc2.d/s20samba /etc/rc2.d/S20samba

24) reboot meteohub

Code: Select all

meteohub:~# init 6

25) Verify the GUI after the reboot

Re: Removing data

Posted: Wed Nov 24, 2010 1:23 pm

by HeinrichH

Thats wat I call a work around!! Beuatifull, thanks, I have something to do this evening!

Starting some time earlier but stuck after the first command...

Re: Removing data

Posted: Thu Nov 25, 2010 7:17 am

by YJB

Looks a typo:
S20loggerd is the file you are going to change drop the '1" in the middle.

Probably best to use cut 'n paste as much as possible to prevent typo's

Re: Removing data

Posted: Thu Nov 25, 2010 9:25 am

by HeinrichH

You're right, I see the mistake ...
I can go on with this exercise over 2 weeks because I have to go abroad for 2 weeks for work, hopeful I can find a good internet connection so I can use VNC to do something else I have to wait until I'm back home.