Redirect weather-data-path to network drive?

[Hathor27]

Hi there,

my Meteohub only runs half a year and its data drive is already 624MB of 2756MB used.
Is there any chance to redirect weather-data-path to an external network device?

Thanks a lot in advance

[admin]

One could mount /data to a nfs share on the LAN by patching /etc/fstab but I would recomment no to do that as it takes reliablity from the solution.

Take a larger card in two years. Lots of time left and and when you discard user generated exports etc from the /data/export folder you should even gain more free space.

[Hathor27]

I think I'm running on a reliable 24/7-raid1-system and network is stable as well.
But I'm not very skilled patching /etc/fstab ...
..would you mind describing this a bit more precise (example), if my external drive is at \\mydrive-ip\data

[Hathor27]

...can anybody help, please?

I tried to mount manually like

Code: Select all

meteohub:/etc# mount -t smbfs -O noatime,_netdev mydatasrvip:/data_meteo /data

and as result I get only

Code: Select all

mount: wrong fs type, bad option, bad superblock on mydatasrvip:/data_meteo,
       missing codepage or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

dmesg | tail says

Code: Select all

meteohub:/etc# dmesg | tail
drivers/usb/serial/pl2303.c: Prolific PL2303 USB to serial adaptor driver
drivers/usb/serial/usb-serial.c: USB Serial support registered for FTDI USB Serial Device
usbcore: registered new interface driver ftdi_sio
drivers/usb/serial/ftdi_sio.c: v1.4.3:USB FTDI Serial Converters Driver
smb_fill_super: missing data argument
NTFS driver 2.1.29 [Flags: R/W MODULE].
smb_fill_super: missing data argument
autofs: called with bogus options
smb_fill_super: missing data argument
smb_fill_super: missing data argument

My data server is a linux server with samba fileshare (Synology DS) and it works perfectly with Windows clients - how do I have to connect the share data_meteo to my Meteohub?

Do I have to install something to get smbfs connected? If yes, what?

[Hathor27]

I found some

Code: Select all

apt-get install smbfs

Is anybody skilled out there to confirm that this is the right way to succeed, please?...

[Hathor27]

OK my friends - I'm gonna get a self-made man...
...and I'm gonna let you know about it:

Code: Select all

apt-get install smbfs

doesn't work on my ALIX, because it claims

Code: Select all

404 Not Found Debian Etch apt-get update

Therefor I found to modify /etc/apt/sources.list, because the packages have moved

Code: Select all

deb http://archive.debian.org/debian-archive/debian etch main
deb http://archive.debian.org/debian-archive/debian-security etch/updates main
deb http://archive.debian.org/debian-archive/debian-volatile etch/volatile main

After that

Code: Select all

apt-get install smbfs

works fine

With

Code: Select all

mount -t smbfs -o username=MyUser,password=WantToKnow,noatime,_netdev //MyDataSRV/data_meteo /data

I'm able to reach my destination samba network drive.

But: As I found now, the whole \data section containing Meteohub application, configs, graph-definitons and so on want to be there. The only thing I wanted to move was the weather-data section, because this part is increasing rapidly.

So I tried again to mount just weather-data as external drive, located at /data/weather. I tried the following:

Code: Select all

mv /data/weather /data/weather.old     # for backup reason
mkdir /data_meteo                      # as new mounting point
mount -t smbfs -o username=meteohub,noatime,_netdev //MyDataSRV/data /data_meteo
ln -s /data_meteo/weather weather      # symlink from old weather-path to mount, subdirectory weather

I agree: It's quite complicated... and is it failsave enough? I'm not shure...

Again: Is anybody out there knowing well this behaviour and is willing to support me? - Thx in advance

[Hathor27]

...as expected: I already encountered some troubles using smbfs

I found to change my mount into cifs

Code: Select all

mount -t cifs -o username=meteohub,password=WantToKnow //MyDataSRV/data /data_meteo

Now it runs fine so far

[Hathor27]

As I recognized so far, it's not really nice to write the actual and all recalculated weatherdata to the external Data-Server: If the server breaks down (for any reason), data logging stops. So I set back my modifications (weatherdata back to Meteohub).

New approach:

• Mount network drive (as described before)
• Move only ancient data to mounted network drive
• Create symlink in original place of weatherdata directing to mounted network drive

My final solution I coded in a bash script, which will be run once every month by cron.

Bash script (free):

Code: Select all

#!/bin/sh
#
# Build History of Weather Data
# - Copy monthly raw-data-folder to Data-Server
# - Verify copy
# - Rename (later delete) orignal folder
# - Create symlink to Data-Server
#
# 2011-12-10/Ha  created under pressure
#

# live-data
path_wdata=/data/weather

# historical data, mounted to Data-Server, symlinked to mountpoint
# > mount -t cifs -o username=meteohub,password=WantToKnow //MyDataSRV/data /data_meteo
# > ln -s /data_meteo/weather /data/weather.history
path_hdata=/data/weather.history

# list of all directories with simple name (without ".ext")
folder_list=$(dir -d1 */ | grep -v "\." | gawk 'BEGIN {FS="/"} {print $1}')

# name of actual data folder
folder_actual=$(date '+%Y%m')

for folder in $folder_list
do
  # if [exists and is a symbolic link] OR [is actual folder]
  if [ -h $path_wdata/$folder ] || [ $folder == $folder_actual ]; then
    echo "Folder "$folder" is not to copy"
  else
    cp -R $path_wdata/$folder $path_hdata/
    # look for differences between original and copy
    verify_cp=$(diff -q $path_wdata/$folder $path_hdata/$folder)
    # if [exists and is a directory] AND [no differences in files]
    if [ -d $path_hdata/$folder ] && [ "$verify_cp" == "" ]; then 
      echo "Folder "$folder" is well copied"
      # rename folder (when approved, change to delete)
      mv $path_wdata/$folder $path_wdata/$folder.obs
      #rm -rf $path_wdata/$folder
      ln -s $path_hdata/$folder $path_wdata/$folder
    else
      echo "Folder "$folder" is corrupted !"
    fi
  fi
done

Suggestions welcome

[admin]

Why don't you do a daily rsync? The rsync-server is already running on the Meteohub,
all you need is to connect with an rsync client from your Linux or windows server and
a data duplicate on your server will be made up in an incremental manner.
This is much better than doing full copies.

Please search for rsync in the Meteohub manual to get details.

[Hathor27]

Why don't you do a daily rsync? The rsync-server is already running on the Meteohub,
all you need is to connect with an rsync client from your Linux or windows server and
a data duplicate on your server will be made up in an incremental manner.
This is much better than doing full copies.

Thank you for your suggestion, admin.
But: Does rsync really do the same thing that I want to do?

I move all ancient weatherdata away from meteohub to save disc-space on its flashcard. To grant access to these data for further recalculation I use symlinks directed to the moved month-folders. This saves diskspace on meteohub for many decades ( ).
Backup is implemented on my Data-Server, at the moment only for the ancient files.

Rsync is good thing, too. I will check it for further issues

[admin]

May be rsync - which is intended mainly as a clever backup procedure - does not fit your plan,
but honestly I don't get what you really intend.

When space is your consideration, buy a 16 GB SD card or USB stick and you can forget about that problem for decades.
Having rsync aditionally in place takes care that you have a daily backup to your server. Your server can create backup sets from
that which allow you to go back to any time in the past. What else do one need?

[Hathor27]

May be rsync - which is intended mainly as a clever backup procedure - does not fit your plan,
but honestly I don't get what you really intend.

Never mind - I feel free with my solution

When space is your consideration, buy a 16 GB SD card or USB stick and you can forget about that problem for decades.

I already own a 4GB card - that's OK for this system. But as I want to log much more data, I was looking for more contingency for future aspects.

Having rsync aditionally in place takes care that you have a daily backup to your server. Your server can create backup sets from that which allow you to go back to any time in the past.

I will have a look at this, too - as soon as I get familiar with rsync - thanks anyway