Evaluation version - WDL upload frequency? - [Solved]

[gm4jjj]

I am currently evaluating meteohub on my Asus Eee PC 701 in a VM. I particularly using it to upload via FTP the clientraw.txt and other WDL files for use with iWDL an iPhone Web Application.

I am not sure, but it appears that the interval between updates appears to get very slow after a few minutes, is this because I using the evaluation version or is there some other reason for this?

As far as I can tell uploads to Wunderground appear to be uploaded consistently.

If I can confirm what is going on and can be convinced that it is going to work then I will purchase a licence.

Many thanks

[admin]

WD live updates (of most recent data in "clientraw.txt") can be scheduled to be done every minute. This should be a constant update interval. I never got notice that this slows down over time.

[gm4jjj]

WD live updates (of most recent data in "clientraw.txt") can be scheduled to be done every minute. This should be a constant update interval. I never got notice that this slows down over time.

OK thanks - I am not sure what is going wrong, the local copy is updating OK, but the copy that is FTP'd up to my webspace sometimes does not get there, though there is no error shown in the logfiles.

I wondered if perhaps it was a restriction in the demo version.

I have it set to send clientraw every 1 minute, right now the latest data I have up on the website is over 13 minutes ago. Eventually it will start working again, though I have frequently rebooted meteohub thinking that it has permanently stalled.

Ca you explain what this setting is for please?
Web Services Project (experimental)
Project Support Upload test data every 10 minutes

Does that help you find bugs?

Details of my system

Software
Linux Kernel: 2.6.24 1266 BogoMIPS, 516 MB RAM, VMware
MeteoHub: 4.6 (Build 1432) ©2007-2009 by Boris Pasternak, info@meteohub.de
System ID: 1XvD87y Limited Demo Version, 37 days left

Network
Hostname: meteohub
Workgroup: HOME
MAC: xxxxxxxxxxxxxx
IP: 192.168.0.110
Gateway: 192.168.0.1
DNS: 192.168.0.1
WAN IP: 212.56.xxxxxxx

Storage
Swap: 0MB of 188MB used (0%)
System: 512MB of 941MB used (54%)
Data: 77MB of 2903MB used (2%)

Date and Time
UTC: 22.01.2010 22:46:30
Time Zone: Europe/London
Local Time: 22.01.2010 22:46:30
Uptime: 2 hours, 26 minutes
System Load: 0.07, 0.07, 0.02

Process
Weather Data Recomputation: completed
Weather Data Logging: running (last data: 5 sec)
Weather Underground "Rapid Fire": running
SSH Login: running
SMB Shares: running
Time Synchonization: running



EDIT

I just noticed this in my WDL directory on website

Clientrawhour.txt.tmp - perhaps some problem there that is stopping further uploads?

[gm4jjj]

individual Schedule
Store local copy in "/public/myweb/uploads/"


I have switched off store local copy and it looks like the FTP is more reliable now. I will leave it running and see if this is the solution.

[gm4jjj]

Have made some changes to my EeePC today with regard to FSB speed and fan speed settings and now FTP seems reliable again. Will test for rest of weekend.

[gm4jjj]

Still getting problems, every day or so with WDL uploads stopping. The local data is OK, it is the FTP that stops. When the uploads stop I notice
Clientrawhour.txt.tmp in my WDL directory on my website.
Rebooting Meteohub fixes it.

[YJB]

Before you do a reboot, can you issue a df -h from the Linux command prompt? I'm wondering if you somehow are running out of space over time.

[admin]

As that problem is caused by your port I moved it to the generic x86 section...

ncftp uploads files as ".tmp" first and then renames these to the real file name. looks like your ncftp is having trouble with that under certain circumstances or you have multiple ncftps running because the inhibiting clause in the cron tab is not working with your setup:
[ -z "`ls /var/run/meteohub/uploads/`" ] || [ -n "`/bin/pidof /usr/bin/ncftpput`" ] || /usr/bin/ncftpput -S .tmp -R -V -DD

[gm4jjj]

FTP of WDL just hung up again after 4 hours of going fine;

meteohub:~# df -h

Code: Select all

Filesystem            Size  Used Avail Use% Mounted on
/dev/hda1             942M  513M  382M  58% /
tmpfs                 253M     0  253M   0% /lib/init/rw
udev                   10M   52K   10M   1% /dev
tmpfs                 253M     0  253M   0% /dev/shm
/dev/hda3             2.9G   83M  2.7G   4% /data
tmpfs                 8.0M     0  8.0M   0% /var/lock
tmpfs                  16M  2.8M   14M  17% /var/run
tmpfs                  16M   20K   16M   1% /var/log

[gm4jjj]

Before you do a reboot, can you issue a df -h from the Linux command prompt? I'm wondering if you somehow are running out of space over time.

Just tried that (thanks) and I don't appear to have a memory problem:

meteohub:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/hda1 942M 513M 382M 58% /
tmpfs 253M 0 253M 0% /lib/init/rw
udev 10M 52K 10M 1% /dev
tmpfs 253M 0 253M 0% /dev/shm
/dev/hda3 2.9G 83M 2.7G 4% /data
tmpfs 8.0M 0 8.0M 0% /var/lock
tmpfs 16M 2.8M 14M 17% /var/run
tmpfs 16M 20K 16M 1% /var/log

[gm4jjj]

admin wrote:
[quote

ncftp uploads files as ".tmp" first and then renames these to the real file name. looks like your ncftp is having trouble with that under certain circumstances or you have multiple ncftps running because the inhibiting clause in the cron tab is not working with your setup:
[ -z "`ls /var/run/meteohub/uploads/`" ] || [ -n "`/bin/pidof /usr/bin/ncftpput`" ] || /usr/bin/ncftpput -S .tmp -R -V -DD[/quote]

It definitely seems to be the .tmp file that is causing this block in updating. Is there any way to force a continuation without rebooting meteohub when this happens?

I have removed the .tmp file manually from my website, but meteohub refuses to update WDL data on the site until I reboot it again I imagine.

[gm4jjj]

Does this logfile help you see what is happening?

Code: Select all

7509 ?        Z<     0:00 [sh] <defunct>
17671 ?        S      0:00 /USR/SBIN/CRON
17672 ?        Ss     0:00 /bin/sh -c sleep 16; [ -z "`ls /var/run/meteohub/uploads/`" ] || [ -n "`/bin/pidof /usr/bin/ncftpput`" ] || /usr/bin/ncftpput -S .tmp -R -V -DD -u "account" -p "password" -P 21 ftp.plus.net /htdocs/gm4jjj/weather/wdl /var/run/meteohub/uploads/* 2>&1 | /usr/bin/logger
17714 ?        S      0:00 /usr/bin/ncftpput -S .tmp -R -V -DD -u ?        -p ?        -P 21 ftp.plus.net /htdocs/gm4jjj/weather/wdl /var/run/meteohub/uploads/clientraw.txt /var/run/meteohub/uploads/clientrawhour.txt
17715 ?        S      0:00 /usr/bin/logger
18786 ?        Ss     0:00 sshd: root@pts/0 
18789 pts/0    Ss+    0:00 -bash
20173 ?        S      0:00 /USR/SBIN/CRON
20174 ?        Ss     0:00 /bin/sh -c sleep 16; [ -z "`ls /var/run/meteohub/uploads/`" ] || [ -n "`/bin/pidof /usr/bin/ncftpput`" ] || /usr/bin/ncftpput -S .tmp -R -V -DD -u "account" -p "password" -P 21 ftp.plus.net /htdocs/gm4jjj/weather/wdl /var/run/meteohub/uploads/* 2>&1 | /usr/bin/logger
20175 ?        S      0:00 sleep 16
20203 ?        Z      0:00 [watchdog] <defunct>
20204 ?        S<     0:00 meteohub.cgi
20206 ?        S<     0:00 /usr/sbin/thttpd -C /etc/thttpd.conf
20243 ?        R<     0:00 ps ax

[admin]

looks like ftp jobs are queuing up for a reason that must be caused by your port.

can you provide your crontab (please without any user/pw credentials in there).

[gm4jjj]

Code: Select all

meteohub:/var/spool/cron/crontabs# cat root.new
*/1 */1 * * *   /bin/pidof -x histeval0 || /usr/bin/nice /home/meteohub/histeval0
*/5 */1 * * *   /bin/pidof -x histeval1 || /usr/bin/nice /home/meteohub/histeval1
13 */6 * * *   /bin/pidof -x histeval2 || /usr/bin/nice /home/meteohub/histeval2
47 4 * * *   /bin/pidof -x histeval3 || /usr/bin/nice /home/meteohub/histeval3
*/1 * * * * /bin/cat /proc/uptime >/var/run/meteohub/alive
22,52 * * * * echo 'wdnow' >/dev/meteohub
*/3 * * * * /bin/pidof wswinimport || /usr/bin/nice /home/meteohub/wswinimport 2>>/data/log/messages
17 */6 * * * rm /var/run/*.idx
26 6 * * * nice /srv/www/meteolog.cgi 0 0
#
# nightly reboot
37 2 * * * /usr/bin/test -f /home/meteohub/autoreboot && echo norecompute >/home/meteohub/norecompute && /sbin/shutdown -r -n now 
#
# keep watchdog happy
#
# restart thttpd if necessary
*/1 * * * * ([ -n "`/bin/pidof thttpd`" ] && [ -n "`/bin/pidof thttpdbackup`" ]) || rcthttpd restart
#
# generate history data
#
# synchronize hw clock
44 */2 * * * /sbin/hwclock -w
#
# compute weather data
8,18,28,38,48,58 * * * * /usr/bin/nice /home/meteohub/wmr928eval -x -c min1 -t -3600 -w /data/weather/ -s /home/meteohub/wmr928eval.conf -e /home/meteohub/wswin.conf -E /home/meteohub/wd.conf
1,11,21,31,41,51  * * * * /usr/bin/nice /home/meteohub/wmr928eval -c min5 -t -3600 -w /data/weather/ -s /home/meteohub/wmr928eval.conf
2,22,42 * * * * /usr/bin/nice /home/meteohub/wmr928eval -c min10 -t -3600 -w /data/weather/ -s /home/meteohub/wmr928eval.conf
3,33 * * * * /usr/bin/nice /home/meteohub/wmr928eval -c min30 -t -7200 -w /data/weather/ -s /home/meteohub/wmr928eval.conf
3 * * * * /usr/bin/nice /home/meteohub/wmr928eval -c hour1 -t -7200 -w /data/weather/ -s /home/meteohub/wmr928eval.conf
4 0,6,12,18 * * * /usr/bin/nice /home/meteohub/wmr928eval -c hour6 -t -50000 -w /data/weather/ -s /home/meteohub/wmr928eval.conf
5 2 * * * /usr/bin/nice /home/meteohub/wmr928eval -c day1 -t -200000 -w /data/weather/ -s /home/meteohub/wmr928eval.conf
6 2 1 * * /usr/bin/nice /home/meteohub/wmr928eval -c month1 -t -7000000 -w /data/weather/ -s /home/meteohub/wmr928eval.conf
#
# test for complete recomputation of weather data
*/5 * * * * /home/meteohub/wmr928fullevalwrapper >/dev/null 2>>/data/log/messages
#
# cut wswin import data into monthly chunks
#
# check for alarms to be mailed to the admin and send them once a day
44 5,11,17,23 * * * /home/meteohub/alarm check "" >/dev/null 2>>/data/log/messages
55 23 * * * /home/meteohub/alarm send "" >/dev/null 2>>/data/log/messages
#
# shorten logfiles
12 0 * * * /usr/bin/tail -n 1000 /data/log/thttpd.log >/data/log/thttpd.log.new; /bin/mv -f /data/log/thttpd.log.new /data/log/thttpd.log
13 0 * * * /usr/bin/tail -n 1000 /data/log/ntp.log >/data/log/ntp.log.new; /bin/mv -f /data/log/ntp.log.new /data/log/ntp.log
14 0 * * * /usr/bin/tail -n 1000 /data/log/smb.log >/data/log/smb.log.new; /bin/mv -f /data/log/smb.log.new /data/log/smb.log
15 0 * * * /usr/bin/tail -n 1000 /data/log/meteohub.log >/data/log/meteohub.log.new; /bin/mv -f /data/log/meteohub.log.new /data/log/meteohub.log
16 0 * * * /usr/bin/tail -n 1000 /data/log/alarm.log >/data/log/alarm.log.new; /bin/mv -f /data/log/alarm.log.new /data/log/alarm.log
17 */6 * * * /usr/bin/tail -n 1000 /var/cron/log >/var/cron/log.new; /bin/mv -f /var/cron/log.new /var/cron/log
#
#
## start section written by meteohub
0 1 * * * /bin/echo "norecompute" >/home/meteohub/norecompute && /sbin/shutdown -r now
0 */3 * * *	sleep 120; /home/meteohub/meteonet
#- "WD-live" "*/1 * * * *" "clientraw.txt" 0
*/1 * * * *	sleep 0; /srv/www/meteohtml.cgi /home/meteohub/clientraw.conf >/var/run/meteohub/clientraw.out; cp /var/run/meteohub/clientraw.out /var/run/meteohub/perm/clientraw.txt; mv /var/run/meteohub/clientraw.out "/var/run/meteohub/uploads/clientraw.txt"
#- "WD-live extra" "*/5 * * * *" "clientrawextra.txt" 0
*/5 * * * *	sleep 4; /srv/www/meteohtml.cgi /home/meteohub/clientrawextra.conf >/var/run/meteohub/clientrawextra.out; cp /var/run/meteohub/clientrawextra.out /var/run/meteohub/perm/clientrawextra.txt; mv /var/run/meteohub/clientrawextra.out "/var/run/meteohub/uploads/clientrawextra.txt"
#- "WD-live hour" "*/1 * * * *" "clientrawhour.txt" 0
*/1 * * * *	sleep 8; /srv/www/meteohtml.cgi /home/meteohub/clientrawhour.conf >/var/run/meteohub/clientrawhour.out; cp /var/run/meteohub/clientrawhour.out /var/run/meteohub/perm/clientrawhour.txt; mv /var/run/meteohub/clientrawhour.out "/var/run/meteohub/uploads/clientrawhour.txt"
#- "WD-live daily" "*/30 * * * *" "clientrawdaily.txt" 0
*/30 * * * *	sleep 12; /srv/www/meteohtml.cgi /home/meteohub/clientrawdaily.conf >/var/run/meteohub/clientrawdaily.out; cp /var/run/meteohub/clientrawdaily.out /var/run/meteohub/perm/clientrawdaily.txt; mv /var/run/meteohub/clientrawdaily.out "/var/run/meteohub/uploads/clientrawdaily.txt"
*/1 * * * *	sleep 16; [ -z "`ls /var/run/meteohub/uploads/`" ] || [ -n "`/bin/pidof /usr/bin/ncftpput`" ] || /usr/bin/ncftpput -S .tmp -R -V -DD -u "eeeeeee" -p "xxxxxx" -P 21 ftp.plus.net /htdocs/gm4jjj/weather/wdl /var/run/meteohub/uploads/* 2>&1 | /usr/bin/logger
## end section written by meteohub

[admin]

could you please try to run this statement in your ssh shell? It is a test if the recognition whether a process is already running works ok on your rig.

Code: Select all

[ -n "`/bin/pidof /home/meteohub/meteoschedule`" ] || echo "found"

As meteoschedule should be running (please check with "ps ax" ) the statement above should return "found".