meteohub.de

While I made several changes, MeteoHub recomputated the weather data, but it is getting slower... :(

wmr928fulleval(21.09.2009 18:40:05): recomputation of weather data finished: 3623 records processed in 3 seconds (1207 records per second)
wmr928fulleval(21.09.2009 18:50:04): recomputation of weather data finished: 3738 records processed in 3 seconds (1246 records per second)
wmr928fulleval(24.09.2009 15:30:46): recomputation of weather data finished: 57564 records processed in 44 seconds (1308 records per second)
wmr928fulleval(24.09.2009 16:00:56): recomputation of weather data finished: 57946 records processed in 55 seconds (1053 records per second)
wmr928fulleval(25.09.2009 08:46:33): recomputation of weather data finished: 71139 records processed in 91 seconds (781 records per second)
wmr928fulleval(25.09.2009 17:24:14): recomputation of weather data finished: 77974 records processed in 1451 seconds (53 records per second)
wmr928fulleval(25.09.2009 18:13:55): recomputation of weather data finished: 78635 records processed in 1132 seconds (69 records per second)

The first recomputations were very good, but now...

Perhaps this problem is connected with the wrong checksum for the sensors(other topic)?

The weatherstation connects correct to meteohub:
logger (25.09.2009 17:51:11): connect station 0 (WMR-200 via USB HID).

The systemload at the moment is the following:
6.72, 6.81, 7.12

And the disk space is the following:
Swap: 0MB von 131MB belegt (0%)
System: 628MB von 838MB belegt (74%)
Daten: 78MB von 2542MB belegt (3%)
Why isn't the swap-partition used?

I have attached the meteohub.log, perhaps this help. [file name=meteohub-80df64b70387a9ee2e0e6ad4a3fa8247.txt size=34925]http://www.meteohub.de/joomla/images/fb ... fa8247.txt[/file]

Vetinari wrote:While I made several changes, MeteoHub recomputated the weather data, but it is getting slower... :(

If you have made several changes it may be that you have multiple simultaneous recomputations taking place. If that's the case each recomputation will appear to be slow.

IIRC when the multi-station feature of Meteohub was first introduced and I was trying out various combinations I somehow managed to have about 14 recomputations going at the same time!

I think at one time I made a suggestion that recomputations should have a way of periodically checking to see if a later recomputation has started, abandoning their own calculation if that's the case...

David

The recomputations weren't at the same time (as shown in the log), but I could solve the problem for now by rebooting the system. Yet it is at the same speed before the recomputations.

Perhaps several recomputations accumulates the system-load and no rebooting after each slows down the whole system.
Another problem is now, that only one graph shows the correct time, the other graphs are 4 hours too late... :huh:

Vetinari wrote:The recomputations weren't at the same time (as shown in the log),

That just shows the time that recomputations finished. It's not showing "recomputation of weather data started.", so there could potentially be overlapping recomputations.

However you are correct, as I wasn't taking into account the "processed in 3 seconds" and "processed in 1132 seconds", etc. I'm more used to recomputations that take several hours...

What I do notice is that the number of records to be processed jumps up in big steps. Were you restoring data from another system?

but I could solve the problem for now by rebooting the system.

Probably not relevant, but that happens to be one way of solving a multiple-recomputation problem (because it stops them all).

You are correct in saying several hours? In this fact I think the recomputation needs an improvement.

Little example:
Lets say I get every day 20.000 records (nearly the amount from the 24.09 (16:00) to 25.09 (17:24)). That makes an overall of 7.300.000 records! :ohmy:

Now the recomputation-progress:

1. if the SheevaPlug can recompute at a speed of ca. 1300 records/second it will take: 5615,4 seconds or 93 minutes
   if the SheevaPlug can recompute at a speed of only 60 records/second it will take: 121.666,7 seconds or 2027,8 minutes or 33,8 hours!!!

And then the data over several years... :(

I would prefer if the recomputation would only calculate by week or by month, all other is very time and source consuming.

By the way, is it normal that I can't change the password of the login-mask? I tried it several times and always came the message of success, but when I want to login with the new password it didn't work.
Perhaps I will format everything and try a new round...

There have been situations where multiple recomputations got stacked up, when playing with sensor and/or weather station definition. Doing a reboot after having done all the settings might be a good idea. I thought, I had extinguished that kind of problem, but cannot guarantee.

The checksum errors of wmr200 can have several causes. 1) noise coming through the USB cable 2) during recomputation system might not interact fast enough with low level usb handling, so some bytes might be missed on certain situations, which results in checksum errors and dropped data packets. 3) When internal storage buffer of wmr200 gets filled, it seems to suffer in USB connectivity. Clearing internal wmr200 data logger ad setting internal logging interval to maximum might help and postpone the next manual delete in the far future.

recomputation might slow down a bit by the end of each month. overall SheevaPlug should have enough power to handle years of data.

Meteohub makes use of ntpd and ntpdate to keep time in sync. So it is hard to understand how bad time settings might occur. Is timezone set correctly? is "use local time" checkbox marked?
Please check at "inspect data" dialog if problems with data/time did result in totally wrong time stamps for logged data. Are there any folders in the drop-down menu to select the month listed, that can't be right (being in far future or past)? If so these have to be deleted, otherwise Meteohub will fill up all month in between when exporting data to WSWIN or WD which require a set of data every minute. This will bring computation effort and storage capacity to the limit.

As said, when an internet connection is there, then ntpdate and ntpd should allign that. But if ther eis not internet connection during startup and the batteries of the systems don't hold rtc happy, strange things regarding date&time might happen.

admin wrote:Meteohub makes use of ntpd and ntpdate to keep time in sync. So it is hard to understand how bad time settings might occur. Is timezone set correctly? is "use local time" checkbox marked?
Please check at "inspect data" dialog if problems with data/time did result in totally wrong time stamps for logged data. Are there any folders in the drop-down menu to select the month listed, that can't be right (being in far future or past)? If so these have to be deleted, otherwise Meteohub will fill up all month in between when exporting data to WSWIN or WD which require a set of data every minute. This will bring computation effort and storage capacity to the limit.

As said, when an internet connection is there, then ntpdate and ntpd should allign that. But if ther eis not internet connection during startup and the batteries of the systems don't hold rtc happy, strange things regarding date&time might happen.

I think all things are working correctly, the collected data are saved with the GMT-timestamp and there are no crazy months which can be selected. An internet-connection is established, the time is correct, but sometimes there is a message in the ntp-protocol:
29 Sep 23:03:44 ntpd[1672]: can't open /var/spool/ntp.drift.TEMP: Permission denied
30 Sep 00:03:44 ntpd[1672]: can't open /var/spool/ntp.drift.TEMP: Permission denied
It seems that this does not harm the process.

I think I understand now, what the graphs are doing. I thought that the graphs were not correct with the time-line, but I am the opinion I got the clou.
When I create a graph which shows the last 6 hours, there will be every half hour a dash/point for the time.

When I create a graph which shows the last 48 hours, there will be every 3 hours a dash/point.

And when I create a graph which shows the last 7 days, there will be every 12 hours a dash for the time.

I hope you understand what I mean. :unsure:
And now my questions is, if I can change the difference between these points?