Page 1 of 1
Help: Meteohub load consistently 5-8 and 100% of cpu for day
Posted: Tue Jun 16, 2009 9:46 pm
by mfemal
I've got an NSLU2 and it is consistently w/4.3d installed. It originally had version 2.x from the vendor (bought 3, all of them had old software versions and they had different versions of old software) so I first went to upgrade to 3.1 and then 4.3.
Now all run the same firmware. However, one of our systems has a consistent load of between 5 and 8 and is almost maxed on memory. I think it runs out as the system then locks up occassionally. Our other two stations (which I did not need the 3.1 interim upgrade) work fine and have a load between .5 and 1.
Looking at top I see:
Mem: 29332K used, 1044K free, 0K shrd, 240K buff, 18684K cached
Load average: 5.77 5.20 5.03 (Status: S=sleeping R=running, W=waiting)
PID USER STATUS RSS PPID %CPU %MEM COMMAND
12355 root R N 836 12354 18.5 2.7 wmr928eval
999 root R N 704 1 18.5 2.3 meteolog.cgi
2787 root R N 704 2783 18.5 2.3 meteolog.cgi
12343 root R N 668 12333 18.5 2.1 wmr928eval
12356 root R N 636 12300 18.5 2.0 wmr928eval
Any ideas on how to debug?? It is running like this for almost a week.
:(
Re:Help: Meteohub load consistently 5-8 and 100% of cpu for
Posted: Thu Jun 18, 2009 5:12 pm
by d_l
Could this be due to a corrupted flash drive? The Meteohub would have to deal with some bad data and this would cause extra CPU load. It does seem strange that the drive would become corrupted right "out of the box" though.
Re:Help: Meteohub load consistently 5-8 and 100% of cpu for
Posted: Fri Jun 19, 2009 6:08 pm
by mfemal
I'm fairly sure it is software related. I'm able to access the USB flash disk and it is working fine (no real delays or other errors in testing). Those wmr928eval processes appear to be the culprits. They are consuming most of the memory and taking lots of CPU time. I just don't know how to debug them. I really don't even know what they're doing and I've looked at all the logs for any potential errors and didn't see anything.
I was hoping someone else out here had ideas.
Re:Help: Meteohub load consistently 5-8 and 100% of cpu for
Posted: Fri Jun 19, 2009 7:21 pm
by sevenless
Some more details would be helpful:
Are all of your Meteohubs hooked up to the same type of weather station? (And what type of stations are they?) Similarly, are they all direct-USB cables, or are any connected by Serial-to-USB converters?
What happens if you swap the units with two different stations and reboot them - does the same Meteohub unit still have problems or does the one you swapped it with now have the problems?
Have you tried performing a fresh installation of the Meteohub 4.3 disk image onto the system with problems? Perhaps there is some old cruft from the version 2.3 installation that wasn't purged by incremental upgrading.
Wmr928eval is the master program that performs most of the Meteohub station-communication processes, and it will be doing the same thing on all three machines. If it's having problems on that unit then there's something underlying such as a faulty weather station, faulty connector, faulty flash memory, or faulty NSLU2 hardware.
Re:Help: Meteohub load consistently 5-8 and 100% of cpu for
Posted: Sat Jun 20, 2009 9:21 am
by Rikken
Hello
Had the same problem last month here. NSLU loading after updating to a Meteohub 4.x - Version was between 5 and 9 all over the rest of May. Since 1st of June the system is working between 1 and 4 - I think thats quite ok.... so perhaps some corrupted RAW Data of May could be the reason for that?
Re:Help: Meteohub load consistently 5-8 and 100% of cpu for
Posted: Mon Jun 22, 2009 3:13 pm
by mfemal
Thanks to all for the help thus far. Our weather stations are connected to a VantagePro2 (USB). Load is still high and some historical values appear to report incorrectly. I took a snapshot of all the data that was on this production unit, and put it on one of the other stations (same configuration) in a test setting. With the same data, the load has now spiked on this other station too with similar process footprint (wmr processes). Not sure if data is corrupt or not, I do not want to lose any data. Since Rikken had the same issue, I'm hoping it eventually goes back to normal. If anyone else has ideas, I'd appreciate it! I'll follow-up myself if I find a resolution.
Re:Help: Meteohub load consistently 5-8 and 100% of cpu for
Posted: Mon Jun 22, 2009 6:18 pm
by sevenless
A few observations:
1. With freshly loaded data, you can expect the processes and system load to spike while data is being recomputed, which will take anywhere from a few hours to nearly a day on an NSLU2. You can verify if this is going on from the System Info Meteohub page, at the bottom in the process section.
2. During and shortly after recomputation you can check if any corrupted data was found by going to the Log Files page and selecting the the 'Meteohub Log'. Look for entries that begin with "wmr928eval (mm.dd.yyyy hh.mm.ss) illegal data record in line..."
An occasional corrupted data packet is to be expected, especially if you're running a VP2 via USB and have it plugged into the AC adapter. If you have a lot of these entries, then you most likely have a bad USB stick and will need to replace it. However, you should be able to salvage the data from that stick and clean up the raw data files in any text editor, using the line numbers provided in the error line numbers to find the invalid data.
3. If you're running your VP2 plugged in, check that the USB cables aren't overlapping the power cable. I and others have had awful issues with the poor design of the VP2 data logger, which causes a lot of RF interference and corrupts the data stream. You can either add ferrite cores to your USB cable, or run the VP2 off of battery power to remedy this situation. It seems to vary between units but all of them have the problem to some degree. Davis has been slow to acknowledge the design flaw, but they now also make that recommendation. The station should run at least six months on a single set of batteries.
4. Are your NSLU2 units the older design running at 133Mhz, or are they the newer ones running at 266Mhz? (Or do you have a mix of both?) This data is available on the System Info page, at the top. Look at the top line, right side, for "133 BogoMIPS" or "266 BogoMIPS". If any of your stations are running at the slower speed, it is
very easy to overclock them by popping out a single resistor. That can make a very big difference in system load.
Re:Help: Meteohub load consistently 5-8 and 100% of cpu for
Posted: Wed Jul 01, 2009 4:44 pm
by mfemal
Things have worked themselves out after another week. I did have the slower model so I assume recalculations needed to occur. I didn't have any bad entries in the log file (illegal records) and didn't want to try and overclock it (thanks for the advice anyway). Sounds like my issue resolved itself much like Rikken's.
Re:Help: Meteohub load consistently 5-8 and 100% of cpu for
Posted: Wed Jul 01, 2009 6:25 pm
by Rikken
Yes yes . sometimes time is the healer. ;)
What i have found out is that Meteohubs Loading is low at the beginning of each month and getting higher day by day - at he end of June Loading was about 5 up to 6 - and today 1st of July its almoust lower than 2. But all is working fine here so it seems to be a normal behaviour...
Have a nice time!
Re:Help: Meteohub load consistently 5-8 and 100% of cpu for
Posted: Wed Jul 01, 2009 10:52 pm
by d_l
You shouldn't think of running the NSLU2 at 266 MHz as overclocking. The IPX420 was rated for 266 MHz by Intel, but Linksys initially designed the NSLU2 to run at half speed (underclocked). When this underclocking was discovered, people began modifying their NSLU2s to run at the full 266 MHz.
News of this modification became widespread and it seemed like everyone was doing it. In the last two years of production, Linksys quietly began issuing 266 MHz version (fully clocked) NSLU2s by not soldering on the resister that the modders had been removing.