Uploaded Data Incrementing

[EW9092]

First post! My Davis Vantage Vue has been reporting to CWOP and Weather Underground a little over a month now, and I have a minor problem with the file time of the data. I can zero the file time manually by rebooting the MeteoBridge at 58 minutes past the hour which will result in the file time being exactly on the hour. I have the transmit time set for every hour to the Weather Underground and every 15 minutes to CWOP. The problem, every 12 hours the file time for both receivers of the data is incrementing by a leap minute. I have checked the raw data that I download periodically into the WeatherLink program and it is consistently on the hour plus every 5 minutes. The pertinent (maybe) local settings on the MeteoBridge Weather Station tab are set to Tolerated age of data: 10 minutes and Save historical data: every 10 minutes.

[EW9092]

Hmm, lots of views but no guesses. I think it might have something to do with a setting at the bottom of the Weather Station tab in the Meteobridge GUI. I have been changing the Local Settings values for Tolerated Age of Data and Save Historical Data in hopes that maybe a certain value is causing the problem and some other setting will make it stop. It doesn't seem likely since Tolerated Age has to do with how long a particular sensor value is considered representative (usually 10 minutes) and the Historical Data is a periodic push of cumulative historical data to the Meteobridge server in case of a local power or network outage. There doesn't appear to be any other settings that could possibly be in conflict. I'll just have to fiddle with settings and watch it for now. No one else has this problem? Am I the only one who cares about maintaining the same file times from day to day?

[admin]

In general the weather networks don't like to get data only sent at every full hour or so.
Most people feed WU in rapid fire every few seconds and there it is not making sense to synchronize
this to dedicated times.

[EW9092]

I think I have the WU network feed enabled for rapid-fire, but otherwise the upload is hourly. I also send to WeatherBug hourly, although they are working on their website so the data isn't available at the moment. Lastly, I send to CWOP, and that was set to upload every 15 minutes. A little while ago I changed that to every 30 minutes to see if that might change this strange behavior. My observations are still gaining a minute periodically, and since I have been changing settings it now gains a minute every 24 hours. Getting there.

[EW9092]

I have been watching the observation file times for a couple of weeks now, trying to get a sense of where the problem originates. I have determined that the one minute increment rate is random. It will add a minute in as little as 51 hours or as much as 66 hours. The fact that it is not regular leads me to believe the Meteobridge is not at fault. So another experiment is in the works. I have disabled the data upload to the Weather Underground site first because of reports of sporadic unresponsiveness of their servers. The reasoning here is that perhaps the number of retries to connect with the Weather Underground server introduces delays in the upload cycle to the remaining receivers. The CWOP data is basically the only feed I can check for timeliness. I have not found the capability for viewing raw data on the Weather Underground site. After I disabled the transmission to Weather Underground I rebooted and zeroed the Meteobridge transmit time again. And I go back to watching.

[EW9092]

Weeks go by and still no solution. I selectively disabled the additional data receivers (Weather Underground and WeatherBug) and it didn't make a bit of difference. I can't find anything useful in the cryptic Meteobridge Messages log on the system page. There are some entries in the log that seem to indicate a communication problem with the Vantage Vue, words like "unexpected 0 bytes" and "input/output error" which have me wondering. But I can't line any of these log entries up with the appearance of an additional minute in the data file time. I am almost resolved to having a periodic rolling file time. It really bugs me though.

[admin]

Can you please share your logs? "input/output errors" are not expected to occur.
Looks like you are using the USB Davis logger. Do you use a USB hub inbetween?

Why do you insist on having data uploaded to WU etc exactly on full hour
margins? No networks expects this and if all clients do so, it will kill the network.
So it is completely fine to have that distributed and no one cares if the update
done exactly every 60 minutes or 61 minutes. Please rethink your approach on
looking to this.

Do you have bandwidth limitations that don't allow you more frequent uploads?

[EW9092]

Thank you for your assistance. The only log I have looked at is the Messages log on the system tab of the Meteobridge GUI. I grabbed everything currently in that log and pasted it below. I have separated it into two sections. The first is the reboot I performed on July 3 to, yet again, zero the data upload (transmit) time to the hour. The second is some message traffic from July 6 that probably shows a couple of proper upload operations. Not sure why those were recorded. I have not seen the I/O error report lately, but I have a snippet captured from June 25 that I will attach below the two other groups. Yes, I have the USB Davis logger and I am using the included USB hub in between the Davis console and the Meteobridge as recommended in the installation instructions. On July 3 at 12:25 I switched from ethernet to WLAN to see if that would change any behavior, and it did not. The reboot was performed shortly after that to zero the data upload time.

I am currently uploading to two networks, CWOP every 30 minutes and Weather Underground every hour. My reasons for wanting a consistent upload time are mainly because I am accustomed to comparing the online data archive at 24 hour (or multiples thereof) intervals. It isn't absolutely necessary that a consistent file time be maintained. A minute off here and there has little statistical impact to my analysis. What you say about server workload around the times I have chosen to upload my data is interesting. I am a weather technician, weather observer and forecaster, and I know there are differences in METAR observation file times between NWS ASOS, Air Force, and Navy weather observation sources specifically to reduce server congestion. I will try changing my half-hour upload to CWOP to 15 and 45 minutes after the hour to see if that might make a difference. Hard to know what time would be best.

I am locally archiving my Davis data on a computer running the WeatherLink software program and it captures a data string every 5 minutes. This is more than enough resolution for quality graphing and data comparison. My choices for internet upload frequency are not based on any bandwidth limitations but were made for the easy construction of an accessible internet database I can look at from anywhere. I rarely use or look at the WU feed, but I do use the CWOP data I get from the MesoWest website. They have the data arranged in a tabular format referenced by time. And it is this database time record that has been incrementing randomly by one minute and causing all this commotion. I think I will contact their tech services and describe this situation and see what they say. Thanks again.


wnetd (03.07.2016 12:59:58): process started.
logger (03.07.2016 12:59:58): Historical data loaded (19 records).
logger (03.07.2016 12:59:58): data logger (version 5.0q, build 10034) started.
logger (03.07.2016 12:59:59): connect station 0 (Vantage via USB serial).
logger (03.07.2016 12:59:59): wakeup sent.
logger (03.07.2016 12:59:59): internal logger activated.
logger (03.07.2016 12:59:59): syncing.
logger (03.07.2016 13:00:09): gettime request sent.
logger (03.07.2016 13:00:09): no Vantage time offset.
logger (03.07.2016 13:00:09): eeprom request sent.
logger (03.07.2016 13:00:09): rain collector size 0.01 inch per tick.
logger (03.07.2016 13:00:09): transmitter 1 active as station type (#0) "ISS".
logger (03.07.2016 13:00:09): firmware version request sent.
logger (03.07.2016 13:00:09): firmware version 4.18
logger (03.07.2016 13:00:09): no reading of Vantage's internal logger.

logger (06.07.2016 09:40:29): 192.168.0.14's request on port 22222 accepted.
logger (06.07.2016 09:44:03): connection on port 22222 closed.
logger (06.07.2016 09:44:09): 192.168.0.14's request on port 22222 accepted.
logger (06.07.2016 09:44:10): connection on port 22222 closed.

logger (25.06.2016 00:31:30): unexpected 0 bytes.
logger (25.06.2016 00:31:30): station 0 (Vantage), can't set attributes of device /dev/ttyMH to former values: Input/output error
logger (25.06.2016 00:31:30): disconnect station 0 (Vantage): response without data
logger (25.06.2016 00:31:32): usb port reset.
logger (25.06.2016 00:32:00): connect station 0 (Vantage via USB serial).
logger (25.06.2016 00:32:00): wakeup sent.
logger (25.06.2016 00:32:00): internal logger activated.
logger (25.06.2016 00:32:00): syncing.
logger (25.06.2016 00:32:10): gettime request sent.
logger (25.06.2016 00:32:10): no Vantage time offset.
logger (25.06.2016 00:32:10): eeprom request sent.
logger (25.06.2016 00:32:10): rain collector size 0.01 inch per tick.
logger (25.06.2016 00:32:10): transmitter 1 active as station type (#0) "ISS".
logger (25.06.2016 00:32:10): firmware version request sent.
logger (25.06.2016 00:32:10): firmware version 4.18
logger (25.06.2016 00:32:10): no reading of Vantage's internal logger.

[EW9092]

Well, I am about resolved to just let this thing run and not get too excited about the file times. I have tried varying the upload transmit time to different times during the hour in hopes of finding a consistently "quiet" period where the upload would not be delayed. A delay in receiving the upload might be what is causing the file time to advance by a minute. It occurs at entirely random times. The CWOP database shows the last four advances occurred after 52, 70, 68, and 59 hours from a reboot that synchronized the transmit at 15 minutes and 45 minutes past the hour. I just checked the Weather Underground file time and it is 2 minutes slower than the CWOP file time, which I assume must mean that the upload receipts are independent of each other and I might be on to where these additional minutes are coming from. But I can't control how the data is received, so that is the end of it then. If anyone else has ideas or suggestions, I'll be keeping an eye on the thread.