Upload historical data to Weather Underground

[bsparks]

After my Internet connection went down for 12 hours, I would love to be able to send the historical data logged by Meteohub to Weather Underground to complete my data online. Currently, I don't know of any way to do this.

[crm0922]

I am very interested in this feature as well.

I could implement this with a bash script if the wunderground URL request lines could be written out to a file instead of actually sent as web requests.

The I could script something to process the file and submit each URL, moving to the next if successful.

Some record locking scheme would probably have to be used, but I can think of a couple ways to do this without needing the meteohub app to comply.

Is there any way to change the URL of the update request or send the line to a file instead?

Does it use wget for the request?

This is a pretty major "whish" for me, because my station is on satellite internet, which "rain fades" pretty easily and thus rain data is surely going to be lost at some point.

Chris

[bsparks]

This is a feature that would thrill me because of the irony of weather stations and data collection--the data that you want most to upload most is that during severe weather/power outages, which make it impossible to upload!

I can run my station for quite a while on UPS battery backup power, but after 6 hours of complete power outage, my internet connection (cable) goes down, because the cable company only has 6-hour batteries on the line. I learned this in the last hurricane that came through here.

Living in Louisiana, a later upload feature would really be top-notch.

[admin]

Does WU really accept data from the past. I know that the data sent is time stamped, but are you sure that they take data from the past? Have not tried this but assumed that this won't work.

[bsparks]

Yes. I'm 99% sure, because the Davis software says it can upload historicals to Wunderground... at least I think it's the Davis software. I will do some more checking.

[crm0922]

I have this basically working with bash scripts. Are we allowed to post code here?

All I did was intercept the "wget" callout to a script that logs every request and then calls them line by line, recycling any that fail.

I haven't tested it on the meteohub itself, but I have it working on my linux box at home with test data.

Chris

PS - and yes, they do take old data, but I only tested filling in the overnight which I logged with my script...

[bsparks]

That's great! I'm not too familiar with Linux, but maybe this wish will soon come true!

[crm0922]

It isn't too tough to implement if I gave you the scripts and some instructions, but let me test them some more first.

And we'll have to see what the boss says about sharing unsupported code here.

Chris

[admin]

no problem, any helpful scripts are welcome... as long as I don't have to support them :cheer:

I have enough fun to fix my own bugs :lol:

[bsparks]

Chris, have you had any success on this? Hurricane season is heating up here in Louisiana, and I want to be ready if a storm hits!

[crm0922]

I have it working, I'm just not convinced that WU accepts all of the updates all of the time. Sometimes when I test it, WU just ignores some of the updates. However, it usually gets enough not to miss the important stuff.

WU seems to respond "success" to pretty much any request that authenticates properly. It doesn't seem to correlate to whether the update was really a "success".

I'll look my stuff over and post something here tomorrow or Monday.

Chris

[crm0922]

Would it be acceptable to only submit non-Rapid Fire updates? You probably aren't aware, but when "Rapid Fire" is enabled, meteohub is sending both standard updates *and* realtime updates.

I forget the timing of each, but I was thinking of only resending the non-realtime updates since it wouldn't be in realtime anyway.

I'm not sure WU even cares about realtime or not, but maybe they do and that's why some data is disappearing.

I'll get something prepared soon, it is a real pain in the balls because I have to work dialing in over satellite which has latency so bad you want to tear out your eyes with a fork.

Chris

[bsparks]

Definitely. I did know that Meteohub was sending both Rapid Fire and "normal" updates, because in Wunderground's FAQ, it details some information about doing just Rapid Fire, and letting the web software log those as often as it wants to for historical purposes.

I send what I call "historicals" in every 5 minutes. Anything more frequent than that to populate older data would be silly.

I appreciate your work. I wish there were something that I could do to help.

--Branson

P.S. Spoons work better ;-)

[crm0922]

Can you link to the FAQ to which you refer?

Also, how are you submitting "historicals"? Manually? With the Davis software? I'm confused.

Chris

[bsparks]

I did not make myself very clear. The "historicals" to which I refer are those that are actually logged as data points and you can pull up for each day... those updates that are sent that are *not* rapid fire.

Make sense?

Here's the FAQ:
http://wiki.wunderground.com/index.php/ ... r_Stations

and specifically for upload protocol:
http://wiki.wunderground.com/index.php/ ... d_Protocol
They call what I call "historicals" standard updates.

Not submitting any stuff with the Davis software; I use a Mac so I can't use it anyway.