Data Import function for MB PRO and NANO SD

This section covers the Meteobridge PRO units exclusively

Moderator: Mattk

mthiessen
Fresh Boarder
Fresh Boarder
Posts: 14
Joined: Fri Oct 30, 2015 12:25 pm

Re: Data Import function for MB PRO and NANO SD

Post by mthiessen » Sat Dec 29, 2018 12:53 am

It’s the direct download using the tool in the link I provided. My AppleScript just tweeks the URL to get different dates. No need to use the form after you send it once. I read somewhere that WU no longer allows free access to their API but this seems to work without a key.

MeteoOudkarspel
Senior Boarder
Senior Boarder
Posts: 69
Joined: Wed Jun 10, 2015 12:53 pm
Location: Oudkarspel
Contact:

Re: Data Import function for MB PRO and NANO SD

Post by MeteoOudkarspel » Sun Jan 06, 2019 1:34 pm

Is it or it will also be possible to import from wunderground.com?

There is again data available from 2014 onwards, that I started the weather station.
Sincerely,

Ron C.P.J Overtoom Meteo Oudkarspel Municipality Langedijk.

Image

http://www.meteooudkarspel-lt.pd0rok.nl

mthiessen
Fresh Boarder
Fresh Boarder
Posts: 14
Joined: Fri Oct 30, 2015 12:25 pm

Re: Data Import function for MB PRO and NANO SD

Post by mthiessen » Sun Jan 06, 2019 1:55 pm

I can download weather data for each day by building this URL which returns CSV data every 5 minutes for 1 day in browser.
https://www.wunderground.com/weathersta ... 9&format=1
I have a clunky AppleScript (I'm a hack) that saves the data then requests the next day.
I found that roughly every 20 days a page with only the header is returned, no data.

However WU is in the middle of retiring their current API for another one ($$). I'm not sure how much this impacts getting my PWS data.
More here: https://apicommunity.wunderground.com/w ... api-update

It seems not very reliable to use the WU direct download link (above) unless you can find a better way with their newer API.

User avatar
ConligWX
Expert Boarder
Expert Boarder
Posts: 136
Joined: Wed Nov 14, 2018 12:48 pm
Location: Conlig, Co. Down, UK
Contact:

Re: Data Import function for MB PRO and NANO SD

Post by ConligWX » Sun Jan 06, 2019 7:30 pm

Lol I just don't trust WU full stop

mthiessen
Fresh Boarder
Fresh Boarder
Posts: 14
Joined: Fri Oct 30, 2015 12:25 pm

Re: Data Import function for MB PRO and NANO SD

Post by mthiessen » Tue Jan 08, 2019 3:06 pm

I agree. Which is why I am pulling a copy of my data from them.

User avatar
staze
Fresh Boarder
Fresh Boarder
Posts: 18
Joined: Fri Jan 18, 2019 2:56 am
Contact:

Re: Data Import function for MB PRO and NANO SD

Post by staze » Sat Jan 19, 2019 9:26 am

Have other thread elsewhere. Wondering if it would be possible to import from WeatherCat? Will add an example export tomorrow (or can email). Should be pretty simple.

Thanks!

--Update--
Attached example data (export of today's data)
WeatherCat Data Export Jan 19 2019.csv.zip
(3.33 KiB) Downloaded 10 times

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 5601
Joined: Mon Oct 01, 2007 10:51 pm

Re: Data Import function for MB PRO and NANO SD

Post by admin » Fri Jan 25, 2019 9:24 am

Thanks, will have a look at this when being back from vacation on Feb 1.

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 5601
Joined: Mon Oct 01, 2007 10:51 pm

Re: Data Import function for MB PRO and NANO SD

Post by admin » Sat Jan 26, 2019 2:13 pm

staze wrote:
Sat Jan 19, 2019 9:26 am
Have other thread elsewhere. Wondering if it would be possible to import from WeatherCat? Will add an example export tomorrow (or can email). Should be pretty simple.

Thanks!

--Update--
Attached example data (export of today's data)
WeatherCat Data Export Jan 19 2019.csv.zip
Is the sequence of data in the rows fixed or does it depend on the station or sensors used? if it varies between individual setups, any generic import function will be hard to realize.

The header of your example lists these sequence of data in a header line:
Date, Conditions, Ext. Temperature,Dew Point,Wind Chill,Pressure,Wind Speed (10 Min. Average),Average Wind Direction,Precipitation Per Hour,Precipitation Total,Ext. Humidity,Int. Humidity,Int. Temperature,Wind Gust (10 Mins),Leaf Wetness 4,Monthly Rain,Annual Rain,Daily E.T.,Monthly E.T.,Annual E.T.

"Leaf Wetness 4" gives me trouble. What if there is not Leaf Wetness Sensor? What if two of those are connected? Will sequence of data change? if yes, how to detect? Automatically analyzing the header is imho too much of a fuzz.

marco
Junior Boarder
Junior Boarder
Posts: 24
Joined: Wed Apr 11, 2012 11:04 am

Re: Data Import function for MB PRO and NANO SD

Post by marco » Sat Jan 26, 2019 7:42 pm

Hi
I tried to import data from Meteohub into the MB-Pro . And I noticed 2 things.
1. on the import tab I can see/browse the folders e.g. "201203" (for march 2013) but I can't see/choose the file in it. So that means i have to copy just the files into the import folder. But the files from meteohub have no extensions and have all the same name, "raw". so renamed the file into "raw.txt".
Its a bit tricky, but if this is way I can live with it :) thats not the problem...

2. Import Test with that "raw.txt" file failed.
Error message in the System Log:
import (26.01.2019 17:59:10): import of file "raw.txt" started (type:1, mode:0).
import (26.01.2019 17:59:10): invalid timestamp in line 1: 20120320204000 thb0 222 40 0080 9699 -1 10346
import (26.01.2019 17:59:10): completed 0 records (0 ignored), 0 db writes

Did I miss something? Is the import-mechanisem still beta?
many thanks for the support
marco

kds
Fresh Boarder
Fresh Boarder
Posts: 18
Joined: Sat Jan 26, 2019 10:13 pm

Re: Data Import function for MB PRO and NANO SD

Post by kds » Sun Jan 27, 2019 12:47 am

What file am I supposed to use to import from Cumulus to Meteobridge? I have the monthly files with every data point in addition to the dayfile.txt file that rolls up each day. I've tried importing both files and I get timestamp errors on each.

User avatar
ConligWX
Expert Boarder
Expert Boarder
Posts: 136
Joined: Wed Nov 14, 2018 12:48 pm
Location: Conlig, Co. Down, UK
Contact:

Re: Data Import function for MB PRO and NANO SD

Post by ConligWX » Sun Jan 27, 2019 1:22 am

kds wrote:
Sun Jan 27, 2019 12:47 am
What file am I supposed to use to import from Cumulus to Meteobridge? I have the monthly files with every data point in addition to the dayfile.txt file that rolls up each day. I've tried importing both files and I get timestamp errors on each.
You export your cumulus database table to then import it to meteobridge.

Are you writing to a cumulus mysql database?

kds
Fresh Boarder
Fresh Boarder
Posts: 18
Joined: Sat Jan 26, 2019 10:13 pm

Re: Data Import function for MB PRO and NANO SD

Post by kds » Sun Jan 27, 2019 1:35 am

ConligWX wrote:
Sun Jan 27, 2019 1:22 am
You export your cumulus database table to then import it to meteobridge.

Are you writing to a cumulus mysql database?
No, I don't think I have a SQL option. This is just vanilla Cumulus, not CumulusMX.

User avatar
ConligWX
Expert Boarder
Expert Boarder
Posts: 136
Joined: Wed Nov 14, 2018 12:48 pm
Location: Conlig, Co. Down, UK
Contact:

Re: Data Import function for MB PRO and NANO SD

Post by ConligWX » Sun Jan 27, 2019 2:47 am

ok so where is your history records stored in Cumulus? must be some sort of db to hold weather data every 5 minutes for years?

My CumulusMX database table I exported was "Monthly". it held every data record collected in 5 minute intervals, for approx 2 years. extracted as a csv, and then imported into Meteobridge.

if you dont have that type of record, you might have sent data to WU who have a record of every internal. you then might be able to export that data from WU and import it.

kds
Fresh Boarder
Fresh Boarder
Posts: 18
Joined: Sat Jan 26, 2019 10:13 pm

Re: Data Import function for MB PRO and NANO SD

Post by kds » Sun Jan 27, 2019 3:40 am

There are two types of files. The first are the monthly files, which contain every single data point at every single interval for that month. I would think that this is similar to your SQL table generated by CumulusMX, but for some reason Meteobridge doesn't like my monthly files.

The second type is the dayfile, which is a singular file that contains only the max/min values of each data point rolled up on a daily basis.

I do not send data to WU.

User avatar
staze
Fresh Boarder
Fresh Boarder
Posts: 18
Joined: Fri Jan 18, 2019 2:56 am
Contact:

Re: Data Import function for MB PRO and NANO SD

Post by staze » Sun Jan 27, 2019 4:17 am

admin wrote:
Sat Jan 26, 2019 2:13 pm
staze wrote:
Sat Jan 19, 2019 9:26 am
Have other thread elsewhere. Wondering if it would be possible to import from WeatherCat? Will add an example export tomorrow (or can email). Should be pretty simple.

Thanks!

--Update--
Attached example data (export of today's data)
WeatherCat Data Export Jan 19 2019.csv.zip
Is the sequence of data in the rows fixed or does it depend on the station or sensors used? if it varies between individual setups, any generic import function will be hard to realize.

The header of your example lists these sequence of data in a header line:
Date, Conditions, Ext. Temperature,Dew Point,Wind Chill,Pressure,Wind Speed (10 Min. Average),Average Wind Direction,Precipitation Per Hour,Precipitation Total,Ext. Humidity,Int. Humidity,Int. Temperature,Wind Gust (10 Mins),Leaf Wetness 4,Monthly Rain,Annual Rain,Daily E.T.,Monthly E.T.,Annual E.T.

"Leaf Wetness 4" gives me trouble. What if there is not Leaf Wetness Sensor? What if two of those are connected? Will sequence of data change? if yes, how to detect? Automatically analyzing the header is imho too much of a fuzz.
Hi Admin!

You're supposed to still be on vacation! =)

Anyway, funny you mention "Leaf Wetness 4". It may be a bug in Meteobridge. I don't have that sensor. And I've currently got a TPlink 3020v3 installed as a Meteobridge on my Weather Envoy and then Weathercat talking to that over normal "Davis WeatherLink IP". That sensor only showed up once I did that. Previous to that, it was null as it doesn't exist (I just have a Vantage Vue). =/

That said, there are columns in Weathercat for each of those sensors and they're just left out if you don't have the sensor. Not sure if Meteobridge supports those sensors, so may just be a matter of ignoring those columns if they're not supported. Or maybe more advanced, a data "picker" that lets you say which columns belong to what data, but that probably could cause complications as well.

Hopefully that helps. Happy to provide more example data before that "Leaf Wetness 4" sensor showed up. Or see if I can ask for a complete list of possible columns from WC dev.

Thanks!

Post Reply