Data Import function for MB PRO and NANO SD

[mthiessen]

It’s the direct download using the tool in the link I provided. My AppleScript just tweeks the URL to get different dates. No need to use the form after you send it once. I read somewhere that WU no longer allows free access to their API but this seems to work without a key.

[MeteoOudkarspel]

Is it or it will also be possible to import from wunderground.com?

There is again data available from 2014 onwards, that I started the weather station.

[mthiessen]

I can download weather data for each day by building this URL which returns CSV data every 5 minutes for 1 day in browser.
https://www.wunderground.com/weathersta ... 9&format=1
I have a clunky AppleScript (I'm a hack) that saves the data then requests the next day.
I found that roughly every 20 days a page with only the header is returned, no data.

However WU is in the middle of retiring their current API for another one ($$). I'm not sure how much this impacts getting my PWS data.
More here: https://apicommunity.wunderground.com/w ... api-update

It seems not very reliable to use the WU direct download link (above) unless you can find a better way with their newer API.

[ConligWX]

Lol I just don't trust WU full stop

[mthiessen]

I agree. Which is why I am pulling a copy of my data from them.

[staze]

Have other thread elsewhere. Wondering if it would be possible to import from WeatherCat? Will add an example export tomorrow (or can email). Should be pretty simple.

Thanks!

--Update--
Attached example data (export of today's data)

WeatherCat Data Export Jan 19 2019.csv.zip

(3.33 KiB) Downloaded 298 times

[admin]

Thanks, will have a look at this when being back from vacation on Feb 1.

[admin]

Have other thread elsewhere. Wondering if it would be possible to import from WeatherCat? Will add an example export tomorrow (or can email). Should be pretty simple.

Thanks!

--Update--
Attached example data (export of today's data)
WeatherCat Data Export Jan 19 2019.csv.zip

Is the sequence of data in the rows fixed or does it depend on the station or sensors used? if it varies between individual setups, any generic import function will be hard to realize.

The header of your example lists these sequence of data in a header line:
Date, Conditions, Ext. Temperature,Dew Point,Wind Chill,Pressure,Wind Speed (10 Min. Average),Average Wind Direction,Precipitation Per Hour,Precipitation Total,Ext. Humidity,Int. Humidity,Int. Temperature,Wind Gust (10 Mins),Leaf Wetness 4,Monthly Rain,Annual Rain,Daily E.T.,Monthly E.T.,Annual E.T.

"Leaf Wetness 4" gives me trouble. What if there is not Leaf Wetness Sensor? What if two of those are connected? Will sequence of data change? if yes, how to detect? Automatically analyzing the header is imho too much of a fuzz.

[marco]

Hi
I tried to import data from Meteohub into the MB-Pro . And I noticed 2 things.
1. on the import tab I can see/browse the folders e.g. "201203" (for march 2013) but I can't see/choose the file in it. So that means i have to copy just the files into the import folder. But the files from meteohub have no extensions and have all the same name, "raw". so renamed the file into "raw.txt".
Its a bit tricky, but if this is way I can live with it thats not the problem...

2. Import Test with that "raw.txt" file failed.
Error message in the System Log:
import (26.01.2019 17:59:10): import of file "raw.txt" started (type:1, mode:0).
import (26.01.2019 17:59:10): invalid timestamp in line 1: 20120320204000 thb0 222 40 0080 9699 -1 10346
import (26.01.2019 17:59:10): completed 0 records (0 ignored), 0 db writes

Did I miss something? Is the import-mechanisem still beta?
many thanks for the support
marco

[kds]

What file am I supposed to use to import from Cumulus to Meteobridge? I have the monthly files with every data point in addition to the dayfile.txt file that rolls up each day. I've tried importing both files and I get timestamp errors on each.

[ConligWX]

What file am I supposed to use to import from Cumulus to Meteobridge? I have the monthly files with every data point in addition to the dayfile.txt file that rolls up each day. I've tried importing both files and I get timestamp errors on each.

You export your cumulus database table to then import it to meteobridge.

Are you writing to a cumulus mysql database?

[kds]

You export your cumulus database table to then import it to meteobridge.

Are you writing to a cumulus mysql database?

No, I don't think I have a SQL option. This is just vanilla Cumulus, not CumulusMX.

[ConligWX]

ok so where is your history records stored in Cumulus? must be some sort of db to hold weather data every 5 minutes for years?

My CumulusMX database table I exported was "Monthly". it held every data record collected in 5 minute intervals, for approx 2 years. extracted as a csv, and then imported into Meteobridge.

if you dont have that type of record, you might have sent data to WU who have a record of every internal. you then might be able to export that data from WU and import it.

[kds]

There are two types of files. The first are the monthly files, which contain every single data point at every single interval for that month. I would think that this is similar to your SQL table generated by CumulusMX, but for some reason Meteobridge doesn't like my monthly files.

The second type is the dayfile, which is a singular file that contains only the max/min values of each data point rolled up on a daily basis.

I do not send data to WU.

[staze]

Have other thread elsewhere. Wondering if it would be possible to import from WeatherCat? Will add an example export tomorrow (or can email). Should be pretty simple.

Thanks!

--Update--
Attached example data (export of today's data)
WeatherCat Data Export Jan 19 2019.csv.zip

Is the sequence of data in the rows fixed or does it depend on the station or sensors used? if it varies between individual setups, any generic import function will be hard to realize.

The header of your example lists these sequence of data in a header line:
Date, Conditions, Ext. Temperature,Dew Point,Wind Chill,Pressure,Wind Speed (10 Min. Average),Average Wind Direction,Precipitation Per Hour,Precipitation Total,Ext. Humidity,Int. Humidity,Int. Temperature,Wind Gust (10 Mins),Leaf Wetness 4,Monthly Rain,Annual Rain,Daily E.T.,Monthly E.T.,Annual E.T.

"Leaf Wetness 4" gives me trouble. What if there is not Leaf Wetness Sensor? What if two of those are connected? Will sequence of data change? if yes, how to detect? Automatically analyzing the header is imho too much of a fuzz.

Hi Admin!

You're supposed to still be on vacation! =)

Anyway, funny you mention "Leaf Wetness 4". It may be a bug in Meteobridge. I don't have that sensor. And I've currently got a TPlink 3020v3 installed as a Meteobridge on my Weather Envoy and then Weathercat talking to that over normal "Davis WeatherLink IP". That sensor only showed up once I did that. Previous to that, it was null as it doesn't exist (I just have a Vantage Vue). =/

That said, there are columns in Weathercat for each of those sensors and they're just left out if you don't have the sensor. Not sure if Meteobridge supports those sensors, so may just be a matter of ignoring those columns if they're not supported. Or maybe more advanced, a data "picker" that lets you say which columns belong to what data, but that probably could cause complications as well.

Hopefully that helps. Happy to provide more example data before that "Leaf Wetness 4" sensor showed up. Or see if I can ask for a complete list of possible columns from WC dev.

Thanks!