Data Import function for MB PRO and NANO SD

[kds]

I have a theory I need to test out. My Cumulus file only had imperial units included, and I'm presuming, like most of the world and the entire scientific community, that Meteobridge is expecting metric units. So I'm running all of my historical data through Excel and converting everything.

[ConligWX]

I think I had to do that with one of my values too. you need to make sure the value is the same as the default unit for Meteobridge afaik.

[kds]

I think I had to do that with one of my values too. you need to make sure the value is the same as the default unit for Meteobridge afaik.

Yeah, so temperature is correct now, but rainfall is way off. Where do I find the default units? I checked on my device and I can't seem to find where that is.

[ConligWX]

https://www.meteobridge.com/wiki/index.php/Templates

These sensors are defined in Meteobridge.

th0temp: outdoor temperature in degrees Celsius
th0hum: relative outdoor humidity as percentage
th0dew: outdoor dew point in degrees Celsius
th0heatindex: outdoor heat index in degrees Celsius
thb0temp: indoor temperature in degrees Celsius
thb0hum: indoor humidity as percentage
thb0dew: indoor dewpoint in degrees Celsius
thb0press: station pressure in hPa
thb0seapress: normalized pressure (computed to sea level) in hPa
wind0wind: wind speed in m/s
wind0avgwind: average windspeed in m/s (time used for average depends on station)
wind0dir: wind direction in degress (0° is North)
wind0chill: wind chill temperature in degrees Celsius
rain0rate: rain rate in mm/h
rain0total: rain fall in mm
uv0index: uv index
sol0rad: solar radiation in W/m^2
sol0evo: evapotranspiration in mm (only supported on Davis Vantage stations)
sun0total: sunshine duration in hours (only supported on Davis Vantages equipped with a 3rd party sunshine detector)
lgt0dist: Distance to lightning in kilometer.
lgt0energy: Energy of lightning strike (no physical unit).
lgt0total: Number of lightning strokes.

[staze]

Hi, thanks for implementing import functionality to Nano SD module. I have 10 years of data in Meteotemplate format which i extracted to csv and sql format, and trying to import into Nano SD. So far i have noticed one error regarding parsing data when import is initated: if there is some field which does not contain any value (export in csv format) or has NULL (export in sql format), MB system tab will display error about invalid parameter sequence as shown on image below:

Workaround for me was to manually input value (i.e. 0.0) in those fields. Import was successfull but i dont think that all data are correctly imported. I have cross-comapred original data in sql db and result of import to Nano SD and found that rain amount is not correctly imported, all rain values are lower than actual ones in original DB. Other sensor data seems correctly imported. (I have compared monthly sums,avg,min/max data via History tab on MB Nano)

<snip>

Any advice on this will be helpful. Upload of 10 years db took me around 8hours, and before i try this again, i want to make sure that data will be uploaded correctly. Is there any documentation regarding import functionality which may explain described behaviour?

Hi rjuricic,

Did you have any luck with this? I have previously imported my WeatherCat data into Meteotemplate, and figured that might be simpler than direct WC import. But giving it my Meteotemplate data still resulted in the same error:

Code: Select all

import (23.02.2019 11:15:44): import of file "alldata2009.csv" started (type:3, mode:1).
import (23.02.2019 11:15:45): invalid parameter sequence in line 1: "2009-03-17 00:57:00","40.6","40.6","40.6","90","37.9","0","0","68","0","0","30.08","37.9"
import (23.02.2019 11:15:45): completed 0 records (0 ignored), 0 db writes

Not sure why... I did have to remove an invalid field (null data in that column).

Interestingly, I told Meteobridge to ignore bad data, but it still bails after that first line.

Thanks!

Update: Nevermind, I'm an idiot. I can't just delete the bad column. So I did a find and replace of ,"", for ,"0",. That seems to get the file importing.

Now I wait... =)

[kds]

can you post an extract of the data here so we can look at it to check the data is in the same format as mine, though I would need to find where I put my exported stuff:)

I have most of it sorted but for some reason the rainfall is not importing correctly.

I'm using June 30, 2018 as an example since we had an extreme rainfall that day of 185.928mm in a matter of hours.

Here's the column order I'm using:

Code: Select all

DateTime AS LogDateTime
,TempOutdoorC AS Temp
,HumidityOutdoor AS Humidity
,DewPointOutdoorC AS DewPoint
,WindMS AS Windspeed
,HighGustMS AS Windgust
,WindBearing AS Windbearing
,RainRateMM AS RainRate
,RainSinceMidnightMM AS TodayRainSoFar
,PressureHPA AS Pressure
,NULL AS Raincounter
,TempIndoorC AS InsideTemp
,HumidityIndoor AS InsideHumidity
,HighGustMS AS LatestWindGust
,WindChillC AS WindChill
,HeatIndexC AS HeatIndex
,NULL AS UV
,NULL AS SolarRad
,NULL AS Evap
,NULL AS EvapTrans
,NULL AS AnnualEvap
,NULL AS ApparentTemp
,NULL AS MaxSolarRad
,NULL as HrsSunshine
,WindBearing AS CurrWindBearing
,NULL AS RG11Rain
,RainSinceMidnightMM AS RainSinceMidnight
,WindDirection as WindSym
,WindDirection as WindSymCurr

Which results in the following output for the rainy hours of June 30, 2018 (just a sample):

Code: Select all

"2018-06-30 21:30:00","20.9","96","20.2","1.3","3.1","1","66.548","154.686","1013",,"28.7","39","3.1","20.9","20.9",,,,,,,,,"1",,"154.686","N","N"
"2018-06-30 21:35:00","20.9","96","20.3","0.9","3.1","11","93.726","161.544","1014",,"28.7","39","3.1","20.9","20.9",,,,,,,,,"11",,"161.544","N","N"
"2018-06-30 21:40:00","21.1","97","20.6","0.4","2.2","154","82.296","167.386","1014",,"28.6","38","2.2","21.1","21.1",,,,,,,,,"154",,"167.386","SSE","SSE"
"2018-06-30 21:45:00","21.1","97","20.6","0.0","0.9","210","80.010","172.212","1013",,"28.5","36","0.9","21.1","21.1",,,,,,,,,"210",,"172.212","SSW","SSW"
"2018-06-30 21:50:00","21.1","97","20.6","0.0","0.4","207","23.114","175.514","1013",,"28.4","37","0.4","21.1","21.1",,,,,,,,,"207",,"175.514","SSW","SSW"
"2018-06-30 21:55:00","21.2","97","20.7","0.0","0.9","191","29.972","177.546","1013",,"28.4","38","0.9","21.2","21.2",,,,,,,,,"191",,"177.546","S","S"
"2018-06-30 22:00:00","21.2","97","20.7","0.0","0.9","74","36.322","183.642","1013",,"28.5","38","0.9","21.2","21.2",,,,,,,,,"74",,"183.642","ENE","ENE"
"2018-06-30 22:05:00","21.2","97","20.7","0.4","0.9","355","16.002","184.658","1013",,"28.5","37","0.9","21.2","21.2",,,,,,,,,"355",,"184.658","N","N"
"2018-06-30 22:10:00","21.2","97","20.7","0.4","0.9","353","14.224","185.674","1013",,"28.4","36","0.9","21.2","21.2",,,,,,,,,"353",,"185.674","N","N"
"2018-06-30 22:15:00","21.2","97","20.7","0.0","0.4","353","5.080","185.928","1013",,"28.3","36","0.4","21.2","21.2",,,,,,,,,"353",,"185.928","N","N"
"2018-06-30 22:20:00","21.3","97","20.8","0.0","0.4","353","2.032","185.928","1013",,"28.3","37","0.4","21.3","21.3",,,,,,,,,"353",,"185.928","N","N"
"2018-06-30 22:25:00","21.3","97","20.8","0.0","0.4","109","1.270","185.928","1013",,"28.3","37","0.4","21.3","21.3",,,,,,,,,"109",,"185.928","ESE","ESE"
"2018-06-30 22:30:00","21.2","97","20.7","0.0","0.4","58","0.000","185.928","1013",,"28.3","38","0.4","21.2","21.2",,,,,,,,,"58",,"185.928","ENE","ENE"
"2018-06-30 22:35:00","21.1","97","20.6","0.4","0.9","60","0.000","185.928","1012",,"28.3","36","0.9","21.1","21.1",,,,,,,,,"60",,"185.928","ENE","ENE"
"2018-06-30 22:40:00","21.1","97","20.6","0.4","1.3","146","0.000","185.928","1012",,"28.1","35","1.3","21.1","21.1",,,,,,,,,"146",,"185.928","SE","SE"
"2018-06-30 22:45:00","21.0","97","20.5","0.0","1.3","73","0.000","185.928","1012",,"28.1","36","1.3","21.0","21.0",,,,,,,,,"73",,"185.928","ENE","ENE"
"2018-06-30 22:50:00","20.9","97","20.4","0.0","0.9","85","0.000","185.928","1012",,"28.1","37","0.9","20.9","20.9",,,,,,,,,"85",,"185.928","E","E"

The extract is correctly showing 185.928mm for total rainfall for the day but Meteobridge show 30.7mm for the daily total.

I'm at a loss for this. I've checked the columns and they match up (9th column is daily total) and they were all imported in metric units.

[ConligWX]

The extract is correctly showing 185.928mm for total rainfall for the day but Meteobridge show 30.7mm for the daily total.

I'm at a loss for this. I've checked the columns and they match up (9th column is daily total) and they were all imported in metric units.

you'll need to get Boris on this one.

[staze]

Okay, finally got all my data imported thanks to this thread.

Took exports from Weathercat, and imported into Meteotemplate. Had to fix the data from my previous weather station (LaCrosse WS2316) producing weird rain totals (spanning multiple days/months/etc) with Excel. Once that was into Meteotemplate, I was able to then export out yearly data, concatenate all that together, then convert my silly imperial units to iso in excel, and remove the unneeded fields (oh, and set all the solar numbers to "0" rather than NULL). Import for about 1,005,000 rows (March 2009 through Jan 2019) took about 7 hours total. All the web pushes (WU, Awekas, CWOP, etc just queued up during those 7 hours then updated after). Super cool (other than emails from services saying my station was offline).

I'm sure with some work this wouldn't be that hard to make a python script that did all of this for me, but for something I'm just having to do one off, not that big of a deal. The biggest wrench was really the rain data from my old station. I had forgotten how much of a piece of crap that thing was.

If anyone has questions, let me know. Otherwise, glad I finally got it. =)

[mchd17]

Question for ya. How'd you setup excel to convert from imperial to iso? That's what I need to do. New to excel.

[ConligWX]

took about 7 hours total. All the web pushes (WU, Awekas, CWOP, etc just queued up during those 7 hours then updated after). Super cool (other than emails from services saying my station was offline).

7 hours of not reporting weather:) thats why I broke my data into 3-4 month sections and imported it like that, as the buffers maxed out at 100% with anymore time length of data.

[ConligWX]

Question for ya. How'd you setup excel to convert from imperial to iso? That's what I need to do. New to excel.

you would need to convert each column from whatever unit it is to the unit you want, using the "convert" function.

PS you would be better googling it rather than waiting on an answer like this.

[staze]

took about 7 hours total. All the web pushes (WU, Awekas, CWOP, etc just queued up during those 7 hours then updated after). Super cool (other than emails from services saying my station was offline).

7 hours of not reporting weather:) thats why I broke my data into 3-4 month sections and imported it like that, as the buffers maxed out at 100% with anymore time length of data.

It all cached. In my case I only ever hit about 65% buffer. I don't have that many services/weather nets it's pushing to... yet. =)

And yeah, totally understand breaking it up. I wanted it to run overnight, rather than have to babysit it. =)

[mchd17]

Question for ya. How'd you setup excel to convert from imperial to iso? That's what I need to do. New to excel.

you would need to convert each column from whatever unit it is to the unit you want, using the "convert" function.

PS you would be better googling it rather than waiting on an answer like this.

Thanks, already did

[plex]

Hi

Can anyone tell me, what wrong here:

Code: Select all

import (26.02.2019 21:52:55): invalid parameter sequence in line 1: "2011-01-01 00:04:00",4.5,4.5,4.5,94.0,3.6,4.3,4.3,225.0,0.800,2.300,1007.300,,0.1

[staze]

Hi

Can anyone tell me, what wrong here:

Code: Select all

import (26.02.2019 21:52:55): invalid parameter sequence in line 1: "2011-01-01 00:04:00",4.5,4.5,4.5,94.0,3.6,4.3,4.3,225.0,0.800,2.300,1007.300,,0.1

Yup. It doesn't like the null on the second to last value (solar radiation, I believe). Open in Excel or something (make sure to convert the date/time back to the same format if you open in excel), and set that column to be "0". Save, and then give it another go.

btw. for Excel, you'll want to set the column type to be "Custom" and then do "yyyy-mm-dd hh:mm:ss" for the custom value.