Data Import function for MB PRO and NANO SD

This section covers the Meteobridge PRO units exclusively

Moderator: Mattk

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 5601
Joined: Mon Oct 01, 2007 10:51 pm

Re: Data Import function for MB PRO and NANO SD

Post by admin » Mon Dec 24, 2018 2:48 pm

Thanks, the files differ from what I have had before that they are with ascending time stamps. Just released update now supports both types (CSV and SQL) and both time stamp directions (ascending or descending).

I will now primarily take care of my family... Have a nice Christmas!

User avatar
ConligWX
Expert Boarder
Expert Boarder
Posts: 136
Joined: Wed Nov 14, 2018 12:48 pm
Location: Conlig, Co. Down, UK
Contact:

Re: Data Import function for MB PRO and NANO SD

Post by ConligWX » Mon Dec 24, 2018 4:28 pm

Thanks for the help Boris, Merry Christmas and a Happy New Year!

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 5601
Joined: Mon Oct 01, 2007 10:51 pm

Re: Data Import function for MB PRO and NANO SD

Post by admin » Tue Dec 25, 2018 10:02 am

I received two completely different looking data sets for Meteotemplate and I wonder if one might not even be a Meteotemplate export.

Example 1 starting with "wind average, wind gust, wind direction, ... :

Code: Select all

01-01-2018,00:00,0.00,3.20,84,0.00,22.20,51.00,1002.20,11.50,0.00
01-01-2018,00:05,0.60,1.40,99,0.00,22.10,52.00,1002.20,11.70,0.00
01-01-2018,00:10,0.50,1.40,100,0.00,22.00,52.80,1002.10,11.90,0.00
01-01-2018,00:15,0.00,1.40,83,0.00,21.90,53.80,1002.00,12.10,0.00
01-01-2018,00:20,0.00,3.20,89,0.00,21.80,54.40,1001.80,12.10,0.00
01-01-2018,00:25,0.00,1.40,106,0.00,21.60,55.00,1001.70,12.10,0.00

Example 2 starting with "T", "Tmax", "Tmin", ...:

Code: Select all

"2010-01-01 00:04:00","-0.4","-0.4","-0.4","81.0","-3.3","9.7","9.7","269.0","0.000","0.000","1005.600","0.0","-4.7"
"2010-01-01 00:09:00","-0.6","-0.6","-0.6","81.0","-3.5","16.7","16.7","265.0","0.000","0.000","1006.000","0.0","-6.3"
"2010-01-01 00:14:00","-0.6","-0.6","-0.6","82.0","-3.3","16.7","16.7","262.0","0.000","0.000","1006.000","0.0","-6.3"
"2010-01-01 00:19:00","-0.7","-0.7","-0.7","82.0","-3.4","13.0","13.0","262.0","0.000","0.000","1006.000","0.0","-5.7"
"2010-01-01 00:24:00","-0.7","-0.7","-0.7","83.0","-3.3","14.8","14.8","262.0","0.000","0.000","1006.000","0.0","-6.0"
"2010-01-01 00:29:00","-0.7","-0.7","-0.7","83.0","-3.3","16.7","16.7","261.0","0.000","0.000","1006.000","0.0","-6.4"
"2010-01-01 00:34:00","-0.7","-0.7","-0.7","83.0","-3.3","11.1","11.1","261.0","0.000","0.000","1005.600","0.0","-5.3"

As I am not running a Meteotemplate instance myself I am a bit lost, which one is right. Any expert advice out there?

stewart45
Fresh Boarder
Fresh Boarder
Posts: 12
Joined: Tue Dec 16, 2014 6:43 pm

Re: Data Import function for MB PRO and NANO SD

Post by stewart45 » Tue Dec 25, 2018 11:02 am

This is an extract from my meteotemplate database similar to your example 2


"2018-01-01 00:04:00","4.2","4.2","4.2","86.0","2.0","19.5","27.4","311.0","0.000","0.000","986.300","0.0","-1.3"
"2018-01-01 00:09:00","4.2","4.2","4.2","86.0","2.0","22.7","33.8","276.0","0.000","0.000","986.300","0.0","-1.9"
"2018-01-01 00:14:00","4.1","4.1","4.1","85.0","1.8","16.3","24.1","265.0","0.000","0.000","986.300","0.0","-0.8"
"2018-01-01 00:18:00","3.9","3.9","3.9","86.0","1.7","20.9","25.7","319.0","0.000","0.000","986.700","0.0","-1.9"
"2018-01-01 00:23:00","4.1","4.1","4.1","85.0","1.8","19.5","30.6","293.0","0.000","0.000","986.700","0.0","-1.4"
"2018-01-01 00:28:00","4.1","4.1","4.1","85.0","1.8","19.5","24.1","257.0","0.000","0.000","986.700","0.0","-1.4"
"2018-01-01 00:30:00","4.0","4.0","4.0","85.0","1.7","11.1","25.7","296.0","0.000","0.000","986.700","0.0","0.1"
"2018-01-01 00:35:00","4.0","4.0","4.0","85.0","1.7","13.0","17.7","300.0","0.000","0.000","987.000","0.0","-0.2"

User avatar
ConligWX
Expert Boarder
Expert Boarder
Posts: 136
Joined: Wed Nov 14, 2018 12:48 pm
Location: Conlig, Co. Down, UK
Contact:

Re: Data Import function for MB PRO and NANO SD

Post by ConligWX » Wed Dec 26, 2018 3:26 am

just for info.

I too ran into some issues with CumulusMX exported csv where each test delimiter was set to " "

I found some erroneous data which was fixed in excel, but the Excel removes the delimiter quotes and then Meteobridge refuses to import the data without them.

I had to then use OpenOffice Calc to put the quotes back in the file since Excel just doesn't work the way you want it too like that.

Perhaps imports of data should accept either Quotes and non Quoted text delimited data as the data after all it is separated by a comma.
Last edited by ConligWX on Wed Dec 26, 2018 2:12 pm, edited 1 time in total.

User avatar
ConligWX
Expert Boarder
Expert Boarder
Posts: 136
Joined: Wed Nov 14, 2018 12:48 pm
Location: Conlig, Co. Down, UK
Contact:

Re: Data Import function for MB PRO and NANO SD

Post by ConligWX » Wed Dec 26, 2018 5:11 am

ok I tried to import a cumulusMX csv for 1 whole year. I noticed in the historic data the ALL indoor temp Max was 734c and climbing. the date listed and that value is the overall Raincounter value.

this is obviously a bug and the indoor temp for "All max " is somehow linked /mixed up to the raincounter value.

when importing the values are:

"LogDateTime","Temp","Humidity","Dewpoint","Windspeed","Windgust","Windbearing","RainRate","TodayRainSoFar","Pressure","Raincounter","InsideTemp","InsideHumidity","LatestWindGust","WindChill","HeatIndex","UVindex","SolarRad","Evapotrans","AnnualEvapTran","ApparentTemp","MaxSolarRad","HrsSunShine","CurrWindBearing","RG11rain","RainSinceMidnight","WindbearingSym","CurrWindBearingSym"

error.jpg
error.jpg (36.9 KiB) Viewed 352 times

"2016-09-11 00:00:00","11.2","81","8.0","3.0","8.0","199","0.0","0.0","1010.13","734.2","24.7","48","3.0","11.2","11.2","0.0","0.0","0.0","7.8","9.8","0.0","0.0","183","0.0","0.0","SSW","S"

in its current state the CumulusMX import values do not copy over to the correct values in Meteobridge.

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 5601
Joined: Mon Oct 01, 2007 10:51 pm

Re: Data Import function for MB PRO and NANO SD

Post by admin » Wed Dec 26, 2018 12:31 pm

Thanks for spotting these errors. I changed the code accordingly. Now lines starting directly with a digit or with " or with (" are recognized as data records. Delimiters can be anything. Decimals need to be separated by "." (German "," notation for decimals is currently not supported).

An Update covering that is just released.

Does anybody have a WU export file? I would like to include this as well as some users might have used WU as their archive.

User avatar
ConligWX
Expert Boarder
Expert Boarder
Posts: 136
Joined: Wed Nov 14, 2018 12:48 pm
Location: Conlig, Co. Down, UK
Contact:

Re: Data Import function for MB PRO and NANO SD

Post by ConligWX » Wed Dec 26, 2018 2:17 pm

Boris, just out of curiousity. I noticed my buffers were getting quite high (85% full) before import finished. What happens when it reaches 100%?

Is new data from console just ignored?

rjuricic
Fresh Boarder
Fresh Boarder
Posts: 1
Joined: Wed Dec 26, 2018 6:52 pm

Re: Data Import function for MB PRO and NANO SD

Post by rjuricic » Wed Dec 26, 2018 9:05 pm

Hi, thanks for implementing import functionality to Nano SD module. I have 10 years of data in Meteotemplate format which i extracted to csv and sql format, and trying to import into Nano SD. So far i have noticed one error regarding parsing data when import is initated: if there is some field which does not contain any value (export in csv format) or has NULL (export in sql format), MB system tab will display error about invalid parameter sequence as shown on image below:
Image
Workaround for me was to manually input value (i.e. 0.0) in those fields. Import was successfull but i dont think that all data are correctly imported. I have cross-comapred original data in sql db and result of import to Nano SD and found that rain amount is not correctly imported, all rain values are lower than actual ones in original DB. Other sensor data seems correctly imported. (I have compared monthly sums,avg,min/max data via History tab on MB Nano)

Is there any possibility to interactively check data in MB database? I wanted to do checks via Graphs tab also, but time period allows me to select as far as 2 years only. I would like to be able to graph all 10 years.

Any advice on this will be helpful. Upload of 10 years db took me around 8hours, and before i try this again, i want to make sure that data will be uploaded correctly. Is there any documentation regarding import functionality which may explain described behaviour?

User avatar
ConligWX
Expert Boarder
Expert Boarder
Posts: 136
Joined: Wed Nov 14, 2018 12:48 pm
Location: Conlig, Co. Down, UK
Contact:

Re: Data Import function for MB PRO and NANO SD

Post by ConligWX » Wed Dec 26, 2018 9:13 pm

Just imported 2.5 years of cumulusmx data. it looks ok now, though I did have to fix some erroneous data.

I broke each year years down into 4 month periods of data which seem to be ok. any bigger and buffers were likely to be near full or at 100% services restarted after buffers were cleared of queues.

Lets hope WeeWX sqlite db can be supported too :)

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 5601
Joined: Mon Oct 01, 2007 10:51 pm

Re: Data Import function for MB PRO and NANO SD

Post by admin » Thu Dec 27, 2018 12:26 am

rjuricic wrote:
Wed Dec 26, 2018 9:05 pm
Is there any possibility to interactively check data in MB database? I wanted to do checks via Graphs tab also, but time period allows me to select as far as 2 years only. I would like to be able to graph all 10 years.
When you look at the URL that generates the charts, you can easily see how it works.

Example: "chart.cgi?chart=hws-temphum-iso.chart&res=min&lang=en&start=D2&stop=D0"
[*] chart=hws-temphum-iso.chart, this specifies which of the chart definitions to use
[*] res=min, this specifies which time resolution to take. "min" is only available for the last 2-3 days. alternatives are "hour", "day", "month"
[*] start=D2, this tells chart should start at the beginning of the day before today.
[*] stop=D0, this tells chart should end at the beginning of tomorrow (equals to end of today).

start/stop parameters can be relative like "Yn" (n years before current year), "Mn" (n months before current month) or Dn (n days before today). But you can also specify the date to start/stop.
Examples:
[*] "chart.cgi?chart=hws-temphum-iso.chart&res=month&lang=en&start=2016&stop=2017" will show 2016 in monthly resolution.
[*] "chart.cgi?chart=hws-temphum-iso.chart&res=day&lang=en&start=201501&stop=201504" will show first quarter of 2015 in daily resolution.
[*] "chart.cgi?chart=hws-temphum-iso.chart&res=hour&lang=en&start=2015011120&stop=2015011223" will show hourly data from Jan 11, 2015 8PM until Jan 12, 2015 11PM.

Remark: As stop date/time is exclusive, the stop date itself will not be part of the chart. End is always one time bucket before the stop date/time. Just in case you are missing the very last time bucket (most users will not even notice).

I hope this helps to fully understand the power of the Meteobridge graphing engine and how I engineered it for flexibility.

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 5601
Joined: Mon Oct 01, 2007 10:51 pm

Re: Data Import function for MB PRO and NANO SD

Post by admin » Thu Dec 27, 2018 3:39 pm

Just released update also supports import from weewx exports (in CSV notation). As with the other import functions, this is still regarded beta.

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 5601
Joined: Mon Oct 01, 2007 10:51 pm

Re: Data Import function for MB PRO and NANO SD

Post by admin » Thu Dec 27, 2018 8:45 pm

Now also WSWIN files can be imported. I think we slowly reach the end of the most requested formats.

mthiessen
Fresh Boarder
Fresh Boarder
Posts: 14
Joined: Fri Oct 30, 2015 12:25 pm

Re: Data Import function for MB PRO and NANO SD

Post by mthiessen » Fri Dec 28, 2018 3:37 pm

I would like to suggest Weather Underground data. I have 4 years of data posted there every 5 minutes using Meteobridge (TP-Link). Here is a tool here that allows you to get a CSV file from WU for a date range or a single day.
http://www.meteor.iastate.edu/~slincoln/stationdata/
The date range data is one line of data per day so it's not every 5 minutes (it doesn't seem to be working at all now). The single day data is every 5 minutes but you can't get a range of data.

I'm working on an AppleScript that will send requests for each day and add the results to a text file. I'm not sure how different WU data is from the other data sets you have created filters for. I've attached 1 days worth of WU data (every 5 minutes).

I will be purchasing a Nano SD in a few months when I visit my out-of-town PWS. I would love to be able to upload this 4 years worth of data to the SD for charting.

Keep up the good work. I love the work you are doing.
Weather Underground export data.txt.zip
(4.76 KiB) Downloaded 12 times

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 5601
Joined: Mon Oct 01, 2007 10:51 pm

Re: Data Import function for MB PRO and NANO SD

Post by admin » Sat Dec 29, 2018 12:37 am

Thanks! The WU file you attached is easy to parse, that will be snap to include to Meteobridge data import features. Is that a file that WU does directly offer for download or did you post-process it somehow?

Post Reply