Data Import function for MB PRO and NANO SD

[Neco]

I have similar issues with rain import. Values are too low (when importing 5 minute intervals) or non-existent (importing one hour sums) after import. I use the „direct“ file format.

[WS Heinenoord]

As indicated in an earlier post, I could not get the right historical rain data imported. To get my "rain total" reporting correct for the running year, I entered rain data data manually. And noted another aspect to keep in mind when entering or correcting historical data. There's a difference between how Weatherlink reports averages per time period and what Meteobridge expects per hour slot.
Weatherlink reports rain total per period PRIOR to the timestamp. No surprise, because the software can only add records about what just happened.
The Meteobridge page for entering / correcting data however has slots numbered from 00:00 to 23:00. Meaning rain drop between 23:00 and 24:00 h needs to be reported in the "23:00 h" box, thus AFTER the apparent time stamp.
If that amount is entered in the "00:00" h slot for the next day, like Weatherlink is reporting at 00:00h h, the rain amount from 23:00 - 24:00 h is incorrectly added to the total for the next day.
In summary, when entering Weatherlink data by hand, the reported data for a certain hour needs to be stored in the Meteobridge slot for one hour earlier.

I checked other weather parameters like outside temperature, that did get successfully imported via the "direct" route. Here the numbers are automatically landing in the correct timeframe slot.

[OE3MNS]

Hi
Is it possible to import AWEKAS export weather data into the MB Pro?

[WS Heinenoord]

@ OE3MNS

I think it's possible to import AWEKAS data using the "Direct" import option.
AWEKAS generates a straightforward EXCEL file, that's need however some transformation into the right MB format.
AWEKAS will download in multiple colums & rows for example outside temperature :
Datum tijd Temperatuur
dd.mm.yyyy hh:mm °C
11.05.2020 08:00 7.6

MB Direct import expects for that datpoint: "11589608800 th0temp 7.6" in a single row, single column.
So you need to rework the Excel data a bit, preferably with LOOKUP and CONCATENATE functions. And with a formula to convert the normal date / time stamp into a UNIX Epoch time stamp in Greenwich time.
Do this for every parameter you want to upload to the Meteobridge. Then for every parameter copy/paste that single column in a simple text file and put that in the "import"-subdirectory of the MB. The file(s) should then be visible in the "select importfile" dropbox.
The upload process may still have some issue, please check the relevant post in this forum.
If you get troubles to get the excel formula's working, please drop me an email and I can try to send an example.

Success!

[OE3MNS]

TNX

Greetings from Austria

[robr57]

Hi,

I installed a new Meteobridge Pro to replace my Meteohub. Meteohub was recording from Davis Vantage through WLIP. The Meteobridge Pro is reading direct RF.
I did import a year of Meteohub raw files and wanted to created historical graphs. This was OK, except for my Soil temp/hum station.

I now have the question how I can change the import for the meteohub raw data for Soil.

The records in the raw file are th1 and th2 , and should have been mapped to th10 and th11 during the import.

There is now a year data for th1temp, th1hum, th2temp and th2hum, which cannot be used as the new recordings in the Meteobridge are on th10 and th11.

Anyone a clue how to do this ?

Or should I reimport all th1 and th2 data and change the sensor into th10 and th11 ? That would be a nightmare job for 10 years of data .....

[admin]

can't you simply do a text replacement in the file to be imported?
If you are on linux "sed" can do that with one command line or you make it by a text editor of your choice in Windows.

[robr57]

can't you simply do a text replacement in the file to be imported?

Sure, but I have raw files since 2011, 10 years, so almost 120 in total, all are named raw and are all about 40 Mb in size. So, I have to open 120 files, one by one, rename file to eg. raw201101 for jan 2011, do text string replace , save file, upload to Meteobridge and start file import, wait 4 hours and do the next one......

Alternative that didnt work was to combine a full year of files into one big 480 Mb file, this failed to upload to meteobridge due to size.

But if there is no import mapping feature, then I will have to do the file by file approach...

Btw. wIth the name change of the sensors in the october release , the graph file allinonesoil-iso.chart is also broken as this does not show the changed physical sensor th10temp values.

[admin]

Meteobridge can now import large files. So concatenating the monthly files to yearly ones and importing the yearly data will work (although import of a year will take a few days). By that the text replacing job will be reduced to 10 files and imho "sed" can anyway do this in one command line.

Please understand that I will not code some elaborated sensor renaming logic into Meteobridge when this import issue can be easily solved with common tools outside Meteobridge.

[robr57]

Thanks for your feedback. Understand that implementing this is too much work, I hoped there was somewhere a hidden feature....

I made a small Python program to read all files, combine them and make the required changes. So, 2020 is now OK and loading slowly........

I am now looking into reducing the full year file by skipping redundant records. I have meteohub data since 2010 and Davis Weatherlink data since 2003.

Is there a simple algorithm that I can implement ?

Eg. report th0 and thb0 only once every xx minutes as these values do not change fast, only include rain0 records when it changes to capture all rain,
but I have no clue how to handle wind0, sol0 and uv0 .

If there is a description for this, I will develop the python script and make it available for others to migrate their historical data from Meteohub.

[admin]

just out of curiosity... how many lines has your Meteohub raw data file for a year?

[robr57]

Update to reflect 2019 records:
Approx 15 milion records per year, some months are more than others.

Code: Select all

2019   	 	Records 
th0		682.241 
th1		566.372 
th2		590.348 
thb0		601.900 
wind0	     10.912.570 
rain0		541.177 
sol0		569.165 
uv0		550.228 

Total    15.014.001 

[robr57]

This week I have been able to load 4 years of data, (2017, 2018, 2019, 2020) total 60 million records, in chucks of 4 million.

I am still interested to get the algorithm to compress the data to load all previous years. Meteobridge aggregates data to daily, monthly and yearly levels. So the algorithm must exist, specifically for wind this would be very beneficial to reduce the load times.

Again, I can implement this in my Python programme and make it available. Currently the program reads all raw files from meteohub SMB directory, translates the additional Soil/Leaf/Temp sensors to the correct new naming and outputs files combined per quarter (eg. raw2017q1 ) to easy the upload to meteobridge and import.

[WS Heinenoord]

I use a set of EXCEL sheets to calculate daily averages and extremes for a year of 30 minutes readings. This can also accommodate corrections, additional calculations like Temperature sum and name changes. The outcome is saved as textfile or csv for use in website operations.

If interested I can help setting up a similar template for your data.

[robr57]

I am certainly interested, but primarily for the algorithm to aggregate the wind data. Can I get your set of Excels to determine this ?