Data Import function for MB PRO and NANO SD

This section covers the Meteobridge PRO, NANO SD and Raspberry Pi units exclusively

Moderator: Mattk

meteosa
Fresh Boarder
Fresh Boarder
Posts: 6
Joined: Fri Oct 25, 2019 8:49 am
Location: Italy
Contact:

Re: Data Import function for MB PRO and NANO SD

Post by meteosa »

Hello,
Is it possible to import data from weather display? I'm 10 years old and I wouldn't want to lose them. Now I have Meteobridge Pro.
Thanks

NFTE
Fresh Boarder
Fresh Boarder
Posts: 9
Joined: Mon Nov 11, 2019 9:52 am

Re: Data Import function for MB PRO and NANO SD Problems

Post by NFTE »

NANO SD:
I have tried to import files in WSWIN format.

This works only in about 60% of the imports.
The effect is that the import of one month works for about 25 days of data (15min resolution).
The MeteoBridge database has been correctly updated till this day of the month.
This has nothing todo with the data itself. If I try to import same file again after reboot it might work without problems.

What happens:
  • The import process (ps) vanishes.
  • It is not possible to load another csv file because system thinks import is still in progress.
  • Recording from VP2 stops completely
  • WebUI is still full operationally so you can recognize the system break only when analyzing actual data
Solution is only to Reboot

Attached: one of the WsWin files which have failed in first attempt.

Greetings
NFTE
Attachments
EXP10_19.ZIP
(8.42 KiB) Downloaded 76 times
MeteoBridge Nano SD
VP2 since 2007

User avatar
staze
Expert Boarder
Expert Boarder
Posts: 143
Joined: Fri Jan 18, 2019 2:56 am
Contact:

Re: Data Import function for MB PRO and NANO SD Problems

Post by staze »

NFTE wrote:
Mon Nov 11, 2019 10:37 am
NANO SD:
I have tried to import files in WSWIN format.

This works only in about 60% of the imports.
The effect is that the import of one month works for about 25 days of data (15min resolution).
The MeteoBridge database has been correctly updated till this day of the month.
This has nothing todo with the data itself. If I try to import same file again after reboot it might work without problems.

What happens:
  • The import process (ps) vanishes.
  • It is not possible to load another csv file because system thinks import is still in progress.
  • Recording from VP2 stops completely
  • WebUI is still full operationally so you can recognize the system break only when analyzing actual data
Solution is only to Reboot

Attached: one of the WsWin files which have failed in first attempt.

Greetings
NFTE
Does the Buffer fill up? How long does it take to process what you give it? When I did 5 minute resolution for 10-ish years, each year took maybe 2 hours to process (I'm going off memory, I think I said earlier in this thread). Do you have the Nano posting to any services while it's doing the import?
Davis Vantage Vue, Weather Envoy, Meteobridge Nano SD

NFTE
Fresh Boarder
Fresh Boarder
Posts: 9
Joined: Mon Nov 11, 2019 9:52 am

Re: Data Import function for MB PRO and NANO SD

Post by NFTE »

During import I can see that database fills up day by day.
After about "25 days" of import process stops.
The linux process for the Import task disappears and VP2 recording stops as descibed above.

I have not measured the time to crash but may be about 15 min.
I have some services and weather nets running - maybe this causes the problem!?
Then import function should stop any tasks untill it finishes.

But weird is that same csv data file can be imported completly without any problems.

I will try again and look for memory leak.

Good luck
NFTE
MeteoBridge Nano SD
VP2 since 2007

User avatar
staze
Expert Boarder
Expert Boarder
Posts: 143
Joined: Fri Jan 18, 2019 2:56 am
Contact:

Re: Data Import function for MB PRO and NANO SD

Post by staze »

Definitely check the "Buffer" state under System in the web interface. I'm guessing it's filling up and dying.

You might pause any service uploads until it's done.

What does the log say once it hangs (or before)?
Davis Vantage Vue, Weather Envoy, Meteobridge Nano SD

NFTE
Fresh Boarder
Fresh Boarder
Posts: 9
Joined: Mon Nov 11, 2019 9:52 am

Re: Data Import function for MB PRO and NANO SD

Post by NFTE »

You might pause any service uploads until it's done.
Thank you staze, this helps.
I have increased service rates and now buffer is max. 12/1% and all imports are now successfull :D

Now I have only the problem that the rain0total is not completely imported.
I think meteobridge imports only the full hour rain ammount and doesn't collect/add in this case the quarter hour measurements.

Here an example of import data of one day: 1.2mm
True summary of rain in csv is 2.8mm (1day WSWIN import file attached)
Rain-Import.jpg
Rain-Import.jpg (19.44 KiB) Viewed 3289 times
Thanks for help
NFTE
Attachments
EXP11_19.ZIP
(1.96 KiB) Downloaded 79 times
MeteoBridge Nano SD
VP2 since 2007

User avatar
staze
Expert Boarder
Expert Boarder
Posts: 143
Joined: Fri Jan 18, 2019 2:56 am
Contact:

Re: Data Import function for MB PRO and NANO SD

Post by staze »

Hi there!

Great to hear! So yes, I believe I posted about that a while back. I had to make all my rain numbers be a running total for rain in a given day (I think)?

viewtopic.php?f=61&t=13303&start=75#p28016

That only addresses my previous station which was really dumb and had totals for weeks/months before rolling over. =/
Davis Vantage Vue, Weather Envoy, Meteobridge Nano SD

robr57
Fresh Boarder
Fresh Boarder
Posts: 15
Joined: Fri Jan 03, 2020 2:37 pm
Contact:

Re: Data Import function for MB PRO and NANO SD

Post by robr57 »

Hi, I am struggling with decision to buy a Meteobridge Pro (with Meteostick embedded).

I did have my hands on a Meteobridge Pro device and tested if it could import Weatherlink .wlk files or raw Meteohub files.
That both failed, so I had to return it, there were also problems with reception of the Davis Vantage signal.
Meanwhile I learned that I can get an external antenna for improved reception, so that seems to fix the reception problem.

But before I order a new one, I am trying to find if it is possible to import Weatherlink .wlk files or Meteohub raw files.
I have data from my Davis Vantage Pro 2 since 2003. In total 196 files, 86 Mb. This is mostly 30min aggregated data.

I also have Meteohub .raw files since 2011, 107 files a hefty 3.69 Gigabytes....

I still want to move everything to a new Meteobridge Pro, but for me it is not clear at all how to either import the .wlk files or how to get 107 files all with the same filename (raw). I studied the Faq and tried several ways, but no file would import.

Any one that can describe how to proceed to get best result ?

Thanks in advance for guidance on how to migrate my Meteohub raw or my Weatherlink files.

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 6671
Joined: Mon Oct 01, 2007 10:51 pm

Re: Data Import function for MB PRO and NANO SD

Post by admin »

I would go with the Meteohub raw files. But import of such an amount of data will really take time.
Rough estimate is 10 minutes per one million Meteohub raw data lines.

NFTE
Fresh Boarder
Fresh Boarder
Posts: 9
Joined: Mon Nov 11, 2019 9:52 am

Re: Data Import function for MB PRO and NANO SD

Post by NFTE »

But take care of the bug described above:
Rain amount will not collect properly during import if measurements are more often than one time per hour!

@admin Any workaround or any plan to fix that problem?

Nfte
MeteoBridge Nano SD
VP2 since 2007

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 6671
Joined: Mon Oct 01, 2007 10:51 pm

Re: Data Import function for MB PRO and NANO SD

Post by admin »

I am not aware of a rain total import bug. When someone shares an import file that shows this problem, I can inspect.

NFTE
Fresh Boarder
Fresh Boarder
Posts: 9
Joined: Mon Nov 11, 2019 9:52 am

Re: Data Import function for MB PRO and NANO SD

Post by NFTE »

Please look at my post here in this thread (viewtopic.php?f=61&t=13303&p=31535#p31096), there is an example of wswin data which causes a wrong import of rain amount...

NFTE
MeteoBridge Nano SD
VP2 since 2007

User avatar
staze
Expert Boarder
Expert Boarder
Posts: 143
Joined: Fri Jan 18, 2019 2:56 am
Contact:

Re: Data Import function for MB PRO and NANO SD

Post by staze »

FWIW, my issue was my data from my lacrosse station was dumb and had a running total over multiple days/weeks/months. So I had to fix that with excel.

Not sure about this issue. It’s been a while since I looked...
Davis Vantage Vue, Weather Envoy, Meteobridge Nano SD

NFTE
Fresh Boarder
Fresh Boarder
Posts: 9
Joined: Mon Nov 11, 2019 9:52 am

Re: Data Import function for MB PRO and NANO SD

Post by NFTE »

I have tested with latest version (24.1.20) again with same wrong result. So nothing has changed.
admin wrote:
Sat Jan 04, 2020 4:53 pm
I am not aware of a rain total import bug. When someone shares an import file that shows this problem, I can inspect.
@admin Do you have checked the MB-import with the file I've posted?

NFTE
MeteoBridge Nano SD
VP2 since 2007

WS Heinenoord
Junior Boarder
Junior Boarder
Posts: 22
Joined: Sun Dec 22, 2019 2:08 pm

Re: Data Import function for MB PRO and NANO SD

Post by WS Heinenoord »

Unfortunately my MB Pro has similar import issues as NFTE reports in earlier posts,

This week I wanted to import historical data from my Davis station, stored in a csv file. From that file I created text files in the "direct" format, meaning <epoch time stamp> <sensor name> <sensor value> followed by newline. My data comes in 30 minutes interval.
All uploads went well, except for Rain0total.
Here I had converted the Davis data to a running sum for the year, but results in de Meteobridge were about half of what the Davis system reported for the sum till date. In an earlier post it's indicated that for summed variables there's a maximum of one measurement per hour. I made a table of total rain only at the hour, but that resulted in all zero values for all days. Then tried a dataset with only summed values for the halfpast hour. Same result.
Finally tried doubled values for all available timestamps. That gave values closer to the Davis data:
Davis MB
Jan: 50.4 42.4 mm
Feb,: 146.4 145.6 mm
Mar.: 62.6 62.0 mm

But still not good. Comparing numbers for a single day I see rainpeaks not always in sync with the Davis timing and missing 1 mm in the day total.

What to do, are there any tips?
Or is there something in the import software to be changed?

Finally a remark for clarity for those converting via the "direct" process: time has to be converted to Unix Epoch on GMT basis, so shifting 1 or 2 hours when based in (most) of the West-Europe continent.

Thanks for any help !

Post Reply