Tips & tricks for migrating data from Cumulus 1.9.4 to new NANOSD?

[tomas21]

Well....... after many years of admiration I have invested in the NANOSD to move away from my tried and true MR3040 that seems to be on its last legs.

Since 2018 I have been logging all my MR3040 to original Weather34 template (MB-UB40-RRW) and the MqSQL backend, at the same time publishing to the newer Weather34 Template Aurora MKII 2024, and the original Cumulus 1.9.4 build 10992.

1. What is the best way to migrate from my MR3040 Meteobridge given all the settings that have been setup there to support the above?
2. What is the best way to migrate all my data from 2018 that is housed in my local data folder for the Cumulus 1.9.4 which seems to be the most thorough data set?

Curious what others have done successfully give my current setup. I did read through the data import option for Cumulus but it seems based on my files I will need to do some munipulation to a lot of data (example below from Nov18log.txt

Code: Select all

29/11/18,22:40,49.3,92,47.1,0.0,0.0,0,0.00,0.00,29.674,3.28,66.9,59,0.0,49.3,49.3,0.0,0,0.000,0.000,48.6,0,0.0,0,0.00,3.28
29/11/18,22:41,49.3,92,47.1,0.0,0.0,0,0.00,0.00,29.674,3.28,66.9,59,0.0,49.3,49.3,0.0,0,0.000,0.000,48.6,0,0.0,0,0.00,3.28
29/11/18,22:42,49.2,92,47.0,0.0,0.0,0,0.00,0.00,29.676,3.28,66.9,59,0.0,49.2,49.2,0.0,0,0.000,0.000,48.5,0,0.0,0,0.00,3.28
29/11/18,22:43,49.2,92,47.0,0.0,0.0,0,0.00,0.00,29.676,3.28,66.9,59,0.0,49.2,49.2,0.0,0,0.000,0.000,48.5,0,0.0,0,0.00,3.28
29/11/18,22:44,49.2,92,47.0,0.0,0.0,0,0.00,0.00,29.673,3.28,66.9,59,0.0,49.2,49.2,0.0,0,0.000,0.000,48.5,0,0.0,0,0.00,3.28
29/11/18,22:45,49.2,92,47.0,0.0,1.0,274,0.00,0.00,29.674,3.28,66.9,59,0.0,49.2,49.2,0.0,0,0.000,0.000,48.5,0,0.0,0,0.00,3.28
29/11/18,22:46,49.2,92,47.0,0.0,1.0,274,0.00,0.00,29.674,3.28,66.9,59,0.0,49.2,49.2,0.0,0,0.000,0.000,48.5,0,0.0,0,0.00,3.28
29/11/18,22:47,49.2,92,47.0,0.0,1.0,274,0.00,0.00,29.675,3.28,66.9,59,0.0,49.2,49.2,0.0,0,0.000,0.000,48.5,0,0.0,0,0.00,3.28
29/11/18,22:48,49.2,92,47.0,0.0,1.0,274,0.00,0.00,29.676,3.28,66.9,59,0.0,49.2,49.2,0.0,0,0.000,0.000,48.5,0,0.0,0,0.00,3.28
29/11/18,22:49,49.2,92,47.0,0.0,1.0,262,0.00,0.00,29.676,3.28,66.9,59,0.0,49.2,49.2,0.0,0,0.000,0.000,48.5,0,0.0,0,0.00,3.28
29/11/18,22:50,49.2,92,47.0,0.0,1.0,262,0.00,0.00,29.675,3.28,66.9,59,0.0,49.2,49.2,0.0,0,0.000,0.000,48.5,0,0.0,0,0.00,3.28
29/11/18,22:51,49.2,92,47.0,0.0,1.0,262,0.00,0.00,29.675,3.28,66.9,59,0.0,49.2,49.2,0.0,0,0.000,0.000,48.5,0,0.0,0,0.00,3.28
29/11/18,22:52,49.2,92,47.0,0.0,1.0,262,0.00,0.00,29.678,3.28,66.9,59,0.0,49.2,49.2,0.0,0,0.000,0.000,48.5,0,0.0,0,0.00,3.28
29/11/18,22:53,49.2,92,47.0,0.0,1.0,258,0.00,0.00,29.678,3.28,66.9,59,0.0,49.2,49.2,0.0,0,0.000,0.000,48.5,0,0.0,0,0.00,3.28
29/11/18,23:30,49.5,92,47.3,1.0,1.0,270,0.00,0.00,29.680,3.28,66.7,59,1.0,49.5,49.5,0.0,0,0.000,0.000,48.3,0,0.0,270,0.00,0.00
30/11/18,00:00,49.3,92,47.1,4.0,4.0,225,0.00,0.00,29.683,3.28,66.6,59,4.0,48.0,49.3,0.0,0,0.000,0.000,46.4,0,0.0,225,0.00,0.00
30/11/18,00:30,49.0,91,46.5,3.0,3.0,23,0.00,0.00,29.682,3.28,66.5,59,3.0,48.5,49.0,0.0,0,0.000,0.000,46.5,0,0.0,22,0.00,0.00
30/11/18,01:00,49.0,91,46.5,5.0,5.0,23,0.00,0.00,29.676,3.28,66.4,59,5.0,47.0,49.0,0.0,0,0.000,0.000,45.4,0,0.0,22,0.00,0.00
30/11/18,01:30,49.0,91,46.5,0.0,0.0,0,0.00,0.00,29.693,3.28,66.2,59,0.0,49.0,49.0,0.0,0,0.000,0.000,48.2,0,0.0,0,0.00,0.00
30/11/18,02:00,48.8,91,46.3,2.0,2.0,68,0.00,0.00,29.695,3.28,66.1,59,2.0,48.8,48.8,0.0,0,0.000,0.000,46.8,0,0.0,67,0.00,0.00
30/11/18,02:30,48.8,91,46.3,1.0,1.0,45,0.00,0.00,29.698,3.28,66.0,60,1.0,48.8,48.8,0.0,0,0.000,0.000,47.4,0,0.0,45,0.00,0.00

Sample log file attached.

Thanks in advance for any pointers for those that have taken a similar path or have some suggestions.

[lightmaster]

Haven't messed with importing Cumulus data in several years, so lemme see what I can come up with. I assume that you do have your new NanoSD logging your current data now? Or are you still logging to the older MR3040 right now?

[lightmaster]

Just looking at your date format for each line, looks like some reformatting will need to be done at least. According to https://wiki.meteobridge.com/wiki/index ... al#Cumulus, the date for "Quorted export format" is expected to be a single string in the format of "2016-04-20 23:15:00", whereas you have it as 2 separate strings: 29/11/18,22:40.

I haven't used Cumulus, but is it possible to export it in a sql dump format? That has the column headers built into to it. If the Cumulus program doesn't have that export format available, you could send me the database file and I'll extract it in monthly batches and send you back the files you'd need to import.

[tomas21]

Hey Lightmaster...

I'm still using MR3040 and the NanoSD is brand new on my desk... i'm waiting to have the data migration plan nailed before I do the migration and cutover...

I have the MR3040 pushing data to my websites and my local Cumulus standalone pc client v1.9.4

Cumulus just has a bunch of log files it is creating which is where i probably have the best historical data..

[tomas21]

@lightmaster can you IM me and i'll send you a link to download my data zip?