wswin import still not working properly. **solved**

[WETRHE]

Hi,
after internal storage crashed totally and has been swapped, I imported wswin data ONLY from 2022.
This took round about 24 hours, while I imported month wise (four csv files).
After import has finished, e.g. rain data is uncomplete:
Wswin has collected 193.2 liters per year 2022 and after import, MB has only 180.4 liters per year.

[WETRHE]

I have investigated this issue while completing the missing rain values manually and thereby I noticed, that ALL missing rain values were reported by wswin on a full hour time value.

Hard to describe that in English, so I will additionally put this in German:
ALLE fehlenden Regenwerte wurden von wswin zur vollen Stunde registriert. Der MB Datenbankimport schein mit Regenwerten zur vollen Stunde nicht klar zu kommen und lässt sie einfach weg.

mb01.JPG (62.22 KiB) Viewed 1708 times

mb00.JPG (39.38 KiB) Viewed 1708 times

[Gyvate]

I have investigated this issue while completing the missing rain values manually and thereby I noticed, that ALL missing rain values were reported by wswin on a full hour time value.

Hard to describe that in English, so I will additionally put this in German:
ALLE fehlenden Regenwerte wurden von wswin zur vollen Stunde registriert. Der MB Datenbankimport schein mit Regenwerten zur vollen Stunde nicht klar zu kommen und lässt sie einfach weg.

I guess first a few questions need to be answered by you to provide a baseline for understanding your issue properly:

0. what does your import file look like ? Can you provide the first hour of import records, headers if used included, csv preferred ?
1. what's your importing interval ?
2. Is your MB set to archive minute data ? (or hourly data only ? see System -> Administration)
3a. I would try to import the hour-sums in a single record with the time stamp of the hour
example: assuming your WSWin data are five-minute based - never mind the format I just, it's only to explain what I mean, the correct format for import may be different !
20220420 17:00 - include the WSWin hourly rain-sum
20220402 17:05 - no rain data (still a , , might be needed depending on the import format)
20220402 17:10 - no rain data
20220402 17:15 - no rain data
......
20220402 17:55 - no rain data
20220402 18:00 - include the WSWin hourly rain-sum

or
3b) import hourly records only in a first run - and then fill the gaps (provided minute based archiving is activated)

Also, as good practise, I would start experimenting with one day first until the import does what you expect and then only go for longer periods.
And, backups will be needed to reset, and MB exports for the time the import was running and from where you will have to start over again cleanly once having restored (to) your "old" database. That's the clean approach.

Another option (a bit dirtier as the data "corpses" will remain in the database even though they won't disturb) would be to create a 2nd MB station (Station -> Station1) with the same weather station and do the experiments there - you can later on remove this station again.
If you know how to use SQL, or SQliteBrowser, you can do the clean up later.

[WETRHE]

Import file has been a csv file from wswin directly. Pls. see screenshot:

MB1.JPG (112.12 KiB) Viewed 1627 times

Importing interval is set to 5 min.


My MB is set to archive minute data (see screenshot).

MB2.JPG (40.54 KiB) Viewed 1627 times

“If you know how to use SQL, or SQliteBrowser, you can do the clean up later.”
Is there a way to reach the MB’s internal database with a SQliteBrowser? Is there a kind of a how to? Would be great.

[Gyvate]

the simplest way would be to use a backup of the MB database.
Copy it to the machine you want to use sqlitebrowser on (Windows, Linux, ...).

If you put a USB device into the USB port (on MB Pro front, on RPi best one of the USB3 if it's a RPi4B), MB will copy the backups you create using the backup script in the services/events tab onto that device.
This device will usually either be /dev/sda1 or /dev/sdb1.
You can find out using e.g. puTTY to connect via SSH to your MB. user and password is the same as for your browser interface.
ls /dev/sd* -l will tell.
The exported portion of the file system which can be found under \\METEOBRIDGE\data as a network share under Windows has the path
/tmp/mnt/data.
you can create a directory under data e.g. mkdir /tmp/mnt/data/USB and then mount the device there
mount /dev/sda1 /tmp/mnt/data/USB (or sdb1 depending on your configuration)
then your backup USB device content will be visible under \\METEOBRIDGE\data\USB
you can either use sqlitebrowser from within Windows directly on one of the backups or on a copy which e.g. copy somewhere onto your hard disk.

attention: the mount will be removed by the backup script next time it runs (unless the USB directory is open in Windows Explorer).
The examples are made for using a Windows PC. If you use a Linux PC, you have to do things accordingly in a linux console window.

Another option could be to install the linux version of sqlitebrowser on MB (openWRT) itself, which would probably be more complicated.

If you want to reach the active MB database (which should be done with extreme caution as it's up and running - it's like surgery on the open heart and you really need to know what you do), you can also export the directory which contains the database.
e.g.
mkdir /tmp/mnt/data/database
mount /tmp/mnt /tmp/mnt/data/database
now the database is viewable and accessible from \\METEOBRIDGE\data\database
its name is "dbase" and it's located in /tmp/mnt as you can easily see from the commands before

I'd first try my activities e.g. running some SQL script on a backup copy beforehand before applying it to the active database.

If the network share \\METEOBRIDGE doesn't appear in Windows Explorer under Windows 10 or 11, you need to enable the SMB 1.0 protocol under "Programs and Features" in the Windows Control Panel as this is by default deactivated.

[WETRHE]

Unfortunately this wswin import bug is still existing … today I needed to import all the data from wswin again from scratch, due to a defective internal USB memory stick and the rain data is not being imported completely. All the rain data which has been measured from wswin “on the hour” (in German: zur vollen Stunde), for example at 17:00 is not being imported. Very annoying! Would it be possible to fix this issue at some time please?

[Gyvate]

as a possible workaround I suggest to change the import data from the full hour to full hour + one minute, e.g. instead of 17:00 use 17:01 and see what it gives.

[WETRHE]

Sounds not to bad for a workaround but how should it be possible to change the import data to full hour + one minute?
From January up to September there are 9 csv files in total. It would be at least the same effort to change all these data instead of editing the imported data in MB Pro afterwards.

[Gyvate]

manually, load the files into an editor and make a find and replace:
find :00; replace with :01; (or , depending on the import format - MB allows commas or semicolons as separators for WSWIN imports)
this sequence :00; or :00, should be unique in a line of the import file(s)

should be rather fast

[WETRHE]

How exactly this is working with an editor is absolutely clear to me, the question is more, how high the effort will be ... but I will give it a try at least for this month in order to see, if this workaround will work. But nevertheless, this bug needs to be fixed.

[Gyvate]

how high the effort will be .... ? - come on
there is a "replace all" function in most editors.
Under Windows the simple Notepad or Editor will do.
load the file in the editor - press CTRL-H - enter :00 and :01 - do one, two, three tests by pressing "replace" - and when it works, press "replace all"
processing time under one second - save under a different name
Do the same with the remaining import files ....

@Boris, @admin could have a look into a permanent solution (if it's a bug and not some other issue behind) - I agree

[WETRHE]

Unfortunately the try "full hour + 1 minute" failed. Even these rain data are not being imported.
I repeated the whole procedure with "full hour + 4 minutes" and even this failed. I guess it is a bug in the import function at all:
Per hour only 11 instead of 12 values / rows are being imported, maybe the array or the import loop counter is to small and needs to be increased.
So please fix this bug unless the wswin import function is useless at all.

[admin]

can you please send the file you cannot import to "info@meteobridge.com".

[WETRHE]

Mail mit Datei ist raus.

Danke & Gruß

[WETRHE]

Wswin import function is now working perfectly in the latest fw release, rain data is being imported correctly - thank you very much!