Import of WSWIN-Files (experimental): Error in line 5

[HWTest]

I've tried this new feature but it isn't able to import even Meteohubs own WSWIN export :-(
meteoplug:/# cat /data/export/EXP04_10.csv | /home/meteohub/wswinload > raw-04-10
Error in line 5: 21.04.2010,10:29,22.7,9.6,14.7,19.7,9.9,9.9,8.3,50.0,39.0,51.0,45.0,45.0,55.0,966.7,0.00,5.4,293,7.6,1013.7

I want to import data from my TFA Nexus application into the Meteohub. I was very happy to see this new feature in the release notes.
After some hassle I was able to import the TFA Nexus DBF to WSWIN, export from WSWIN was easy but now I'm stuck.
To rule out any mistake during import/export in WSWIN, I've tried to import the data from Meteohubs WSWIN Data Export, but again the Error in line 5.
So I think there is something wrong with wswinload.

[admin]

Yes it is still experimental and I just discover how different WSWIN files can be. So you are part of the experiment. I am confident this will be stable in very near future.

Could you please send the file you have trouble to import to "info(at)meteohub.de".

[HWTest]

Done.

[admin]

new version of wswinload for x86 systems. [file name=wswinload.gz size=5553]http://www.meteohub.de/joomla/images/fb ... winload.gz[/file]

[admin]

new version of wswinload for nslu2 systems. [file name=wswinload-dc2cc2c8977e74c3248c02f18feae40a.gz size=8215]http://www.meteohub.de/joomla/images/fb ... feae40a.gz[/file]

[HWTest]

I've tried both files I've sent you by email.

meteoplug:/# cat /data/export/EXP03_10.CSV | /home/meteohub/wswinload > raw-03-10
Error in line 5: 01.03.2010,00:02,23.2,8.2,20.7,8.5,6.3,6.4,6.9,3.7,0.0,48,60,48,88,64,67,995.8,0.000,9.3,202,12.9,4
meteoplug:/# cat /data/export/EXP04_10.CSV | /home/meteohub/wswinload > raw-04-10
Error in line 5: 26.04.2010,10:30,21.8,17.1,15.0,19.0,10.6,16.3,18.0,46.0,59.0,46.0,82.0,49.0,35.0,974.9,0.00,7.2,180,11.5,1021.4

[admin]

What kind of architecture do you use?
x86? SheevaPlug? NSLU2?

[HWTest]

SheevaPlug

[admin]

I found an error. new version of wswinload for SheevaPlug systems. [file name=wswinload-a19b1dfff09de24ee3ed96aea33e13f3.gz size=7965]http://www.meteohub.de/joomla/images/fb ... 33e13f3.gz[/file]

[HWTest]

It works, but there are some flaws:
Data were imported to WSWIN from the TFA Nexus dbf, exported from WSWIN and imported to meteohub via wswinload.
- missing date0, I don't know if it is a problem
- local and sea level pressure are reversed, I've tried to change 133 -> 33, but the I got a Segmentation fault
- the local pressure values don't match, only sea level pressure is exported from wswin, maybe some recomputation error, meteohub is set to "based on on stations altitude and temperature th0" - I think it is more accurate.
- sensors TH0+TH4 are mixed up - solvable by search and replace
- rain total doesn't match, I don't know if it is a problem?
- T0 should be T6
- wind gust a average don't match, I think the weather station and meteohub samples and computes the values differently, maybe not a problem
- windchill in wind doesn't match, meteohub is set to "compute from wind and temprature th0 readings0", the values are more realistic
- T5, T8 I didn't find a match to these sensors

Sample of raw file recorded by meteohub:
20100428050154 date0 2010 4 28 7 1
20100428050155 thb0 210 49 0099 9813 3 10289
20100428050155 th4 103 40 -027
20100428050155 th1 185 50 0079
20100428050155 th2 105 84 0079
20100428050155 th3 51 74 0008
20100428050155 th0 36 84 0012
20100428050155 wind0 180 7 1 0037
20100428050155 t0 37
20100428050155 rain0 0 0 8625

Sample of raw from the (almost) same time imported by wswinload:
20100428050100 thb0 210 49 099 10278 6 9823
20100428050100 th0 103 40 -027
20100428050100 th1 185 50 079
20100428050100 th2 105 84 079
20100428050100 th3 51 74 09
20100428050100 th4 36 84 012
20100428050100 t5 81
20100428050100 t6 37
20100428050100 t8 0
20100428050100 rain0 0 0 218
20100428050100 wind0 180 3 0 0103

I've also attached a sample of the export csv from WSWIN. [file name=Sample.zip size=469]http://www.meteohub.de/joomla/images/fb ... Sample.zip[/file]

[admin]

- missing date0, I don't know if it is a problem

That is just a time stamp without meaning, no problem not to have it.

local and sea level pressure are reversed

Yes, will be corrected.

sensors TH0+TH4 are mixed up - T0 should be T6 -

Import functions just maps sensors from WSWIN one after the other. It does not take into account what IDs you might have given those sensors in Meteohub. Please live with assignment as is or use a text editor to change IDs in the entire file.

T5, T8 I didn't find a match to these sensors

They are in your export file. I can't do nothing against this ;-)

your export file has "" around the data, which is quite unusual (I never have seen this with WSWIN exports) and gives another arae of errors :-(
Is that a true WSWIN-Export or an export by any other 3rd party program? I wonder how time stamp recognition could ever go ok with that kind of data.

[HWTest]

sensors TH0+TH4 are mixed up - T0 should be T6 -
Import functions just maps sensors from WSWIN one after the other. It does not take into account what IDs you might have given those sensors in Meteohub. Please live with assignment as is or use a text editor to change IDs in the entire file.

No problem, search and replace solves this very easy.

T5, T8 I didn't find a match to these sensors
They are in your export file. I can't do nothing against this ;-)

Solved by deleting these columns

your export file has "" around the data, which is quite unusual (I never have seen this with WSWIN exports) and gives another arae of errors :-(
Is that a true WSWIN-Export or an export by any other 3rd party program? I wonder how time stamp recognition could ever go ok with that kind of data.

I've cut the sample in Open Office and Open Office saves csv with "" around the data. Now I'm using a csv editor, that doesn't do that.
The exports I'm using are true WSWIN V2.95.0 exports.

[skyewright]

I've cut the sample in Open Office and Open Office saves csv with "" around the data. Now I'm using a csv editor, that doesn't do that.

You have a solution, but...

When you "Save As" CSV in Open Office, if you tick "Edit Filter Setting" (at the bottom of the Save As window) you get to choose the "Text delimiter". The drop down probably only shows " and ', but you can type in other alternatives, including nothing at all (i.e. delete whatever is offered as default).

[HWTest]

I've found another problem, this time with the Rain data.
The export from WSWIN contains the rain data as "Regenmenge", I think it is something as "rain rate".
When I import the monthly exports from WSWIN into the meteohub, it recomputes the "Regenmenge" into "rain fall total (since beginning of recording)" but only for each month, so it is more since beginning of the month.
Is it possible to merge the WSWIN monthly exports into one file and import that file at once and then cut the import raw file into individual month raw files?
This could solve the problem ...

[admin]

"Regenmenge" is "total rain fall" measured in mm.
"rain rate" is a not so well defined measure of how intensive rainfall was, it comes as mm/h. So mapping is correct.

When importing a chunk of WSWIN data Meteohub resets rain fall counter to zero and increments this according to the rainfall info in the WSWIN file. WSWIN does not report rain fall counter numbers but just the increments per time frame.

When you import next month it again starts with zero. But this should not be a problem, because Meteohub handles this the same way as if you have replaced batteries in your rain gauge which also sets counter to zero. Meteohub is using the increments for computation of rain fall per time frame.

You can build one large WSWIN import file, as long as you strip the headers of subsequent data. But do you really have wrong rain total numbers or are you just guessing there might be problems?