How to repair corrupt weather Data ?

Moderator: Mattk

Post Reply
User avatar
ON1DQD
Expert Boarder
Expert Boarder
Posts: 93
Joined: Fri Dec 05, 2008 8:46 pm
Location: Belgium (Genk)
Contact:

How to repair corrupt weather Data ?

Post by ON1DQD » Wed Oct 14, 2015 2:55 am

Hi,

When I had a sd-card error I installed meteohub again and put my weather data back to /data/weather/YYYYMM/*

Now I had the problem meteohub freezes every time while it's recompute the data, so when it freezes I cycled the power and freezes again after 3.5 hours and so on.

I removed the weather data and meteohub does not freeze anymore :D

But I realy like to but back the usefull data I gaterred in these six years of data.

Is there a way I can repair these raw data files?
Regards / Grüß,
Geert - ON1DQD

User avatar
ON1DQD
Expert Boarder
Expert Boarder
Posts: 93
Joined: Fri Dec 05, 2008 8:46 pm
Location: Belgium (Genk)
Contact:

Re: How to repair corrupt weather Data ?

Post by ON1DQD » Sun Nov 22, 2015 6:12 pm

Well I realy was hoping for some input here as I started this thread on 14 Oct 2015 .

As spacially if you know that corrupted weather data wil freeze your Meteohub.

When you bootup Meteohub it begins a Weather Data Recomputation and as soon it read the corrupt data it freezes and you have to power cycle it again.

A tool to correct or delete these corrupted data or simply ignored by meteohub so that Meteohub does not freeze anymore would be nice.
Regards / Grüß,
Geert - ON1DQD

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 5365
Joined: Mon Oct 01, 2007 10:51 pm

Re: How to repair corrupt weather Data ?

Post by admin » Mon Nov 23, 2015 2:36 am

Meteohub is not freezing. It might stop computation of aggregated data when raw data is corrupted.
"Inspect Data" page has a function to cleanup raw data of a month. Did you try this?

Please also understand that recomputation of 6 years of raw data might take up to a day, depending how slow your system is and how much load you put on it to generate graphs, upload stuff etc. RPI, NSLU2, etc are know to be extremely poor performers.

User avatar
ON1DQD
Expert Boarder
Expert Boarder
Posts: 93
Joined: Fri Dec 05, 2008 8:46 pm
Location: Belgium (Genk)
Contact:

Re: How to repair corrupt weather Data ?

Post by ON1DQD » Wed Nov 25, 2015 4:31 am

Yes, it freezes if you ask me, when it freezes web-if wil become unavailable and ping wil fail to 192.168.1.77.

If I remove all the raw data folders /data/weather/ meteohub wil no freeze and only than it continious to function .

The cleanup function I did not dare to use, wil I loose all data from a whole month?
Regards / Grüß,
Geert - ON1DQD

User avatar
admin
Platinum Boarder
Platinum Boarder
Posts: 5365
Joined: Mon Oct 01, 2007 10:51 pm

Re: How to repair corrupt weather Data ?

Post by admin » Fri Nov 27, 2015 9:28 pm

you will not loose data by cleanup. But his freezing is not normal by any means.
Is cpu load going up when it starts "freezing"?
Do you have tons of uploads defined?

User avatar
ON1DQD
Expert Boarder
Expert Boarder
Posts: 93
Joined: Fri Dec 05, 2008 8:46 pm
Location: Belgium (Genk)
Contact:

Re: How to repair corrupt weather Data ?

Post by ON1DQD » Wed Dec 02, 2015 2:25 am

I have cleanup the data for every each month from 2009 till 2015 and still it freezes.
Also there is no much uploads defined in my opinion, see screenshot.
About the cpu load after the Weather Data Recomputation is done:
System Load: 0.03, 0.02, 0.03
Attachments
Meteohub FTP Uploads.JPG
Meteohub FTP Uploads.JPG (66.55 KiB) Viewed 5376 times
Regards / Grüß,
Geert - ON1DQD

User avatar
ON1DQD
Expert Boarder
Expert Boarder
Posts: 93
Joined: Fri Dec 05, 2008 8:46 pm
Location: Belgium (Genk)
Contact:

Re: How to repair corrupt weather Data ?

Post by ON1DQD » Thu Dec 03, 2015 3:12 pm

I have removed all folders in /data/weather exept folders 201510, 201511, 201512 and reboot the system.
The all other files in this folder I left them as is.
Now meteohub is up more than 24 hour without freeze.

How can I trail and error this issue, I realy want to now what cause this freezes?
Regards / Grüß,
Geert - ON1DQD

User avatar
ON1DQD
Expert Boarder
Expert Boarder
Posts: 93
Joined: Fri Dec 05, 2008 8:46 pm
Location: Belgium (Genk)
Contact:

Re: How to repair corrupt weather Data ?

Post by ON1DQD » Sat Dec 05, 2015 11:55 pm

meteohub is now up and runnin for 4 days and 8 hours :D

however I realy like to find the problem of this cuase.
Regards / Grüß,
Geert - ON1DQD

User avatar
ON1DQD
Expert Boarder
Expert Boarder
Posts: 93
Joined: Fri Dec 05, 2008 8:46 pm
Location: Belgium (Genk)
Contact:

Re: How to repair corrupt weather Data ?

Post by ON1DQD » Wed Dec 09, 2015 3:45 pm

It freezes again!
The only thing I did was Defined Weather Graphs in Management of Weather Graphs.
I delete the two I have added and see how it goes and report back here.
Regards / Grüß,
Geert - ON1DQD

User avatar
ON1DQD
Expert Boarder
Expert Boarder
Posts: 93
Joined: Fri Dec 05, 2008 8:46 pm
Location: Belgium (Genk)
Contact:

Re: How to repair corrupt weather Data ?

Post by ON1DQD » Sat Dec 12, 2015 6:48 pm

Well two days agao meteohub started to freeze again.
When I Reset Settings in maintenance I noticed that the settings of Network (LAN), Settings, Defined Weather Graphs & Weather Networks are not resetted.
I started to install meteohub from scratch (again) and dont use any backup or settings-backup files.

At the moment no Weather Networks or Defined Weather Graphs are defined and all works okay till now.

I wil try to add every day some older raw weather data and look how it goes.
Regards / Grüß,
Geert - ON1DQD

User avatar
ON1DQD
Expert Boarder
Expert Boarder
Posts: 93
Joined: Fri Dec 05, 2008 8:46 pm
Location: Belgium (Genk)
Contact:

Re: How to repair corrupt weather Data ?

Post by ON1DQD » Wed Dec 16, 2015 10:45 pm

All weather data from 2009 to 2015 are all transfer back to /data/wheater/.
Recomputations now finish and no freezes anymore.
Uptime: 6 days, 1 hours, 37 minutes :D :D :D :D
Regards / Grüß,
Geert - ON1DQD

User avatar
ON1DQD
Expert Boarder
Expert Boarder
Posts: 93
Joined: Fri Dec 05, 2008 8:46 pm
Location: Belgium (Genk)
Contact:

Re: How to repair corrupt weather Data ?

Post by ON1DQD » Mon Dec 28, 2015 9:44 pm

Well it's up and running for 18 days now and no freezes anymore :D
Regards / Grüß,
Geert - ON1DQD

Post Reply