Page 1 of 1

Bug Report - Log trim cronjob stops logger from writing logs

Posted: Thu Dec 13, 2012 11:27 am
by gcams
Hi Boris,

I have discovered a bug in the current version of meteohub (v4.9q), although I suspect it's been present for a while.

The bug is that at 00:15, when a cron job is called to trim the meteohub.log file, it causes the "logger" process to stop writing it's errors, etc to the meteohub.log file. Perhaps "logger" doesn't like having the file moved/removed/recreated in-flight?

I have proven this behaviour, as by disabling that line in the crontab, it prevents the problem from occurring.

I noticed the problem as the RF errors from my RFXCOM always seemed to stop logging at the same time each night (00:15). I would never see the errors again until I restarted the data logging, and then it would stop again at 00:15 the next day. I've managed to track it down to the log-trimming job.

Hope this is of some help!

Cheers,
Graham

Re: Bug Report - Log trim cronjob stops logger from writing

Posted: Thu Dec 13, 2012 3:01 pm
by YJB
Hmm, never noticed that behavior and I also see older entries in my log from a few days ago.

Never the less daemons don't like this method as it changes the file descriptor of the log file, and the daemon most likely points stll to the old file (which has been deleted).

This is probably a smarter way of doing things

Code: Select all

sed -i -e :a -e '$q;N;1001,$D;ba' /data/log/meteohub.log
And of course this will need to be done for the other trim functions as well, although not all of them are constantly kept open, but I like things consistent.

Code: Select all

sed -i -e :a -e '$q;N;1001,$D;ba' /data/log/thttpd.log
sed -i -e :a -e '$q;N;1001,$D;ba' /data/log/ntp.log
sed -i -e :a -e '$q;N;1001,$D;ba' /data/log/smb.log
sed -i -e :a -e '$q;N;1001,$D;ba' /data/log/alarm.log
sed -i -e :a -e '$q;N;1001,$D;ba' /var/cron/log

Ysbrand

Re: Bug Report - Log trim cronjob stops logger from writing

Posted: Thu Dec 13, 2012 4:01 pm
by gcams
Thanks Ysbrand, I like your proposed solution and i'll give that a go!

Re: Bug Report - Log trim cronjob stops logger from writing

Posted: Thu Dec 13, 2012 4:57 pm
by gcams
Well I'm afraid it looks like the above doesn't work either. I manually ran the sed command on the meteohub.log file to trim it. That was an hour ago, and since then logger has not written any more entries to the log file. Prior to me running the sed command, it was logging various RF messages every 5-10 minutes.

So whatever logger doesn't like about the previous method using tail, it doesn't like about the sed solution either. Worth a shot, but back to the drawing board! :)

Re: Bug Report - Log trim cronjob stops logger from writing

Posted: Thu Dec 13, 2012 5:29 pm
by YJB
In that casde Boris will need to give the answer, he wrote the code. Apparently the logger process is doing something funny with it's open files.

You can check if the log file is still open by going to:

Code: Select all

/proc/< pid of logger process > /fd
Ysbrand

Re: Bug Report - Log trim cronjob stops logger from writing

Posted: Thu Dec 13, 2012 8:21 pm
by gcams
Thanks again YJB, that's a handy little tip.. have been using linux since 2000, but still learn new stuff (like this) every day. Normally I would just use lsof, but I don't think meteohub has that binary installed.

Hopefully Boris can shed some light on why logger stops logging errors to the meteohub.log file when it's trimmed down.

Cheers,
Graham