[Libwebsockets] logrotate puzzle
andy at warmcat.com
Tue Apr 26 13:37:55 CEST 2016
On 04/26/2016 07:25 PM, Jaco Kroon wrote:
> Hi Andy,
> On 26/04/2016 12:32, Andy Green wrote:
>> On April 26, 2016 4:27:20 PM GMT+08:00, Denis Osvald <denis.osvald at sartura.hr> wrote:
>>> Have you looked at the following two options for logrotate?
>>> copy Make a copy of the log file, but don't change the original at
>>> all. This option can be used, for instance, to make a snapshot
>>> of the current log file, or when some other utility needs to
>>> truncate or parse the file. When this option is used, the cre‐
>>> ate option will have no effect, as the old log file stays in
>>> Truncate the original log file to zero size in place after cre‐
>> That sounds like it might work, thanks.
> It'll likely work, but note that you may very well lose log entries
> here. The copytruncate and signal to truncate both results in the same
> race conditions. The best (and only) reliable log rotation mechanism
> (external to the app handling rotation completely internally) is to:
> 1. Rename the log file (logrotate), keeping the inode intact.
> 2. Signal the application to re-open it's log files (which may not
> always be possible).
> 3. delaycompress (or else logrotate can compress the log file before
> the app has re-opened it's log files, again, losing log entries).
> I highly recommend that you rather implement the signal mentioned below
> to re-open log files rather than to perform truncation.
>>> ating a copy, instead of moving the old log file and optionally
>>> creating a new one. It can be used when some program cannot be
>>> told to close its logfile and thus might continue writing
>>> (appending) to the previous log file forever. Note that there
>>> is a very small time slice between copying the file and truncat‐
>>> ing it, so some logging data might be lost. When this option is
>>> used, the create option will have no effect, as the old log file
>>> stays in place.
>>> Perhaps it's better to combine the 'copy' option and a signal upon
>>> lwsws truncates the log itself, rather than 'copytruncate', in order to
>>> prevent the data loss situation described in copytruncate.
>> Alright, I'll try it... it gets me out of having to keep the credentials around just for this.
> To sort this specific problem out, you need to have the lws able to
> write to the file. You can look at acls for the folder /var/log/httpd
> (just need x if you're pre-creating the log files with logrotate, worst
> case chmod o+x /var/log/httpd). Possibly rather log to a separate log
> file other than /var/log/httpd, eg, /var/log/lwsws (surely lwsws can run
> on a system without httpd installed?).
Yes I just tried it and reached the same conclusion, I moved it to
/var/log/lwsws and I can see there's a logging race when it rotates the
logs by copy, but then later fires the signal to truncate the log.
> If those are not possible the combination of copy and copytruncate is
> your best options, but not ideal. No need to perform any special actions
> inside the application, it just needs to initially open the log files
> for appending write.
Yes that's the situation, he opens under root credentials with O_CREAT
and O_APPEND, then switches to weaker credentials.
This stuff is surprisingly tricky... I think it's better to keep root
thoroughly dead after startup as it is now and put up with this very
small race for a few ms every week where we might lose logs that happen
in that window.
I'll unpick my SIGHUP handler and see what happens if we just tell
logrotate to do the truncate.
> I hope that helps.
> Kind Regards,
>> Thanks for the idea.
>>> On 04/26/2016 09:33 AM, Andy Green wrote:
>>>> Hi -
>>>> Logging is working well on master, pointed to /var/log/httpd/...
>>>> That dir tree is like this on my Fedora box
>>>> drwxr-xr-x. 21 root root 4096 Feb 5 08:33 /var
>>>> drwxr-xr-x. 21 root root 4096 Apr 25 03:45 /var/log
>>>> drwx------. 2 root root 4096 Apr 25 03:45 /var/log/httpd
>>>> but it's OK because we start lwsws as root, open the log file, and
>>>> drop permissions to apache.
>>>> Today I noticed that logrotate had come along and renamed (archived)
>>>> log... I was still logging to it okay because the inode was the same.
>>>> But if you went and looked for the original log file, logrotate had
>>>> moved it and there's nothing there. And this "archived" log would
>>>> I thought about it (too briefly) and added a patch that will reopen /
>>>> recreate the log file if it's been open for 5 mins. No logs will be
>>>> lost because they'll get appended to the moved / archived / renamed
>>>> for up to 5 minutes.
>>>> But this fails, because after 5 minutes, there is no way he can
>>>> the log file with apache permissions, he can't get back in
>>>> Anyone have a good idea about the Right Way to solve this? The
>>>> seem to be cop out and require apache permissions level able to
>>>> the log dir (I guess it's a good idea how it is, since it won't let
>>>> hackers with apache permission level cover their tracks), or somehow
>>>> leave a root permission fork around that redoes the logs on SIGHUP.
>>>> logrotate options for regenerating a log with apache permissions
>>>> help, because even if we regenerate it, we can't get back in the
>>>> /var/log/httpd/ with apache permissions.
>>>> It looks a bit messy whatever we do, since the forked process can't
>>>> nicely reopen files in the apache permissions process. It'd have to
>>>> become responsible for logging (vhosts pass logs to this root process
>>>> get logged without having the fd open themselves).
>>>> Is that really the best way or some better way?
>>>> Libwebsockets mailing list
>>>> Libwebsockets at ml.libwebsockets.org
>>> Libwebsockets mailing list
>>> Libwebsockets at ml.libwebsockets.org
>> Libwebsockets mailing list
>> Libwebsockets at ml.libwebsockets.org
> Libwebsockets mailing list
> Libwebsockets at ml.libwebsockets.org
More information about the Libwebsockets