# Apache eats space between restarts

## audiodef

Restarting apache brings my df usage down by 10 percent or more, even though /var/log/apache2 has nothing over 600k in it due to my syslog-ng/logrotate config. Why is apache eating up space and what files/dirs are taking up that kind of space between apache restarts? When I restart, I have not deleted anything from /var/log/apache2, so it has to be elsewhere. /tmp? /var/tmp?

----------

## Carnildo

When you rotate your log files, do you tell Apache to close the old one and start writing to the new one?

----------

## audiodef

As far as I can tell. I have this in /etc/logrotate.d/syslog-ng:

```

/var/log/apache2/* {

    compress

    size 1M

    rotate 5

    missingok

    sharedscripts

    postrotate

        /etc/init.d/syslog-ng reload > /dev/null 2>&1 || true

    endscript

}

```

Although, come to think of it I should fork that off to a separate logrotate.d file for apache so I can manage individual web sites.

----------

## audiodef

I just restarted apache - and did absolutely nothing else - and that freed up about 2 GB of space. 

Why?

----------

## faemin

...Last edited by faemin on Sun Dec 02, 2012 9:38 pm; edited 1 time in total

----------

## audiodef

I found out about htcacheclean and now have this running:

```

htcacheclean -d 1440 -p /var/cache/apache2/ -l 5M

```

But I still see df usage creeping up, and it's due to Apache somehow. 

I didn't have lsof, but I emerged it. I see some output, but am not sure what to do with it.

/etc/apache2/httpd.conf:

```

ServerRoot "/usr/lib64/apache2"

LoadModule actions_module modules/mod_actions.so

LoadModule alias_module modules/mod_alias.so

LoadModule auth_basic_module modules/mod_auth_basic.so

LoadModule authn_alias_module modules/mod_authn_alias.so

LoadModule authn_anon_module modules/mod_authn_anon.so

LoadModule authn_dbm_module modules/mod_authn_dbm.so

LoadModule authn_default_module modules/mod_authn_default.so

LoadModule authn_file_module modules/mod_authn_file.so

LoadModule authz_dbm_module modules/mod_authz_dbm.so

LoadModule authz_default_module modules/mod_authz_default.so

LoadModule authz_groupfile_module modules/mod_authz_groupfile.so

LoadModule authz_host_module modules/mod_authz_host.so

LoadModule authz_owner_module modules/mod_authz_owner.so

LoadModule authz_user_module modules/mod_authz_user.so

LoadModule autoindex_module modules/mod_autoindex.so

<IfDefine CACHE>

LoadModule cache_module modules/mod_cache.so

</IfDefine>

LoadModule cgi_module modules/mod_cgi.so

LoadModule cgid_module modules/mod_cgid.so

<IfDefine DAV>

LoadModule dav_module modules/mod_dav.so

</IfDefine>

<IfDefine DAV>

LoadModule dav_fs_module modules/mod_dav_fs.so

</IfDefine>

<IfDefine DAV>

LoadModule dav_lock_module modules/mod_dav_lock.so

</IfDefine>

LoadModule deflate_module modules/mod_deflate.so

LoadModule dir_module modules/mod_dir.so

<IfDefine CACHE>

LoadModule disk_cache_module modules/mod_disk_cache.so

</IfDefine>

LoadModule env_module modules/mod_env.so

LoadModule expires_module modules/mod_expires.so

LoadModule ext_filter_module modules/mod_ext_filter.so

<IfDefine CACHE>

LoadModule file_cache_module modules/mod_file_cache.so

</IfDefine>

LoadModule filter_module modules/mod_filter.so

LoadModule headers_module modules/mod_headers.so

LoadModule include_module modules/mod_include.so

<IfDefine INFO>

LoadModule info_module modules/mod_info.so

</IfDefine>

LoadModule log_config_module modules/mod_log_config.so

LoadModule logio_module modules/mod_logio.so

<IfDefine CACHE>

LoadModule mem_cache_module modules/mod_mem_cache.so

</IfDefine>

LoadModule mime_module modules/mod_mime.so

LoadModule mime_magic_module modules/mod_mime_magic.so

LoadModule negotiation_module modules/mod_negotiation.so

LoadModule rewrite_module modules/mod_rewrite.so

LoadModule setenvif_module modules/mod_setenvif.so

LoadModule speling_module modules/mod_speling.so

<IfDefine SSL>

LoadModule ssl_module modules/mod_ssl.so

</IfDefine>

<IfDefine STATUS>

LoadModule status_module modules/mod_status.so

</IfDefine>

LoadModule unique_id_module modules/mod_unique_id.so

<IfDefine USERDIR>

LoadModule userdir_module modules/mod_userdir.so

</IfDefine>

LoadModule usertrack_module modules/mod_usertrack.so

LoadModule vhost_alias_module modules/mod_vhost_alias.so

User apache

Group apache

Include /etc/apache2/modules.d/*.conf

Include /etc/apache2/vhosts.d/*.conf

CustomLog /var/log/apache2/access_log combined

# vim: ts=4 filetype=apache

ServerName (ip_address)

```

/etc/conf.d/apache2:

```

# /etc/conf.d/apache2: config file for /etc/init.d/apache2

APACHE2_OPTS="-D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -D AUTH_IMAP -D PERL"

```

(I deleted all the commented out lines that are there when installed.)

I have several vhosts, and they all more or less look like:

```

<VirtualHost *:80>

ServerAdmin (email_address)

DocumentRoot "/path/to/location"

ServerName (domain.com)

ServerAlias (www.domain.com)

<Directory "/path/to/location">

    Options Indexes FollowSymLinks

    AllowOverride All

    Order allow,deny

    Allow from all

</Directory>

</VirtualHost>

CustomLog /var/log/apache2/access_log_domain combined

```

Does anything look like it would cause apache to fill up disk space somewhere?

----------

## faemin

...Last edited by faemin on Sun Dec 02, 2012 9:50 pm; edited 1 time in total

----------

## BitJam

As Carnildo mentioned, as long as a file is open in a program it is not actually deleted even though you've removed all directories entries for that file with the rm command.

The simplest thing is to simply restart Apache after rotating the log files.  This is generally done by sending it a HUP signal.  See here:

https://httpd.apache.org/docs/2.2/stopping.html *Quote:*   

> Signal: HUP
> 
>     apachectl -k restart
> 
> Sending the HUP or restart signal to the parent causes it to kill off its children like in TERM, but the parent doesn't exit. It re-reads its configuration files, and re-opens any log files. Then it spawns a new set of children and continues serving hits.

 

----------

## faemin

...Last edited by faemin on Sun Dec 02, 2012 9:50 pm; edited 1 time in total

----------

## BitJam

 *faemin wrote:*   

> He posted that he has already restarted apache several times....

 The OP actually said: *Quote:*   

> Restarting apache brings my df usage down by 10 percent or more, even though /var/log/apache2 has nothing over 600k in it due to my syslog-ng/logrotate config.

 

This is exactly what I was talking about.  There was nothing really wrong,   The OP just didn't understand what was happening.  Once you realize inodes for open files don't magically disappear when the files are deleted then it all makes sense.  It's like hardlinks.  You need to remove every hardlink to a file before the actual file gets deleted and frees up its disk space.  I believe there is an accompanying problem of losing log information between the time the logs are rotated and when Apache is restarted.

You just need to restart Apache after you rotate the logs.  This is pretty standard stuff.

----------

## faemin

...Last edited by faemin on Sun Dec 02, 2012 9:50 pm; edited 1 time in total

----------

## BitJam

I didn't say it was a hard link.  I said it was like a hard link.  I was trying to explain how it works.  If you have N hard links to one file then all N need to be removed before you can free up space from the file (barring truncation).  Likewise when N programs have a particular file open then all N programs need to close the file before you can free up its disk space (again, barring truncation).

It is true that copy-and-truncate is an alternative approach to log rotation but is is usually only used when the program that controls the logs cannot be told to re-open its log files.  It is less safe and less efficient than the standard approach.  As the logrotate man page explains:

```
copytruncate

    Truncate the original log file in place after creating a copy,

    instead  of moving  the old log file and optionally creating a new

    one.  It can be used when some program cannot be told to close its

    logfile and thus  might  continue  writing  (appending)  to  the

    previous log file forever.   Note that there is a very small time

    slice between copying the  file  and  truncating it, so some

    logging data might be lost.  When this option is used, the cre-

    ate option will have no effect, as the old log file stays in

    place.
```

Apache is not broken in this way.  It was designed to respond to the HUP signal by re-reading its config files and re-opening its log files which was explained on the Apache documentation page I linked to.  Furthermore, I didn't say restarting Apache was the only way to deal with the problem, I said it was the simplest.

I didn't post here to get in a series of fights with you.  I simply wanted to help audiodef understand and fix his problem.

----------

## faemin

...Last edited by faemin on Sun Dec 02, 2012 9:50 pm; edited 1 time in total

----------

## cach0rr0

This thread will get back on topic ASAP, or posts will start disappearing. The tangents are annoying. Really, really, really annoying. 

Address the original poster's questions or shut up. 

audiodef - sorry I have nothing useful to add, just popping in to address this current exercise in stupidity.

----------

## audiodef

 *cach0rr0 wrote:*   

> This thread will get back on topic ASAP, or posts will start disappearing. The tangents are annoying. Really, really, really annoying. 
> 
> Address the original poster's questions or shut up. 
> 
> audiodef - sorry I have nothing useful to add, just popping in to address this current exercise in stupidity.

 

No worries. I appreciate that people were trying to help.   :Smile: 

I don't mind having Apache restart after log rotation. Could I simply put apache2ctl restart in the postrotate section of /etc/logrotate.d/apache2?

Two things: 

1. I noticed that -k is not a valid option for apache2ctl.

2. Whether I use apache2ctl or /etc/init.d/apache2 restart, I have to do it twice because the first time, I get "start-stop-daemon: /usr/sbin/apache2 is already running" and the second time I get "apache2 not running (no pid file)" but it starts up and all is well again. If I restart immediately after restarting, there is no such problem - this only happens after apache has already been running for a while.

----------

## BitJam

I don't have Apache installed here but you just need to get the pid of the parent httpd process.  The Apache page I linked to suggests using `cat /usr/local/apache2/logs/httpd.pid` to get the pid in which case the command would be:

```
kill -HUP `cat /usr/local/apache2/logs/httpd.pid`
```

I'd bet the pid file on your system lives elsewhere.

----------

## cach0rr0

 *BitJam wrote:*   

> 
> 
> I'd bet the pid file on your system lives elsewhere.

 

easy enough to figure out

```

pidof apache2

```

There should already be a

```

  /etc/init.d/apache2 reload > /dev/null 2>&1 || true

```

in /etc/logrotate.d/apache2

the reload() function does an apachectl -k restart already

IOW, this *should* already be functional/working. Why it isn't, I'm not sure.

----------

## audiodef

 *cach0rr0 wrote:*   

> 
> 
> There should already be a
> 
> ```
> ...

 

There wasn't, and I had to create /etc/logrotate.d/apache2 manually. 

That brings up another question, though: is "reload" enough for apache to release its hold on files and clear things up, versus an outright restart?

While I'm at it, I'm wondering if I can make my logrotate config for apache more efficient. It goes on like this:

```

/var/log/apache2/access_log {

    compress

    size 1M

    rotate 2

    missingok

    sharedscripts

    postrotate

        /etc/init.d/syslog-ng reload > /dev/null 2>&1 || true

        /etc/init.d/apache2 reload > /dev/null 2>&1 || true

    endscript

}

/var/log/apache2/access_log_abusedmen {

    compress

    size 2M

    rotate 5

    missingok

    sharedscripts

    postrotate

        /etc/init.d/syslog-ng reload > /dev/null 2>&1 || true

        /etc/init.d/apache2 reload > /dev/null 2>&1 || true

    endscript

}

/var/log/apache2/access_log_audiodef {

    compress

    size 1M

    rotate 2

    missingok

    sharedscripts

    postrotate

        /etc/init.d/syslog-ng reload > /dev/null 2>&1 || true

        /etc/init.d/apache2 reload > /dev/null 2>&1 || true

    endscript

}

```

There are several more similar entries. Basically, the only thing I want done differently for each vhost is the number of rotates.

----------

## BitJam

 *audiodef wrote:*   

> That brings up another question, though: is "reload" enough for apache to release its hold on files and clear things up, versus an outright restart?

 The "reload" is a very good way to do it.  It does a graceful restart which is explained here:

https://httpd.apache.org/docs/2.2/stopping.html#graceful

----------

## audiodef

 *BitJam wrote:*   

>  *audiodef wrote:*   That brings up another question, though: is "reload" enough for apache to release its hold on files and clear things up, versus an outright restart? The "reload" is a very good way to do it.  It does a graceful restart which is explained here:
> 
> https://httpd.apache.org/docs/2.2/stopping.html#graceful

 

Thanks.   :Smile:  That bypasses whatever is causing apache to stutter when I try a hard restart after it's been running for a while.

----------

