# cron.weekly scripts don't send by email until I reboot

## Robert S

I have a a number of cron.weekly scripts analyse logs, back up my system and update a few things (eg sa-learn).  At the end is a script called zz_logrotate that runs logrotate - so that it runs last.  The scripts produce output which is emailed to me.

Since I have had this setup I haven't been getting sent any email notification of the output of cron.weekly scripts.  Looking in my /var/log directory, my logs have been rotated, indicating that the last (zz_logrotate) script has been run.  If I reboot my system however I get sent all the output my cron.weeky scripts.  My cron.daily scripts get sent fine.

I assume that none of my cron.weeky scripts are hanging because the last script is run.  Also `ps ax' and top do not indicate any hanging scripts.

I use sendmail.

I nobody can provide an answer can somebody suggest where I can start looking?  /var/spool/cron and /var/spool/at don't help.

----------

## lxg

If you execute the scripts manually, do they run and work as expected? If so, can you please post the output of:

```
ls -la /etc/cron.*

cat /etc/crontab

cat /usr/sbin/run-crons
```

Btw, which cron implementation are you using, vixie-cron?

----------

## Robert S

Hi.

Here it all is.  I use vixie-cron.  Its not possible that something in cron.daily is happening before cron.weekly??  The run-crons script seems to indicate that weekly runs after hourly and daily.

I've just run all of the scripts and they all finish cleanly.  There are however some errors in on of them - which I will investigate.

```
# ls -la /etc/cron.*

-rw-r--r-- 1 root root  220 May 27 14:31 /etc/cron.deny

/etc/cron.d:

total 12

drwxr-xr-x  2 root root 4096 May 30 12:39 .

drwxr-xr-x 66 root root 4096 Jul 24 20:07 ..

-rw-r--r--  1 root root    0 May 27 14:31 .keep_sys-process_vixie-cron-0

-rw-r--r--  1 root root  157 May 30 12:39 logcheck_reboot

/etc/cron.daily:

total 76

drwxr-x---  2 root root 4096 Jul 13 23:21 .

drwxr-xr-x 66 root root 4096 Jul 24 20:07 ..

-rwx--x--x  1 root root  962 Jun  2 10:19 aide.daily

-rwx--x--x  1 root root 2565 Jun  2 09:00 backup.daily

-rwx------  1 root root  382 Jun  2 10:09 chkrootkit.cron

-rwx------  1 root root  630 Jun  2 10:46 chksuid.daily

-r-xr-xr-x  1 root root 7386 May 27 21:34 dccd

-rwxr-xr-x  1 root root   32 Jun  2 10:12 eix

-rwx------  1 root root 1272 Jun  2 11:31 filesec.daily

-rw-r--r--  1 root root    0 May 14 20:05 .keep

-rw-r--r--  1 root root    0 May 27 14:28 .keep_sys-process_cronbase-0

-rwxr-xr-x  1 root root  220 Jun  2 16:52 logsentry.cron

-rwxr-xr-x  1 root root  115 May 15 04:11 makewhatis

-rwxr-xr-x  1 root root 4050 May 27 21:36 rkhunter

-rwxr-xr-x  1 root root  955 Jun 11 20:45 sa-update

-rwx------  1 root root  127 Jun  2 10:13 settime.cron

-rwxr-xr-x  1 root root  152 May 27 14:31 slocate

-rwxr-xr-x  1 root root  565 Jun  2 10:16 syncphotos

-rwxr-xr-x  1 root root 2178 Jul 13 23:21 tmpwatch

/etc/cron.hourly:

total 20

drwxr-x---  2 root root 4096 Jun  2 17:04 .

drwxr-xr-x 66 root root 4096 Jul 24 20:07 ..

-rwx--x--x  1 root root  988 Jun  2 16:53 clamdchk.cron

-rwxr-xr-x  1 root root  453 Jun  2 16:53 diskfull.cron

-rw-r--r--  1 root root    0 May 14 20:05 .keep

-rw-r--r--  1 root root    0 May 27 14:28 .keep_sys-process_cronbase-0

-rw-r--r--  1 root root  169 Jun  2 16:53 logsentry.cron

/etc/cron.monthly:

total 16

drwxr-x---  2 root root 4096 Jul  4 10:06 .

drwxr-xr-x 66 root root 4096 Jul 24 20:07 ..

-rwxr-xr-x  1 root root  356 May 29 17:57 geoip-update

-rw-r--r--  1 root root    0 May 14 20:05 .keep

-rw-r--r--  1 root root    0 May 27 14:28 .keep_sys-process_cronbase-0

-rwxr-xr-x  1 root root  299 Jul  4 10:00 mysql-delete-logs

/etc/cron.weekly:

total 52

drwxr-x---  2 root root 4096 Jul 12 20:53 .

drwxr-xr-x 66 root root 4096 Jul 24 20:07 ..

-rwx--x--x  1 root root 2966 Jun  7 07:38 backup.cron

-rwxr-xr-x  1 root root 1072 May 31 12:36 browscap

-rwxr-xr-x  1 root root   78 Jun  3 07:13 chkrootkit

-rwx--x--x  1 root root  768 May 31 12:36 glsa.cron

-rwxr-xr-x  1 root root  265 May 30 13:19 iptables-by-country

-rw-r--r--  1 root root    0 May 14 20:05 .keep

-rw-r--r--  1 root root    0 May 27 14:28 .keep_sys-process_cronbase-0

-rwxr--r--  1 root root 2045 Jun 20 08:58 spamlearn

-rwxr-xr-x  1 root root 2484 Jul 12 20:53 spam-stats

-rwxr-xr-x  1 root root  143 May 27 21:27 squid.cron

-rwxr-xr-x  1 root root  250 May 31 12:36 test-botnet

-rwxr-xr-x  1 root root 3934 Jun  2 17:46 virus_report

-rwxr-xr-x  1 root root   52 May 27 14:45 zz_logrotate.cron

```

```
# cat /etc/crontab

# for vixie cron

# $Header: /var/cvsroot/gentoo-x86/sys-process/vixie-cron/files/crontab-3.0.1-r4,v 1.2 2009/05/12 09:13:46 bangert Exp $

# Global variables

SHELL=/bin/bash

PATH=/sbin:/bin:/usr/sbin:/usr/bin

MAILTO=root

HOME=/

# check scripts in cron.hourly, cron.daily, cron.weekly and cron.monthly

59  *  * * *    root    rm -f /var/spool/cron/lastrun/cron.hourly

9  3  * * *     root    rm -f /var/spool/cron/lastrun/cron.daily

19 4  * * 6     root    rm -f /var/spool/cron/lastrun/cron.weekly

29 5  1 * *     root    rm -f /var/spool/cron/lastrun/cron.monthly

*/10  *  * * *  root    test -x /usr/sbin/run-crons && /usr/sbin/run-crons

```

```
LOCKDIR=/var/spool/cron/lastrun

LOCKFILE=${LOCKDIR}/lock

mkdir -p ${LOCKDIR}

# Make sure we're not running multiple instances at once.

# Try twice to lock, otherwise give up.

for ((i = 0; i < 2; i = i + 1)); do

        ln -sn $$ ${LOCKFILE} 2>/dev/null && break

        # lock failed, check for a running process.

        # handle both old- and new-style locking.

        cronpid=$(readlink ${LOCKFILE} 2>/dev/null) ||

        cronpid=$(cat ${LOCKFILE} 2>/dev/null) ||

        continue        # lockfile disappeared? try again

        # better than kill -0 because we can verify that it's really

        # another run-crons process

        if [[ $(</proc/${cronpid}/cmdline) == $(</proc/$$/cmdline) ]] 2>/dev/null; then

                # whoa, another process is really running

                exit 0

        else

                rm -f ${LOCKFILE}

        fi

done

# Check to make sure locking was successful

if [[ ! -L ${LOCKFILE} ]]; then

        echo "Can't create or read existing ${LOCKFILE}, giving up"

        exit 1

fi

# Set a trap to remove the lockfile when we're finished

trap "rm -f ${LOCKFILE}" 0 1 2 3 15

for BASE in hourly daily weekly monthly ; do

        CRONDIR=/etc/cron.${BASE}

        test -d $CRONDIR || continue

        if [ -e ${LOCKDIR}/cron.$BASE ] ; then

                case $BASE in

                hourly)

                        #>= 1 hour, 5 min -=> +65 min

                        TIME="-cmin +65" ;;

                daily)

                        #>= 1 day, 5 min -=> +1445 min

                        TIME="-cmin +1445"  ;;

                weekly)

                        #>= 1 week, 5 min -=> +10085 min

                        TIME="-cmin +10085"  ;;

                monthly)

                        #>= 31 days, 5 min -=> +44645 min

                        TIME="-cmin +44645" ;;

                esac

                find ${LOCKDIR} -name cron.$BASE $TIME -exec rm {} \; &>/dev/null || true

        fi

        # if there is no touch file, make one then run the scripts

        if [ ! -e ${LOCKDIR}/cron.$BASE ] ; then

                touch ${LOCKDIR}/cron.$BASE

                set +e

                for SCRIPT in $CRONDIR/* ; do

                        if [[ -x $SCRIPT && ! -d $SCRIPT ]]; then

                                $SCRIPT

                        fi

                done

        fi

done

# Clean out bogus cron.$BASE files with future times

touch ${LOCKDIR}

find ${LOCKDIR} -newer ${LOCKDIR} -exec /bin/rm -f {} \; &>/dev/null || true

```

Last edited by Robert S on Sat Jul 24, 2010 10:14 pm; edited 1 time in total

----------

## lxg

Ouch, you used quote instead of code, so it's a bit hard to read.  :Wink: 

However, everthing looks ok so far. All scripts seem to be executable. Your run-scripts looks different than mine, but I assume you just cut the Changelog away. Crontab looks ok, too.

Is it possible that something in your scripts doesn't work when they're executed from a non-login shell? For example, do they rely on something that is set/loaded in ~/.bash_profile?

Maybe you could post an example script, in order to have a look – without invading your privacy. (So, don't post the backup script for your p*** collection.  :Wink: )

----------

## Robert S

I made the following with crontab -e (as root):

```
# Global variables

SHELL=/bin/bash

PATH=/sbin:/bin:/usr/sbin:/usr/bin

MAILTO=root

HOME=/

# check scripts in cron.hourly, cron.daily, cron.weekly and cron.monthly

#20 8 25 7 7  root    test -x /usr/sbin/run-crons && /usr/sbin/run-crons

43 8 * * *  for SCRIPT in /etc/cron.weekly/*; do [ -x $SCRIPT ] && $SCRIPT; done

```

It ran and emailed me the output - so far so good   :Confused: 

Here's a bit of my woeful code.  It had a few errors in it that I've fixed.  I've made changes today that make the script run much more quickly.  Despite this, the old script DID run on Saturday morning (looking at the custom error logs).  Maybe its fixed the problem??:

```
#!/bin/bash

## /etc/cron.weekly/spamlearn

LOGFILE=/var/log/sa-learn/sa-learn.log

ERROR_LOGFILE=/var/log/sa-learn/sa-learn-error.log

MAILDIR=.maildir

## Set up log

echo -n "Spamlearn started at " >> $LOGFILE

echo `date` >> $LOGFILE

echo -n "Spamlearn started at " >> $ERROR_LOGFILE

echo `date` >> $ERROR_LOGFILE

## Per-user spam learning

for n in `cat /etc/passwd |grep -v root |sed s/:.*//`

  do

## Only do this if user has set up Spam folder

  if [ -d /home/$n/$MAILDIR/.Spam ]

   then

   echo Spamlearn started at `date` for $n 2>> $ERROR_LOGFILE >> $LOGFILE

   [ ! -d /home/$n/.razor ] && su $n -c "razor-admin -create"

## Learn ham

   echo "Learning ham" 2>> $ERROR_LOGFILE >> $LOGFILE

   LIST=''

   for N in `find /home/$n/$MAILDIR/cur/ -mtime +7 -and -mtime -14 -type f `

      do

      LIST+="$N "

      done

   [ "" != "$LIST" ] && su $n -c "sa-learn --ham $LIST 2>> $ERROR_LOGFILE >> $LOGFILE"

## Learn spam

   echo "Learning spam" 2>> $ERROR_LOGFILE >> $LOGFILE

   LIST=''

   for N in `find /home/$n/$MAILDIR/.Spam{,-12}/{cur,new}/ -mtime +7 -and -mtime -14 -type f`

      do

      LIST+="$N "

      done

   [ "" !=  "$LIST" ] && su $n -c "sa-learn --spam $LIST 2>> $ERROR_LOGFILE >> $LOGFILE"

## Fix autowhitelist

   echo "Removing entries from auto-whitelist for $n" 2>> $ERROR_LOGFILE >> $LOGFILE

   for M in $(for N in `find /home/$n/$MAILDIR/.Spam{,-12}/{cur,new}/ -mtime +7 -and -mtime -14 -type f`; do grep ^From: $N|sed "s/.*From: //; s/.*<//; s/>$//"; done | sort |uniq)

      do

      [ "" != "$M" ] && su $n -c "spamassassin --remove-addr-from-whitelist $M 2>> $ERROR_LOGFILE >> $LOGFILE"

      done

   fi

  done

## Complete log

echo -n "Spamlearn completed at " >> $LOGFILE

echo `date` >> $LOGFILE

echo -n "Spamlearn completed at " >> $ERROR_LOGFILE

echo `date` >> $ERROR_LOGFILE

```

Can you think of any other way I can test this without waiting for next Saturday?  Note - my cron.daily scripts DO produce an email of the output, so I assume that they are working fine.

----------

## lxg

Can you move the scripts into cron.hourly? Then you would get the chance to test more often than once a week and you would see if it has really to do with the cron.weekly part.

Apart of that, I don't know. What I would do, is some good old try'n'error … Commenting out the entire script, and only run specific sections, to see if they work. Or insert one line for testing purposes (e.g. to mail you something).

What I usually do when I'm totally desperate  :Wink:  with a script's misbehaviour, I insert some “break points”:

```

#!/bin/bash

…

TESTFILE="/root/mytestfile.txt"

echo '' > $TESTFILE

…

echo "Step 1 at $(date +%s)" >> $TESTFILE

…

echo "Step 2 at $(date +%s)" >> $TESTFILE

…

echo "Step 3 at $(date +%s)" >> $TESTFILE

… 

```

This way, you can see if it works at all, how long it takes between the steps and if it breaks at a certain step.

----------

## Robert S

I have discovered, after a long time, that vixie-cron is the likely problem.

 *Quote:*   

> # ps ax |grep cron
> 
> 14022 ?        S      0:00 /usr/sbin/cron
> 
> 29512 ?        Ss     0:00 /usr/sbin/cron

 

 *Quote:*   

> # /etc/init.d/vixie-cron restart
> 
>  * Stopping vixie-cron ...       [ ok ]
> 
>  * Starting vixie-cron ...        [ !! ]

 

 *Quote:*   

> # # lsof |grep cron
> 
> cron      14022      root  cwd       DIR              254,0     4096     133918 /var/spool/cron
> 
> cron      14022      root  rtd       DIR                8,1     4096          2 /
> ...

 

 *Quote:*   

> # ps ax |grep cron
> 
> 14022 ?        S      0:00 /usr/sbin/cron
> 
> # kill 14022

 

Then the message gets sent.

Does anyone know what is going on here?

----------

