# Stopping Syslogd/Syslog-ng from repeating itself...

## humbletech99

I've got a stack of server machines running either syslogd or syslog-ng all logging to a central server but I want to know how to suppress repeated lines. Does anybody know how to do this in either syslog or syslog-ng. The central logger is syslog-ng so perhaps I can't just do it there?

----------

## think4urs11

What exactly do you mean?

Having identical messages logged from the same machine

OR

having e.g. the same message logged from machine A (which also collects from B) and the same message from machine B?

OR

do you mean you get the same message logged into different log files on the central machine because it matches more than one log statement?

In first case - check the client which generates the logs and why he does so.

In second case - reconfigure your clients to only log to the central server (no 'staged' loggers)

In third case - see : https://forums.gentoo.org/viewtopic-p-3234672.html#3234672

----------

## humbletech99

I mean having the same message repeated several times from the same machine. Usually you wouldn't care all that much, it's not like it takes the world of space but on a central syslog collecting from many machines, it can add up. But worse, when you have a tty displaying real time logs then the screen moves up to quickly with duplicate messages that are of no importance, you only need to see it once....

I'm pretty sure that loggers have ways of suppressing dupilcate messages from their logs, I could do this on each machine, or I could do this on the central server (or best to do it on all if I can).

----------

## Antimatter

I'm interested in this also, but I haven't heard of any option/setting that does what you're asking, so at the moment I think the best way to delete the duplicate lines, is to run the log though a shell script processor that will grep the log and then cut/remove duplicate lines that are next to each others or something alikin to this.

----------

## humbletech99

Fair enough, but I want to change to logging into a mysql database on the central syslog server. 

Also, my console displays logs in real time and duplicate lines send the logs off screen faster so I'd like to get rid of this...

----------

## Antimatter

 *humbletech99 wrote:*   

> Fair enough, but I want to change to logging into a mysql database on the central syslog server.
> 
> Also, my console displays logs in real time and duplicate lines send the logs off screen faster so I'd like to get rid of this...

 

To be frank, I wouldn't know how, but I would suspect that if you are going to implement logging into a mysql database, i would assume that it would include scripts/programs that can convert the syslog stuff into the sql database format, so perhaps it maybe possiable to edit some of those script/program to delete repeative logs before its added into the database or something alikin to that?

Could be something like this snipet of pseduo code:

```

var = get a line of log

prev = get the last line of log from the SQL database

if ( var == prev )  then skip to the end and don't instert this line of log into the database

else instert this line of log into the database

```

now repeat that for each line of log that is entered into the sql database.

But for the console, i have no idea how to deal with that one.

----------

## humbletech99

I'm sure that syslog-ng should be able to do this kind of thing... without needing the mysql bits or anything else external like scripts. I just have to find it - then it' help with the tty output...

----------

## JeliJami

on some of our debian servers, the /var/log/syslog file contains lines such as:

```
last message repeated 107 times
```

that's probably what humbletech99 is looking for?

the servers that expose that behaviour are:

- Debian 3.1

- syslogger: sysklogd-1.4.1-17

the config file does not seem to contain a special setting for this behaviour, so it is probably a default setting

maybe give that one a try. it is available in portage

----------

## humbletech99

I've also seen the same thing in some logs before which is why I'm sure it can be done (haven't seen 107 repeats though!)

I've kinda already settled on syslog-ng, I don't know if sysklogd is any better or if there is any reason to consider switching, especially since I've spent the time to set up the central server on syslog-ng and half of the clients...

----------

## think4urs11

 *Quote:*   

> #   Many syslog programs, when configured to relay messages on to another syslog program on another host, will leave out certain parts of the syslog message - complicating proper identification of certain fields.
> 
> Imagine this common scenario:
> 
> The syslogd program on a Solaris machine is configured to send all logs to a central syslog server. When Solaris syslogd sends logs over the network, it doesn't include the hostname field at all. This means that a "last message repeated" message looks more like this:
> ...

 

In other words - from point of view as beeing the central syslog server this is more of a bug than a feature.

No matter if or if not you're able to surpress doubled messages you'll (depending on number of syslog 'clients') never come to a point where a real time console view is useful at all.

Better idea to log everything into a sql database and create appropriate views there. (php-syslog-ng is *very* handy there)

----------

## humbletech99

Thanks for the recommendation, I'm actually considering this. On the point of the console view though, every single host is sending it's hostname and timestamp info. All the gentoo servers are now on syslog-ng (got rid of that weak metalog) but the rest of the suse servers <spit> are on the old syslogd which also seems to be fine with sending the headers to the central server.

----------

