Introduction

 
Every time an issue/bug is reported at a customer’s place on a staging or production environment, the numero uno question asked from any developer or supporting guy is the application log. In Access Gateway(AG), managing logs is done using Advanced options. From the AG documentation severity of logs are configured using option:

LogLevel <loglevel>
Possible loglevel values are emergalertcriterrorwarnnotice, info or debug

The most favored loglevel for most of the developers is “debug”, but the recommended or default loglevel is “warn” for the obvious reason of negative impact on application performance. AG’s logging in debug mode not only increases the size of the log files quickly, in a highly loaded system, it can also lead to request or connection timeout and can slow down the response time.

The biggest drawback of setting AG’s advanced option is the automatic restart of the AG process/services.

The Solution

 
This cool solution will allow the user to change the severity level of logs on the fly, without restarting AG. This will help developers to debug an intermittent issue by enabling the debug level of logs only when needed without restarting Access Gateway process/services and avoid adverse impact of setting debug log level all the time. It would also help in the situation where the issue disappears as soon as services are restarted.

Steps to follow:

 
On an SLES/RHEL environment, Access Gateway sends logs to syslog. This cool solution takes help of syslogs’s filter mechanism. Depending on which syslog (syslog-ng or rsyslog) is being used, the steps will differ. Access Gateway on SLES-11 uses syslog-ng whereas on SLES-12 & RHEL it is rsyslog.

First and foremost, make sure the AG advance option for logging is set to:

LogLevel debug

For syslog-ng

  1. Login to AG system
  2. Open syslog conf file
    vim /etc/syslog-ng/syslog-ng.conf
  3. Comment out all occurrences of the following line
    #log { source(src); filter(f_user); destination(agsmessages); flags(final); };
  4. Go to MAG section update as follows. Newly added lines are between #cool-solution-start & #cool-solution-end. The rest are existing lines.
    #MAG
    filter f_user { facility(user); };
    destination agsmessages { file("/var/log/novell-apache2/error_log"); };
    #log { source(src); filter(f_user); destination(agsmessages); flags(final); };
    #cool-solution-start
    filter f_user_defined { level(warn, error, crit, alert, emerg) and facility(user); }; 
    log { source(src); filter(f_user_defined); destination(agsmessages); flags(final); };
    #cool-solution-end
    
    filter f_local6 { facility(local6); };
    destination httpheaders { file("/var/log/novell-apache2/httpheaders"); };
    log { source(src); filter(f_local6); destination(httpheaders); flags(final); };
    
    filter f_local5 { facility(local5); };
    destination soapmessages{ file("/var/log/novell-apache2/soapmessages"); };
    log { source(src); filter(f_local5); destination(soapmessages); flags(final); };
    
    #
    # Enable this and adopt IP to send log messages to a log server.
  5.  The above setting is the same as the default setting where severity level is set to warn and below. Anytime user is looking for logs with better verbosity,  replace the corresponding filter line (between #cool-solution-start & #cool-solution-end) with any one of the following:
    filter f_user_defined { level(notice, warn, error, crit, alert, emerg) and facility(user) };
    filter f_user_defined { level(info, notice, warn, error, crit, alert, emerg) and facility(user) };
    filter f_user_defined { level(debug, info, notice, warn, error, crit, alert, emerg) and facility(user) };
  6. Similarly, the opposite is also possible when the user wants to reduce the verbosity of logs. Replace the line with lesser log level. For e.g.
    filter f_user_defined { level(crit, alert, emerg) and facility(user) };
  7. Restart syslog service
     service syslog restart

    The above steps allow the user to change the severity of logs on the run without restarting AG processes and services.

Another useful filter that could help to block redundant logs based on string matching is:

filter f_user_defined { level(debug, info, notice, warn, error, crit, alert, emerg) and facility(user) and not match('CFilter'); };

This filter does not allow logs containing ‘CFilter’ string.

For rsyslog

  1. Login to AG box
  2. Open rsyslog’s nam specific conf file
    vim /etc/rsyslog.d/nam.conf
  3. Comment out 2 lines as follows:
    local0.* @@164.99.162.16:1290;ForwardFormat
    & ~
    #user.* -/var/log/novell-apache2/error_log
    #& ~
    local6.* -/var/log/novell-apache2/httpheaders
  4. Open rsyslog conf file
    vim /etc/rsyslog.conf
  5. Search for following line
    if $syslogfacility-text == 'user' then /var/log/novell-apache2/error_log
    

    and replaced with

    if (\
     ($syslogfacility-text == 'user') and \
     ($syslogseverity <= 4) \
    ) \
    then /var/log/novell-apache2/error_log

    Here is the severity code for each log level.

    emerg

    0

    alert

    1

    crit

    2

    error

    3

    warn

    4

    notice

    5

    info

    6

    debug

    7

    So by default $syslogseverity is set to <= 4 (warn). To enable maximum verbosity (debug), set it to $syslogseverity <=7

  6. Restart syslog service
    service syslog restart

    The above steps allow the user to jump between verbosity of AG logs on the fly without impacting AG service/process.

Similar to syslog-ng, to block logs based on string matching, the following example can be taken for reference. To block logs having string “a.com:8181/IDPUpdate” use:

if (($syslogfacility-text == 'user') and ($syslogseverity <= 4) and not ($msg contains 'a.com:8181/IDPUpdate')) then /var/log/novell-apache2/error_log

Please note that searching logs for string would have some impact on performance. More filter options can be found at the syslog documentation. Please remember to restart syslog service every time its configuration is changed.

The syslog configuration guide describes many more rich sets of filter options. Please refer to the official documentation for more detailed info.

Reference

  1. syslog-ng documentation https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide
  2. rsyslog documentation https://www.rsyslog.com/doc/v8-stable/configuration/index.html
1 vote, average: 5.00 out of 51 vote, average: 5.00 out of 51 vote, average: 5.00 out of 51 vote, average: 5.00 out of 51 vote, average: 5.00 out of 5 (1 votes, average: 5.00 out of 5)
You need to be a registered member to rate this post.
Loading...

Disclaimer: As with everything else at NetIQ Cool Solutions, this content is definitely not supported by NetIQ, so Customer Support will not be able to help you if it has any adverse effect on your environment.  It just worked for at least one person, and perhaps it will be useful for you too.  Be sure to test in a non-production environment.

Leave a Reply

No Comments
By: phimdeep
Feb 28, 2019
8:09 am
Reads:
266
Score:
5
Active Directory Authentication Automation Cloud Computing Cloud Security Configuration Customizing Data Breach DirXML Drivers End User Management Identity Manager Importing-Exporting / ICE/ LDIF Intelligent Workload Management IT Security Knowledge Depot LDAP Monitoring Open Enterprise Server Passwords Reporting Secure Access Supported Troubleshooting Workflow