Logging of appliance caching activity can be useful for a number of reasons, such as billing for services rendered. The iChain Proxy Server lets you specify how often a new log file is started (rolled over), how long old log files are retained, and the format of the log files.
iChain offers the following logging services:
You can turn on logging for reverse proxy as well as for URL filtering.
You can have the appliance automatically download files to an FTP server and automatically delete downloaded files.
You can control the deletion of old log files based on an older-than-x time period or the number of log files in the system.
The appliance can create logs using both the common and extended log formats. A wide variety of tools exists for manipulating and processing these files.
iChain Proxy Services provides a high performance proxy cache system capable of handling thousands of transactions per second. Although the iChain Proxy Server can log extensive details for each transaction, and although the disk space reserved for log files is quite generous on most appliances, if transaction volume is high and log entries consume a few hundred bytes each, iChain Proxy Services can fill up the available disk space in a matter of minutes.
This section explains how appliance logging works and presents management options you can use to ensure optimal use of the available log file disk space and timely migration (downloading) of log files to other storage devices.
The following table shows the transactions the appliance can log and the formats available for each service type.
Turning on logging for a given service increases system overhead and causes some degradation of performance. Therefore, logging should be used only when service transactions must be tracked for customer billing purposes or other compelling reasons.
Transaction volume and log entry size can cause available log disk space to fill up quickly. Proxy cache disk space is unaffected by log files.
See Planning Step 3: Calculating Log Rollover Requirements for formulas you can use to estimate how quickly your logging disk will fill.
To plan a logging strategy you must know the capacity and limitations of your appliance.
Logging disk space is not user-configurable; it is preset to 1 gigabyte on the iChain Proxy Server. Plan to download and delete log files before the disk space is filled.
iChain Proxy Services does not allow the deletion of active log files (files that are currently in use by the caching system). Only log files that have been rolled over and closed can be deleted.
You can ensure that there are closed files on the system by scheduling regular rolling over of log files. During each rollover, the current log file is closed and a new log file is opened.
You must plan for log files to roll over on a schedule that coincides with your download and deletion schedule. This is to ensure that there is at least one closed log file per service when the download and delete cycle starts.
NOTE:Although you can download active log files, this is normally useful only for periodic administrative checks.
Active files contain only the transaction data up to the moment of the download and are incomplete from customer-billing and other business standpoints.
When the appliance encounters a log disk full condition, it stops logging and closes all active log files. Information that would have been logged after that point is lost. Other appliance functions continue without interruption.
The appliance automatically generates log filenames as follows:
Six numbers representing the year, month, and day the file was created
A dash separating the date from a single letter identifier. The dash is not included after the letters double.
Letter identifiers running from A through ZZ
This naming convention accommodates up to 702 log files per service per day. If the rollover options are set so that all the possible filenames are used in one day, the log file with the ZZ letter identifier is not closed until the start of the next day unless the logging disk becomes full.
Appliance log rollover options let you specify when the appliance closes active log files and opens new files so that the closed files can be downloaded and deleted.
Because of the limitations explained in this section, it is essential that you develop a solid log file rollover plan. This ensures that your appliance doesn't run out of logging disk space or overwrite log files before they are downloaded and deleted.
You can have the appliance roll log files over according to time or file size as explained in the following table:
As explained in The Costs of Logging and System Constraints for Logging, logging of caching transactions involves system and maintenance overhead. If your situation requires logging, you should plan carefully so that the information you are tracking aligns with specific requirements. This ensures optimal use of appliance resources.
Because logging requirements and transaction volume vary widely, it is impossible to make recommendations regarding specific logging strategies.
The following sections step you through the logging strategy planning process. We recommend you record the information you gather on a planning sheet of some kind.
To plan a logging strategy, you should first determine the requirements behind the need for logging.
Identify the business and other reasons for tracking service transactions.
Possible examples might include customer billing requirements, statistical analysis, growth planning, etc.
Determine which services you need to track.
Record this information for further reference.
If you use the common log format, log entry size is fixed. If you use the extended log format, log entry size depends on the number of log fields selected.
Complete the following steps for each service you need to track:
Record which log fields must be tracked for each service to be logged.
Carefully scrutinize the information you plan to track to ensure that the log data collected is essential.
For example, if you select URI, selecting URI-stem and URI-query would be redundant because URI = URI-stem + URI-query. Also, logging cookie information can consume a lot of space and might not provide critical information.
A few bytes can add up quickly when the appliance is tracking thousands of hits every second.
As explained in Log Rollover Options, you can have the appliance roll over log files based on time or on size, but not both.
If you already know which option you want to use, scan this section and then complete the calculations pertinent to your choice.
If you don't know which option best matches your situation, completing the calculations in this section should help you decide.
The following variables are used in the formulas:
logvol_size: The total disk capacity reserved for log files on your appliance.
The default size is 1 gigabyte.
logentry_size: The average log entry size.
You can determine this by configuring your appliance to track the required information, generating traffic to the appliance, downloading the log files, determining how large each entry is, and calculating the average.
request_rate: The peak rate of requests per second.
You can estimate this rate or place your appliance in a service and get more accurate data by accessing the browser-based management tool's Monitoring pages.
num_services: The number of services for which you plan to enable logging.
logs_per_service: The number of log files, both active and closed, that you want the appliance to generate for each service before the disk fills.
You must plan to have at least two log files per service before the disk is filled. See Rolling Over Log Files Before Deletion.
Using the following formula, you can calculate how long it will take the appliance to fill your logging disk space:
diskfull_time seconds = logvol_size / (request_rate * logentry_size * num_services)
For example, if you assume the following:
logvol_size = 1 GB
request_rate = 1000 requests per second
logentry_size = 1 KB
num_services = 1
Then diskfull_time = (1 GB) / (1000 * 1KB * 1) = 1048 seconds (17.47 minutes).
The logging disk space will fill up every 17.47 minutes.
If this time is too short, you must reduce the log entry size by configuring the appliance to log less information per transaction. This is because you can’t increase the disk space or limit the requests being logged.
To calculate the diskfull_time for your appliance:
Determine the values of the four variables listed in the bullet list above.
For more information, refer to Variable Definitions.
Using the diskfull_time formula, calculate how often you can expect your logging disk to fill, then use the result in Calculating MAX_ROLL_TIME.
Using the following formula, you can calculate the maximum roll-over time value you should specify in the Rollover Every field of the Log Options dialog box.
max_roll_time = diskfull_time / logs_per_service
For example, if you assume the following:
diskfull_time = 12 hours
logs_per_service = 2
Then max_roll_time = 12 / 2 = 6 hours.
If you roll over your logs by time intervals, the maximum time should be less than six hours. Otherwise scheduling the download and deletion of log files is much more complicated and the window in which this can be done is narrower.
To calculate the max_roll_time for your appliance:
Determine how many log files you want the appliance to generate per service before log space fills.
The minimum number is two.
Using the max_roll_time formula and the diskfull_time value obtained in Calculating DISKFULL_TIME, calculate how often you should have the appliance roll over the log files.
Record the max_roll_time result on your planning sheet.
Using the following formula, you can calculate the maximum log file size you should specify in the Rollover When File Size Reaches field of the Log Options dialog box.
max_log_roll_size = logvol_size / (num_services * logs_per_service)
For example, if you assume the following:
logvol_size = 600 MB
num_services = 2
logs_per_service = 3
Then max_log_roll_size = 600 MB / (2 * 3) = 100 MB.
If you roll over your logs when they reach a specific size, the file size must be no more than 100 MB. Otherwise, the system runs out of disk space before you have three complete log files, and scheduling the download and deletion of log files is much more complex.
To calculate the max_log_roll_size for your appliance:
Determine the values of the three variables in the bulleted list above.
Using the max_log_roll_size formula, calculate the maximum size a log file should reach before the appliance rolls it over.
Record the max_roll_time result on your planning sheet.
Based on the planning you have completed in Planning Your Logging Strategy, you must now configure the log options for each affected service.
For each service you are logging, open the Log Options dialog box in the browser-based management tool.
Refer to the services you selected in Planning Step 1: Determining Your Logging Requirements.
The path for the Web Server Accelerator service is Configure > Web Server Accelerator > Insert > Enable Logging for This Accelerator > Log Options.
The following sections discuss each of the areas within the Log Options dialog box.
In the Log Options dialog box, specify the log format for the service based on the planning you did in Planning Step 2: Selecting a Log File Format and Optimizing the Log Entry Size.
Remember that each bit of information you log increases the size of each log entry, and affects the rate at which logging disk space is used.
In the Log Options dialog box, specify how the appliance rolls over the log files for the service based on the planning you did in Planning Step 3: Calculating Log Rollover Requirements.
You must schedule regular download and deletion of log files to avoid running out of log disk space.
Whenever possible, we recommend you use the FTP log push feature for this task (see About the FTP Log Push Feature). However, you can also manage log files manually using the browser-based management tool or the appliance's Mini FTP Server. See Manually Downloading and Deleting Log Files.
The appliance also provides three options for dealing with old files as a failover precaution.
Limit Number of Files To: This option lets you limit the total number of log files retained for each service. After the limit for each is reached, the oldest file for the service is deleted each time a new file is created. All logging data in deleted files is lost.
Delete Files Older Than: This option lets you configure the appliance to delete files when they are older than the time you specify. All logging data in deleted files is lost.
Do Not Delete: This option is not recommended because it can lead to a disk full condition if files are not manually downloaded and deleted. If, however, the older logging data has more value for some reason, this option preserves the oldest log files unless you manually delete them or specify their deletion in the FTP Log Push Configuration dialog box.
To specify how the appliance handles older files:
In the Log Options dialog box, select an old file option that matches your requirements. (To review option specifics, see the bullet list above.)
As with log format and rollover options, you can specify different old file options for each service. We recommend, however, that you avoid potential confusion by using the same old file settings for each service.
(One time only) Click FTP Log Push > configure the FTP log push options.
For help with setting options in the FTP Log Push Configuration dialog box, refer to Using FTP Push to Automatically Download and Delete Log Files, then return to this procedure.
In the Log Options dialog box (see Configuring Logging Options), double-check the Old File Options settings against either your FTP log push configuration or your schedule for manual download and deletion to ensure that log files won't reach the deletion threshold (number or age) prior to a scheduled download and deletion.
If you need to configure log options for other services, return to Configuration Step 1: Opening the Appropriate Log Options Dialog Box; otherwise, continue with the next section.
Ideally, iChain Proxy Services will never actually use the old file option you select because you scheduled the downloading and deleting of log files so that the system never becomes full. Two of these options automatically dispose of older files to avoid the disk full condition. The third option is not recommended for most situations.
As with all appliance operations, you should monitor what is happening with your logging strategy over time and make adjustments and refinements if necessary.
Ensure that all the logging information you are gathering is being used. If not, you might be able to further reduce your logging record size.
Ensure that your log file sizes match the estimated averages you used to plan your log file roll-over strategy. If not, you might need to adjust the frequency or even the method used to trigger log file rollover.
Ensure that your logging strategy is leaving a buffer of free log disk space adequate for possible surges in appliance traffic.
Ensure that the external storage capacity (FTP server or other storage) is adequate.
Ensure that all aspects of your logging strategy are keeping pace with increases in traffic through the appliance.
The FTP Log Push feature lets you configure the appliance to push log files to an FTP server at specified intervals: on the first day of the month and/or on specified days of the week. However, log files cannot be pushed more often than once a day.
The feature operates within the following parameters:
iChain Proxy Services tries as many times as necessary to establish one connection with the FTP server during the hour of the scheduled push. When the hour changes, iChain Proxy Services stops trying until the next interval you have specified.
When a connection with the FTP server is established, iChain Proxy Services assumes that the pushing of log files is successful. Any errors that prevent the successful pushing of log files are not detected by iChain Proxy Services.
For example, you specify that log files are to be pushed on every day of the week at 12 midnight. When the system clock reaches the target hour, iChain Proxy Services begins trying to establish a connection with the FTP server.
If a connection cannot be established before the hour changes to 1 a.m., iChain Proxy Services stops trying to connect and doesn’t try again until 12 midnight the next day.
If a connection is established but an error occurs that prevents a successful push, the error is not detected, and iChain Proxy Services doesn’t try to connect again until 12 midnight the next day.
To configure your appliance to use the FTP Log Push feature:
In the browser-based management tool, access the FTP Log Push Configuration dialog box by clicking FTP Log Push on any of the Log Options dialog boxes.
Paths to the dialog boxes are summarized under Configuring Logging Options.
IMPORTANT:Although the FTP Log Push Configuration dialog box is accessed through one of the service-specific Log Options dialog boxes, it is unaffected by the path used to reach it. The settings you specify affect all the log types you select in the box.
This lets you set the FTP push options for all log types in a single place.
In conformance with your logging strategy, specify the following information:
Which log file types to push (all of the log types to be managed through FTP push must be selected).
Your FTP server information.
The method the appliance uses for determining when to push log files.
If your FTP server is always available, we recommend using the Push Logs When the Logs Roll Over option rather than setting specific download times. This protects your appliance from sudden surges in traffic, which can fill the disk sooner than expected.
Whether the appliance should delete the files from the log disk after they have been pushed.
We recommend deleting log files after they have been pushed unless there is a compelling reason for manually deleting them. Automatic deletion also protects your appliance from sudden surges in traffic.
When you have configured your FTP log push options, click OK to return to the Log Options dialog box of the service you are configuring.
Whenever possible, we recommend that you use the FTP Log Push feature to automatically download and delete log files. See Using FTP Push to Automatically Download and Delete Log Files.
If you need to manage your log files manually, we recommend that you establish a regular schedule and ensure that all those responsible for downloading and deleting log files know the following information:
When log files are to be downloaded and deleted
How to determine the name of each log file to be downloaded and deleted
Where to save the log files
You should develop specific procedures for your situation. The following sections contain general ideas for accomplishing these tasks.
The primary consideration is that log files must be downloaded and deleted before the logging disk space fills up.
Before you can download or delete a log file you must know its exact name.
Appliance log filenames can be listed in the browser-based management tool in Monitoring > Cache Logs. They can also be listed from the command line, or through a Telnet session using the get command.
The appliance automatically generates log filenames as follows:
Six numbers representing the year, month, and day the file was created
A dash separating the date from a letter identifier
Letter identifiers running from A through ZZ
This naming convention accommodates up to 702 log files per day. If the rollover options are set so that all the possible filenames are used in one day, the log file with the ZZ letter identifier is not closed until the start of the next day.
To list log files using FTP, you must know the path to the files. Use the following table to determine the paths to various log files.
You can most easily view log filenames in the browser-based management tool. To do so, click Monitoring > click Cache Logs > select a log format > select a service.
The Mini FTP Server supports the CWD command for changing to the target log directories. You can also use the LS command in connection with full paths to list log files.
For example, the following command lists transparent and forward proxy log files in common format:
ls log:etc/proxy/data/logs/forward/common/
For a complete list of log file directory paths, see Getting Log Filenames.
You can also see a list of log filenames from the command line. However, you cannot download files from the command line.
The following table presents some command line/Telnet examples.
You can download the files in the browser-based management tool as you view them. After you click Download, when the browser asks what you want to do with the file, save it to your designated log file storage location.
You can use FTP from the storage location to retrieve the files with the get command. You must first obtain each filename by using one of the options explained in Getting Log Filenames.
After you have the log filename, you can transfer the file to your workstation. For example, to download a forward proxy common format log file, you would use the following command after starting an FTP session with the appliance:
get log:/etc/proxy/data/logs/forward/common/filename.log
The filename variable is the name of the log file you have previously obtained.
You can also use the mget command, but this command also downloads active log files that are not complete.
The appliance doesn't currently support the FTP server put command.
The following is a list of supported FTP commands:
TYPEUSERDELEQUITPWDSYST
After the log files have been downloaded and saved to another location, delete the files using one of the following options:
The Delete button in the browser-based management tool
The del command in FTP
The following information about field values in extended log files might help you interpret the content of the files.
Fields within the file are delimited by the tab character.
A field can be one of two types: string or non-string.
String fields are enclosed in quotation marks (").
If a string field contains a quotation mark, that character is repeated once for every occurrence to enable unambiguous file parsing.
If a string field has no value, it is represented by two quotation marks (“”).
Non-string fields containing no value are represented by a hyphen (-).
Field headers starting with s- are associated with the appliance.
Field headers starting with c- are associated with the client/browser.
Field headers starting with sc are associated with flow from the appliance to the client/browser.
Field headers starting with cs are associated with flow from the client/browser to the appliance.
The information in the following table is supplementary to the W3C Extended Log Format Specification found on the Extended Log File Format Web site. You might find it useful for interpreting the content of extended log field headers.
The logs for authorized access attempts and unauthorized access attempts can be turned on or off globally. The rules for logging authorized access attempts for an individual access control list (ACL) can also be turned on or off. However, because unauthorized access attempts are usually the result of a user not being defined in any ACL rule, logging of unauthorized access attempts cannot be turned on or off for individual ACL rules.
To enable or disable ACL rule logging on a global level:
Access the URL of the iChain Proxy Server where you installed the iChain Proxy Services software to launch the proxy server browser-based administration tool.
For example, http://xx.xx.xx.xx:1959/appliance/config.html where xx.xx.xx.xx is the IP address.
Click Configure > Web Server Accelerator > Modify.
Select the Enable Logging check box.
To enable or disable ACL rule logging for an individual ACL rule:
In ConsoleOne, right-click an ACL Rule object.
Select Properties.
Check the Authorized Logging check box.
The ACL log files for each 24-hour period are saved to log:\etc\proxy\data\logs\reverse\extended\aclcheck\yymmdd-a.log, where yymmdd is today's date represented by two digits for the year, month and date. The default maximum size of the file is 1 MB. The default size can be changed; see Using ACLCHECK options for more information. If a log exceeds the maximum size, a new file named yymmdd-b.log is created.
Each file contains the following fields:
Date
Time
Source IP address
Destination IP address
Protocol
Source port
Destination port
TCP flag
Access (Allow=1, Deny=0)
IP headers
IP payload
Username
Destination host name
URL (the user requested)
Rule Object name (if access was allowed; if denied, the field displays a —)