Web Server Log File Analysis -- Basics

After reading this material, you will be able to:

It is easy to fall into the “hits” trap. Hit counters have been around for some time now and hits are an easy statistic to quote. When managing a web site you don’t need to know all the details of log files, but understanding what information is available to you and making your analysis consistent over time is important.

The type and depth of log file analysis you perform depends on the type of Web site you manage. If you are simply providing information, knowing who and how many are visiting your site is interesting information, but not vital to the company. If, however, you site is an e-commerce site with a significant impact on the bottom line, you may want to know how users are moving through the site, especially just before they decide to make a purchase.

Information in Server Logs

Web server logs are plain text (ASCII) files, independent of server platform. There are some differences between server software, but traditionally there are four types of server logs:

1. Transfer (access) log
2. Error log
3. Referer log
4. Agent log

The first two types of log files are standard. The referer and agent logs may or may not be “turned on” at the server or may be added to the transfer log file to create an “extended” log file format. Each HTTP protocol transaction, whether completed or not, is recorded in the logs, and some transactions are recorded in more that one log. For example, most (but not all) HTTP errors are recorded in the transfer log and the error log. Lets take a look at the type of information collected in an “extended format” transfer log file.

Transfer (access) log

The following is an example of a single line in a common transfer log. This typically displays as one long line of ASCII text, separated by tabs and spaces (useful for importing it into a spreadsheet program).

Lets look at each section of this entry.

Address or DNS

This is the address of the computer making the HTTP request. The server records the IP and then, if configured, will lookup the Domain Name Server (DNS). However, with all the dynamically assigned IP addresses these days, you don’t learn as much as you’d expect from the domain name. In this case the visitor seems to be a customer of an ISP, which is located in Santa Monica, California.

RFC931 (or identification)

-

Rarely used, the field was designed to identify the requestor. If this information is not recorded, a hyphen (-) holds the column in the log.

Authuser

-

List the authenticated user, if required for access. This authentication is sent via clear text, so it is not really intended for security. This field is usually filled by a hyphen (-).

Time Stamp

The date, time, and offset from Greenwich Mean Time (GMT x 100) are recorded for each hit. The date and time format is: DD/Mon/YYYY HH:MM:SS. The example above shows that the transaction was recorded at 12:13 pm on May 9, 1999 at a location 7 hours behind GMT.

By comparing time stamps between entries, we can also determine how long a visitor spent on a given page. From the following excerpts from the log, we see that the visitor spent about a minute on the page before moving to the detailed page.

One of three types of HTTP requests is recorded in the log. GET is the standard request for a document or program. POST tells the server that data is following. HEAD is used by link checking programs, not browsers, and downloads just the information in the HEAD tag information. The specific level of HTTP protocol is also recorded.

Status Code

There are four classes of codes

1. Success (200 series)
2. Redirect (300 series)
3. Failure (400 series)
4. Server Error (500 series)

A status code of 200 means the transaction was successful. Common 300-series codes are 302, for a redirect from http://www.mydomain.com to http://www.mydomain.com/, and 304 for a conditional GET. This occurs when the server checks if the version of the file or graphic already in cache is still the current version and directs the browser to use the cached version. The most common failure codes are 401 (failed authentication), 403 (forbidden request to a restricted subdirectory), and the dreaded 404 (file not found) messages. Sever errors are red flags for the server administrator.

Transfer Volume

9887

For GET HTTP transactions, the last field is the number of bytes transferred. For other commands this field will be a hyphen (-) or a zero (0).

The transfer volume statistic marks the end of the common log file. The remaining fields make up the referer and agent logs, added to the common log format to create the “extended” log file format. Lets look at these fields.

Referer URL

The referrer URL indicates the page where the visitor was located when making the next request. The actual request is shown in the last field of the entry

and is duplicated from the HTTP Request, the fifth field in this log.

If you were looking at just the referer log, not integrated into the transfer log, it would be made up of just two fields. The left field is the starting URL and the right field is where the reader went from the URL. Transfers within your site would also show in the transfer log. For example, movement from one page to another within a web site might show in the referrer log as:

The visitor went from the top-level page to information about the next SSI conference through a link on the page.

User Agent

The user agent is information about the browser, version, and operating system of the reader. The general format is:

The confusion comes from the word “Mozilla,” which is the original code name for Netscape. Now almost all browsers compatible with Netscape use the Mozilla code. The following are entries into a recent agent log: 

The one entry not identified by Mozilla is a robot used to test a specific page. Search engine robots are often similarly identified in the agent log. This is one way to find out who is indexing your site. You can also find out how a reader is reaching your site from a referrer log. The following is a typical entry from a user who entered your site from a category-based index, in this case Yahoo. From this information you can detect how your site is categorized.

The following is an entry from a search engine, in this case Alta Vista. From this data you can see the key words used by the visitor to find your site. 

Error Log

The following is a typical example of an error log transaction:

Compared to the access log, the error log is designed for human eyes to decode. It starts with a time stamp, in an entirely different format than the access log, and is followed by a textual description of the error. The time stamp is in the following format:

A entry is made to the error log when there are server startups and shutdowns, access failures (such as 404 error), lost connections, timeouts, and cancellations by the visitor. There is no information about the user. 

Hits, Views, and Visits

Hit counters continue to be popular features on web pages, but they, in fact, have little value. First, most hit counters can be adjusted to start at any number. So any number you see in on a hit counter may be artificial. Second, just what is defined as a hit? In fact, requesting a single web page can result in multiple hits to the server. First, if the requested URL does not have a following slash (such as http://www.mydomain.com), the server will first redirect to the URL with a slash (such as http://www.mydomain.com/). Then the page of html will show as a hit and each graphic on the page will also record as a hit in the log. So a page of html with six graphics in a navigation bar could record eight individual hits in the log. This set of eight hits is described as a view, all the hits necessary to display a web page. The next analysis is to look through the views to recreate the user’s visit to the site – how they got to your site, where they went in the site and how long they spent. 

Exercises

1. Interpreting Transaction Log File Data

Take a look at the following excerpt from a real extended-format web server log file.

Transaction #1

Transaction #2

Transaction #3

1. How many visitors are reflected in these web transactions? Explain.
2. On what date did these transactions take place?
3. How many minutes passed between the first and last entry?
4. Were all three transactions successful? If not, explain.
5. Which transaction requested the largest file? What size was the file?
6. What browsers are being used to access these pages?
7. What platforms are being used to access these pages?
8. Can you determine the path the visitor from 202.41.102.153 took through the website? here

2. Hits, Views, and Visits

Dowload a “sample.log.” This file, in plain ASCII text, is real extended transfer log data from a single visitor to a website we manage. This visit is made up of ninety records. Open the file in a tool of your choice (any text editor or spreadsheet program will open it) and answer the following questions. As supplied, the entries are sorted by time stamp.

1. How many hits could this visit record to a simple hit counter?
2. How many unique pages were viewed during the visit?
3. Were any pages viewed more than once?
4. How long did the visitor spend on the site during that single visit?
5. From this data can you easily track the path the visitor took through your site?
6. As the manager of a web site, which of these bits of data is most useful to know?

3. Analyzing Data From Web Log Analysis Tools

Large web sites can have web files with more than 100,000 daily transactions. Analyzing this data by hand is a difficult task. Consider the following data, collected using WebTrends Professional on one week of data from the Stanford Linear Accelerator Center’s Unix web server (the web site contained over 420,000 pages in June 1999).

1. How many transactions (hits) were recorded for this site for the week? What is the daily range (low and high, excluding error)?
2. How many actual page views were reflected in these hits?
3. How many visits were reflected in these views?
4. How many views per day, on average?
5. How long does the average visit last?
6. If the server needs to be shut down for servicing, what would be the best day of the week to perform that work?
7. Does the number of hits on the least active day fit with the remaining statistics? Explain.

Review Questions

1. Site visitors are identified in which field of the Access Log?

a)_____ Authuser
b)_____ HTTP Request
c)_____ IP Address or DNS
d)_____ Status Code

2. Which of the following indicates a successful transaction in the web log?

a)_____ 200
b)_____ 302
c)_____ 404
d)_____ 500

3. The referer URL shows?

a)_____ the browser, version, and operating system of the reader
b)_____ number of bytes transferred
c)_____ page where the visitor was located when making the next request
d)_____ address of the computer making the HTTP request

4. Errors are recorded only in the error log.

_____ True  
_____ False

5. Which of the following is probably a higher number?

a)_____ Hits 
b)_____ Views
c)_____ Visits
d)_____ Hits = Views = Visits

6. Which of the following is the best description of a site visit?

a)_____ A list of all requests made to a web server on a given date
b)_____ All the pages and graphics needed to view a given page on the web
c)_____ The set of recorded transactions by a given IP address or DNS entry, from site entry to exit.
d)_____ Only the image downloads necessary to use a page effectively.

Answers

mcdunn
02/23/01 08:41:38 AM