Web Log Analysis: Who's Doing What, When?
Part 3
BYTECOUNT
#!/usr/local/bin/perl
require 'ctime.pl';
$root = "/usr/logs/";
if ($ARGV[0]) { $x = $ARGV[0]; }
else { print "File name? "; $x = <STDIN>; chop $x; }
while (!-e "$root/$x") {
print "Bad file name\nFile name? ";
$x = <STDIN>;
chop $x;
}
if ($x =~ /\.gz$/i) {
open (IN, "gunzip -c $root/$x |");
} else {
open (IN, "< $root/$x");
}
while (<IN>) {
/()()()()/;
/\"(GET|POST|HEAD)
\/([^\/\s\-A-Z]+)\/[^\"]*\" ([0-9]*) ([0-9]*)/;
$clients{$2} += $4;
$i++;
}
close IN;
open (OUT, ">> bytereport");
select (OUT);
print &ctime(time) . "\n";
foreach (sort keys %clients) {
if ($clients{$_} > 100000)
{
printf "%-15.15s : %15.15s\n", $_, $clients{$_};
}
}
select (STDOUT);
close OUT;
QUICKDIRTY
#!/usr/local/bin/perl
$root = "/usr/logs/";
$clientthres = 1000;
$refthres = 10;
if ($ARGV[0]) { $x = $ARGV[0]; }
else { print "File name? "; $x = <STDIN>; chop $x; }
while (!-e "$root/$x") {
print "Bad file name\nFile name? ";
$x = <STDIN>;
chop $x;
}
/()/;
$x =~ /httpd\-log\.([^\.]*)\./;
if ($1) { $nomain ="www.${1}"; }
else { $nomain = "niente"; }
open (ENV, "< $x") || die "Can't open file\n";
while (<ENV>) {
/()()/;
chop;
/\"([^\"]*)\" \"([^\"]*)\" \"[^\"]*\"$/;
$ref = $1;
$cli = $2;
/()()/;
/\"(GET|POST|HEAD) \/([^\/]*)\//;
$head = $2;
if ($ref ne "-" & $ref =~ /http/i & \
$ref !~ /$nomain/i)
{ $url{$ref}++; }
if ($cli) { $browser{$cli}++; }
}
if (!$ARGV[0]) {
open (OUT, "> env.temp");
select (OUT);
}
print "\nREFERERS:\n";
foreach (sort keys %url)
{ $urlnum{$url{$_}} .= "$_\n"; }
for $i (0..($refthres - 1)) { $urlnum{$i} = ""; }
foreach $num (sort numerically keys %urlnum) {
foreach $val (split('\n', $urlnum{$num})) {
printf ("%-70.70s : %5.5s\n", $val, $num);
}
}
print "\n\nBROWSERS:\n";
foreach (sort keys %browser) {
if ($browser{$_} > $clientthres)
{
printf ("%-60.60s : %10.10s\n", $_, $browser{$_});
}
}
select (STDOUT);
sub numerically { $a <=> $b; }
Common Log Format
The common log format appears exactly as follows:
host/ip rfcname logname [DD/MMM/YYYY:HH:MM:SS -0000]
"METHOD /PATH HTTP/1.0" code bytes
|
host/ip
|
If reverse DNS works and DNS lookup is enabled, the hostname of
the client is dropped in; otherwise the IP number displays.
|
|
RFC name
|
If you enable identd (see Web Developer® Spring 1996 issue, p. 23), you
can retrieve a name from the remote server for the user. If no value is
present, a "-" is substituted.
|
|
logname
|
If you're using local authentication and registration, the user's log
name will appear; likewise, if no value is present, a "-" is substituted.
|
|
datestamp
|
The format is day, month (three-letter abbreviation), year, hour in 24-
hour clock, minute, second, and the offset from Greenwich Mean
Time (for example, Pacific Standard Time is -0800).
|
|
retrieval
|
Method is GET, PUT, POST, or HEAD; path is the path and file
retrieved; HTTP/1.0 defines the protocol.
|
|
code
|
HTTP completion code. 200 is successful, 304 is a reload from cache,
404 is file not found, and so forth.
|
|
bytes
|
number of bytes in file retrieved.
|
Here's an example:
sniksnak.foobar.org - - [30/Feb/1996:06:03:24 -0800]
"GET /film/logos/the.movies.main.gif HTTP/1.0" 200 278
[ < Web Log Analysis: Who's Doing What, When?:
Part 2 ] |
[ Web Log Analysis: Who's Doing What, When?:
Part 1 > ] |
|
