Improve Log Messages

Many programs either contain poor or nonexistent logging, or log so much that any useful messages drown in the noise. This post concerns improving logs generated by scripts running on Unix, which usually suffer from the poor or nonexistent logging. Witness the succinct ! log message from the original ed(1) in contrast to Java stack trace barfs. Middle ground for usable logs can be found.

Technorati Tags: coding, Perl, Unix

Hard to Match Messages that lack Critical Information

# Perl to open a named file, # or exit script with error open my $fh, '<', $filename or die "Couldn't open $filename\n";

The above failure log exhibits two problems: the error message is difficult to match using Unix command line tools, and worse, the log omits the system error message contained in the special $! variable (errno in C). Improve the log by omitting the apostrophe, and including the system error message:

open my $fh, '<', $filename or die "could not open $filename: $!\n";

Now the error message can be grepped without special handling required for the apostrophe due to Unix shell quoting problems, and the reason the file cannot be opened will be included.

# the awkward $ grep '^Couldn'\''t open ' script.log # becomes instead $ grep '^could not open ' script.log

Missing Metadata

The use of a script must be considered, as the context a script is run in will change the format of the logs. If the script will run in a Unix pipe chain, include the name of the script in all logs:

open my $fh, '<', $filename or die "$0: could not open $filename: $!\n";

Without this information, the error would be difficult to tie to the specific command:

$ foo-util input | bar-util | zot-util > output bar-util: Could not open /var/empty: no such file or directory

I also recommend each log contain a severity level, such as debug, info, notice, warn, error, and fatal. This information simplifies log analysis, and allows rules whereby all fatal logs page someone, and so forth.

open my $fh, '<', $filename or die "$0: fatal: could not open $filename: $!\n";

Regroup Metadata

Consider moving the $filename away from the log message, and into a metadata section. This improves log message portability, if translating to other languages, and will help future searches of the data by providing more structure to match against.

open my $fh, '<', $filename or die "$0: fatal: could not open: " . "file=$filename, errno=$!\n";

This format suits a log handling subroutine that accepts the severity, log message, and a hash of metadata to display. The log handling subroutine can then add the program name as appropriate, direct or duplicate the logs to syslog(1), or other needs as appropriate. Use the example subroutine as a starting point.

sub remark { my $priority = shift; # string my $message = shift; # string my $attributes = shift; # hash reference chomp $message; my $attr_str; if ($attributes) { $attr_str = join ', ', map { $attributes->{$_} ||= q{}; "$_=$attributes->{$_}" } sort keys %$attributes; } print STDERR "$0: $priority: $message" . ( $attr_str ? ": $attr_str" : q{} ) . "\n"; return 1; }

Example invocation of this subroutine:

remark('debug', 'test message', { pid => $$, line => __LINE__ } );

Jeremy Mates’s Blog

Improve Log Messages

Hard to Match Messages that lack Critical Information

Missing Metadata

Regroup Metadata