Jeffrey Palermo 2:45 pm on March 11, 2008

Parse IP addresses out of a web server log file

I had written a quick script off the top of my head, but my astute reader reminded me of the fully-featured LogParser tool that can not only extract IP addresses but can also pull out a whole host of information from your log files:

http://www.microsoft.com/downloads/details.aspx?FamilyID=890cd06b-abf8-4c25-91b2-f8d975cf8c07&displaylang=en

Ayende Rahien 3:16 pm on March 11, 2008

This is horribly inefficient for large files or large amount of files.
It would be better to do this on a line by line basis.
Jeffrey Palermo 3:22 pm on March 11, 2008

@Ayende,
Perhaps, but it works great for 10-20 files. 🙂 Just enough to find out what IP addresses have accessed the server in the past few days.
Sam Smoot 3:40 pm on March 11, 2008

You asked for it. 😉

require ‘set’
require ‘pp’

results = SortedSet.new

Dir[‘*.log’].each do |path|
File.read(path).scan(
/b(d{1,3}.d{1,3}.d{1,3}.d{1,3})b/
).flatten.each { |ip| results << ip }
end

pp results

It took me 2 minutes to write this (I'm slow 😉 ). Not that you took any longer. I've just been thinking lately how it feels like I'm at the point where I can constantly throw away little solutions like this and not get wrapped up in reusability for a utility because they're one of the few things I can (almost) write faster in my head than I can type.

That level of proficiency or approach to a problem doesn't get much press in programming circles it seems. Then again, not much more to say, so that's probably just because it's not a very interesting topic. Just thinking out loud here. 🙂
Peter Mounce 3:45 pm on March 11, 2008

Walk, don’t run, to http://www.logparser.com, and query your logs like you would if they were SQL tables … 😉
Eric Hexter 3:51 pm on March 11, 2008

Log parser is the way to go… especially when you are talking about large log files.

http://www.microsoft.com/downloads/details.aspx?FamilyID=890cd06b-abf8-4c25-91b2-f8d975cf8c07&displaylang=en
Aaron Lerch 4:05 pm on March 11, 2008

1 word: PowerShell! 🙂
Matt (MSFT) 4:51 pm on March 11, 2008

Jeffrey — Ayende’s right here. The performance implications of the following lines are really terrible:

string text = “”;
…
foreach (var filePath in filePaths) {
text += File.ReadAllText(filePath);
}

You’re allocating:
1.) A string to hold the entire contents of 1 text file
2.) Another string that holds the entire contents read up to this point PLUS the data from step #1
3.) And you’re repeating this for every log file

Like he said, you could solve this by processing a file at a time, line-by-line, but if you don’t want to do that, you could improve things tremendously by using a StringBuild that you append to.
Steven Rigney 5:03 pm on March 11, 2008

I can’t believe you haven’t heard of “x”. There are what? 600 different things that could have worked? You did one that you know well and wrote quickly, that’s what works well. There a couple things I don’t agree with.

1. loading all of the files into a single combined text string. I would have opened one file, processed the matches then moved on to the next file.

2. Writing and running this as a unit test. The test framework brings nothing to your solution. You aren’t using a setup or any assertions. You have a console application that you use NUnit to run. I’d have just written it as a console application.

I probably would have also output the results to a file instead of just the console window, but that’s a very minor change.
Joshua Flanagan 1:56 am on March 12, 2008

I’m just left wondering what your intention with this post was…
You knew there were better tools and method out there to do the job, and you knew people would call them out. So that rules out the “providing a solution for others to follow” motive.
But then you seem a bit defensive when people suggest alternatives, so that rules out the “information gathering” motive.

Comments are closed.

Programming with Palermo

Jeffrey Palermo, Microsoft MVP, Author, Speaker, Clear Measure Chief Architect, Azure DevOps Expert

Parse IP addresses out of a web server log file

Related