Continuously Parse file in Python

Question

I'm writing a script that parses a file with HTTP traffic lines, and takes out the domains and currently just prints them to the screen. I'm using httpry to continuously write the traffic to a file. Here is the script I'm using to strip out the domain names

#!/usr/bin/python

import re

input = open("results.txt","r")

for line in input:
    domain = line.split()[6]
    if domain != "-":
        print domain

While this script works great, I'd like a way to continuously run this script so that as new traffic gets added to the input file, the script is able to strip it out. I can't just run awk on the output of httpry, as I'm eventually going to be entering these domains into a Mongo database, and I'll need the script to do that as well. If anyone could give me some ideas how to constantly run this python script on the output, but not reprint previous entries, it would be much appreciated. Thanks.

Matt Williamson · Accepted Answer · 2012-11-30 03:16:32Z

6

Try this tail -f implementation as found at http://code.activestate.com/recipes/157035-tail-f-in-python/

import time

while 1:
    where = file.tell()
    line = file.readline()
    if not line:
        time.sleep(1)
        file.seek(where)
    else:
        print line, # already has newline

answered Nov 30, 2012 at 3:16

Matt Williamson

40.3k10 gold badges67 silver badges73 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

jches · Accepted Answer · 2012-11-30 03:44:41Z

0

Node.js has a nice readline module that should handle this nicely:

var readline = require('readline')
  , fs = require('fs')

var input = process.stdin; // or: fs.createReadStream('input.txt');
var output = process.stdout; // or: fs.createWriteStream('output.txt')

var reader = readline.createInterface({
  input: input,
  output: output
});

reader.on('line', function(line) {
  this.write(line.split(/[ ]+/)[6]);
});

Save this in a .js file and do node domains.js, or whatever you named it. Or cat file | node domains.js.

It should integrate nicely with mongodb in the future, too :)

answered Nov 30, 2012 at 3:44

jches

4,52729 silver badges37 bronze badges

Collectives™ on Stack Overflow

Continuously Parse file in Python

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related