I've got a log file on external computer in my LAN network. Log is an XML file. File is not accessible from http, and is updating every second. Currently i'm copying log file into my computer and run parser, but I want to parse file directly from external host.
How can I do it in Python? Is it possible, to parse whole file once, and later parse only new content added to the end in future versions?
You can use paramiko and xml.sax's default parser, xml.sax.expatreader, which implements xml.sax.xmlreader.IncrementalParser.
I ran the following script on local virtual machine to produce XML.
#!/bin/bash
echo "<root>" > data.xml
I=0
while sleep 2; do
echo "<entry><a>value $I</a><b foo='bar' /></entry>" >> data.xml;
I=$((I + 1));
done
Here's incremental consumer.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import time
import xml.sax
from contextlib import closing
import paramiko.client
class StreamHandler(xml.sax.handler.ContentHandler):
lastEntry = None
lastName = None
def startElement(self, name, attrs):
self.lastName = name
if name == 'entry':
self.lastEntry = {}
elif name != 'root':
self.lastEntry[name] = {'attrs': attrs, 'content': ''}
def endElement(self, name):
if name == 'entry':
print({
'a' : self.lastEntry['a']['content'],
'b' : self.lastEntry['b']['attrs'].getValue('foo')
})
self.lastEntry = None
def characters(self, content):
if self.lastEntry:
self.lastEntry[self.lastName]['content'] += content
if __name__ == '__main__':
# use default ``xml.sax.expatreader``
parser = xml.sax.make_parser()
parser.setContentHandler(StreamHandler())
client = paramiko.client.SSHClient()
# or use ``client.load_system_host_keys()`` if appropriate
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect('192.168.122.40', username = 'root', password = 'pass')
with closing(client) as ssh:
with closing(ssh.open_sftp()) as sftp:
with closing(sftp.open('/root/data.xml')) as f:
while True:
buffer = f.read(4096)
if buffer:
parser.feed(buffer)
else:
time.sleep(2)
I am assuming that another process of which you don't have access to is maintaining the xml as an object being updated every so often, and then dumping the result.
If you don't have access to the source of the program dumping the XML, you will need a fancy diffing between the two XML versions to get an incremental update to send over the network.
And I think you would have to parse the new XML each time to be able to have that diff.
So maybe you could have a python process watching the file, parsing the new version, diffing it (for instance using solutions from this article), and then you can send that difference over the network using a tool like xmlrpc. If you want to save bandwidth it'll probably help. Although I think I would send directly the raw diff via network, patch and parse the file in the local machine.
However, if only some of your XML values are changing (no node deletion or insertion) there may be a faster solution. Or, if the only operation on your xml file is to append new trees, then you should be able to parse only these new trees and send them over (diff first, then parse in the server, send to client, merge in client).
