python tcp put connection reset

Question

def put_data(line = '', dest_host = 'server', dest_port = '876'):
    ''' This method once invoked pushes data to dest_host and dest_port  '''
    if line:
        cmd = '/bin/echo put ' + line + '| ' + '/bin/nc -w 15 server port'
        print('TCP put using command: ' + cmd)
        os.system(cmd)

I am using the above method to TCP put lines of data to an opentsdb driver on top of a HBASE.

This method as can be seen, calls the the netcat utility for each line of data that is being pushed out.

Questions: 1.) Is there a better way to accomplish this task by not having to open a TCP connection each time for every line of data I want to push? I run this every minute and this is causing a large number of connections in TIME_WAIT state.

2.) Can I push say 1000 lines at a time using a persistent TCP socket? I have looked at the netcat implementations in python but they seem to cause a connection reset from the peer when I push all the lines

For example: When I try to use the below method, I keep getting the error: error(104, 'Connection reset by peer')

import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(server, port)
for line in data_list:
    s.sendall(line)
s.close()

error(104, 'Connection reset by peer')

Sample lines of data being pushed:

line1 = 'fin.wait2 1385428269 0.0 host=srv1 testname=tcp source=nag03 product=rebuild'
line2 = 'established 1385428269 11.0 host=srv2 testname=tcp source=nag03 product=rebuild'
line3 = 'fin.wait1 1385428269 0.0 host=srv3 testname=tcp source=nag03 product=rebuild'
line4 = 'last.ack 1385428269 0.0 host=srv4 testname=tcp source=nag03 product=rebuild'
line5 = 'unknown 1385428269 0.0 host=srv5 testname=tcp source=nag03 product=rebuild'
line6 = 'syn.recv 1385428269 0.0 host=srv6 testname=tcp source=nag03 product=rebuild'

yes the each line starts on a new line on them. So the second method works for say up to 20 lines and then I get a connection reset from the opentsdb driver. — cog_n1t1v3
– cog_n1t1v3, Commented Nov 26, 2013 at 1:03
OK, please (a) show us some sample data, so people don't have to guess what it looks like, and (b) update the answer to explain that it works for 20 or so lines before failing, because as written it implies that it doesn't work at all. — abarnert
– abarnert, Commented Nov 26, 2013 at 1:04
Meanwhile, if OpenTSDB for some reason doesn't like getting a whole bunch of separate commands in a row, what about just concatenating them together and sending one big batch? In other words, does s.sendall(''.join(data_list)) work for a whole 100 lines when your loop fails after 20? — abarnert
– abarnert, Commented Nov 26, 2013 at 1:05
Hmm - I added an explcit '\n' after each line it seems to be working...thanks so much. Since your suggestion/answer was in the comments I am unable to accept it — cog_n1t1v3
– cog_n1t1v3, Commented Nov 26, 2013 at 1:24

abarnert · Accepted Answer · 2013-11-26 01:30:54Z

The problem is that your lines don't have newlines on the end.

Your netcat-based code is passing the lines to netcat using echo, which, without any arguments, will add a newline to the end, so it works.

But your Python socket-based code is not adding any newlines.

So, you wanted to send it five commands like this:

put fin.wait2 1385428269 0.0 host=srv1 testname=tcp source=nag03 product=rebuild
put established 1385428269 11.0 host=srv2 testname=tcp source=nag03 product=rebuild
put fin.wait1 1385428269 0.0 host=srv3 testname=tcp source=nag03 product=rebuild
put last.ack 1385428269 0.0 host=srv4 testname=tcp source=nag03 product=rebuild
put unknown 1385428269 0.0 host=srv5 testname=tcp source=nag03 product=rebuild
put syn.recv 1385428269 0.0 host=srv6 testname=tcp source=nag03 product=rebuild

… but instead, you're sending it one big line like this:

put fin.wait2 1385428269 0.0 host=srv1 testname=tcp source=nag03 product=rebuildput established 1385428269 11.0 host=srv2 testname=tcp source=nag03 product=rebuildput fin.wait1 1385428269 0.0 host=srv3 testname=tcp source=nag03 product=rebuildput last.ack 1385428269 0.0 host=srv4 testname=tcp source=nag03 product=rebuildput unknown 1385428269 0.0 host=srv5 testname=tcp source=nag03 product=rebuildput syn.recv 1385428269 0.0 host=srv6 testname=tcp source=nag03 product=rebuild

Either it's getting confused by your data and hanging up, rejecting your overly long line as a DoS attempt, or just waiting around for the newline that's supposed to end the line and never getting it and eventually timing out.

Whatever the underlying problem, the root cause is that lack of newlines, and the fix is to put them in there.

While we're at it, your netcat code appends a put at the start of each command, and your socket code doesn't, and you'll probably need to fix that as well. So:

for line in data_list:
    s.sendall('put {}\n'.format(line))

yes it was important to put the 'put' as well - thanks for pointing that out too.

Collectives™ on Stack Overflow

python tcp put connection reset

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related