How to read multiple files from multiple remote servers in Python?

Question

I have a list of AWS ubuntu servers e.g

[email protected]
[email protected]
[email protected]
...

On each of these servers, I have a folder with variable number of files, the path is the same for each server e.g /roth/files/

I want to write a Python script that would fetch the contents of those files and merge them locally on my machine.

How do I go about fetching contents of those files on remote servers?

The way I login to these servers is

ssh -i  path/aws.pem [email protected]

e.g using a key

I found answer on similar question here

sftp_client = ssh_client.open_sftp()
remote_file = sftp_client.open('remote_filename')
try:
    for line in remote_file:
        # process line
finally:
    remote_file.close()

But I do not see where you provide a server name, and the key...

EDIT: As a small correction to Ganesh's answer you need to do the following to fetch every file or otherwise you get an error complaining that you try to fetch directory:

lobj = sftp.listdir_attr(target_folder_remote)
    for o in lobj:
        name = o.filename
        sftp.get(os.path.join(target_folder_remote, name), os.path.join(target_folder_local, name))

Ganesh Kathiresan · Accepted Answer · 2019-11-09 04:31:09Z

3

aws_host_list = [] # Your list here

client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
dest_list = list()
for host in aws_host_list:
    client.connect(host, username=<User Name>, key_filename=<.PEM File path)
    sftp = client.open_sftp()
    lobj = sftp.listdir_attr(target_folder_remote)
    for o in lobj:
        name = o.filename
        sftp.get(os.path.join(target_folder_remote, name), os.path.join(target_folder_local, name))
        dest_list.append(os.path.join(target_folder_local, name))
    sftp.close()
    client.close()

# Then process you files
for f in dest_list:
    # Combine it

edited Nov 9, 2019 at 4:31

answered Nov 8, 2019 at 17:07

Ganesh Kathiresan

2,1082 gold badges25 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

YohanRoth Over a year ago

I am a bit unclear what to provide in <Destination>+host

YohanRoth Over a year ago

Also looking at docs it says that .put is used to "Copy a local file (localpath) to the SFTP server as remotepath". I want the reverse right... Copy remote to local

YohanRoth Over a year ago

I think it should be .get() instead of .put()?

Ganesh Kathiresan Over a year ago

My bad yeah, I'll fix it

YohanRoth Over a year ago

so when I use .get() on a directory it throws error: IsADirectoryError: [Errno 21] Is a directory: How do I fetch a list of files? I think I need to call get() on every file spearately

|

Collectives™ on Stack Overflow

How to read multiple files from multiple remote servers in Python?

1 Answer 1

7 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related