Passing a list as variable to python script from bash script

Question

I have a python script like below.

# Import necessary libraries and functions
import sys
import traceback

y = '2020'
querysting = "select {} from table where Year={}".format(col_list,y)
df = pd.read_sql(querystring,db)


if __name__ == '__main__':
    if len(sys.argv) != 8:
        print('Invalid number of args......')
        print('Usage: file.py Input_path Output_path')
        exit()
    _, col_list, FinalDB, final_table, host, dsl_tbl_name, username, password = tuple(sys.argv)

    data_load(col_list, FinalDB, final_table, host, tbl_name, username, password)

Now I am calling this python script inside a shell script like below

#!/bin/bash

col_list='id, name, address, phone_number'
FinalDB='abc'
final_table='xyz'
host='xxxxxx.xxxx'
tbl_name='test'
username='USER'
password='test_pass'


python python_script.py ${col_list} ${FinalDB} ${final_table} ${host} ${tbl_name} ${username} ${password}

Now when I run this bash script I am getting Invalid number of args...... error

I am pretty much sure that it is some thing to do with col_list variable being passed to the python script.

Because instead of having columns in a list if I just pass select * in the query and removing col_list variable then I don't get any errors

What am I doing wrong here and how can I resolve this issue.

You should almost always put double-quotes around variable references (e.g. ... "${col_list}" "${FinalDB}" "${final_table}" ... instead of just ... ${col_list} ${FinalDB} ${final_table} ...). This is a mostly-duplicate of the question: When is double-quoting necessary?. — Gordon Davisson
– Gordon Davisson, Commented May 18, 2020 at 0:38

Chiheb Nexus · Accepted Answer · 2020-05-17 23:14:22Z

The problem comes from how you pass your variables that contains spaces from Bash to Python.

You may note that:

In shell scripts command line arguments are separated by whitespace, unless the arguments are quoted.

Let's have this simple Python script:

python_script.py:

import sys

if __name__ == '__main__':
    print(sys.arv)

Then in your Terminal:

$> export var1="hello"
$> python python_script.py $var1
['python_script.py', 'hello']

However:

$> export var1="hello, hi"
$> python python_script.py $var1
['python_script.py', 'hello,', 'hi']  # Note Here: Two args were passed
$> export var1="hello,hi"
$> python python_script.py $var1
['python_script.py', 'hello,hi']  # One arg passed

So, a solution to your problem, is to pass your Bash args as a string even with spaces, like this example:

$> export var1="hello, hi, hola"
$> python python_script.py "$var1"
['python_script.py', 'hello, hi, hola']

For more informations see this article

Collectives™ on Stack Overflow

Passing a list as variable to python script from bash script

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related