engine = create_engine("")
df = pd.read_csv('in.csv', chunksize=1000)
for chunk in df:
list= tuple(list(chunk["column2"]))
sql = "SELECT * from table where value in {};".format(list)
found = pd.read_sql(sql, engine)
found.to_csv('out.csv', mode='a', header ['column2'], index=False)
an error appeared and I'm not sure why and how to fix:
list= tuple(list(chunk["column2"]))
TypeError: 'tuple' object is not callable
Your issue is that you are overwriting what list is. You are assigning it to a tuple, and then calling it again, but instead of calling Python's builtin list, you are trying to call a tuple:
list = tuple(list((1,2))) # this will work since you haven't reassigned list yet
list = tuple(list((2,3))) # this will throw an error.
There is a better approach - save your DF as a temporary (additional) table in your DB and use it for the subquery:
df[['column2']].to_sql('tmp', engine, if_exists='replace')
sql = """
SELECT * from table
where value in (select column2 from tmp)
"""
found = pd.read_sql(sql, engine)
or save it directly to CSV:
(pd.read_sql(sql, engine)
.to_csv('out.csv', mode='a', header=['column2'], index=False))
list. It overrides a builtin python name and will lead to hard to find errors.tuple(list(chunk["column2"]))works and assigns its result to the namelist. On the second iterationlistis now a tuple, so the inner partlist(chunk["column2"])is calling a tuple, not the builtinlist.