-3

I currently try to merge two DataFrames by using pandasql, but I don't know how to create a new column for my DataFrame. I tried the code "ALTER TABLE crsp ADD monthid INT()", and it told me that there is no such table: crsp. But I have created a crsp DataFrame by using pd.read_sas, so I'm not sure how to solve this error.

Does someone know how to solve this problem?

import pandas as pd
from pandasql import sqldf
mysql = lambda q: sqldf(q, globals())
crsp = pd.read_sas("crsp_nasdaq100.sas7bdat", encoding= "ISO-8859-1 ")
query = "ALTER TABLE crsp \
         ADD monthid INT()"
mysql(query)
Error message:
OperationalError                          Traceback (most recent call last)
~/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py in _execute_context(self, dialect, constructor, statement, parameters, execution_options, *args, **kw)
   1770                 if not evt_handled:
-> 1771                     self.dialect.do_execute(
   1772                         cursor, statement, parameters, context

~/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/default.py in do_execute(self, cursor, statement, parameters, context)
    716     def do_execute(self, cursor, statement, parameters, context=None):
--> 717         cursor.execute(statement, parameters)
    718 

OperationalError: no such table: crsp

The above exception was the direct cause of the following exception:

OperationalError                          Traceback (most recent call last)
~/opt/anaconda3/lib/python3.9/site-packages/pandasql/sqldf.py in __call__(self, query, env)
     60             try:
---> 61                 result = read_sql(query, conn)
     62             except DatabaseError as ex:

~/opt/anaconda3/lib/python3.9/site-packages/pandas/io/sql.py in read_sql(sql, con, index_col, coerce_float, params, parse_dates, columns, chunksize)
    627     else:
--> 628         return pandas_sql.read_query(
    629             sql,

~/opt/anaconda3/lib/python3.9/site-packages/pandas/io/sql.py in read_query(self, sql, index_col, coerce_float, parse_dates, params, chunksize, dtype)
   1578 
-> 1579         result = self.execute(*args)
   1580         columns = result.keys()

~/opt/anaconda3/lib/python3.9/site-packages/pandas/io/sql.py in execute(self, *args, **kwargs)
   1423         """Simple passthrough to SQLAlchemy connectable"""
-> 1424         return self.connectable.execution_options().execute(*args, **kwargs)
   1425 

~/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py in execute(self, statement, *multiparams, **params)
   1247 
-> 1248             return self._exec_driver_sql(
   1249                 statement,

~/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py in _exec_driver_sql(self, statement, multiparams, params, execution_options, future)
   1546         dialect = self.dialect
-> 1547         ret = self._execute_context(
   1548             dialect,

~/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py in _execute_context(self, dialect, constructor, statement, parameters, execution_options, *args, **kw)
   1813         except BaseException as e:
-> 1814             self._handle_dbapi_exception(
   1815                 e, statement, parameters, cursor, context

~/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py in _handle_dbapi_exception(self, e, statement, parameters, cursor, context)
   1994             elif should_wrap:
-> 1995                 util.raise_(
   1996                     sqlalchemy_exception, with_traceback=exc_info[2], from_=e

~/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/util/compat.py in raise_(***failed resolving arguments***)
    206         try:
--> 207             raise exception
    208         finally:

~/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py in _execute_context(self, dialect, constructor, statement, parameters, execution_options, *args, **kw)
   1770                 if not evt_handled:
-> 1771                     self.dialect.do_execute(
   1772                         cursor, statement, parameters, context

~/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/default.py in do_execute(self, cursor, statement, parameters, context)
    716     def do_execute(self, cursor, statement, parameters, context=None):
--> 717         cursor.execute(statement, parameters)
    718 

OperationalError: (sqlite3.OperationalError) no such table: crsp
[SQL: ALTER TABLE crsp          ADD monthid INT()]
(Background on this error at: https://sqlalche.me/e/14/e3q8)

During handling of the above exception, another exception occurred:

PandaSQLException                         Traceback (most recent call last)
/var/folders/jr/xg1lg5x50w18f16_0bh805l80000gn/T/ipykernel_933/1396149349.py in <module>
      1 query = "ALTER TABLE crsp \
      2          ADD monthid INT()"
----> 3 mysql(query)

/var/folders/jr/xg1lg5x50w18f16_0bh805l80000gn/T/ipykernel_933/285284892.py in <lambda>(q)
----> 1 mysql = lambda q: sqldf(q, locals())

~/opt/anaconda3/lib/python3.9/site-packages/pandasql/sqldf.py in sqldf(query, env, db_uri)
    154     >>> sqldf("select avg(x) from df;", locals())
    155     """
--> 156     return PandaSQL(db_uri)(query, env)

~/opt/anaconda3/lib/python3.9/site-packages/pandasql/sqldf.py in __call__(self, query, env)
     61                 result = read_sql(query, conn)
     62             except DatabaseError as ex:
---> 63                 raise PandaSQLException(ex)
     64             except ResourceClosedError:
     65                 # query returns nothing

PandaSQLException: (sqlite3.OperationalError) no such table: crsp
[SQL: ALTER TABLE crsp          ADD monthid INT()]
(Background on this error at: https://sqlalche.me/e/14/e3q8)
10
  • show minimal working code. As I know pandasql send data to SQLite and run query on data in database - but it will not send dataframe crsp if you don't use this dataframe as parameter. But without code we can't help you. We can't read in your mind. You have to show all details in question. Commented May 18, 2022 at 2:09
  • Please provide enough code so others can better understand or reproduce the problem. Commented May 18, 2022 at 3:43
  • Remember that crsp is not an SQL table. It's a DataFrame. You add a new table with, for example, crsp['monthid'] = 0. Commented May 19, 2022 at 3:10
  • @TimRoberts What if I'm only allowed to use pandasql. How can I create a SQL table that includes the data in crsp. Commented May 19, 2022 at 3:22
  • Is that EXACTLY your code? So you KNOW that crsp is actually a global, and not local within some function? Commented May 19, 2022 at 4:42

1 Answer 1

0

The problem here is a fundamental limitation in pandasql. For pandas DataFrames, it only supports SELECT statements. If it does not find FROM or JOIN clauses in the statement, then it falls back to a regular database operation with the connection string you passed, and the default is an empty sqlite3 database.

You will have to do this operation the normal way, using pandas. That's much more efficient anyways. pandasql is a cute idea, but not very practical.

Sign up to request clarification or add additional context in comments.

1 Comment

sorry, I did create mysql by using the sqldf, but I forgot to show it. I think the main problem is that it can not recognise crsp as a table, but I don't know how to fix that

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.