Spark - Operation not allowed: alter table replace columns

Question

Looks like hive replace columns is not working with spark 2.2.1 and also with 2.3.1

alterSchemaSql : alter table myschema.mytable replace columns (a int,b int,d int)
Exception in thread "main" org.apache.spark.sql.catalyst.parser.ParseException: 
Operation not allowed: alter table replace columns(line 2, pos 6)

Looks like it's been fixed for ADD COLUMNS 2.2 version onwards. It also works for me but replace columns still fails.

Operation not allowed: alter table add columns(line 1, pos 0)

Following docs says it should be supported. Not sure why it's failing for me. https://spark.apache.org/docs/2.2.0/sql-programming-guide.html#supported-hive-features

https://docs.databricks.com/spark/latest/spark-sql/language-manual/alter-table-or-view.html#replace-columns

Do you mean rename column or replace column?

Anurag Sharma
– Anurag Sharma

2018-12-18 05:44:35 +00:00
Commented Dec 18, 2018 at 5:44 — Anurag Sharma
– Anurag Sharma, Commented Dec 18, 2018 at 5:44
replace column.

nir
– nir

2018-12-18 16:32:34 +00:00
Commented Dec 18, 2018 at 16:32 — nir
– nir, Commented Dec 18, 2018 at 16:32

Gooseman · Accepted Answer · 2019-05-29 20:56:12Z

0

Sadly, it seems ALTER TABLE table REPLACE is not implemented by Spark.

Take a look at SparkSqlParser.scala: SparkSqlParser.scala

There, you can see, what SQL statements work with Spark.

edited May 29, 2019 at 20:56

answered May 29, 2019 at 20:46

Gooseman

2,25118 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Spark - Operation not allowed: alter table replace columns

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related