Pyspark KMeans clustering features column IllegalArgumentException

Question

pyspark==2.4.0

Here is the code giving the exception:

LDA = spark.read.parquet('./LDA.parquet/')
LDA.printSchema()

from pyspark.ml.clustering import KMeans
from pyspark.ml.evaluation import ClusteringEvaluator

kmeans = KMeans(featuresCol='topic_vector_fix_dim').setK(15).setSeed(1)
model = kmeans.fit(LDA)

root
|-- Id: string (nullable = true)
|-- topic_vector_fix_dim: array (nullable = true)
| |-- element: double (containsNull = true)

IllegalArgumentException: 'requirement failed: Column topic_vector_fix_dim must be of type equal to one of the following types: [struct < type:tinyint,size:int,indices:array < int >,values:array < double > >, array < double >, array < float > ] but was actually of type array < double > .'

I am confused - it does not like my array <double>, but says that it may be the input.
Each entry of the topic_vector_fix_dim is a 1d array of floats

Sokolokki · Accepted Answer · 2019-03-14 13:03:41Z

7

containsNull of the features column should be set to False:

new_schema = ArrayType(DoubleType(), containsNull=False)
udf_foo = udf(lambda x:x, new_schema)
LDA = LDA.withColumn("topic_vector_fix_dim",udf_foo("topic_vector_fix_dim"))

After that everything works.

answered Mar 14, 2019 at 13:03

Sokolokki

8631 gold badge9 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

LN_P · Accepted Answer · 2019-11-05 11:28:40Z

2

The containsNull answer didn't work for me, but this did:

vectorAssembler = VectorAssembler(inputCols = ["x1", "x2", "x3"], outputCol = "features")
df = vectorAssembler.transform(df)
df = df.select(['features', 'Y'])

answered Nov 5, 2019 at 11:28

LN_P

1,4984 gold badges23 silver badges37 bronze badges

2 Comments

Sokolokki Over a year ago

In the question the input feature is a single column already. The question is not about converting multiple columns to a single one.

ciurlaro Over a year ago

@ArturSokolovsky But It did actually solve the problem. The only thing that makes sense is that the library doesn't explicit it, but it internally only recognize VectorAssembler's arrays when training Estimators.

Collectives™ on Stack Overflow

Pyspark KMeans clustering features column IllegalArgumentException

2 Answers 2

Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related