Check if two pyspark Rows are equal

Question

I am writing unit tests for a Spark job, and some of the outputs are named tuples: pyspark.sql.Row

How can I assert their equality?

actual = get_data(df)
expected = Row(total=4, unique_ids=2)
self.assertEqual(actual, expected)

When I do this, the values are rearranged in an order I can not determine.

pault · Accepted Answer · 2018-03-27 19:34:33Z

1

Your code should work as written because according to the docs:

the fields will be sorted by names.

Nevertheless, another way is to use the asDict() method of the pySpark.sql.Row and compare them as dictionaries:

actual = get_data(df)
expected = Row(total=4, unique_ids=2)
self.assertEqual(actual.asDict(), expected.asDict())

answered Mar 27, 2018 at 19:34

pault

43.7k17 gold badges121 silver badges161 bronze badges

Sign up to request clarification or add additional context in comments.

1 Answer 1