SQL-Functions with schemaRDD using language integrated SQL

Question

I want to filter a schemaRDD using language integrated SQL based on SQL functions. For example I want to run

SELECT name FROM people WHERE name LIKE '%AHSAN%' AND name regexp '^[A-Z]{20}$'

How can I use such SQL functions in people.where()?

Reference:

For language integrated SQL, I am following the example given here.

val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext._
val people: RDD[Person] = ... // An RDD of case class objects, from the first example.
// The following is the same as 'SELECT name FROM people WHERE age >= 10 AND age <= 19'
val teenagers = people.where('age >= 10).where('age <= 19).select('name)
teenagers.map(t => "Name: " + t(0)).collect().foreach(println)

Thanks in advance!

zsxwing · Accepted Answer · 2014-12-03 04:11:23Z

0

You can use SQL functions like numeric operators. E.g.,

people.where('name like "%AHSAN%").where('name rlike "^[A-Z]{20}$").select('name)

There is no regexp in Spark SQL but it's same as rlike.

answered Dec 3, 2014 at 4:11

zsxwing

21k4 gold badges42 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

BdEngineer Over a year ago

can we have two where conditions ?

zsxwing Over a year ago

Yep. Multiple where conditions are okay.

Collectives™ on Stack Overflow

SQL-Functions with schemaRDD using language integrated SQL

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related