How to use MLlib in spark SQL

Question

Lately, I've been learning about spark sql, and I wanna know, is there any possible way to use mllib in spark sql, like :

select mllib_methodname(some column) from tablename;

here, the "mllib_methodname" method is a mllib method. Is there some example shows how to use mllib methods in spark sql?

Thanks in advance.

Currently I don't think , SQL is mainly meant for dataware housing and pre processing the data , you can surely build the dataset using SQL and then run in MLlib , but I couldn't find the other way around — Abhishek Choudhary
– Abhishek Choudhary, Commented Jun 25, 2015 at 14:50
I think I can customize the function in the sql to call method in MLlib — ldl
– ldl, Commented Jun 26, 2015 at 10:42
That will be great and you may check spark buglist , if its not there you may contribute — Abhishek Choudhary
– Abhishek Choudhary, Commented Jun 26, 2015 at 11:08

Xiangrui Meng · Accepted Answer · 2015-06-26 19:10:03Z

1

The new pipeline API is based on DataFrames, which is backed by SQL. See

Or you can simply register the predict method from MLlib models as UDFs and use them in your SQL statement. See

answered Jun 26, 2015 at 19:10

Xiangrui Meng

461 bronze badge

Sign up to request clarification or add additional context in comments.

1 Answer 1