Polars Python select based on dtype pl.list

Question

Hi I want to select those cols of a polars df that are of the dtype list.
Selecting by dtypes works ususally fine with df.select(pl.col(pl.Utf8)).

However for the type list this does not seem to work...

MRE

import polars as pl

df = pl.DataFrame({"foo": [[c] for c in 
    ["100CT pen", "pencils 250CT", "what 125CT soever", "this is a thing"]]}
)

df

Output:

foo
list[str]
["100CT pen"]
["pencils 250CT"]
["what 125CT soever"]
["this is a thing"]


df.select(pl.col(pl.List))

Output:

shape: (0, 0)

roganjosh · Accepted Answer · 2023-10-31 17:32:53Z

3

You need to provide the type of the items in the List unlike primitive types (where print(df.select(pl.col(pl.Int64))) would work in the below example).

import polars as pl

df = pl.DataFrame({
    "foo": [[c] for c in 
        ["100CT pen", "pencils 250CT", "what 125CT soever", "this is a thing"]],
    "bar": [1, 2, 3, 4]
    }
)
print(df.select(pl.col(pl.List(str))))

I can't seem to find anything that's generic across types that the List contains. There is a NESTED_DTYPES here and this answer suggests that you might be able to use it in a more "catch-all" manner, but it doesn't seem to work if you want to grab columns that contain a nested type regardless of the type of data it contains.

Thanks to @jqurious for pointing out that this seems to be a requested feature in an open ticket. This has an interesting use-case for me in that, the only reason I've switched dfs back to pandas recently is that polars refuses to write List to CSV so I either filter out all such columns by name or, if this is implemented, I could drop them in one go. I didn't create those columns and I don't want them in the output.

edited Oct 31, 2023 at 17:32

answered Oct 31, 2023 at 16:13

roganjosh

13.3k4 gold badges33 silver badges53 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

jqurious Over a year ago

It is on the todo list I believe: github.com/pola-rs/polars/issues/9971#issuecomment-1642162835 - The information is in df.schema.items() by checking the .base_type() but that's rather awkward (and only works for the top-level)

Collectives™ on Stack Overflow

Polars Python select based on dtype pl.list

MRE

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

MRE

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related