Extract values from numpy array with condition on strings

Question

So this is my first post here and English is not my native language, I'll try to be as clear as possible.

I have a numpy array coming frome a shape (basically a table of data) containing the following :

[('information1',   'identifier1',              length1)
('information2',    'identifier2',              length2)
('information3',    'identifier3,identifier4',  length3)
....
]

where :

informationx is a string,
identifier is a string containing one or multiple ids in one string,
length is a float.

I need to extract from this array all the rows containing information about one identifier.

In SQL I would do

select * from array where id like "%identifier1%"

It's easy when there's only one identifier :

extract = array[array[id_header] == identifier1]

Is there any elegant and pythonic way to do so (maybe through extract, select or where) ?

I would actually use pandas for that, so that the floats stay floats. So what is the issue you have? This should work, doesn't it? — Matthieu Brucher
– Matthieu Brucher, Commented Oct 16, 2018 at 13:47
The issue I have is I don't know how to 'translate' the SQLish statement into python to extract the information from my array. This will work in sql, but I'm looking for the trick in python. I'll look into pandas though. — Seb
– Seb, Commented Oct 16, 2018 at 13:49
I'm looking for the equivalent (if it exists) of "select * from array where id like "%identifier1%"" in python using numpy. I know the equivalent of "select * from array where id = 'identifier1'" is array[array[id_header] == identifier1]. — Seb
– Seb, Commented Oct 16, 2018 at 13:58

WagnerAlbJr · Accepted Answer · 2018-10-16 14:35:33Z

1

This is a easy task in pandas,considering you can use pandas, translate your array to a pandas dataframe, using
import pandas as pd df = pd.dataFrame([your_array]) #creating data_frame df.columns = ['col_1','col_2','col_3'] #setting column names

considering you had set names col_1,col_2,col_3 to your columns.

subselect the columns that you want with this code.

df_subset = df[ df['col_2'].str.contains('identifierx') ] #subselecting the data frame.

Considering you can't use pandas, only using numpy.

new_lis = []
for idx in range(0,len(your_array)):
    if( 'identifierx' in your_array[idx][1]):
        new_lis.append(your_array[idx])

edited Oct 16, 2018 at 14:35

answered Oct 16, 2018 at 14:16

WagnerAlbJr

3403 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Seb Over a year ago

Yeah that looks great ! Except I did not manage to install pandas in my Arcgis10.2 python. I'm still on it.

Seb Over a year ago

It works, thank you. Similar to Jacob's proposal but this version with len() works.

WagnerAlbJr Over a year ago

ok, mark my answer as the right if that helped you. Thank you!

Jacob Sharp · Accepted Answer · 2018-10-16 14:08:16Z

0

you could do a loop to go through each index to see if the identifier is the one you want:

 lengths = []
 for i in range(array.size[0]): #this should iterate through each row in the table 
      if array[i][1] == "identifierx":
           lengths.append(array[i][2]) #adds the lenghts to a list containing all the lengths from the identifier you want

answered Oct 16, 2018 at 14:08

Jacob Sharp

334 bronze badges

2 Comments

Seb Over a year ago

I get the following error : for i in range(arr_support.size[0]): TypeError: 'int' object has no attribute 'getitem'.

Jacob Sharp Over a year ago

@Seb this is because array.size shouldn't return multiple values, I shouldn't have used it. it was just supposed to represent the number of rows in the table/array. However many rows are in the table, you should put that in for the range.

jameshollisandrew · Accepted Answer · 2020-07-02 17:36:32Z

0

That's a beautiful numpy solution! Just wanted to add the list comp version:

Ran these over a (1000012, 3) array with the above values padded in to query the search and got the below timings:

%%time
new_lis = []
for idx in range(0,len(huge_data)):
    if('identifier3' in huge_data[idx][1]):
        new_lis.append(huge_data[idx])

Returns Wall time: 875 ms

For the list comp:

new_lis = [idx for idx in range(0, len(huge_data)) if ('identifier3' in huge_data[idx][1])]

Returns Wall time: 772 ms

But yeah - I tried to solve with list comp + numpy indices but to catch the string I used a regular expression so it slowed it down to ~4.5s wah waaaah.

Good question, good answers!

answered Jul 2, 2020 at 17:36

jameshollisandrew

1,36110 silver badges10 bronze badges

Collectives™ on Stack Overflow

Extract values from numpy array with condition on strings

3 Answers 3

3 Comments

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related