Python pandas iterating rows with complicated calculation faster way of current code

Question

I've implemented some sort of object stability calculator in pandas. But performance time is horrible. Can someone help me, please.

def calculate_stability(ind, df, sidx, max_k):
    indexes = sidx[:, ind]
    indexes = np.delete(indexes, np.where(indexes == ind))

    d = 0
    last_crtit_obj_count = 0

    for j in range(max_k):
        if df.at[ind, "Class"] == df.at[indexes[j], "Class"]:
            d = d + 1
        if d / (j+1) > 1/2:
            last_crtit_obj_count = (j+1)

    print(f'\t Object {ind} = {last_crtit_obj_count / max_k}')
    return last_crtit_obj_count / max_k

df.iloc was very slow. That's why I changed to df.at.

Code is here

Need to vectorized version of loop.

I don't see df.iloc in your code snippet.

Kate
– Kate

2021-10-30 17:13:38 +00:00
Commented Oct 30, 2021 at 17:13 — Kate
– Kate, Commented Oct 30, 2021 at 17:13

Alexander Volkovsky · Accepted Answer · 2021-10-30 20:44:16Z

2

Here is the version without the loop:

def calculate_stability(ind, df, sidx, max_k):
    indexes = sidx[:, ind]
    indexes = indexes[indexes != ind][:max_k]

    # `d` contains all values from the first condition from the original loop:
    d = (df["Class"][ind] == df["Class"][indexes]).cumsum()
    # `j` contains all values from the original `range` + 1:
    j = np.arange(1, len(d) + 1)
    # select `last_crtit_obj_count` values:
    crtit_objs = j[(d / j > 1 / 2)]
    # calculate `last_crtit_obj_count / max_k`
    result = crtit_objs[-1] / max_k if len(crtit_objs) else 0
    print(f"\t Object {ind} = {result}")
    return result

answered Oct 30, 2021 at 20:44

Alexander Volkovsky

2,9681 gold badge12 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python pandas iterating rows with complicated calculation faster way of current code

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related