How to extract values from specific keys from list of dictionaries in python?

Question

If I have the following type of data - a list of dictionaries, how can I extract some key values from it?

comps = [
{
    "name":'Test1',
    "p_value":0.02,
    "group0_null": 0.0,
    "group1_null": 0.0,
},{
    "name":'Test2',
    "p_value":0.05,
    "group0_null": 0.0,
    "group1_null": 0.0,
},{
    "name":'Test3',
    "p_value":0.03,
    "group0_null": 0.0,
    "group1_null": 0.0,
},{
    "name":'Test4',
    "p_value":0.07,
    "group0_null": 0.0,
    "group1_null": 0.0,
},{
    "name":'Test5',
    "p_value":0.03,
    "group0_null": 0.0,
    "group1_null": 0.0,
},{
    "name":'Test6',
    "p_value":0.02,
    "group0_null": 0.0,
    "group1_null": 0.0,
},{
    "name":'Test7',
    "p_value":0.01,
    "group0_null": 0.0,
    "group1_null": 0.0,
}]

Result

From the data above, let's say I only want name and p_value. How can I get this result.

[{
    "name":'Test1',
    "p_value":0.02,
},{
    "name":'Test2',
    "p_value":0.05,
},{
    "name":'Test3',
    "p_value":0.03,
},{
    "name":'Test4',
    "p_value":0.07,
},{
    "name":'Test5',
    "p_value":0.03,
},{
    "name":'Test6',
    "p_value":0.02,
},{
    "name":'Test7',
    "p_value":0.01,
}]

this shows everything

[c for c in comps]

This shows only the names [c['name'] for c in comps]

But if I do this:

[c['name','p_value'] for c in comps ]

I get the error:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-94-b29459f7b089> in <module>
----> 1 [c['name','p_value'] for c in comps['continuous_explainers'] ]
      2 
      3 # cont_comps = []
      4 
      5 # for c in comps['continuous_explainers']:

<ipython-input-94-b29459f7b089> in <listcomp>(.0)
----> 1 [c['name','p_value'] for c in comps['continuous_explainers'] ]
      2 
      3 # cont_comps = []
      4 
      5 # for c in comps['continuous_explainers']:

KeyError: ('name', 'p_value')

The real data dictionary is much larger than this. I want to do this so that I can have a list of things that are need.

UPDATE

Since some pointed out that the structure of the data that I showed is different from what I receive from the server, here's the code that I used to pull the data.

# get all comparisons
comps = source.get_comparison(name='Pr1 vs. Rest')

# only take the continuous explainers 
comps['continuous_explainers'][1:5]

DATA

[{'name': 'Gender',
  'column_index': 2,
  'ks_score': 0.0022329709328575142,
  'p_value': 1.0,
  'quartiles': [[0.0, 0.0, 1.0, 1.0, 2.0], [0.0, 0.0, 1.0, 1.0, 2.0]],
  't_test_p_value': 0.8341377317414621,
  'diff_means': 0.0014959875249118681,
  'primary_group_mean': 0.6312769010043023,
  'secondary_group_mean': 0.6297809134793905,
  'ks_sign': '+',
  'group0_percent_null': 0.0,
  'group1_percent_null': 0.0},
 {'name': 'Gender_Missing_color',
  'column_index': 3,
  'ks_score': 2.220446049250313e-16,
  'p_value': 1.0,
  'quartiles': [[1.0, 1.0, 1.0, 1.0, 1.0], [1.0, 1.0, 1.0, 1.0, 1.0]],
  't_test_p_value': 1.0,
  'diff_means': 0.0,
  'primary_group_mean': 1.0,
  'secondary_group_mean': 1.0,
  'ks_sign': '0',
  'group0_percent_null': 0.9966523194643712,
  'group1_percent_null': 0.9959153360564427},
 {'name': 'Gender_Missing',
  'column_index': 4,
  'ks_score': 0.0007369834078797544,
  'p_value': 1.0,
  'quartiles': [[0.0, 0.0, 0.0, 0.0, 1.0], [0.0, 0.0, 0.0, 0.0, 1.0]],
  't_test_p_value': 0.40301091478187256,
  'diff_means': -0.0007369834079284866,
  'primary_group_mean': 0.0033476805356288893,
  'secondary_group_mean': 0.004084663943557376,
  'ks_sign': '-',
  'group0_percent_null': 0.0,
  'group1_percent_null': 0.0},
 {'name': 'Male',
  'column_index': 5,
  'ks_score': 0.0029699543407862294,
  'p_value': 0.9999999999915384,
  'quartiles': [[0.0, 0.0, 1.0, 1.0, 1.0], [0.0, 0.0, 1.0, 1.0, 1.0]],
  't_test_p_value': 0.6740956861786738,
  'diff_means': 0.0029699543407684104,
  'primary_group_mean': 0.6245815399330444,
  'secondary_group_mean': 0.621611585592276,
  'ks_sign': '+',
  'group0_percent_null': 0.0,
  'group1_percent_null': 0.0}]

This is the output I get. As mentioned above, I only need some data from this list of dictionaries.

[(c['name'], c['p_value']) for c in comps ]? What is the result you expect? — juanpa.arrivillaga
– juanpa.arrivillaga, Commented Jan 17, 2021 at 21:03
[{'name': item['name'], 'p_value': item['p_value']} for item in comps] — roganjosh
– roganjosh, Commented Jan 17, 2021 at 21:04
@juanpa.arrivillaga they gave the expected result? Just a list of trimmed-down dicts? — roganjosh
– roganjosh, Commented Jan 17, 2021 at 21:06
Ha, brilliant. So you don't have that structure at all, it's probably serialized as JSON and so one giant string — roganjosh
– roganjosh, Commented Jan 17, 2021 at 21:11
@roganjosh, I've updated the question to show the actual data that I receive from the server. But the question remains, even if you think the structure is not the same, and since I'm not as experienced in python, let's say it's not. How do I still solve this problem that I have? — user1828605
– user1828605, Commented Jan 17, 2021 at 21:50

Ori David · Accepted Answer · 2021-01-17 21:05:35Z

2

You could create a new dict for each object in comparisons, and initialize it only with name and p_value keys.

ex = [{'name': d['name'], 'p_value': d['p_value']} for d in comparisons]

answered Jan 17, 2021 at 21:05

Ori David

3723 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

user1828605 · Accepted Answer · 2021-01-18 15:27:15Z

1

I'm still not sure how to make the answers above work for me. However, I figured another way to do this:

test = [(c['name'],c['p_value'], c['group0_percent_null']) for c in comps]
pd.DataFrame(test)

    0   1   2
0   ID  5.374590e-13    0.000000
1   Gender  1.000000e+00    0.000000
2   Gender_Missing_color    1.000000e+00    0.996652
3   Gender_Missing  1.000000e+00    0.000000
4   Male    1.000000e+00    0.000000
... ... ... ...

It gave me the result I was looking for.

edited Jan 18, 2021 at 15:27

answered Jan 17, 2021 at 23:04

user1828605

1,7352 gold badges29 silver badges71 bronze badges

Comments

Hadrian · Accepted Answer · 2021-01-17 21:06:28Z

-1

try

[{'name':c['name'], 'p_value':c['p_value']} for c in comps]

edited Jan 17, 2021 at 21:06

answered Jan 17, 2021 at 21:04

Hadrian

9477 silver badges11 bronze badges

1 Comment

ppwater Over a year ago

While this code may answer the question, including an explanation of how or why this solves the problem would really help to improve the quality of your post. Remember that you are answering the question for readers in the future, not just the person asking now. Please edit your answer to add explanations and give an indication of what limitations and assumptions apply.

Collectives™ on Stack Overflow

How to extract values from specific keys from list of dictionaries in python?

3 Answers 3

Comments

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related