pandas dataframe columns to hierarchical data structure?

Question

I have a pandas dataframe with a number of columns. Some columns are hierarchically groupable. I would like to use this groupability to turn the column structure into a hierarchical structure to be used in a machine learning environment.

Example:

my pandas frame has columns run, obj_id and data and it can look as follows:

Index    run    obj_id    data1    data2
0        0      0         1.3134   3.4943
1        0      0         2.3311   5.4434
2        1      0         1.3345   6.9942
3        1      0         3.4422   3.5353
4        0      1         4.2233   0.3112

and so on. What I would like to do here is first of all train a separate model for each obj_id. Then I would like to turn the run into batch, that is, each run should be seen as a batch. And then the data columns should be the features.

The result would probably look like this:

X = [ # obj_id: model
      [ # run: batch
        [ # data_: features
          [1.3134, 3.4943], 
          [2.3311, 5.4434]
        ], 
        [
          [1.3345, 6.9942], 
          [3.4422, 3.5353]
        ]
      ]

Is there an easy way to do that transformation?

Quang Hoang · Accepted Answer · 2019-12-05 15:18:38Z

2

Not the best solution, but you can do:

(df.groupby('obj_id')
   .apply(lambda x: x.groupby('run')['data1','data2']
                     .apply(lambda y: y.values.tolist() )
                     .to_list()
         )
   .to_list()
)

Output:

[
    [
        [
            [1.3134, 3.4943], 
            [2.3311, 5.4434]
        ], 
        [
            [1.3345, 6.9942], 
            [3.4422, 3.5353]
        ]
    ],
    [
        [
            [4.2233, 0.3112]
        ]
    ]
]

answered Dec 5, 2019 at 15:18

Quang Hoang

151k11 gold badges64 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Hendrik Wiese Over a year ago

Is that different from df.groupby(["obj_id", "run"]).aggregate(list).values.tolist()?

Quang Hoang Over a year ago

I didn’t try that, but that does look much better, if it gives the desired output.

Hendrik Wiese Over a year ago

I think the problem with my suggestion is that it combines the grouping of obj_id and run into one hierarchical layer instead of handling them separately...

Quang Hoang Over a year ago

Then my solution, which gives the correct hierarchy, is different.

Hendrik Wiese Over a year ago

I'll give it a shot. Though, maybe there's a also more elegant solution. Thanks

|

Collectives™ on Stack Overflow

pandas dataframe columns to hierarchical data structure?

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related