concat Datasets along multiple dimensions

Question

I have multiple data with the same coords and similar values for them. I.e., there is no single unique ID but rather a combined ID of (index, split). Ideally, I would just want to append all the datasets one after another, but I haven't found the right way to do so.

import xarray as xr
from xarray import Dataset
import numpy as np

datasets = []
for split in range(3):
    dim2_len = 4
    dim1_len = 3
    data_len = 5
    d = Dataset({'data1': (['index', 'dim1'], np.random.rand(data_len, dim1_len)),
                 'data2': (['index', 'dim2'], np.random.rand(data_len, dim2_len)),
                 'data3': (['index', 'dim2'], np.random.rand(data_len, dim2_len))},
                coords={'index': range(data_len),
                        'dim2': range(dim2_len),
                        'dim1': range(dim1_len),
                        'split': split})
    datasets.append(d)

xr.auto_combine(datasets)
# ValueError: too many different dimensions to concatenate: {'dim1', 'dim2', 'index'}

The current xr.concat / xr.auto_combine function does not allow you to concatenate datasets along multiple dimensions (https://github.com/pydata/xarray/blob/4b8339b53f1b9dcd79f2a9060933713328a13b90/xarray/core/combine.py#L358). I guess, I could manually take all the data and coords and throw them into a new dataset but it seems like there should be a better way in the library.

Is there a way to concatenate datasets along multiple dimensions?

Can you provide some code explaining what you've tried and a summary of how your datasets are structured? This question is not complete in its current form. — jhamman
– jhamman, Commented Nov 29, 2017 at 16:58
I just answered a similar question: stackoverflow.com/questions/47545138/…, can you try breaking your concat operations into pieces, one dimension at a time? — jhamman
– jhamman, Commented Nov 29, 2017 at 23:55

ThomasNicholas · Accepted Answer · 2023-09-19 16:40:43Z

0

Xarray does now have this feature - you can use combine_nested or combine_by_coords to concatenate along an arbitrary number of dimensions. You can also use either of these algorithms from within open_mfdataset. The various functions for combining datasets are covered in the documentation page here.

answered Sep 19, 2023 at 16:40

ThomasNicholas

1,39212 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

concat Datasets along multiple dimensions

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related