I have a lot of netcdf files, how would I use xarray to upload all the files into one python notebook?

Question

I have 1 folder of 12 netcdf files. How might I use xarray to combine all the netcdf files into one data array?

The folder represents year 2015 and the 12 netcdf files represent data for each month in 2015.

I thought maybe I could try and manipulate the strings by running a for loop and change each number of the file in the string, since my files are organized (for 2015) by:

EN.4.2.1.f.analysis.g10.201501.nc
EN.4.2.1.f.analysis.g10.201502.nc
EN.4.2.1.f.analysis.g10.201503.nc
....

I am loading the one netcdf file like:

import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
import netCDF4 as s
import glob
import xarray as xr
from datetime import datetime

inpath='../../Data/EN.4.2.1_analyses/EN.4.2.1.analyses.g10.2015/'

# just loading in the first month (january) of 2015
en4_2015 = xr.open_dataset(inpath+'EN.4.2.1.f.analysis.g10.201501.nc')

How would I combine all the netcdf files into one array rather than 12 different xarrays?

I tried:

import glob
import xarray as xr
from datetime import datetime

# List all matching files
files = glob.glob(inpath+'*.nc')

# Create list for 
individual_files = []

# Loop through each file in the list
for i in files:
    
    # Load a single dataset
    timestep_ds = xr.open_dataset(i)
    
    # Create a new variable called 'time' from the `time_coverage_start` field, and 
    # convert the string to a datetime object so xarray knows it is time data
    timestep_ds['time'] = datetime.strptime(timestep_ds.time_coverage_start, 
                                           "%Y-%m-%dT%H:%M:%S.%fZ")
    
    # Add the dataset to the list
    individual_files.append(timestep_ds)

# Combine individual datasets into a single xarray along the 'time' dimension
modis_ds = xr.concat(individual_files, dim='time')

print(modis_ds)

AttributeError                            Traceback (most recent call last)
<ipython-input-36-34685efc1691> in <module>
     10     # Create a new variable called 'time' from the `time_coverage_start` field, and
     11     # convert the string to a datetime object so xarray knows it is time data
---> 12     timestep_ds['time'] = datetime.strptime(timestep_ds.time_coverage_start, 
     13                                            "%Y-%m-%dT%H:%M:%S.%fZ")
     14 

~/miniconda3/envs/py3_std_maps/lib/python3.8/site-packages/xarray/core/common.py in __getattr__(self, name)
    226                 with suppress(KeyError):
    227                     return source[name]
--> 228         raise AttributeError(
    229             "{!r} object has no attribute {!r}".format(type(self).__name__, name)
    230         )

AttributeError: 'Dataset' object has no attribute 'time_coverage_start'

While it may seem that a downvote is an "overreaction", you have to remember that I come here to help people and instead end up slogging through mounds of lazily written questions from users asking for "teh codez". Instead of telling me that my downvote is "obnoxious", which will not change my mind about casting it, I suggest that you don't post sloppy questions that waste my time to begin with, which will. — Mad Physicist
– Mad Physicist, Commented Feb 1, 2021 at 12:35
Is individual_files.append(timestep_ds) an in-place operation in xarray, or does it make a new array, as in numpy? — Mad Physicist
– Mad Physicist, Commented Feb 1, 2021 at 12:38
Take a careful look at that error. I don't have access to your data, so you'll have to fix that one yourself. Also, use triple backticks around your code and error blocks. That will make them easier to separate. — Mad Physicist
– Mad Physicist, Commented Feb 1, 2021 at 12:39

wabash · Accepted Answer · 2021-02-02 15:38:23Z

1

Easy fix! Use:

ds = xr.open_mfdataset(file_path_folder...time??/*nc')

where time instead of being say 2015, it would be 20?? to get all the files in the 21st century, same for 19?? to get all files in 20th century etc.

answered Feb 2, 2021 at 15:38

wabash

8791 gold badge12 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

I have a lot of netcdf files, how would I use xarray to upload all the files into one python notebook?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related