read 2-D binary data in Python and write in NetCDF format

Question

I have binary data with following header information:

DSET ../prcp/cmorph_test
OPTIONS template little_endian
UNDEF  -999.0
TITLE  Precipitation estimates
XDEF 1440 LINEAR    0.125  0.25
YDEF  480 LINEAR  -59.875  0.25
ZDEF   01 LEVELS 1
TDEF 999999 LINEAR  00z01jan2014  3hr
VARS 1
cmorph   1   99 RAW CMORPH integrated satellite precipitation estimates [mm/3hr]
ENDVARS

Please suggest if it is possible to read (and any sample code snippet would be really helpful) and convert it into NetCDF using Python?

You may want, Ibe, to read whathaveyoutried.com & show some respect to the StackOverflow Community, which strongly encourages to post high quality questions, altogether with a MCVE ( a Minimum-Complete-Verifiable-Example of code ) showing what-you-have-tried so far. You may want to update your post, so as to meet this minimum reasonable level of quality & to show your will to respect other StackOverflow contributing members. They are professionals who love to answer good questions on MCVE-related issues. Enjoy being StackOverflow Member & do support this Community Netiquette — user3666197
– user3666197, Commented Oct 13, 2014 at 16:57

DopplerShift · Accepted Answer · 2014-10-13 15:28:47Z

3

I can't help with the read of the binary data, but you can read/write netcdf files with: https://github.com/unidata/netcdf4-python

answered Oct 13, 2014 at 15:28

DopplerShift

5,8731 gold badge24 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Dharman · Accepted Answer · 2020-10-20 13:27:15Z

The Nasa guys don't seem to respond any emails.

It downloads the data and plots it to compare what can be found in ftp://ftp.cpc.ncep.noaa.gov/precip/global_CMORPH/3-hourly_025deg/cmorph-precip_3hrly-025deg.gif

import numpy as np
import os
import datetime
import bz2
import urllib.request as request
from contextlib import closing
import shutil
import matplotlib.pyplot as plt


def get_closest_file_index(hour=None, dataHours=np.array([0,3,6,9,12,15,18,21,24])):
    """
    hour is in decimal format: that is 7:40 am is 7.66666
    assuming here we have 8 data files/ 24h == 0,3,6,9,12,15,18,21
    (24 is with to get datapoints from the next 24h period)
    """
    return (np.argmin(np.abs(float(hour)-dataHours)))

def cmorph_download(year=None, month=None, day=None, hour=None,
                    urlStart='ftp://ftp.cpc.ncep.noaa.gov/precip/CMORPH_V1.0/CRT/0.25deg-3HLY/',
                    ext='.bz2'):
    """
    Returns rain-rate in single grid, first dimension is lat, second lon
    first data is from data[lat=-59.875, lon=0.125] , step is 0.25

    download cmorph water data from
    ftp://ftp.cpc.ncep.noaa.gov/precip/CMORPH_V1.0/CRT/0.25deg-3HLY/

    data is real*4 little endian binary

    ftp://ftp.cpc.ncep.noaa.gov/precip/CMORPH_V1.0/CTL/CMORPH_V1.0_CRT_0.25deg-3HLY.ctl
    There it says:
    OPTIONS template little_endian
    UNDEF  -999.0
    TITLE  Precipitation estimates
    XDEF 1440 LINEAR    0.125  step 0.25 last is 359.875
    YDEF  480 LINEAR   59.875  step 0.25 last is -59.875
    ZDEF   01 LEVELS 1

    motivation:
  
    The CMORPH technique provides a passive
    microwave (PMW) rain-rate dataset that is derived from
    low-Earth-orbit microwave satellite observations, and
    these observations are translated via spatial propagation
    information obtained from geostationary IR1 data.
    """
    fileNumber = get_closest_file_index(hour=hour)
    if fileNumber == 8:
        # data is from the next day...
        # so we take it next day at 0am
        fileNumber = 0
        minute = round((float(hour) - int(hour)) * 60.)
        dataDate = datetime.datetime(int(year), int(month), int(day), int(hour), int(minute))
        year = dataDate.year
        month = dataDate.month
        day = dataDate.day
    print("fileNumber:", fileNumber)
    file = 'CMORPH_V1.0_ADJ_0.25deg-3HLY_' \
           + str(year) + str(month).rjust(2,'0') + str(day).rjust(2,'0') \
           + ext

    ftppath = urlStart \
              + '/' + str(year) \
              + '/' + str(year) + str(month).rjust(2,'0') \
              + '/' + file


    fileName = file
    if not os.path.exists(fileName):
        with closing(request.urlopen(ftppath)) as r:
            print(f"Downloading... {ftppath}")
            with open(fileName, 'wb') as f:
                shutil.copyfileobj(r, f)
    else:
        print("using old file:", fileName)

    with bz2.open(fileName, "rb") as f:
        # Decompress data from file
        data = f.read()
    data = np.fromstring(data, dtype='<f4')
    data = np.reshape(data, (8, (120 * 4), (360 * 4)))
    data = data[fileNumber, ...]
    data = np.squeeze(data)

    return data

data = cmorph_download(year=2012, month=3, day=19, hour=3)
# to compare with
# ftp://ftp.cpc.ncep.noaa.gov/precip/global_CMORPH/3-hourly_025deg/cmorph-precip_3hrly-025deg.gif
plt.contour(data)
plt.show()

Collectives™ on Stack Overflow

read 2-D binary data in Python and write in NetCDF format

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related