How to extract data from NetCDF file using C# and Microsoft Scientific Data Set?

Question

I am trying to extract data from a NetCDF file using Microsoft's Scientific Data Set libraries (SDS). I'm restricted to using MS SDS and C#. I'm not a programmer by trade so I'm struggling to get the basics working. To begin with I'd like to write a simple script to extract data and write it to a .csv file. Using the introduction doc and codeplex tutorials. I've tried to write a simple C# console app which just reads the file and writes it out to the console or ideally to a .csv file.

using SDS 1.3 command line I can view the contents of my test file:

D:\NetCDF>sds list test.nc
[2] ACPR of type Single (Time:85) (south_north:213) (west_east:165)
[1] Times of type SByte (Time:85) (DateStrLen:19)
D:\NetCDF>

My script looks like this:

using System;
using System.IO;
using sds = Microsoft.Research.Science.Data;
using Microsoft.Research.Science.Data.Imperative;


namespace NetCDFConsoleApp
{
    class Program
    {
        static void Main(string[] args)
        {
            /// Gets the path to the NetCDF file to be used as a data source.
            var dataset = sds.DataSet.Open("D:\\NetCDF\\test.nc?openMode=readOnly");

            SByte[,] times = dataset.GetData<SByte[,]>("Times");
            //Int32[,] times = dataset.GetData<Int32[,]>("Times");

            //Single[] longitudes = dataset.GetData<Single[]>("west_east");
            //var latitudes = dataset.GetData<Single[]>("south_north");
            Single[,,] dataValues = dataset.GetData<Single[,,]>("ACPR");

            for (int iTime = 50; iTime < 60; iTime++)
            {
                for (int iLongitude = 130; iLongitude < 150; iLongitude++)
                 {
                    for (int iLatitude = 130; iLatitude < 140; iLatitude++)
                    {
                        // write output data
                        float thisValue = dataValues[iTime,iLatitude,iLongitude];

                        Console.WriteLine(iTime);
                        Console.WriteLine(iLatitude);
                        Console.WriteLine(iLongitude);
                        Console.WriteLine(thisValue);

                    }
                 }
            }
        }
    }
}

If I comment out the var Times... line it runs. But I'm struggling to get SDS to read Time Dimension. If I use SByte it complains that the variable doesn't exist. If I use Int32 it complains about converting to string.

System.InvalidOperationException was unhandled
  HResult=-2146233079
  Message=Requested variable does not exist in the data set
  Source=ScientificDataSet
  StackTrace:
       at Microsoft.Research.Science.Data.Imperative.DataSetExtensions.FindVariable(DataSet dataset, Func`2 predicate)
       at Microsoft.Research.Science.Data.Imperative.DataSetExtensions.GetData[D](DataSet dataset, String variableName)
       at NetCDFConsoleApp.Program.Main(String[] args) in \\gdc-fs01\user$\prm\Visual Studio 2015\projects\NetCDFConsoleApp\Program.cs:line 16
       at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
       at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
       at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
       at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading.ThreadHelper.ThreadStart()
  InnerException:

What am I missing?

For what I can understand (maybe I'm wrong) west_east and south_nort aren't variables of the dataset but part of the ACPR variable. — Gusman
– Gusman, Commented Sep 10, 2017 at 21:52
@Gusman Thanks for your reply. I've updated my code and post. Somehow my code is treating Time as a variable rather than a dimension. I now have the script spewing numbers out to the console. (Yay!) but still not reading and using the dimensions correctly. — Sir Swears-a-lot
– Sir Swears-a-lot, Commented Sep 11, 2017 at 4:55
This gets a response from the command line: E:\temp>sds data test.nc ACPR[1:10] — Sir Swears-a-lot
– Sir Swears-a-lot, Commented Dec 18, 2017 at 1:36

Sir Swears-a-lot · Accepted Answer · 2017-09-13 04:38:26Z

I think I have solved the puzzle. The problem is that the 2nd variable is empty. I'm not sure if that is by accident or design.

I have found the Date of when data is supposed to start in a meta data field. I've modified my code now so that it retrieves this, and writes it all to the console.

using System;
using System.IO;
using sds = Microsoft.Research.Science.Data;
using Microsoft.Research.Science.Data.Imperative;


namespace NetCDFConsoleApp
{
    class Program
    {
        static void Main(string[] args)
        {
            /// Gets the path to the NetCDF file to be used as a data source.
            var dataset = sds.DataSet.Open("D:\\NetCDF\\test.nc?openMode=readOnly");

            string dt = (string)dataset.Metadata["START_DATE"];

            Single[,,] dataValues = dataset.GetData<Single[,,]>("ACPR");

            for (int iTime = 50; iTime < 60; iTime++)
            {
                for (int iLongitude = 130; iLongitude < 150; iLongitude++)
                {
                    for (int iLatitude = 130; iLatitude < 140; iLatitude++)
                    {
                        // write output data
                        float thisValue = dataValues[iTime,iLatitude,iLongitude];                                                
                        Console.WriteLine(dt.ToString() + ',' + iTime.ToString() + ',' + iLatitude.ToString() + ',' + iLongitude.ToString() + ',' + thisValue.ToString());

                    }                 
                }
            }            
            Console.ReadLine();
        }           
    }
}

I've really struggled with this so I'm sharing this in the hope it will be of use to someone else.

One thing I really found useful was the discussion tab on Codeplex as it has lots of useful code snippets.

Collectives™ on Stack Overflow

How to extract data from NetCDF file using C# and Microsoft Scientific Data Set?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related