1

I have to premise that I a SQL guy and not a C# guy.

I have to ingest such a JSon:

[
{
    "gameId": "a_string_id",
    "name": "A string with the name",
    "width": 1280,
    "height": 720,
    "description": "A lot of mumble jumble A lot of mumble jumble A lot of mumble jumble A lot of mumble jumble A lot of mumble jumble A lot of mumble jumble A lot of mumble jumble A lot of mumble jumble A lot of mumble jumble A lot of mumble jumble",
    "themeUrl": "/address1/address2/address3/filename.jpg",
    "thumbnailUrl": "/address1/address2/address3/filename.jpg",
    "verticalThumbnailUrl": "",
    "helpUrl": "",
    "trivia": [],
    "traits": [
      "aString"
    ],
    "seoName": "a-long-name",
    "friendlyName": "a-friendly-name"
  },
  {
    "gameId": "a_string_id",
    "name": "A string with the name",
    "width": 1600,
    "height": 878,
    "description": "",
    "themeUrl": "/address1/address2/address3/filename.jpg",
    "thumbnailUrl": "/address1/address2/address3/filename.jpg",
    "verticalThumbnailUrl": "",
    "helpUrl": "",
    "trivia": [],
    "traits": [],
    "seoName": "a-long-name",
    "friendlyName": "a-friendly-name"
  }
]

I need to do this with a script task source. I need to put one entity of the json in every row, preferably already separated into columns, because the total json is very long and there is no such variable that can contain it

My code is the following:

using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
using System.Collections.Generic;
using System.Net;
using System.Web.Script.Serialization;
using System.Collections;

/// <summary>
/// This is the class to which to add your code.  Do not change the name, attributes, or parent
/// of this class.
/// </summary>
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{


    /// <summary>
    /// This method is called once, before rows begin to be processed in the data flow.
    ///
    /// You can remove this method if you don't need to do anything here.
    /// </summary>
    public override void PreExecute()
    {
        base.PreExecute();
    }

    /// <summary>
    /// This method is called after all the rows have passed through this component.
    ///
    /// You can delete this method if you don't need to do anything here.
    /// </summary>
    public override void PostExecute()
    {
        base.PostExecute();
    }

    public override void CreateNewOutputRows()
    {
        /*
          Add rows by calling the AddRow method on the member variable named "<Output Name>Buffer".
          For example, call MyOutputBuffer.AddRow() if your output was named "MyOutput".
        */

        String json = DownloadJson("http://someapi.com");


        // Convert json string to .net object using the old school JavaScriptSerializer class
        //JavaScriptSerializer serialize = new JavaScriptSerializer();
        //Root righe = (Root)serialize.Deserialize(json, typeof(Root));

        // Loop through array of earthquakes, outputing desired values to SSIS buffer
        //foreach (var feature in righe.MyArray)
        //{
        //    Output0Buffer.AddRow();
        //    Output0Buffer.gameid = feature.gameid;
        //}
        Output0Buffer.AddRow();
        Output0Buffer.gameid = json;


    }


    public static string DownloadJson(string downloadURL)
    {
        using (WebClient client = new WebClient())
        {
            return client.DownloadString(downloadURL);
        }
    }


    // Root myDeserializedClass = JsonConvert.DeserializeObject<Root>(myJsonResponse); 
    public class MyArray
    {
        public string gameId { get; set; }
        public string name { get; set; }
        public int width { get; set; }
        public int height { get; set; }
        public string description { get; set; }
        public string themeUrl { get; set; }
        public string thumbnailUrl { get; set; }
        public string verticalThumbnailUrl { get; set; }
        public string helpUrl { get; set; }
        public List<string> trivia { get; set; }
        public List<object> traits { get; set; }
        public string seoName { get; set; }
        public string friendlyName { get; set; }

        internal static IEnumerator GetEnumerator()
        {
            throw new NotImplementedException();
        }

        public string gameid { get; set; }
    }

    public class Root
    {
        public List<MyArray> MyArray { get; set; }
    }

    public class MyArrayList : IEnumerable<MyArray>
{
        private List<MyArray> carbootsales;



        public IEnumerator<MyArray> GetEnumerator()
    {
        return carbootsales.GetEnumerator();
    }

    IEnumerator IEnumerable.GetEnumerator()
    {
        return MyArray.GetEnumerator();
    }
}


}

For now I am just trying to work with 1 column, GameId.

The error that I receive is the following:

at Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer.SetString(Int32 columnIndex, String value) at Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer.set_Item(Int32 columnIndex, Object value) at ScriptMain.CreateNewOutputRows() at UserComponent.PrimeOutput(Int32 Outputs, Int32[] OutputIDs, PipelineBuffer[] Buffers, OutputNameMap OutputMap) at Microsoft.SqlServer.Dts.Pipeline.ScriptComponentHost.PrimeOutput(Int32 outputs, Int32[] outputIDs, PipelineBuffer[] buffers)

This error is arabic for me.

If someone colud help, even proposing a new approach to the problem, it would be great.

Thanks

7
  • Which version of SQL Server will you be working with? Commented Jan 12, 2021 at 17:09
  • the version is visual studio 2010 Commented Jan 12, 2021 at 17:16
  • JavaScriptSerializer serialize = new JavaScriptSerializer(); it should be this class that now is commented out. I copied most of this from stack-overflow since this is not really my field. Commented Jan 12, 2021 at 17:20
  • What are you using MyArray, Root and MyArrayList for? It looks like you don't use them at all and just set Output0Buffer.gameid = json;. Does your problem reproduce with just Output0Buffer.gameid = json;? Commented Jan 12, 2021 at 17:20
  • Now it simply runs indefinitely without errors, which is an improvement. I deleted everything after the class myarray. It is strange tough, because they were simply unused classes. Commented Jan 12, 2021 at 17:33

1 Answer 1

4

This block of code throws an exception

String json = DownloadJson("http://someapi.com");
Output0Buffer.AddRow();
Output0Buffer.gameid = json;

It's likely overflowing whatever length you have defined for the string column. But, I had a lunch hour so you have a solution, of sorts ;)

What I did different

I defined a member variable myArray which is an old-school array of type MyArray. When we turn the json into objects, this will be the list that will be at the per item level (game1, game2, etc).

In my PreExecute method, I will download the json and convert it into an in instance of our class typed as an array.

I struck the Root, MyArrayList definitions as I was getting ambiguous definition errors trying to unpack the json.

I removed the enumerator and the lowercased gameid from your MyArray class as the json indicates it should be the first definition of gameId

In my CreateNewOutputRows, I am going to enumerate through the member array and for each item I find, I am going add a new row to the output buffer and then populate it with data.

using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
using System.Collections.Generic;
using System.Net;
// Add reference to System.Web.Extensions
using System.Web.Script.Serialization;
using System.Collections;

// Root myDeserializedClass = JsonConvert.DeserializeObject<Root>(myJsonResponse); 
public class MyArray
{
    public string gameId { get; set; }
    public string name { get; set; }
    public int width { get; set; }
    public int height { get; set; }
    public string description { get; set; }
    public string themeUrl { get; set; }
    public string thumbnailUrl { get; set; }
    public string verticalThumbnailUrl { get; set; }
    public string helpUrl { get; set; }
    public List<string> trivia { get; set; }
    public List<object> traits { get; set; }
    public string seoName { get; set; }
    public string friendlyName { get; set; }


}

/// <summary>
/// This is the class to which to add your code.  Do not change the name, attributes, or parent
/// of this class.
/// </summary>
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
    //Root rootDocument;
    MyArray[] myArray;

    public string DownloadJson(string downloadURL)
    {
        return @"[
{
            'gameId': 'a_string_id',
    'name': 'A string with the name',
    'width': 1280,
    'height': 720,
    'description': 'A lot of mumble jumble A lot of mumble jumble A lot of mumble jumble A lot of mumble jumble A lot of mumble jumble A lot of mumble jumble A lot of mumble jumble A lot of mumble jumble A lot of mumble jumble A lot of mumble jumble',
    'themeUrl': '/address1/address2/address3/filename.jpg',
    'thumbnailUrl': '/address1/address2/address3/filename.jpg',
    'verticalThumbnailUrl': '',
    'helpUrl': '',
    'trivia': [],
    'traits': [
      'aString'
    ],
    'seoName': 'a-long-name',
    'friendlyName': 'a-friendly-name'
  },
  {
    'gameId': 'a_string_id',
    'name': 'A string with the name',
    'width': 1600,
    'height': 878,
    'description': '',
    'themeUrl': '/address1/address2/address3/filename.jpg',
    'thumbnailUrl': '/address1/address2/address3/filename.jpg',
    'verticalThumbnailUrl': '',
    'helpUrl': '',
    'trivia': [],
    'traits': [],
    'seoName': 'a-long-name',
    'friendlyName': 'a-friendly-name'
  }
]";
        //using (WebClient client = new WebClient())
        //{
        //    return client.DownloadString(downloadURL);
        //}
    }

    public override void PreExecute()
    {
        base.PreExecute();
        // You likely want to have this as a Variable and passed in to the method
        string json = DownloadJson("http://someapi.com");

        JavaScriptSerializer serialize = new JavaScriptSerializer();
    myArray = serialize.Deserialize<MyArray[]>(json);

}

    /// <summary>
    /// This method is called after all the rows have passed through this component.
    ///
    /// You can delete this method if you don't need to do anything here.
    /// </summary>
    public override void PostExecute()
    {
        base.PostExecute();
    }

    public override void CreateNewOutputRows()
    {
        foreach (var item in this.myArray)
        {
            Output0Buffer.AddRow();
            Output0Buffer.gameid = item.gameId;
            Output0Buffer.height = item.height;
            Output0Buffer.width = item.width;
            Output0Buffer.name = item.name;
        }
    }

}

In the Inputs and Outputs tab, I have defined 4 of the N columns you have in the MyArray class. Be sure to adjust the DataType and the Length of these to match the maximum expected length. If you need to deal with unicode data, you might need to switch that to Unicode string/DT_WSTR data type.

enter image description here

You can see here I have the two rows flowing through and I have data.

enter image description here

Not addressed in this solution

MyArray has two lists within it: trivia and traits. You won't be able to add them as-is to the buffer because that's not a thing. Pedantic answer is yes you can but realistic answer is don't from the guy with 15 years of doing SSIS.

For these two columns, the answer is driven by how are you going to store this data? Maybe it's going to a table with a poor design decision like storing a delimited list. That's the "easiest" to deal with as you could just zip the data together like

Output0Buffer.trivia = string.Join(";", item.trivia);

That code is approximately correct but it doesn't check for those collections being null or whether the delimiter already exists in it. At any rate, that is for you the data knowledge worker to decide.

A different approach might be that you create an OutputTrivaBuffer and send gameId and a single trivia value down a separate path.

There's no error handling at all so you'll want to fortify this code.

You'll likely want to add a Variable to the SSIS package with the API's url in it and then pass that as a ReadOnly parameter to the Script Component so you can point at different end points (dev/prod) without editing the package.

Sign up to request clarification or add additional context in comments.

1 Comment

the solution you proposed worker, thank you very much.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.