Run Stream Analytic Transformation Query Programmatically using .NET

Question

I am trying to use .NET to automate the running of Stream Analytic Transformation Query. I've read the documentation of the two framework

Microsoft.Azure.Management.StreamAnalytics

Microsoft.Azure.Management.StreamAnalytics.Models

and also this guide on how to run a Stream Analytic Job using .NET.

They explain how to Create, Run and Stop a Job but what I need is a bit different. In relation to the following picture

I manually, upload a json file (upload sample Input in the image), Test the query (Test query button in the image) and download the result (button not shown in the image).

I want to do all this steps programmatically using .NET but not sure on what objects I need to use. The following shows how to create a Transformation, in my case I have one and want to run it. (I don't want to run the Job because I do not have authorization to do this in my Azure subscription but have authorization to run the query).

// Create a transformation
Transformation transformation = new Transformation()
{
    Query = "Select Id, Name from <your input name>", // '<your input name>' should be replaced with the value you put for the 'inputName' variable above or in a previous step
    StreamingUnits = 1
};
Transformation createTransformationResult = streamAnalyticsManagementClient.Transformations.CreateOrReplace(transformation, resourceGroupName, streamingJobName, transformationName);

The ITransformationsOperations Interface doesn't have any method to run a tranformation. Is it possible to do it, can anyone help?

Florian Eiden · Accepted Answer · 2021-06-18 22:09:21Z

1

To my knowledge the SDK only allows to manage remote resources deployed in a subscription that you own. The query testing experience available through the portal is not part of it, as the backend is an internal service.

If you want to run a query locally and programmatically, you can use the npm package available as part of the CI/CD tooling.

answered Jun 18, 2021 at 22:09

Florian Eiden

8425 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

vinx Over a year ago

Thanks a lot @Florian, I' ve tried the npm package and it seems just the tool I needed. Quite strange, in my view, there is no such a library available in .NET.

vinx Over a year ago

I am using the azure-streamanalytics-cicd tool but and it works fine generally. Anyway, I am not able to configure the "ExpectedOutputs" correctly. I mean, can it compare the generated output with an expected one? Is it correct to set "Required" to true and pass a "FilePath"? I do not get any failing test, even when the output is not generated at all. Am I missing something?

Florian Eiden Over a year ago

Hey Vinx, you can start another question if it's more comfortable than using these comments ;) Without more details I'm not sure what's going wrong. The process is to first create a test case with addtestcase, create an expected result file by yourself, and point to it in the test case file (with required set to true yes). Then you run the test via the test command. The whole thing is described here

vinx Over a year ago

Thanks Florian, I think I've figured out the problem. In order for the tool to say if a test pass or not you need to provide an expected output in any case. Even when you do not expect an output. I am setting it this way and my results are consistent.

Florian Eiden Over a year ago

I checked and apparently an empty ExpectedOutputs.FilePath in testConfig.json will trigger an "Object reference not set to an instance of an object" error. You can put any kind of string in there when the output is not required, it doesn't need to be an actual filepath at all, it just needs to not be an empty string "".

Collectives™ on Stack Overflow

Run Stream Analytic Transformation Query Programmatically using .NET

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related