1

I am defining a Stream Analytics Job via an Azure Resource Manager (ARM) template.

I need to persist some messages from an Event Hub input into a SQL Server database output.

when running this command:

az group deployment create \
  --name "deployStreamAnalyticsJobs" \
  --resource-group "CiTestRG" \
  --template-file ./templates/stream-analytics-jobs-definition.json 

then I see this output:

Deployment failed. Correlation ID: <ONE_GUID>. {
  "code": "422",
  "message": "The required property 'type' is missing from the request.",
  "details": {
    "code": "422",
    "message": "The required property 'type' is missing from the request.",
    "correlationId": "<ANOTHER_GUID>",
    "requestId": "<YET_ANOTHER_GUID>"
  }
}

but according to this documentation: https://learn.microsoft.com/en-us/azure/templates/microsoft.streamanalytics/streamingjobs#MicrosoftSqlServerDatabaseOutput

I should only be providing the database output details as:

"datasource": {
    "type": "Microsoft.Sql/Server/Database",
    "properties": {
        "server": "string",
        "database": "string",
        "user": "string",
        "password": "string",
        "table": "string"
    }
}

I think what the error message of the az command is trying to say is that those properties are not defined as string, but then why the docs don't specify the type but just pass those properties as string values?

This is the JSON definition of the ARM template:

{
    "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "location": {
            "type": "string",
            "defaultValue": "westeurope"
        },
        "hubName": {
            "type": "string",
            "defaultValue": "myIoTHub"
        },
        "eventhubs_messages_name": {
            "defaultValue": "myEhName",
            "type": "String"
        },
        "namespaces_oecollector_name": {
            "defaultValue": "myEventHub",
            "type": "String"
        },
        "streamAnalyticsJobName": {
            "type": "string",
            "defaultValue": "myStreamAnalyticsJob"
        }
    },
    "resources": [{
        "type": "Microsoft.StreamAnalytics/StreamingJobs",
        "apiVersion": "2016-03-01",
        "name": "[parameters('streamAnalyticsJobName')]",
        "location": "[resourceGroup().location]",
        "properties": {
            "sku": {
                "name": "standard"
            },
            "outputErrorPolicy": "Drop",
            "eventsOutOfOrderPolicy": "adjust",
            "eventsOutOfOrderMaxDelayInSeconds": 0,
            "eventsLateArrivalMaxDelayInSeconds": 86400,
            "inputs": [{
                "Name": "EventHubOutputLable",
                "Properties": {
                    "DataSource": {
                        "Type": "Microsoft.ServiceBus/EventHub",
                        "Properties": {
                            "eventHubName": "[parameters('eventhubs_messages_name')]",
                            "serviceBusNamespace": "[parameters('namespaces_oecollector_name')]",
                            "sharedAccessPolicyKey": "[listKeys(resourceId('Microsoft.EventHub/namespaces/eventhubs/authorizationRules', parameters('namespaces_oecollector_name'), parameters('eventhubs_messages_name'), 'super-user'),'2017-04-01').primaryKey]",
                            "sharedAccessPolicyName": "RootManageSharedAccessKey"
                        }
                    },
                    "Serialization": {
                        "Properties": {
                            "Encoding": "UTF8"
                        },
                        "Type": "Json"
                    }
                }
            }],
            "outputs": [{
                "Name": "myOutputDb",
                "Properties": {
                    "DataSource": {
                        "Type": "Microsoft.Sql/Server/Database",
                        "Properties": {
                            "Server": "my-sql-server-name",
                            "Database": "my-database-name",
                            "User": "my-sa-user",
                            "Password": "my-password",
                            "Table": "MySchema.MyTable"
                        }
                    }
                }
            }],
            "transformation": {
                "name": "Transformation",
                "properties": {
                    "streamingUnits": 1,
                    "query": "WITH data AS (\r\n      SELECT\r\n        <THE_FIELDS_OF_THE_JSON_INPUT_AS_COLUMNS>\r\n      FROM\r\n        input\r\n      WHERE\r\n        topic = 'FOO'\r\n    )\r\n    \r\n    SELECT * INTO myOutputDb FROM data"
                }
            }
        }
    }]
}

How can I make this Stream Analytics Job picking messages from an input Event Hub and storing them into an output SQL Server database?

1 Answer 1

2

Your Input properties object has "DataSource" and "Serialization" but is missing "type". Without this, there is no way of knowing if the input is stream input or reference data input. Once you add it, your input properties object should look like:

"Properties": {
                "type": "stream",
                "DataSource": {
                    "Type": "Microsoft.ServiceBus/EventHub",
                    "Properties": {
                        "eventHubName": "[parameters('eventhubs_messages_name')]",
                        "serviceBusNamespace": "[parameters('namespaces_oecollector_name')]",
                        "sharedAccessPolicyKey": "[listKeys(resourceId('Microsoft.EventHub/namespaces/eventhubs/authorizationRules', parameters('namespaces_oecollector_name'), parameters('eventhubs_messages_name'), 'super-user'),'2017-04-01').primaryKey]",
                        "sharedAccessPolicyName": "RootManageSharedAccessKey"
                    }
                },
                "Serialization": {
                    "Properties": {
                        "Encoding": "UTF8"
                    },
                    "Type": "Json"
                }
            }
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.