XML To Custom SQL Tables

Question

I have a large XML data feed (5GB) dropped to me daily (I need to get it via HTTP). This feed is a representation of some tables which are too complex for my needs, I want to trim them down as much as possible. I have refined the table design to include only what I absolutely need, but I am not sure of what the quickest technology would be.

I am doing this in Azure, using C# .NET 4.5 and using SQL Server (2012 if my memory is correct). I would expect SSIS to be the best option, but I do not believe Azure supports it at the moment. If it does (I can't find any docs to support this), could you point me towards some examples/documentation?

Thanks!

Brendan Green · Accepted Answer · 2014-03-05 21:57:40Z

1

You could upload the XML data feed into Azure Table Storage, and then process it via a web job (see: http://www.windowsazure.com/en-us/documentation/articles/web-sites-create-web-jobs/).

Since you mention that you receive a daily feed, Web Jobs can be scheduled.

answered Mar 5, 2014 at 21:57

Brendan Green

11.9k5 gold badges47 silver badges82 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Faraday Over a year ago

How would I go from table storage to SQL? Also, what would put the feed in table storage? That part would need to be timed, not the processing part.

Brendan Green Over a year ago

You say "I have a large XML data feed (5GB) dropped to me daily", with no mention on how this is done. Can the sender drop it directly to Azure instead? Your Web Job would read the XML data feed and do whatever processing it is that you need to do to "trim it down" and insert the result into SQL.

Faraday Over a year ago

I used the wrong term. I need to do a get via HTTP (question updated with that detail). So I need to be able to ping a http/ftp URL, when the file is there, get it, then throw it into SQL Azure.

Brendan Green Over a year ago

It all still applies. See this additional resource: asp.net/aspnet/overview/developing-apps-with-windows-azure/…. Create a scheduled job to ping the server and download via HTTP and store in Azure Table Storage.

Faraday Over a year ago

Hey, how would you put it into SQL? I know I'd have to stream it via the streamreader to do it via C#. But is there really no way at all of invoking SSIS in Azure for this?

Collectives™ on Stack Overflow

XML To Custom SQL Tables

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related