0

Having issue with exporting each line item from an SQL server database into it's own XML file.

All the questions I've read on site all have various knowledge points that don't solve the answer for me and I am having one or two pointed gaps in knowledge letting me down. Basically for each item (~350 rows in a table called BPAProcess which exists across 6 databases, each database is basically a different version they're independent of each other. so ideally I can execute this query 6 times manually once on each db to extract all the rows from this table) I want to take XML for each row and save it locally, the xml contents of the cell are already suitable for xml structure they just need to export. So far I've tried creating a loop mechanism to loop over a temp table I created then for each row item extract it into an SQL command to save the details out. I have gotten this far where BCP is failing because it can't see the temporary table ive created, after reading up on this I think its going to fail once i resolve this issue in some other design because there is a character limit of 8k on fiel output but these files are ~300k long each.

    USE [BP6.8.1]
GO
 
-- Save XML records to a file:
DECLARE @fileName VARCHAR(175)
DECLARE @filePath VARCHAR(175)
DECLARE @sqlStr VARCHAR(1000)
DECLARE @sqlCmd VARCHAR(1000)

DECLARE @MaxRowsCount INT
DECLARE @Iter INT



select ROW_NUMBER() OVER (order by processid) row_num, name, processxml into #Process from BPAProcess
SET @MaxRowsCount = (SELECT MAX(row_num) FROM #Process)
SET @Iter = 1


WHILE @Iter <= @MaxRowsCount
BEGIN
 
SET @fileName = (select name from #Process where row_num = +@Iter)
SET @filePath = 'C:\Temp\sql queries\'+ @fileName +'.xml'
SET @sqlStr = 'select processxml from #Process where row_num ='+cast(@Iter as varchar(3))
 
SET @sqlCmd = 'bcp "' + @sqlStr + '" queryout "' + @filePath + '" -w -T'
 
--EXEC xp_cmdshell @sqlCmd


SET @Iter = @Iter + 19090

-- +19090 just to execute first iteration only
END
Drop TABLE #Process

I feel what I need is some way to loop the #Process table and for each item at the row_num then export it out via stdout/similar or create a looping mechanism to do a bpc command with the sql pointing at the xml cell I want but I don't know how to make bpc see the table I'm creating.

Few caveats, this is not my DB its a db used by an application and changing anything is not an option. The task is to take each individual XML and store it on a file drive saved as [name].xml where name is in the table. I know this isn't (from reading comments around everywhere) the correct way to use SQL, admittedly im not an SQL developer nor do we have one to hand but Ive been tasked with exporting this code and the manual way on the GUI will take several weeks as it's a long a laborious process wheras this would be much faster. The XML contents are quite long like i said 300k->500k in some instances.

Any help is appreciated, and if that help is 'this is not appropriate for SQL to be executing' that would be fine I could go explore it in C# or some other language potentially if this really isn't the way it should be done.

11
  • "~350x6 databases"? You have over 2,000 databases? What do you for each of the databases you want to export the row? Are you saying you want to iterate through every table is all 2,000+ of your databases and put it into XML? With, respect, really? Smells like an XY Problem Commented Nov 22, 2021 at 11:53
  • 1
    Neither bcp.exe nor any other external process can see #-temp tables because they are scoped to the connection that created them. Have you considered an external process, such as a PowerShell script, that queries the BPAProcess tables directly and can write the file data itself? Commented Nov 22, 2021 at 11:55
  • haha sorry that should read 350s in about 6 databases each so ~2k rows in all., I don't want to iterate through each database I can execute the query 6 times manually that's fine Commented Nov 22, 2021 at 12:27
  • @AlwaysLearning I dont know much about powershell but if that better then I could progress down that route alright Commented Nov 22, 2021 at 12:28
  • @Larnu I have updated the question syntax now. Think that should clear up the misunderstanding Commented Nov 22, 2021 at 12:31

2 Answers 2

1

When you call someone rude, should you expect an actual answer? A developer with any significant experience (you) should be able to find possible solutions and evaluate their usability. As an example, searching "sql server export one row to file" finds first entry using a cursor.

Quite frankly you seem to have significant experience in different languages so I question why you chose a TSQL based solution rather than one involving whatever language that is your strong suit. But that's a very different issue. No matter - RBAR is still RBAR regardless of language / development platform.

Your code converted to a simple cursor is below. There are some things you assume and I again suggest you use a language that you are proficient in.

declare @sql varchar(500);
declare @name varchar(20);
declare c1 cursor  FAST_FORWARD for 
    select name from dbo.mytable order by name;
    
open c1;
fetch next from c1 into @name;
while @@fetch_status = 0
begin
   --print @name;

   set @sql = 'select processxml from dbo.mytable where name = ''' + @name + ''''; 
   set @sql = 'bcp "' + @sql + '" queryout "' + 'C:\Temp\sql queries\'+ @name +'.xml " -w +T'
   print @sql; 
   
   fetch next from c1 into @name;
end;

close c1;
deallocate c1;

You might need to adjust this to connect to the correct instance and database using a specific login. The documentation has cursor examples - always a good starting point. Perhaps the first thing you should try is to use BCP from a command prompt to export that column in a single specific row to a specific file to validate your assumptions and expectations first.

With just a little work you could add some additional logic to run this for multiple databases within a given instance. Doing that would require handling name collisions in the set of files generated.

Lastly, your code as is but without the reliance on the temp table with the BCP command. Note the small adjustment because of the simplified table in the fiddle.

--select ROW_NUMBER() OVER (order by processid) row_num, name, processxml 
select ROW_NUMBER() OVER (order by name) row_num, name 
   into #Process 
   from dbo.mytable
SET @MaxRowsCount = (SELECT MAX(row_num) FROM #Process)
SET @Iter = 1

WHILE @Iter <= @MaxRowsCount
BEGIN
 
   SET @fileName = (select name from #Process where row_num = +@Iter)
   SET @filePath = 'C:\Temp\sql queries\'+ @fileName +'.xml'
   SET @sqlStr = 'select processxml from dbo.mytable where row_num ='+cast(@Iter as varchar(3))
 
   SET @sqlCmd = 'bcp "' + @sqlStr + '" queryout "' + @filePath + '" -w -T'
   print @sqlCmd;
   --EXEC xp_cmdshell @sqlCmd

   SET @Iter = @Iter + 1
END
Drop TABLE #Process

fiddle to demonstrate both.

Sign up to request clarification or add additional context in comments.

1 Comment

RBAR isn't RBAR in every language. A lot of languages have enumerator or pipeline capabilities. Either way, TSQL is just the wrong tool
1

This is really not a job for T-SQL, it is not a generalized scripting language, it is only meant for querying.

Instead, use Powershell to extract the XML as separate rows, then output them into files

Invoke-Sqlcmd
  -ServerInstance "YourServer"
  -Database "YourDB" -Username "user" -Password "pass"
  -Query "select name, processxml from dbo.mytable;"
| % {
    Out-File -FilePath ("Downloads\temp\" + $_.name + ".xml") -InputObject $_.processxml
}

You could also use any other client app to do the same, far more easily than in T-SQL.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.