Fastest way to migrate from sql server to sqlite for large datasets

Question

I have a database with more than 32 million records, I have to migrate it from SQL Server to Sqlite.

I have tried SSIS (SQL Server Integration Services) with the help of this article http://dbauman.blogspot.com/2009/03/connecting-to-sqlite-through-ssis.html

but the process is very very slow, how can I migrate this data?

How much time does it take for you to dump the data into Sqlite using SSIS ? By enhancing the performance of DFT ,you can some how increase the speed — praveen
– praveen, Commented Jun 9, 2012 at 11:17
In the article ODBC is used, which usually is a poor choice if you have performance in mind; it just adds another abstraction layer. I found SQLite to be quite fast (I don't have any statistics at hand) when executing loads of prepared INSERT commands within a transaction, I never worked with SSIS unfortunately. — C.Evenhuis
– C.Evenhuis, Commented Jun 9, 2012 at 11:23
Just give a try :-If you have indexes in your sqlite table then try to drop them and load the data and then rebuilt it again — praveen
– praveen, Commented Jun 9, 2012 at 11:36

Damith · Accepted Answer · 2012-06-09 12:29:23Z

21

There is a C# utility to automatically do conversion from a SQL Server DB to a SQLite DB by liron.

Here is the code project article.

answered Jun 9, 2012 at 12:29

Damith

63.2k13 gold badges106 silver badges154 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Mare Infinitus · Accepted Answer · 2012-06-09 11:29:17Z

1

I suggest to use transactions in SQLite, else the Indices are built with every new record added, not when one block of work is done.

As i don't know about the structure of your data, it is hard to give concrete advice, but writing a small application that can read from SQLServer and write to SQLite is often a practical start for migration. With the bonus of having a full wrapper after migration, which you can use in future.

answered Jun 9, 2012 at 11:29

Mare Infinitus

8,1799 gold badges70 silver badges113 bronze badges

1 Comment

Saw Over a year ago

As @Damith mentioned the solution in this article: codeproject.com/Articles/26932/…

Collectives™ on Stack Overflow

Fastest way to migrate from sql server to sqlite for large datasets

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related