Persist POJO in CSV files with Hibernate

Question

Heej,

I'm working on a project where I should persist a lot of POJO files in to an Oracle database. The problem is that for each POJO Hibernate creates a new insert JDBC request. When persisting over 45.000.000 objects, this will take a really long time. I have to save data to a database and that data will be spread over 15 different tables. So for each set of data, I have to do 15 different insert statements. When I have to persist 3.000.000 data sets, 45.000.000 insert statements are needed.

When I was discussing this issue with a colleague, he suggested using CSV files instead of inserting each POJO with an insert statement. Now I'm wondering, is there an easy way of letting Hibernate write for each table a CSV file?

Or maybe someone does know a better solution?

In the end, the CSV files will need to be loaded and inserted in the database as well. What would you gain by first writing to files and then write the contents of the files to the database? — JB Nizet
– JB Nizet, Commented Jun 22, 2012 at 12:16
This bulk load utility will have to use insert statements as well. And I don't see why Hibernate would be faster at writing to an imaginary database using CSV files than writing to a real database. Where do these 3,000,000 data sets come from, BTW? — JB Nizet
– JB Nizet, Commented Jun 22, 2012 at 12:28
I thought using bulk load, you can write directly to the file system of the database. Letting Hibernate write to a file, I think, must me much faster than using a JDBC request, over the network and using SQL. — JordyOnrust
– JordyOnrust, Commented Jun 22, 2012 at 12:39
Possible duplicate of Using Hibernate to work with Text Files — Dave Jarvis
– Dave Jarvis, Commented Jun 3, 2016 at 3:02

MikePatel · Accepted Answer · 2012-06-22 13:53:13Z

-1

Suggest you read hibernate chapter on batch processing : http://docs.jboss.org/hibernate/orm/3.3/reference/en/html/batch.html

answered Jun 22, 2012 at 13:53

MikePatel

2,7022 gold badges27 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

JordyOnrust Over a year ago

I did, many times. But there is nothing about doing one big insert. It tells only about controlling multiple actions.

MikePatel Over a year ago

Oh maybe i've misunderstood. I thought you were asking how to batch the insert statements, which you can do with the session. I guess you already tried that though ?

JordyOnrust Over a year ago

Yeah that's how it works now. But even if I batch all the insert statements, still for every object there will be in insert statement.

MikePatel Over a year ago

And what Id generation strategy are you using ( if using one ?? )

JordyOnrust Over a year ago

I select a sequence from the Oracle db. I hope I can change this to. But this sequence is used also by other users.

|

Collectives™ on Stack Overflow

Persist POJO in CSV files with Hibernate

1 Answer 1

6 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related