1

What is the better way to insert large data into multiple tables in DB from CSV with foreign keys?

I tried to import large csv file into multiple tables and I use the normal Mysql query with looping csv data, but it was very slow. Then I tried 'LOAD DATA' method.But it is only possible to insert data into a single table with same column order.

However, it will not work for because I need to insert data to multiple tables and need to use last

inserted id of one table to join another table.

Can anyone suggest a better way to upload data into multiple tables?

Here, I am using Zend Framework with Doctrine for handling database operations. However, from my understanding Doctrain is taking more time than normal MySQL query, that is the reason why I used direct MySQL Query.

2
  • 1
    Possible duplicate of Best practices for importing large CSV files Commented May 30, 2018 at 13:06
  • However, above link does not discussing about foreign keys, that is 'id' of one table need to insert as foreign key in another table. Commented May 31, 2018 at 10:18

2 Answers 2

1

You can import large files around millions of records in one shot from below method, please use mysql Load DATA

for ex-

LOAD DATA LOCAL INFILE  
'/PATH_TO_YOUR_CSV/YOUR_CSV.csv'
INTO TABLE MY_TABLE
FIELDS TERMINATED BY ',' 
ENCLOSED BY '"'
LINES TERMINATED BY '\n';

Please remember your table structure and your csv file structure (no of columns) should be same

Sign up to request clarification or add additional context in comments.

1 Comment

Thanks for your replay. However I need to insert data to more than one table with foreign keys. ie, Last insert Id of one table should need to insert as foreign key in another table.
0
-- import the csvfile to this temp table , the column just like your csv header
create table IF NOT EXISTS my_db.csv_file (

    id int auto_increment primary key,
    vendor_name varchar(200),
    product_name varchar(200),
    product_price double
    
);

-- table one
create table if not exists my_db.product(
    id int auto_increment primary key,
    vendor_id int, -- foreign
    name varchar(200),
    price double
)

-- table two
create table if not exists my_db.vendor(

    id int auto_increment primary key,
    name varchar(200)
)

-- import csv data ,you can use LOAD command 
insert into my_db.csv_file(vendor_name,product_name,product_price)values
('A','book',1.00),
('B','computer',2.00),
('C','phone',3.00);


-- step 1:

insert into my_db.vendor(name)
(select vendor_name from my_db.csv_file group by vendor_name);

-- step 2:
insert into my_db.product(vendor_id , name , price)
(select vendor.id,temp.product_name,temp.product_price from my_db.csv_file as temp left join my_db.vendor as vendor on vendor.name = temp.vendor_name);

1 Comment

Please share more details. If the initial queries were too slow, how does your approach handle this better?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.