How to read multiple csv files in node.js

Question

I'm trying to read multiple csv files and then add them as properties to an object, but the async of node.js really prevents me from doing this.

This is the code I tried.

var Converter=require("csvtojson").core.Converter; 
var fs = require('fs');
var csvConverter=new Converter();
var data = {};

for (var i = 0; i < 10; i++)
{   
    var csvFileName="./data/String(i)".csv";

    csvConverter.on("end_parsed",function(jsonObj){
    data['file'+String(i)] = json;


    //read from file
    fs.createReadStream(csvFileName).pipe(csvConverter);
});

}

This leads to a horrible mess, as the for-loop finishes before any of the events are triggered. I would really prefer a synchronous solution to this, but I understand that node.js simply isn't built for that.

So I would be pleased if someone could help me understand how to fix this async. However, this is the first time I've had such a problem. An hour ago I didn't know about the concept asynchronous code.

You have quite a few bugs/formatting issues in your example that should be cleaned up to remove any confusion as to what may be the source of your issue. I submitted an edit to correct them. — Jason Cust
– Jason Cust, Commented Jun 8, 2015 at 20:17
Learn to use promises. Bluebird is a great promise library and will let you make portions of your code synchronous — Tyler
– Tyler, Commented Jun 8, 2015 at 21:26

beautifulcoder · Accepted Answer · 2015-06-09 17:03:05Z

1

i is problematic since you can't predict what it is going to be inside the end_parsed callback function, so try:

var fileIndex = 0;
csvConverter.on('end_parsed', function (json) {
    data['file' + fileIndex] = json;
    if (fileIndex === 9) {
       // It is done, call the callback to finish processing
       callback(data);
    }
    fileIndex += 1;
});

var callback = function (data) {
    // Put callback code here
    console.log(data.file0);
};

Really the best way to solve this is to use Promises.

edited Jun 9, 2015 at 17:03

answered Jun 8, 2015 at 21:24

beautifulcoder

11.4k3 gold badges21 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Eivind Over a year ago

How does the 'end parsed' event know about i?

beautifulcoder Over a year ago

As long as it's inside the for loop, it knows about i. Actually JavaScript is function scoped, so the inner function has access to all the outer function's variables

Eivind Over a year ago

What I experienced was that the forloop completed long before any of the "end parsed" events, so i was of no use.

Eivind Over a year ago

I don't understand this. I'm not familiar with callback functions. Can you please write the exact code?

Collectives™ on Stack Overflow

How to read multiple csv files in node.js

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related