send large csv file from NodeJS to python

Question

I need to send large csv file from node to python. this code is working for small file but not for large files. i have tried spawn process too. i am not understanding whats the problem. if anyone knows the correct code do share

Code:

const express=require('express')
const app=express()    
let p = require('python-shell');
const fs = require('fs');
let filledarray=[]

fs.createReadStream('data.csv')

.pipe(csv())

.on('data', (row) => {



filledarray.push(row)

})

.on('end', () => {

   console.log('CSV file successfully processed');

});

app.get('/send',(req,res)=>{



  var options = {
       args:
       [
           JSON.stringify(filledarray)
       ]
  }
  p.PythonShell.run('hello.py', options, function  (err, results)  {

         if(err) {
           console.error(err)
         }
         else{
            console.log(results)
            res.send(results)
         }

  });

})

app.listen('5000')

Error

 Error: spawn ENAMETOOLONG at ChildProcess.spawn (internal/child_process.js:394:11) at Object.spawn 
 (child_process.js:535:9)

Marcos Casagrande · Accepted Answer · 2019-12-20 18:46:16Z

You are sending a lot of data as arguments to the script hello.py, that's why you're getting ENAMETOOLONG.

You need to change your Python script to receive data from stdin, and use pyshell.send(data);

let pyshell = new PythonShell('hello.py', { mode: 'text' });

// sends a message to the Python script via stdin
pyshell.send('hello');

You can use one of the 3 following modes:

use text mode for exchanging lines of text
use json mode for exchanging JSON fragments
use binary mode for anything else (data is sent and received as-is)

In your particular case, you can use json and send each row individually. Then in your python script you can use the following, taken from python-shell examples.

I don't know any python

import sys, json

# simple JSON echo script
for line in sys.stdin:
  print(json.dumps(json.loads(line)))

let pyshell = new PythonShell('hello.py', { mode: 'json' });

fs.createReadStream('data.csv')
.pipe(csv())
.on('data', (row) => {
   pyshell.send(row);
})

Collectives™ on Stack Overflow

send large csv file from NodeJS to python

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related