NodeJS csv data dump into s3 object

Question

I'm a beginner with AWS, JavaScript and I'm trying to get data from an oracle database using the following code in a AWS Lambda function.

 let conn = await this.getConnection(userName, password, connectString);
 let results = await conn.execute(query, {}, {});
 let data = results.rows.map(el => `${el}`);
 console.log(data);

which gives me the following output on the console

[ "TEST_SESSION_0,121,High,Warning,7",
"TEST_SESSION_2,122,High,Error,11",
"TEST_SESSION_3,123,Low,Error,431,
"TEST_SESSION_4, ASMT_,103,Low,Fatal,10" ]

Now, my goal is to convert this output into CSV data stream and then upload into S3 bucket object output.csv for further processing. i tried below to convert into CSV but having issue to connect it back to S3 using AWS SDK and required NPM modules, when i test the below code i dont get any error but at the same time dont see data getting uploaded to S3. not sure where i'm doing wrong, appreciate any help..

stringify(results.rows, function(err, output){
  if(err){
    console.log('some error occured '+ err);
  } else {
     console.log(output); //gives me below output
//TEST_SESSION_0,121,High,Warning,7
//TEST_SESSION_2,122,High,Error,11
//TEST_SESSION_3,123,Low,Error,431
//TEST_SESSION_4, ASMT_,103,Low,Fatal,10

below code is to take CSV output to upload to S3 object

     var param = {
       Bucket: 'Test-Bucket',
       Key: 'output.csv',
      Body: new Buffer(fs.createWriteStream(output)) //can i pass the csv data that i captured above in "output"?
      };  
    s3bucket.putObject(param, function(err, output){
   if(err) console.log(err);
  else console.log(output);
});
}
})

Can I know if what are you trying to achieve via this?

Shivang Agarwal
– Shivang Agarwal

2019-03-18 17:35:19 +00:00
Commented Mar 18, 2019 at 17:35 — Shivang Agarwal
– Shivang Agarwal, Commented Mar 18, 2019 at 17:35

madebydavid · Accepted Answer · 2019-03-18 17:22:00Z

3

I would start by separating out the code for the S3 upload and working on that part by itself - trying to get it to create a file in the bucket containing the text from a string.

The putObject function expects a Buffer as the Body param. To construct this from a string you can use the Buffer.from method:

var AWS = require('aws-sdk');

var output = 'This is a test';
var myBody = Buffer.from(output);

var param = {
  Bucket: 'Test-Bucket',
  Key: 'output.csv',
  Body: myBody
};

var s3bucket = new AWS.S3({ signatureVersion: 'v2' });
s3bucket.putObject(param, function(err, output) {
  if(err) {
    console.error(err);
  } else {
    console.log(output)
  }
});

Once you have that part working perfectly, plug it into the other code and change output to be the contents of the CSV file.

NB you will need the following environment variables set:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

edited Mar 18, 2019 at 17:22

answered Mar 17, 2019 at 14:35

madebydavid

6,5172 gold badges25 silver badges30 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

madebydavid Over a year ago

Are you initialising the s3bucket var? I will update my code with an example of this.

user1411254 Over a year ago

actually, the problem was since i'm testing the Lambda locally on my PC i didn't update the env variable that will have 'output.csv' and that is the reason i was getting bucket undefined error. i was able to test it successfully after making that change. thank you so much!!

Collectives™ on Stack Overflow

NodeJS csv data dump into s3 object

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related