How to write a numpy array as a csv to S3

Question

I have a numpy ndarray with 2 columns that looks like below

[[1.8238497e+03 5.2642276e-06]
 [2.7092224e+03 6.7980350e-06]
 [2.3406370e+03 6.6842499e-06]
 ...
 [1.7234612e+03 6.6842499e-06]
 [2.1071147e+03 2.1332115e-05]
 [2.6937273e+03 6.2854801e-06]]

I want to write it as a csv with column names in S3 directly (without creating a local csv file) using s3fs. I know how to do it if I had a dataframe and I can convert my numpy to dataframe. But I would like a way to do it without the need to converting the numpy array to a dataframe.

Below is the code I have

output = [[1.8238497e+03 5.2642276e-06]
     [2.7092224e+03 6.7980350e-06]
     [2.3406370e+03 6.6842499e-06]
     ...
     [1.7234612e+03 6.6842499e-06]
     [2.1071147e+03 2.1332115e-05]
     [2.6937273e+03 6.2854801e-06]]
## write the outpout in csv at s3
s3 = s3fs.S3FileSystem(anon=False)
dftowrite = pd.DataFrame({'col1': output[:, 0], 'col2': output[:, 1]})

with s3.open(f"{'s3://bucket'}/result.csv",'w') as f:
     dftowrite.to_csv(f)

I would like to do it without converting the NumPy array output to a dataframe dftowrite. Any suggestion?

jellycsc · Accepted Answer · 2021-07-31 23:35:23Z

1

Try this

output = [[1.8238497e+03 5.2642276e-06]
     [2.7092224e+03 6.7980350e-06]
     [2.3406370e+03 6.6842499e-06]
     ...
     [1.7234612e+03 6.6842499e-06]
     [2.1071147e+03 2.1332115e-05]
     [2.6937273e+03 6.2854801e-06]]
## write the outpout in csv at s3
s3 = s3fs.S3FileSystem(anon=False)

with s3.open(f"{'s3://bucket'}/result.csv",'w') as f:
    numpy.savetxt(f, output, delimiter=",")

answered Jul 31, 2021 at 23:35

jellycsc

12.5k2 gold badges21 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to write a numpy array as a csv to S3

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related