Insert large blob into vitess through node js

Question

I'm using this method to insert a large blob into my mySQL database (Blob size ~40MB) After ~15 seconds it throws a exception or CPU utilisation goes to 100% and remains 100% upto ~25-50 seconds. The whole thing here works with blobs smaller then ~7MB.

I am using node js, mysql

Your file size is too large it is better to storing the file in a directory and store location of the file in the db. — Gorkhali Khadka
– Gorkhali Khadka, Commented Jan 2, 2020 at 10:28
I am using docker containers, so I want to access that blob into another container which is not possible if i'm storing it into file. Also, on restarting container, it is quite possible that I will loose files. — Vikas
– Vikas, Commented Jan 2, 2020 at 10:30
Additional information request. RAM size, # cores, any SSD devices on MySQL Host server? Post on pastebin.com and share the links. A) complete (not edited) my.cnf or my.ini From SSH login root, Text results of: B) SHOW GLOBAL STATUS; after minimum 24 hours UPTIME C) SHOW GLOBAL VARIABLES; D) FLUSH LOGS; then post LAST 400 lines of your error log. E) complete MySQLTuner report for server blob sizing analysis to provide suggestions. — Wilson Hauck
– Wilson Hauck, Commented Jan 2, 2020 at 15:37
Hi wilson-hauck, Currently I'm trying it on my development machine which is quad core 256GB SSD with 16GB of RAM. — Vikas
– Vikas, Commented Jan 3, 2020 at 5:34
@Vikas Your development machine should be FINE. Any chance you can post additional requested info? — Wilson Hauck
– Wilson Hauck, Commented Jan 3, 2020 at 13:05

Sugu Sougoumarane · Accepted Answer · 2020-01-02 17:26:40Z

1

You'll need to bump the grpc_max message_size for vtgates and vttablets: https://github.com/vitessio/vitess/blob/master/go/vt/grpccommon/options.go#L28-L32. Even otherwise, it should have returned an error.

answered Jan 2, 2020 at 17:26

Sugu Sougoumarane

4161 gold badge3 silver badges7 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Vikas Over a year ago

Already done, it's working with other languages like JAVA, I tried in JAVA, but not with the node js.

Sugu Sougoumarane Over a year ago

This isn't something we've seen before. When it hangs, can you look at '/debug/pprof/goroutine?debug=1' for the vtgate? It may give us a clue. If the stack trace is too big, you could post it as an issue on github.

Collectives™ on Stack Overflow

Insert large blob into vitess through node js

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related