I have a python script which reads the csv file,create table in bigquery based on data in csv and loads the data into the table on runtime. I want to trigger this script whenever a event occurs like a file arriving on a specified bucket. My Python script is present on vm instance(till now,used to run the script from vm instance). Is there any way to execute this script based on a event trigger ?
1 Answer
A feature of Google Cloud Storage called 'Object Change Notifications' could come in handy here. This will push a notification to a web hook, where you can use your processing code to handle the event (an example to run this on AppEngine is given here, but you could also implement this as a Flask endpoint for example).
Another option is to use Pub/Sub Notifications in combination with Cloud Storage, in case you are more comfortable with that. You can then use the Pub/Sub Python SDK to listen to a specific topic your events arrive in, and use the code you already wrote to handle these events.
12 Comments
Shikha
Thanks a lot for providing documents link. Going through it and hopefully it will be helpful.
Shikha
Hi ..Is there any way to directly call to python script within an event trigger ? My index.js has following code in event trigger :exports.helloGCS = function (event, callback) { const file = event.data; if (file.resourceState === 'not_exists') { console.log(
File ${file.name} deleted.); } else if (file.metageneration === '1') { console.log(File ${file.name} uploaded.); } else { console.log(File ${file.name} metadata updated.); } callback(); }; Can I call python script which is placed on vm instance in any way from this file only ?
Matthias Baetens
Can you give some more details on what you are trying to implement? Is the code above a Google Cloud Function you are deploying to handle the events? Are you using Object Change Notifications or Pub/Sub?
Shikha
Trying to call Python script from CloudFunction code. Trying to follow this link :github.com/extrabacon/python-shell Wrote below code in index.js: var PythonShell = require('python-shell'); PythonShell.run('my_script.py', function (err) { if (err) throw err; console.log('finished'); }); but getting error:Deployment failure: Function load error: Code in file index.js can't be loaded. Did you list all required modules in the package.json dependencies? Detailed stack trace: Error: Cannot find module 'python-shell' Can you please help.
Matthias Baetens
Hi Shikha. Unfortunately that will be impossible. The Cloud Function is a one-off stand-alone unit that performs the code you've written; completely serverless. It has no way to access the Python script you specify. You can either implement the logic you have in Python in the Cloud Function (in NodeJS) or you can develop a Python server that will handle the events sent to a certain endpoint in the cloud (specified when deploying the Object Change Notification).
|