File tree Expand file tree Collapse file tree 1 file changed +25
-1
lines changed Expand file tree Collapse file tree 1 file changed +25
-1
lines changed Original file line number Diff line number Diff line change 11# schedule-python-script-using-Google-Cloud
2- Schedules a Python script to append data into Bigquery using Google Cloud's App Engine with a cron job
2+ Use Case: Automates live Chicago traffic data and flows it into Bigquery for interactive real-time analysis
3+ Technical Concept: Schedules a Python script to append data into Bigquery using Google Cloud's App Engine with a cron job.
4+
5+ Reference: http://zablo.net/blog/post/python-apache-beam-google-dataflow-cron
6+
7+ Shout out to Mylin Ackerman for all his help. Saved me weeks of research with his personal touch.
8+ https://www.linkedin.com/in/mylin-ackermann-25a00445/
9+
10+ Order of Operations
11+ 1 . Develop scripts with Google cloud shell or SDK
12+ 2 . Deploy on appengine
13+ 3 . Deploy cron job
14+ 4 . Check BigQuery
15+ 5 . Connect with dataviz tool such as Tableau
16+
17+ Specific Instructions:
18+ 1 . Copy github repository into SDK or Google cloud shell(thankfully it has persistent storage, so you don't have to recopy the folder structure)
19+ 2 .
20+
21+
22+
23+ Document Context:
24+ 1 . app.yaml contains definition of App Engine app, which will spawn Dataflow pipeline
25+ 2 . cron.yaml contains definition of App Engine CRON, which will ping one of the App endpoints (in order to spawn Dataflow pipeline)
26+ 3 . appengine_config.py adds dependencies to locally installed packages (from lib folder)
You can’t perform that action at this time.
0 commit comments