File tree Expand file tree Collapse file tree 1 file changed +9
-2
lines changed Expand file tree Collapse file tree 1 file changed +9
-2
lines changed Original file line number Diff line number Diff line change 11# schedule-python-script-using-Google-Cloud
22Use Case: Automates live Chicago traffic data and flows it into Bigquery for interactive real-time analysis
3+
34Technical Concept: Schedules a Python script to append data into Bigquery using Google Cloud's App Engine with a cron job.
45
56Reference: http://zablo.net/blog/post/python-apache-beam-google-dataflow-cron
67
78Shout out to Mylin Ackerman for all his help. Saved me weeks of research with his personal touch.
89https://www.linkedin.com/in/mylin-ackermann-25a00445/
910
10- Order of Operations
11+ Order of Operations:
11121 . Develop scripts with Google cloud shell or SDK
12132 . Deploy on appengine
13143 . Deploy cron job
14154 . Check BigQuery
15165 . Connect with dataviz tool such as Tableau
1617
17- Specific Instructions:
18+ Setup Prerequisites:
19+
20+ Development Instructions:
18211 . Copy github repository into SDK or Google cloud shell(thankfully it has persistent storage, so you don't have to recopy the folder structure)
19222 .
2023
24+ Deploy Instructions:
25+ 1 . Install all required packages into local lib folder: pip install -r requirements.txt -t lib
26+ 2 . To deploy App Engine app, run: gcloud app deploy app.yaml
27+ 3 . To deploy App Engine CRON, run: gcloud app deploy cron.yaml
2128
2229
2330Document Context:
You can’t perform that action at this time.
0 commit comments