Skip to content

Commit a11a761

Browse files
authored
Update README.md
1 parent f4c662d commit a11a761

File tree

1 file changed

+25
-1
lines changed

1 file changed

+25
-1
lines changed

README.md

Lines changed: 25 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,26 @@
11
# schedule-python-script-using-Google-Cloud
2-
Schedules a Python script to append data into Bigquery using Google Cloud's App Engine with a cron job
2+
Use Case: Automates live Chicago traffic data and flows it into Bigquery for interactive real-time analysis
3+
Technical Concept: Schedules a Python script to append data into Bigquery using Google Cloud's App Engine with a cron job.
4+
5+
Reference: http://zablo.net/blog/post/python-apache-beam-google-dataflow-cron
6+
7+
Shout out to Mylin Ackerman for all his help. Saved me weeks of research with his personal touch.
8+
https://www.linkedin.com/in/mylin-ackermann-25a00445/
9+
10+
Order of Operations
11+
1. Develop scripts with Google cloud shell or SDK
12+
2. Deploy on appengine
13+
3. Deploy cron job
14+
4. Check BigQuery
15+
5. Connect with dataviz tool such as Tableau
16+
17+
Specific Instructions:
18+
1. Copy github repository into SDK or Google cloud shell(thankfully it has persistent storage, so you don't have to recopy the folder structure)
19+
2.
20+
21+
22+
23+
Document Context:
24+
1. app.yaml contains definition of App Engine app, which will spawn Dataflow pipeline
25+
2. cron.yaml contains definition of App Engine CRON, which will ping one of the App endpoints (in order to spawn Dataflow pipeline)
26+
3. appengine_config.py adds dependencies to locally installed packages (from lib folder)

0 commit comments

Comments
 (0)