6

I want to use different API keys for data scraping each time my program is run.

For instance, I have the following 2 keys:

apiKey1 = "123abc"
apiKey2 = "345def"

and the following URL:

myUrl = http://myurl.com/key=...

When the program is run, I would like myUrl to be using apiKey1. Once it is run again, I would then like it to use apiKey2 and so forth... i.e:

First Run:

url = "http://myurl.com/key=" + apiKey1

Second Run:

url = "http://myurl.com/key=" + apiKey2

Sorry if this doesn't make sense, but does anyone know a way to do this? I have no idea.

EDIT:

To avoid confusion, I've had a look at this answer. But this doesn't answer my query. My target is to cycle between the variables between executions of my script.

Improve this question
9
  • 1
    for key in itertools.cycle((apiKey1, apiKey2)):? When should it stop switching between them? Commented Apr 23, 2017 at 18:20
  • 4
    Your program needs to keep state. Usually this is done by writing the information to a file. Also, you're almost certainly violating the terms of service of the API you're (ab)using. Commented Apr 23, 2017 at 18:21
  • @jonrsharpe i was thinking of cycle as well, but I have a hunch that OP wants to cycle between the variables between executions of his script. Commented Apr 23, 2017 at 18:21
  • @JonathonReinhart Don't worry, I'm not ;-) Commented Apr 23, 2017 at 18:21
  • @timgeb You are right =) Commented Apr 23, 2017 at 18:22

3 Answers 3

Reset to default
2

I would use a persistent dictionary (it's like a database but more lightweight). That way you can easily store the options and the one to visit next.

There's already a library in the standard library that provides such a persistent dictionary: shelve:

import shelve

filename = 'target.shelve'

def get_next_target():
    with shelve.open(filename) as db:
        if not db:
            # Not created yet, initialize it:
            db['current'] = 0
            db['options'] = ["123abc", "345def"]

        # Get the current option
        nxt = db['options'][db['current']]
        db['current'] = (db['current'] + 1) % len(db['options'])  # increment with wraparound

    return nxt

And each call to get_next_target() will return the next option - no matter if you call it several times in the same execution or once per execution.

The logic could be simplified if you never have more than 2 options:

db['current'] = 0 if db['current'] == 1 else 1

But I thought it might be worthwhile to have a way that can easily handle multiple options.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

1
+50

Here is an example of how you can do it with automatic file creation if no such file exists:

import os
if not os.path.exists('Checker.txt'):
    '''here you check whether the file exists
    if not this bit creates it
    if file exists nothing happens'''
    with open('Checker.txt', 'w') as f:
        #so if the file doesn't exist this will create it
        f.write('0')

myUrl = 'http://myurl.com/key='
apiKeys = ["123abc", "345def"]

with open('Checker.txt', 'r') as f:
    data = int(f.read()) #read the contents of data and turn it into int
    myUrl = myUrl + apiKeys[data] #call the apiKey via index

with open('Checker.txt', 'w') as f:
    #rewriting the file and swapping values
    if data == 1:
        f.write('0')
    else:
        f.write('1')
Improve this answer

5 Comments

This is great! Thanks very much, if you could just quickly explain the code that would be great
You are welcome! I added comments to help you understand the process. Do ask if you need anything further clarified :)
Thanks! That's great ;-)
Oh, I do have 1 question, does os.path.exists refer to the folder the script is in?
Yes, but you can add full path and check for other folders as well.
0

I would rely on an external process to hold which key was used last time, or even simpler I would count executions of the script and use a key if execution count is an odd number, or the other key for an even number.

So I would introduce something like redis, which will also help a lot for other (future ?) features you may want to add in your project. redis is one of those tools that always give benefits at almost no cost, it's very practical to be able to rely on an external permanent storage - it can serve many purposes.

So here is how I would do it:

  1. first make sure redis-server is running (can be started automatically as a daemon, depends on your system)
  2. install Python redis module
  3. then, here is some Python code for inspiration:

    import redis

    db = redis.Redis()

    if db.hincrby('execution', 'count', 1) % 2:
      key = apiKey1
    else:
      key = apiKey2

That's it !

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.