How to randomize Header with Users Agent list?

Question

In my script .py i have:

    headers = {'Connection': 'keep-alive',
              'Cache-Control': 'max-age=0',
              'Upgrade-Insecure-Requests': '1',
              'User-Agent': {agents},
              'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
              'Accept-Encoding': 'gzip, deflate',
              'Accept-Language': 'en-US,en;q=0.9,fr;q=0.8',
              'referer': 'bing.com'}

On the .json file

    { 
      "user_agent_list": [
          "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36",
          "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"
      ]
    }

and when i send the request:

find_file = (grequests.get(url_a, headers={headers}, timeout=None, stream=True, verify=False, allow_redirects=False) for url_a in [f'{site_link}{x}' for x in file_rootl])

how i can, randomize headers={headers} for each requests ? of course " for agents in user_agent_list "

more better:

'User-Agent': {agents}, this i need randomize inside headers...

Any idea?

Learning is a mess · Accepted Answer · 2022-06-13 20:54:00Z

1

You can have a random one by having it passed by a random function:

import random

user_agent_list = [
          "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36",
          "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"
      ]

def get_headers():
    headers = {'Connection': 'keep-alive',
              'Cache-Control': 'max-age=0',
              'Upgrade-Insecure-Requests': '1',
              'User-Agent': random.choice(user_agent_list),
              'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
              'Accept-Encoding': 'gzip, deflate',
              'Accept-Language': 'en-US,en;q=0.9,fr;q=0.8',
              'referer': 'bing.com'}
    return headers

#then
find_file = (grequests.get(url_a, headers=get_headers(), timeout=None, stream=True, verify=False, allow_redirects=False) for url_a in [f'{site_link}{x}' for x in file_rootl])

where user_agent_list has the values imported from the json.

edited Jun 13, 2022 at 20:54

answered Jun 13, 2022 at 12:10

Learning is a mess

8,3178 gold badges42 silver badges82 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Massimo B. Over a year ago

Also must to correct, need just get_headers(), without { }

Collectives™ on Stack Overflow

How to randomize Header with Users Agent list?

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related