Random User Agent
26 April, 2017 - 18:52
Hello all,
So kind of separate from another project I am working on I threw together a quick script to randomize the User Agent in the HTTP header when making web requests. I'm going to have to integrate it into my networking project at some point. That way when I'm doing things like crawling websites I'm able to mask my identity. This will most likely morph into a software proxy of some kind. Still unclear on that. Feel free to check it out and let me know what you think. I'm still working on printing out the HTTP header information, so it's not completely done.
Keep in mind there is a companion file with 1000 different user agents all imported as strings.
import random
import requests
user_agents = []
user_agents_file = open("user_agents.txt", "r") # Text file of user agents
for agents in user_agents_file:
# cycles through the list adds them to the user_agents array
user_agents.append(agents)
random_agents = random.choice(user_agents)[1:-2] # Takes out parens and new line character, also randomizes array choice
url = "http://quotes.toscrape.com"
headers = {"User-Agents" : random_agents}
r = requests.get(url, headers=headers)
print r.content
import requests
user_agents = []
user_agents_file = open("user_agents.txt", "r") # Text file of user agents
for agents in user_agents_file:
# cycles through the list adds them to the user_agents array
user_agents.append(agents)
random_agents = random.choice(user_agents)[1:-2] # Takes out parens and new line character, also randomizes array choice
url = "http://quotes.toscrape.com"
headers = {"User-Agents" : random_agents}
r = requests.get(url, headers=headers)
print r.content