When parsing in python3 via beautifulsoup error 400 bad request what to do?

Question

When parsing in python3 via beautifulsoup error 400 bad request what to do?

Parsed the online store with authorization and through BeautifulSoup, but stably after 226 requests, every second request gives an error 400 bad request, what to do?

0

python python-3.x requests beautiful-soup

Author: dmitriyff, 2020-11-24

Source

1 answers

score 1 · Answer 1

If the problem is that the site thinks that you are a robot because you have a user agent of the python/requests type, then try using the fake_useragent package.

Installation: pip install fake-useragent.

Application:

import requests
import fake_useragent
    
url = "https://ru.stackoverflow.com/"
    
# данная строчка возвращает рандомный юзер-агент
user_agent = fake_useragent.UserAgent().random
headers = dict(user_agent=user_agent)
    
response = requests.get(url, headers=headers)