Can selenium automation be used with BS4?
I am using
selenium for both automation and scraping. Now I found that it’s too slow on some of the sites. If i use
beautifulSoup then I can scrape them faster but the automation can’t be done.
Is there anyway where I can automate the website (button click events etc.) and can also scrape websites with it on
Can you give me an example of button/search automation with
Any help would be appreciated…
from bs4 import BeautifulSoup as Soup from selenium import webdriver driver = webdriver.Chrome() driver.get("https://stackoverflow.com/questions/tagged/beautifulsoup+selenium") page = Soup(driver.page_source, features='html.parser') questions = page.select("#questions h3 a[href]") for question in questions: print(question.text.strip())
import requests from bs4 import BeautifulSoup as Soup url = 'https://stackoverflow.com/questions/tagged/beautifulsoup+selenium' response = requests.get(url=url) page = Soup(response.text, features='html.parser') questions = page.select("#questions h3 a[href]") for question in questions: print(question.text.strip())
Remember to read https://stackoverflow.com/robots.txt
Answered By – 0m3r