Can selenium automation be used with BS4?

0

Issue

I am using selenium for both automation and scraping. Now I found that it’s too slow on some of the sites. If i use beautifulSoup then I can scrape them faster but the automation can’t be done.

Is there anyway where I can automate the website (button click events etc.) and can also scrape websites with it on beautifulSoup?

Can you give me an example of button/search automation with bs4 + selenium?

Any help would be appreciated…

Solution

Example

from bs4 import BeautifulSoup as Soup
from selenium import webdriver

driver = webdriver.Chrome()
driver.get("https://stackoverflow.com/questions/tagged/beautifulsoup+selenium")
page = Soup(driver.page_source, features='html.parser')
questions = page.select("#questions h3 a[href]")

for question in questions:
    print(question.text.strip())

Or Just

import requests
from bs4 import BeautifulSoup as Soup


url = 'https://stackoverflow.com/questions/tagged/beautifulsoup+selenium'
response = requests.get(url=url)
page = Soup(response.text, features='html.parser')
questions = page.select("#questions h3 a[href]")

for question in questions:
    print(question.text.strip())

Remember to read https://stackoverflow.com/robots.txt

Answered By – 0m3r

This Answer collected from stackoverflow, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0

Leave A Reply

Your email address will not be published.

This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Accept Read More