Selenium How to scrape website table when website pagination relies on Javascript?

0

Issue

I’m pretty new to webscraping and As of Now I only came across scraping website that had pagination link in their next button. But I came to situation where website pagination button link doesn’t change when I’m clicking on the next button.

driver = webdriver.Chrome(ChromeDriverManager().install())
driver.maximize_window()
driver.implicitly_wait(30)
driver.get("https://www.nepalstock.com.np/floor-sheet?fbclid=IwAR0-V615fp0ujUH8BRr3Zu4erfAtsCz0-RHUgjMIvhjpGsaya-S9v7xI_-8")

but this website doesnot change link on loading the next button.
Any help or hint I can do to grab these pagination table data?

Ps: Right click is disabled on above website (I used ctrl+shift+c) to enable it

Solution

I just tried the below-mentioned java code for the above-mentioned website and I am able to click the next button and also able to get the data.

driver.get("https://www.nepalstock.com.np/floor-sheet?fbclid=IwAR0-V615fp0ujUH8BRr3Zu4erfAtsCz0-RHUgjMIvhjpGsaya-S9v7xI_-8");
 
int rowSize=driver.findElements(By.xpath("//tbody//tr")).size();
int pageSize=Integer.parseInt(driver.findElement(By.xpath("//li[@class='ellipsis']/following-sibling::li//span[@class='show-for-sr']/following-sibling::span")).getText());
for(int page=1;page<pageSize;page++)
{

for (int i=1;i<rowSize;i++)
{
    
    for(int col=1;col<=8;col++)
    {
    System.out.print(driver.findElement(By.xpath("//tbody//tr["+i+"]/td["+col+"]")).getText()+" | ");
    }
    System.out.println();
    }
driver.findElement(By.xpath("//li[@class='pagination-next']")).click();
}

Answered By – Yosuva Arulanthu

This Answer collected from stackoverflow, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0

Leave A Reply

Your email address will not be published.

This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Accept Read More