Ever since Google Web Search API deprecation in 2011, I’ve been searching for an alternative. I need a way to get links from Google search into my Python script. So I made my own, and here is a quick guide on scraping Google searches with requests and Beautiful Soup.
Can you add something to the code if you want to scrape all the pages from the google search, and not only the first?
Yes. However Google limits the search results to return a maximum of 100. But you can add a loop to query the next 100. You need to add an additional parameter
start to index between the results. For example to get the results 500-600. They query string will have
it seems to me that this script worked for me two months ago, but now it doesn’t.
print(results) return empty .
Do you can check it if this works for you?
Google code changed. I’ve updated the repo. Give it a try now.
Thank you for your quick answer!
I see you changed the user agent to mobile, but nevertheless unfortunately, the results return still empty (I used the code form this page: https://hackernoon.com/how-to-scrape-google-with-python-bo7d2tal )
Oh I see. I updated the GitHub repo (https://github.com/getlinksc/scrape_google). I can’t find a way to edit old posts.
Oh, now it works!
Thank you very much!