How To Scrape Google With Python

How To Scrape Google With Python

Ever since Google Web Search API deprecation in 2011, I’ve been searching for an alternative. I need a way to get links from Google search into my Python script. So I made my own, and here is a quick guide on scraping Google searches with requests and Beautiful Soup.

1 Like

Can you add something to the code if you want to scrape all the pages from the google search, and not only the first?

Yes. However Google limits the search results to return a maximum of 100. But you can add a loop to query the next 100. You need to add an additional parameter start to index between the results. For example to get the results 500-600. They query string will have q=keyword&start=500&num=100.

Hello,

it seems to me that this script worked for me two months ago, but now it doesn’t.
print(results) return empty [].
Do you can check it if this works for you?
Koliak

Google code changed. I’ve updated the repo. Give it a try now.

Thank you for your quick answer!
I see you changed the user agent to mobile, but nevertheless unfortunately, the results return still empty :frowning: (I used the code form this page: https://hackernoon.com/how-to-scrape-google-with-python-bo7d2tal )

Oh I see. I updated the GitHub repo (https://github.com/getlinksc/scrape_google). I can’t find a way to edit old posts.

1 Like

Oh, now it works!

Thank you very much!

1 Like