How to scrape Google News results

The free RSS route, the Python route, and what changes once you need this every day without failures.

The short answer

Start with the RSS feed, because it is free and it is right there. Move to something sturdier when you hit one of the three walls below. Most people meet the first wall within a week.

Route 1: the RSS feed

Google News publishes search results as RSS. Add a query to the feed URL and you get back a list of headlines, links and publication dates, no key and no account. For a hobby project, a newsletter or a quick look at a topic, this is the correct answer and anything fancier is a waste.

Where it stops: the feed returns a limited slice rather than everything, the article link is usually a Google redirect rather than the publisher URL, and the fields are thin. There is no reliable source name, no snippet worth analysing, and no control over the time window beyond what the query syntax supports.

Route 2: Python and a parser

The next step most people take is fetching the results page and parsing it. It works on the first afternoon. Then three things happen, in this order.

Route 3: a dataset that just returns articles

If the news feed is an input to something else, a workflow, a dashboard, a model, then the scraping itself is not the work and should not be your problem. Our news dataset returns one row per article with the fields filled in, billed per article listed at $3.50 per 1,000, and runs the same way from a scheduler as it does by hand.

Things to get right whichever route you pick

The datasets behind this

Google News Scraper (bulk queries, lightweight)

News articles from Google News per search term, topic or country headlines: title, source, publication date, link and snippet. Up to 200 queries per run. Only delivered articles are charged.