Guide

How to scrape Google News results

The free RSS route, the Python route, and what changes once you need this every day without failures.

Start with the RSS feed

It is free and it is already there. For a hobby project or a weekly newsletter it is the whole answer, and anything fancier is wasted effort.

Then you hit a wall. There are three of them below, and most people meet the first one inside a week.

Route 1: the RSS feed

Google News publishes search results as RSS. Add a query to the feed URL and you get back a list of headlines, links and publication dates, no key and no account. For a hobby project, a newsletter or a quick look at a topic, this is the correct answer and anything fancier is a waste.

Where it stops: the feed returns a limited slice rather than everything, the article link is usually a Google redirect rather than the publisher URL, and the fields are thin. There is no reliable source name, no snippet worth analysing, and no control over the time window beyond what the query syntax supports.

Route 2: Python and a parser

The next step most people take is fetching the results page and parsing it. It works on the first afternoon. Then three things happen, in this order.

  • The markup changes. Google restructures the page and your selectors return empty lists. Not once, repeatedly.
  • The address gets blocked. Consistent automated requests from one server address start returning consent pages and challenges instead of results.
  • The results stop matching. Google News is personalised and localised. The results your server sees are not the results you saw in your browser, and if you do not pin language and country explicitly you will spend a day confused.

Route 3: a dataset that just returns articles

If the news feed is an input to something else, a workflow, a dashboard, a model, then the scraping itself is not the work and should not be your problem. Our news dataset returns one row per article with the fields filled in, billed per article listed at $3.50 per 1,000, and runs the same way from a scheduler as it does by hand.

Things to get right whichever route you pick

  • Pin language and country. Always, explicitly. Otherwise your results drift with whatever the server looks like it is.
  • Deduplicate. The same story appears from many outlets, often with near identical headlines. Decide whether you want the story or the coverage.
  • Respect the publishers. A headline, a link and a date are a citation. The full article text belongs to the publisher and copying it wholesale is a different decision with different rules.
  • Store the raw result. When your parsing turns out to be wrong in a month, the raw response is the only thing that lets you fix history.

The datasets behind this