Simple Scraping With Lynx

November 24, 2009

A scraper is simply an automated Web browser.

By automated, we mean that a scraper can be driven either by you directly (usually via the command line) or by a script (or batch file). Scraping at its most basic, is simply the act of running a program that automatically retrieves information from a remote Web page or Web service.

If we accept that simple definition of scraping, then the very simplest of all scrapers, is Lynx.

How To Use Lynx to Quickly List All the URLs On A Web Page

November 17, 2009

When used with the --dump option, Lynx parses an HTML document and builds a list of hyperlinks in that document.

If you have tried either of our previous Lynx recipes, then you may have noticed that, in addition to parsing out the textual content from Web pages. Lynx prints all of the URLs referenced in a document, as a numbered list at the bottom of the output.

Lynx As A Simple (but Fast!) HTML Parser

November 10, 2009

There are numerous useful things that can be done with Lynx. For instance, you can quickly scrape a weather report for any US zip code.

Another little-known use for Lynx, is to preview HTML documents.

Using Lynx to Scrape a Weather Report

November 03, 2009

Lynx As A Simple HTML Parser

One great and often-overlooked feature of Lynx is its --dump option.

lynx --dump <url> dumps the text content of the Web page at URL, followed by a numbered list of URLs that were referenced in that page.