A scraper is simply an automated Web browser.
By automated, we mean that a scraper can be driven either by you directly (usually via the command line) or by a script (or batch file). Scraping at its most basic, is simply the act of running a program that automatically retrieves information from a remote Web page or Web service.
If we accept that simple definition of scraping, then the very simplest of all scrapers, is Lynx.
Did you know that you can install the same CSS validation tool on your
local machine? Follow the simple instructions below, to set yourself up to
run the
Web developers are all familiar with the W3's CSS validation service.
CSS validation is an essential part of developing a Web interface. The same goes for XHTML validation and running JavaScript through JSLint.
When used with the --dump option, Lynx parses an HTML document and
builds a list of hyperlinks in that document.
If you have tried either of our previous Lynx recipes, then you may have noticed that, in addition to parsing out the textual content from Web pages. Lynx prints all of the URLs referenced in a document, as a numbered list at the bottom of the output.
There are numerous useful things that can be done with Lynx. For instance, you can quickly scrape a weather report for any US zip code.
Another little-known use for Lynx, is to preview HTML documents.
One great and often-overlooked feature of Lynx is its --dump
option.
lynx --dump <url> dumps the text content of the Web page at URL,
followed by a numbered list of URLs that were referenced in that page.
When last we posted about tiny Web servers built with Ruby, Face-book friend Michael Cottingham immediately protested that Gary Robinson has blogged about a Python Web server that is in fact even smaller.
And the Python version seems to be more featureful than the Ruby version as well. For instance it is a bit more secure (and much more intuitive), in that "server root" defaults to the directory where SimpleHTTPServer was invoked. By contrast, the Ruby version blindly serves up everything in the root of the drive where it was invoked.
w00t!!1! Major congratulations to aram, who deciphered our very
first code
puzzler!
cksum file1 file_n | perl -ane 'warn if $F[0] != ($x ||= $F[0])'
Aram explained the convoluted code above as follows:
> Just a guess, it looks like it does a comparison of 2 files.
> The chksum file file2 calculates and outputs the checksum which is
> piped into a perl program.
> perl -a auto splits the output from checksum and puts it into array $F,
> so F$[0] is the first element.
> the perl -n loops through all the piped input.
> $F[0] is the checksum of one file.
> $F[0] != ($x ||= $F[0]
> is true if $f[0] is not equal to $x or $F[0].
> it seems $x is set to $F[0] if not equal
Awesome, Aaram. Comparing the checksum of two files is exactly what this little one-liner is meant to do.
Hey Aram -- DM us on Twitter and let us know how we can get in touch with you! We'd like to send you a Nerdabout laptop sticker.
Here is how to set up passwordless login for SSH access to a remote server (including SVN over SSH) on Windows.
Getting this right has sadly eluded us for some time, even though the internets contain some really good articles on how passwordless login works as well as step-by-step instructions on how to set up passwordless login.
Still, putting it all together, on Windows, we found a bit tricky. But we've finally banged our heads up against a solution.
This past Wednesday night was the 2nd Dorkbot-NYC of the season at Location One in Soho. The theme of the evening was the interaction between humans and technology.
Take advantage of the convenience of Emacs keyboard commands, even if you aren't using Emacs. It turns out that quite a few pieces of software support the basic Emacs text-editing command set. All this is made possible of the GNU Readline library.
There's a complete list of keyboard navigation commands in the Readline documentation. Or if you prefer, there is the bash/Readline cheatsheet :-;)
If you aren't already, you'll want to familiarize yourself with the the basic Emacs text-editing command set before trying these out.
Firefox, Safari and many other Mac OS X applications.
The Ruby's use Emacs commands in the IRB interactive shell.
Cygwin, the Unix terminal emulator for Windows, runs bash and so supports Emacs keyboard navigation shortcuts
When I wrote about bash support for the basic Emacs text-editing command set, I should have mentioned Cygwin, the bash emulator for Windows. Cygwin ships with a Linux terminal emulator, that runs bash and so supports Emacs keyboard navigation shortcuts.
If you haven't, try it. Especially if you are a Windows user and haven't tried they GNU/Linux tools yet.
Cygwin can change your life by enabling you to do things that you generally just can't do on Windows, at least not out-of-the box.
It's so awesome to have a free tool that will let you, for instance, search for files on your hard drive by name and by content, at the same time. And it's free. I'm consistently surprised by the preponderance of ten-to-fifty-dollar shareware programs, whose functionality can be duplicated with a 5 or 10 line shell script, and perhaps a little Perl.
I'm always looking for new mashups and was quite happy to discover Twyric via digg. When I first started to use Twitter, I tried to write only in haiku. That fell by the wayside for a bit but I'm working on picking it back up which is why Twyric--with its minimalistic approach--is so appealing to me.
With the assistance of Twitter's real time search, Twyric collects tweets that include the hashtags #haiku, #twyric, #twly, #gedicht, #poesie, #lyric, #poetry, #limerick, #poetic and #poem. Keywords are parsed out and using the Flickr API, an image is matched up with the tweet. I've seen the same poems fade in and out. Sometimes the image paired with the poem remains the same, other times it changes.
I did my first haiku this morning and it took about 1/2 hour to be on Twyric. Two things I learned: 1) reread tweets for typos and word repeats (as you can see I make many in the morning) 2) don't put a title (in this case "Tea") before your hashtag otherwise it appears in your haiku (minus the hashtag because that's removed).
In 2008, Michael Neumann published a Ruby one-liner that leverages the power of Ruby's standard RSocket::TCPServer library, to start up a fully functional HTTP server; in just a single line of Ruby.
ruby -rsocket -e's=TCPServer.new(5**5);loop{_=s.accept;_<<"HTTP/1.0 \
200 OK\r\n\r\n#{File.read(_.gets.split[1])rescue nil}";_.close}'
cartoon: goopymart
Although there's no fancy stuff like 404 pages or redirects, you'll find that all the basic HTTP transactions work (eg you can follow links to other documents).
Once started, the server's root directory is the root of your local filesystem.
Relatively speaking, there are few things more satisfying than a good hack. Hacking pranks prove useful year round. We dug back through our archives and picked out a few of our favorites.
GNU Linux's bash shell supports the Emacs command set out of the box. Like IRB as well as many Mac OS X applications, you can navigate your way around the Bourne-Again SHell.
This is because Bash takes advantage of a library called GNU Readline
Here is a Readline Cheatsheet. Bash Emacs Editing Mode (readline) Cheat Sheet
As you can see, there are tons of features potentially available. That's because bash is a really large, featureful tool. You can work in bash for 10 or 20 years, and still find interesting things to learn, and find out about things you didn't know you could do.
Of course, many people are more interested in just getting some work done. In that case just stick to the the basic Emacs text-editing command set and you'll likely find that just mitigating your need to take your fingers off the home keys, can be a big boost to your productivity.
Ruby's interactive debugger, the IRB is a friendly and helpful shell environment for experimenting with and learning about Ruby.
Cartoon: _why
The IRB has many nice features, not least of which is support for Emacs-style editing and keyboard navigation.
Not only is the basic Emacs text-editing command set supported, but the IRB also supports
C-SPCC-t