Simple Scraping With Lynx

November 24, 2009

A scraper is simply an automated Web browser.

By automated, we mean that a scraper can be driven either by you directly (usually via the command line) or by a script (or batch file). Scraping at its most basic, is simply the act of running a program that automatically retrieves information from a remote Web page or Web service.

If we accept that simple definition of scraping, then the very simplest of all scrapers, is Lynx.

How To Install the W3 CSS Validator

November 23, 2009

Did you know that you can install the same CSS validation tool on your local machine? Follow the simple instructions below, to set yourself up to run the W3 CSS validator from the command line!

CSS Validation on the Command Line

Web developers are all familiar with the W3's CSS validation service.

CSS validation is an essential part of developing a Web interface. The same goes for XHTML validation and running JavaScript through JSLint.

Set Up Passwordless SSH Login on Windows with Cygwin and SSH-Keygen

October 13, 2009

Here is how to set up passwordless login for SSH access to a remote server (including SVN over SSH) on Windows.

Getting this right has sadly eluded us for some time, even though the internets contain some really good articles on how passwordless login works as well as step-by-step instructions on how to set up passwordless login.

Still, putting it all together, on Windows, we found a bit tricky. But we've finally banged our heads up against a solution.

Cygwin (the Linux emulator for Windows): Emacs' Commands Without Emacs, part 4

September 29, 2009

When I wrote about bash support for the basic Emacs text-editing command set, I should have mentioned Cygwin, the bash emulator for Windows. Cygwin ships with a Linux terminal emulator, that runs bash and so supports Emacs keyboard navigation shortcuts.

photo: wili_hybrid

If you haven't, try it. Especially if you are a Windows user and haven't tried they GNU/Linux tools yet.

Cygwin can change your life by enabling you to do things that you generally just can't do on Windows, at least not out-of-the box.

It's so awesome to have a free tool that will let you, for instance, search for files on your hard drive by name and by content, at the same time. And it's free. I'm consistently surprised by the preponderance of ten-to-fifty-dollar shareware programs, whose functionality can be duplicated with a 5 or 10 line shell script, and perhaps a little Perl.

Twyric: A Twitter Flickr Mashup

September 28, 2009

By Joanna Burgess

I'm always looking for new mashups and was quite happy to discover Twyric via digg. When I first started to use Twitter, I tried to write only in haiku. That fell by the wayside for a bit but I'm working on picking it back up which is why Twyric--with its minimalistic approach--is so appealing to me.

With the assistance of Twitter's real time search, Twyric collects tweets that include the hashtags #haiku, #twyric, #twly, #gedicht, #poesie, #lyric, #poetry, #limerick, #poetic and #poem. Keywords are parsed out and using the Flickr API, an image is matched up with the tweet. I've seen the same poems fade in and out. Sometimes the image paired with the poem remains the same, other times it changes.

I did my first haiku this morning and it took about 1/2 hour to be on Twyric. Two things I learned: 1) reread tweets for typos and word repeats (as you can see I make many in the morning) 2) don't put a title (in this case "Tea") before your hashtag otherwise it appears in your haiku (minus the hashtag because that's removed).

How To Run the World's Smallest Web Server: one line of Ruby!

September 22, 2009

In 2008, Michael Neumann published a Ruby one-liner that leverages the power of Ruby's standard RSocket::TCPServer library, to start up a fully functional HTTP server; in just a single line of Ruby.

ruby -rsocket -e's=TCPServer.new(5**5);loop{_=s.accept;_<<"HTTP/1.0 \
200 OK\r\n\r\n#{File.read(_.gets.split[1])rescue nil}";_.close}'

cartoon: goopymart

Although there's no fancy stuff like 404 pages or redirects, you'll find that all the basic HTTP transactions work (eg you can follow links to other documents).

Once started, the server's root directory is the root of your local filesystem.

Top 10 Nerdy Blow Offs

September 10, 2009

By Joanna Burgess

photo: Francis Storr

A good PvPer knows you have to be two steps ahead of the masses. The same holds true for social situations. "Sorry I can't. It's patch day and I have to re-spec my druid." has been a favorite, mid-week defense of mine. As I believe it prudent to be prepared at all times, for all uncomfortable moments, be it at the bar or library, I called on the good people of Twitter and Facebook to help me round out my Top 10 Nerdy Blow Offs list and as usual our geeky fans did not disappoint.

Use Emacs' Commands in the IRB

September 08, 2009

Ruby's interactive debugger, the IRB is a friendly and helpful shell environment for experimenting with and learning about Ruby.

Cartoon: _why

The IRB has many nice features, not least of which is support for Emacs-style editing and keyboard navigation.

Not only is the basic Emacs text-editing command set supported, but the IRB also supports

selecting text with

C-SPC

or transposing characters with

C-t

as well as many other Emacsian keyboard shortcuts.

I Can Haz Spayez?

September 02, 2009

Lol cats come and go but space is infinite which means perfect fodder for, well, spaceLOL. We're looking for the most stellar lolz out there. Check out what's been done so far and then email your spaceLOL submission to nerdabout@discovery.com. And please remember our mantra when wielding your Photoshop tools: be a nerd.

photo:polandeze

The Sun Sets on Pirate Bay?

August 25, 2009

Updated 9:45 PM EST on Tuesday, August 25 2009 Yes, we know the Pirate Bay is back up. We confess to writing the post last night and letting the auto-post script thingy deal with it this morning. Didn't realize PB was back up until @kellbot alerted us (thanks). In the future we promise not to delegate vital tasks to robots…

In the meantime please enjoy the digital masterpiece below.

---

It's like when the Grateful Dead broke up and everyone started to listen to Phish.

Farewell and adieu Pirate Bay. Maybe we'll see you on the upside but we're sure you'll be asking for money. Until then, we won't be downloading anything from mininova.org.

For more info on what went down, read this from PCmag. Tip o' the buccaneer's hat to Ben for the link.

photo: Rob

Ohio is a Piano: An Audio Data Visualization

August 18, 2009

By Joanna Burgess

Once again I am entertaining myself with Google Maps Mania while drinking my tea. This morning's awesomeness? Ohio is a Piano, created by cartographer Andy Woodruff. He's turned the science of statistics into a musical experiment. Here's what the map does: Ohio has 88 counties, a piano has 88 keys. So basically this is an audio data visualization of the state. Each county represents a piano key and you program the map to play a song by assigning a county a key based upon data like kids under 5, median age, avg sz97 and sqmi (which I read as "sqee me" at this early hour and was briefly baffled until I realized this actually meant square miles).

Click here to begin composing.

9 Essential Emacs Commands for Beginnners

August 17, 2009

By Noah Sussman

For many hackers, Emacs is one of those mysterious, crufty applications that pops up from time to time but has no real use; a leftover from the bad old days when computers didn't have graphical interfaces and most programs were written with a magnetic needle. And that's understandable because most chance encounters with Emacs occur in an ugly little terminal window with no mouse support.

But in fact Emacs is no relic. It's still widely used and has a living, vibrant community of developers and plugin writers. Modern Emacsen (that's the plural of Emacs) have mouse support, and a plethora of other features besides.

Emacs has a rich (some would say notoriously rich) keyboard command set. That shouldn't be surprising, since it was developed 30 years ago, before computers had mice.

photo: fhisa

3D Google Maps

August 03, 2009

This just in...Google Maps Flash API 3D now available. The mashup demos posted are very cool and the developers have provided excellent DYI documentation if you're interested in creating your own map--which I am although it might take me some time considering I'm not a programmer. Still--totally doable. Our insect expert Lonesome Polecat's partner in crime loves arcade driving games so this one is included here especially for her: the work-in-progress Driving Simulator by Katsuomi K.

Since NYC is becoming like London with our massive amounts of torrential rain as of late, I've been pouring over weather maps. Trice Designs has a good one with cloud overlays. Here's my pick because it makes me feel like I'm in a cartoon: UK Weather Map by Ian Watson.

As always, a tip of the hat to Google Maps Mania.

Google vs. Privacy. Who Will Win?

July 24, 2009

By Joanna

photo: matt hammond

I've played around with, a few different e-books from one of the first prototypes 9 years ago (when I worked in publishing) to Sony Reader to the Kindle. I haven't bought one yet mainly due to the cost factor. I get most of my books for free at the library. Or I download them and read them on my computer or Blackberry. But whichever way I get my information, and why I choose to read what I read is nobody's business but my own.

This week has been a big shake-up in digital privacy. Google is pushing to radically redefine how we get and use digital content. According to the Electronic Frontier Foundation, Google wants to monitor what you read, when you read it, and even how long you stay on any given page. The info on what you choose to read could possibly be combined with other personal information from Google apps you've used in past. It's a virtual library of who you are and what you do. The outcome? A bunch of people having knowledge about you that's none of their business.

One word: creepy. For more information on what's happening,visit EFF.org.

Google Map Mashups

July 23, 2009

By Joanna

My obsession with Google map mashups continues, thanks mostly to the fine blog Google Maps Mania which collects mashups from people all over the world. I've found a few lately that I'm really enamored with like the most excellent Topography-to-Topography made by Aussie Rhett Dashwood who used Google Earth images to create the alphabet. This one attempts to help besieged folks in Toronto deal with their garbage strike by mapping out places to dump trash. As someone who loves to travel and preferably read plaques on the side of the road while traveling, the Wikifieldtrip map is a trivia dork's dream. It maps Wikipedia articles to locations based on the map you're looking at. The TwitterTravel map shows you traffic in major cities around the world. Scribblemaps is one I actually found some months back but have only recently gotten around to playing with it. I'm making a map of all the places I've traveled in the world, with some of the more interesting sites I've seen called out. And who doesn't love Digg A Big Hole? You pick a spot to dig and the map shows you where you'd end up, complete with latitude and longitude coordinates.

I'm putting together my Top 10 Google Map Mashups. In the meantime, if you find an interesting map, post it here or Tweet @nerdabout on Twitter with your find.

Are You A Word Nerd? Then Check Out Wordbirds.

July 17, 2009

By Joanna Burgess

Neologism is the ancient science of creatively combining words together to make a brand new word or phrase. I've seen this art perfected at work by programmers turning innocent words into verbal weapons spewed over monitors. (This is especially true once their rubber band guns have broken.) Now I've found Wordbirds, the ultimate neologism site. Liesl Schillinger, the word nerd behind Wordbirds, took a break from crafting our lexicon to chat about her inspirations and share some of her minted words, like "idoloathe" and "droidan slip."

photo:BrianScott

NERDABOUT

N 'nur-da-bowt Someone whose jokey, self-deprecating attitude about his hobby can't disguise the fact that for him the hobby is actually a dead-serious, magnificent obsession. Usage: Liz was a basset hound nerdabout: she made fun of her bassetmania, and pretended it was just a phase, but as she branched into selling basset hound paintings, plates, scarves and hood ornaments, her neighbors sensed that the passion went deeper than she let on.

Book Worm + Tech Nerd = The Future (?)

July 14, 2009

By Liz Suman

I spent the last two and a half hours in the Columbia School of Journalism (on the sixth floor in a really uncomfortable chair). Had I thought to tweet, the message I would have shot into cyberspace would have read something like: “Listening to the Director of the Institute for the Future of the Book Bob Stein tell a room of 100 aspiring book editors, writers, publicists and agents that the print publishing industry as we know it is dead or at least will be soon.” After over two hours of lecture and discussion, half the audience wanted to run and the other half wanted to stay and fight. * Traditional print book publishing should move away from the printed page and onto the screen in an online, subscriber-based community of readers. * “A book is a place where readers (and sometimes authors) congregate” (Stein). * Authors should be paid not for selling physical copies of books, but for turning on their monitors and engaging with readers about their work via online comment boards and paragraph-by-paragraph annotations.

These are some of the new roles Stein envisions for readers, writers and books within Comment Press, a user-based reading concept he has developed at the Institute for the Future of the Book, “a small think-and-do tank investigating the evolution of intellectual discourse as it shifts from printed pages to networked screens.”

Doris Lessing’s The Golden Notebook, one of the Institute’s reading projects, in which seven readers discussed a digital text with one another online. Photo Courtesy of Apt.

Stein’s ideas are timely and provocative as well as jarring and controversial. We all know that the Kindle, Sony Reader, eBooks, and other eReaders are quickly catching on with consumers (Stephen King’s latest novel was originally released straight-to-Kindle), but no one who loves books wants to be told that their tattered copies of James and the Giant Peach and Where the Wild Things Are aren’t sacred let alone obsolete. Stein’s argument that his online, interactive book publishing model will replace the book as a physical object is fundamentally at odds with the traditional, definitive reading experience of sitting down (alone) to read a book (that we can hold in our hands), an experience tonight’s audience was quick to defend with a sea of hands right around the time Stein prophesized that, “Our great grandchildren will think of reading as a social experience.”

Comment Press isn’t the only project Bob has pioneered. Click here for a sample of Comment Press and visit the Institute’s website to learn more about what is probably the most complex, nerdy, and sophisticated set of alternatives to traditional printed matter out there ("Sophie", for example, includes multiple versions of the same text (i.e. The user can view original and revised drafts of the Gettysburg Address just by clicking different links).

I don’t believe physical books will ever become obsolete, whether digital books come to dominate the marketplace or not - The experiences are simply too different. But while online books may not have won the populist reader vote, they definitely give book nerds a lot to think about.

Bob Stein, Director of the Institute for the Future of the Book. Photo Courtesy of New York Magazine.

Jonathan Coulton Machinima

July 10, 2009

Dave's post on Jonathan Coulton is pretty popular so we're leaving it at the top of Nerdabout today. A few months back Joanna interviewed Spiffworld who makes World of Warcraft machinima videos using Coulton's songs. Mythbusters Adam Savage says Spiff's video using Coulton's,"The Future Soon" is "maybe the best piece of machinima art evar!" We've got that one and a few others. Click here to see 'em.

John Sundman, Science Fiction Geek Author

July 01, 2009

By Noah Sussman

Last week at Ajax World, I was psyched to stumble across John Sundman, blogger for Wetmachine, science fiction author, former truck driver, and former Silicon Valley guy. His self-published books, Acts of the Apostles, Cheap, Complex Devices, and The Pains deal with the complex relationships between humans and technology. You can download any or all of them for free. However, since the art is awesome we suggest you buy one for yourself. John's work has been described as "immoral and deeply ugly" and he was the recipient of Brazil's Rei do Lixo award and the Society for Technical Communication's Award of Distinguised Technical Communication.

Etymology: It's Not Just for Nerds

June 23, 2009

By Joanna Burgess

Etyomologists have a nerdy rep. But with the advent of web specific words and phrases, all that's changing. Here's a few sites we like to visit when we want to know not only what new internet slang means means (but might be too embarassed to ask least we seem old school) or the history of a word. These sites also endlessly amusing time vampires.

Wordnik is a pretty cool site I discovered as I was cruisin' metafilter. You type a word into their search and get definition(s), a history and examples of how the word is used, including literary references, some of which are quite funny. If the word isn't in Wordnik, it scrapes Twitter to pull examples of people who've used it. If you want to add a word, just register as a user.

Urban Dictionary bills itself as the "dictionary you wrote" with "4,060,135 definitions written since 1999." It's the place to go to discover the meaning of words, such as slunker, or phrases such as tilly wacker

Web_Friend is another go to for geeky slang, tech words and chat acronyms. Their chat slang section comes in quite handy, especially when playing World of Warcraft with ten year olds. Just in case all these acronyms have made you giddy, here's a silly and yet apparently serious guide to web slang etiquette.

Tweeting Cat Door and Other Home Defenses

June 09, 2009

By Joanna Burgess

Everyone loves cats right? You just might not love the little dead critters they drag in or friends they invite in to feast upon your cat food. If you have a cat that regularly shows up with guests, you might find the Tweeting Cat Door just the thing to guarantee that only the cat which you want in your house...gets in your house. Invention and step by step directions available here.

Does it bother you if your cat jumps on the counter? Then check out Brain Gaut's Blender Defender. It looks startling.

Maker Faire 2009 Revisted

June 04, 2009

Guest blogger: Alyssa Wodtke

The 4th Annual Maker Faire in San Mateo, CA was this weekend. My husband and I have gone every year and it keeps getting bigger. We have to go both days of the fair weekend just to see everything. For those who don't know, the Maker Faire is put on by Make Magazine, a magazine for Do-It-Yourselfers, a growing movement. Well before the economy crashed, geeks of all sorts discovered the joys of hacking Roombas to make spirographs and turning trash into art. The Maker Faire celebrates those who make, build, sew, craft, hack, and teach. It's a brilliant concept.

Creativity reigns at the Maker Faire, both in the products that are displayed and demonstrated and in the people who display and demonstrate. Walking through the fairground, you can't look at a crowd without seeing people in costume (Steampunk, furry, or superhero), people representing the rebellious fringes of society (punks, eccentrics, hippies), or those just expressing a wild individuality (hair of every hue and length, squid hats, stilts). And, of course, the ever-present Utilikilt.

The displays vary wildly. There are always a lot of robots. They're a real crowd pleaser and people clearly love to make them. There are science teachers talking about demos they do in class. There are electronics for bikes, Christmas, and entertainment. There are always lots of things that shoot fire, such as the Crucible truck and the Flaming Lotus Girls' display. And let's not forget the over-the-top displays of making, like a life-sized Mousetrap game or the big metal giraffe that tools around the fairground or the human powered amusement park rides or the giant mechanical hand that picks up oil drums as if they're soda cans.

Ecology is a big influence for many of the makers. It shows up in big things like electric cars (my husband loves the Tesla) and in smaller ones like using recycled plastic bottles to make art. There's also a group called Engineers Without Borders which helps build wind turbines and provide water to countries in Africa.

A new feature this year was a Food Makers tent. A little sparse but promising, it offered cheese tastings, home beer brewing kits, tips on pickling, great chocolate, mushroom growing, and information on vegetarianism and local food. Since this Maker Faire takes place in the Bay Area, land of serious foodies and a huge local food movement, I expect this tent to grow in future years.

It's often hard to take time to sit down and listen to the presentations by makers since there’s so much to see. But there are talks and music all day long. We always make time for Adam Savage, co-host of Mythbusters, who spins a good yarn about what it's like to be a maker. This year he talked about failure in a funny and touching way. My husband caught a talk by a 14 year-old CEO who created a chemistry-based game called Elementeo.

As ever, a good time was had by all and a great deal of money was spent. On the way out, you can stop by the Maker Shed where they sell books and kits and materials so you can try your hand at being a maker or expanding your repertoire. I always walk out inspired to create. If I got nothing else out of the Maker Faire, that's definitely worth the price of admission.

Don't miss all of Alyssa's Maker Faire photos on flickr.

Book Review: A Quarter Century of Unix

June 02, 2009

Noah Sussman

Most diff implementations remain outwardly unchanged since 1975.
                -- Wikipedia

It's been over a year since david23 lent me his copy of A Quarter Century Of Unix by Peter Salus (part of the Unix and Open Systems series from Addison-Wesley) and it has sat on my shelf throughout a good 18 months of learning to hack Perl, Ruby and the GNU toolset. I should mention that, 19 months ago, I thought I knew how to manage an HTML source tree with the GNU tools and Perl; but now I know differently (and I know slightly more than I did then as well).

One thing I've learned is that, although yes some of the "old" Unix tools were (apparently) perfected back when I was coding my first lines of BASIC, there's still a lot going on in the world of command-line tools. In fact, one of the big revelations for me in the last year and a half, is that despite the growing ubiquity of multitouch displays (a friend just built one in his garage) and iPhones; a lot of people still work in the shell.

So there's really just a great, healthy, growing community of open-source command-line hackers out there, and as I've joined their ranks, I've grown more and more interested in their (our) history. After all, I wrote my first lines of code on an Osborne computer running CP/M, an OS descended directly from the DEC operating systems that ran on some of the mainframes where Unix was spawned.

Photo: Digital PDP-11 by Laughing Squid

So it was great fun to finally get around to reading A Quarter Century Of Unix. For one thing, the book contains lots of relevant technical information (now I finally understand that the parenthetical numbers in the names of manpages, refer to volumes of the Unix manual). And, as with any history of the open-source movement, there are plenty of anecdotes about pranks and late nights and spur-of-the-moment experiments that turned into tools that are still powering the Internet today.

This book also helped me to appreciate that the "viral" spreading of Unix systems didn't start with Linux. Salus takes a couple of chapters to fully describe the dissemination of unlicensed or quasi-licensed Unixes, on great reels of magnetic tape, from MIT and Stanford to Wollongong university in Australia.

The thing that really got me about this book though, is all the details about the tools. It's good to reflect occasionally that "basic" tools like the text editor were invented, and not that long ago. And I didn't know that one of the first industrial uses of Unix, was in preparing documents for print. It's always good to be reminded that the shell predates the GUI. As my dad told me when, at the age of 10 I asked how a computer works: pictures may show up on the screen, but underneath, its all text (OK, he really said "underneath, it's all ones and zeroes," but close enough.)

Anyway, this book is a fun, fast read, and while it contains a lot of information found online, the writing is enjoyable, and, being written by Salus, who was present at many of the events he describes, there's a lot of interesting details.

Geocaching in Hawaii

June 01, 2009

Guest blogger Carol Campbell shares how she uses GPS to turn a vaction into an adventure.

So you’ve booked your vacation to the Hawaiian Islands but you’re wondering what to do when you get there. Of course there are all of the usual trappings…luaus, pineapple tours, and black sand beaches. If you are going to be driving from one side of the island to the other every day, you might as well make the most of it and see as many beautiful places as possible. That’s not always so easy to do because many of them are hidden away and not put on the map that your concierge will give you at the hotel. Our first visit to Kauai was fun but we saw so much more our second time because we had some help with finding locations that we didn’t even know existed.

Better Know A Blogger

April 29, 2009

By Joanna

Nerdabout would like to introduce part 1 of our 6 billion part series, Better Know a Blogger. Kevin, whose blog Rumproast won the 2008 Weblog Best Small Blog award, reveals what makes a blog not suck. (Dark footage intentional as a half hearted attempt to conceal identities. A couple pints of beer remedied that.)

Rule # 1: No Expectations

Rule # 2: Don't Tell Your Friends

Rule # 1 & 2 from Noah Sussman on Vimeo.