Scrape and save Google catalog search results to a comma-delimited file.
The December-Holiday-of-Your-Choice comes but once a year, but catalog shopping is a year-round joy. And Google Catalogs [Hack #33] makes it easier than ever.
Of course, just because you spend a rainy winter afternoon finding the perfect shawl for Aunt Prunella doesn’t mean that you’ll be able to replicate the search when you need it. Or maybe you want to do a search for several things and browse them at your leisure later.
Because Google Catalogs aren’t supported by the Google API, this hack scrapes finds from the HTML of a Google Catalogs results page. It saves the catalog title, date or season, page number, and even a link to an image of the page itself at Google. Results are saved in CSV format, ready for import into a database or spreadsheet application.
Because Google’s Terms of Service prohibits the automated access of their search engines except through the Google API, this hack does not actually connect to Google. Instead, it works on a page of results that you’ve saved from a Google Catalogs search you’ve run yourself. Simply save the results page as HTML source using your browser’s File → Save As... command.
As with the Google News hack [Hack #47], you can optimize the
effectiveness of this hack by changing the results URL ever so
slightly to tweak the order of any data displayed. By adding a
&num=100
to the end of the catalog search
results URL, you’ll get up to 100 results instead of
only the first.
For example, Figure 4-3 shows results of a query
for the perfect shawl
for that aunt.
#!/usr/bin/perl
# catalogs2csv.pl
# Google Catalogs Results exported to CSV suitable for import into Excel
# Usage: perl catalogs2csv.pl < catalogs.html > catalogs.csv
print qq{"title","link","date","page"
};
my($results) = join '', <>;
while ( $results =~ m!<td>(.+?) <font size=-1>(.+?) - Pa
ge (w+?) -.+?<a href="(/catalogs.+?)"><img src=.+?></a> !migs ) {
my($title, $date, $page, $url) = ($1||'',$2||'',$3||'',$4||''),
$title =~ s!"!""!g; # double escape " marks
my $output = qq{"$title","$url","$date","$page"
};
$output =~ s! ! !g; # clean spaces
print $output;
}
Run the script from the command line, specifying the Google Catalogs
results HTML filename and name of the CSV file you wish to create or
to which you wish to append additional results. For example, using
catalogs.html
as our input and
catalogs.csv
as our output:
$ perl catalogs2csv.pl < catalogs.html > catalogs.csv
Leaving off the >
and CSV filename sends the
results to the screen for your perusal.
You (and your aunt) appear to be in luck; there’s an
abundance of perfect shawl
s to be found in the
Google Catalogs:
"title","link","date","page" "Isabella","http://catalogs.google.com/catalogs?num=100 &hl=en&lr=&ie=UTF-8&issue=13655&catpage=cover","Fall 2002","3" "Sovietski Collection","http://catalogs.google.com/catalogs?num=100 &hl=en&lr=&ie=UTF-8&issue=9447&catpage=cover","Summer 2002","37" "Rego","http://catalogs.google.com/catalogs?num=100&hl=en &lr=&ie=UTF-8&issue=12484&catpage=cover","2002","21" "Crazy Crow Trading Post","http://catalogs.google.com/catalogs?num=100 &hl=en&lr=&ie=UTF-8&issue=12346&catpage=cover","2002","39" "Winter Silks - Sale","http://catalogs.google.com/catalogs?num=100 &hl=en&lr=&ie=UTF-8&issue=10002&catpage=cover","Summer 2002","11" ... "Previews","http://catalogs.google.com/catalogs?num=100 &hl=en&lr=&ie=UTF-8&issue=14468&catpage=cover","Oct 2002","381"
(Each listing actually occurs on its own line; lines are broken and occasionally shortened for the purposes of publication.)
The output format may be altered to suit your fancy; see the Google News Scraper [Hack #47] for details on hacking the hack.