Is it possible to call scrapers in scrapers? For example I have a web
site with search results of books - if someone already wrote a scraper
for the books detail page I could just plug it in in a loop. Following
that idea there could be comment/reviews which again could be scraped of
the book detail page and so on.
The first idea is to reuse the scrapers and maybe it may even be useful
to allow an automated discovery and crawling of sub pages - e.g. if a
scraper exists it could offer me not only to scrape the initial page but
also the detail pages with out me actually writing it into the code.
That would save a lot of coding because it often is 'scrape this page
and follow all sub pages and then scrape them'. Not very flexible
especially considering multiple and changing concatenations.
I know it is not a fully developed idea but I just ran into it and
thought it would be worth mentioning but I wouldn't be surprised if you
guys already thought of it and have it on your to-do list.
Cheers,
Christian Prokopp
SAP Research CEC Brisbane
SAP Australia Pty. Ltd
Level 12, 133 Mary Street
Brisbane QLD 4000
Australia
T + 61 7 3259 9 0
E christian.prokopp_at_sap.com
http://www.sap.com/australia
Received on Mon Jan 16 2006 - 01:34:25 EST