Re: Solvent - modularize - automation

From: David Huynh <dfhuynh_at_csail.mit.edu>
Date: Thu, 19 Jan 2006 02:16:08 -0500

Currently there's no easy way for one scraper to call another. But it is
an interesting idea. Let us know if you flesh out this idea...

David


Prokopp, Christian wrote:

>Is it possible to call scrapers in scrapers? For example I have a web
>site with search results of books - if someone already wrote a scraper
>for the books detail page I could just plug it in in a loop. Following
>that idea there could be comment/reviews which again could be scraped of
>the book detail page and so on.
>The first idea is to reuse the scrapers and maybe it may even be useful
>to allow an automated discovery and crawling of sub pages - e.g. if a
>scraper exists it could offer me not only to scrape the initial page but
>also the detail pages with out me actually writing it into the code.
>That would save a lot of coding because it often is 'scrape this page
>and follow all sub pages and then scrape them'. Not very flexible
>especially considering multiple and changing concatenations.
>
>I know it is not a fully developed idea but I just ran into it and
>thought it would be worth mentioning but I wouldn't be surprised if you
>guys already thought of it and have it on your to-do list.
>
>Cheers,
>
>Christian Prokopp
>SAP Research CEC Brisbane
>SAP Australia Pty. Ltd
>Level 12, 133 Mary Street
>Brisbane QLD 4000
>Australia
>T + 61 7 3259 9 0
>E christian.prokopp_at_sap.com
>http://www.sap.com/australia
>
>
>
>
Received on Thu Jan 19 2006 - 07:17:44 EST

This archive was generated by hypermail 2.3.0 : Thu Aug 09 2012 - 16:39:18 EDT