Re: Solvent - modularize - automation

From: David Huynh <>
Date: Thu, 19 Jan 2006 02:16:08 -0500

Currently there's no easy way for one scraper to call another. But it is
an interesting idea. Let us know if you flesh out this idea...


Prokopp, Christian wrote:

>Is it possible to call scrapers in scrapers? For example I have a web
>site with search results of books - if someone already wrote a scraper
>for the books detail page I could just plug it in in a loop. Following
>that idea there could be comment/reviews which again could be scraped of
>the book detail page and so on.
>The first idea is to reuse the scrapers and maybe it may even be useful
>to allow an automated discovery and crawling of sub pages - e.g. if a
>scraper exists it could offer me not only to scrape the initial page but
>also the detail pages with out me actually writing it into the code.
>That would save a lot of coding because it often is 'scrape this page
>and follow all sub pages and then scrape them'. Not very flexible
>especially considering multiple and changing concatenations.
>I know it is not a fully developed idea but I just ran into it and
>thought it would be worth mentioning but I wouldn't be surprised if you
>guys already thought of it and have it on your to-do list.
>Christian Prokopp
>SAP Research CEC Brisbane
>SAP Australia Pty. Ltd
>Level 12, 133 Mary Street
>Brisbane QLD 4000
>T + 61 7 3259 9 0
Received on Thu Jan 19 2006 - 07:17:44 EST

This archive was generated by hypermail 2.3.0 : Thu Aug 09 2012 - 16:39:18 EDT