Re: Using Solvent with Javascript rendered HTML pages

From: David Huynh <dfhuynh_at_csail.mit.edu>
Date: Wed, 01 Feb 2006 07:54:24 -0500

Barney wrote:

>Ok thanks, looks like I misdiagnosed the problem. Is there an
>alternative way to deal with sites that use IFRAMES?
>
>
In Firefox, I think you can right-click on anywhere inside an IFRAME and
select Show This Frame Only. Then you can use Solvent on that. But then
if you want your scraper to run on the original page, you need to get
from that page's DOM into the IFRAME's DOM first before applying the
XPath you've captured:

    for (var i = 0; i < doc.frames.length; i++) {
       if (frames[i] /* is the one you want */) {
          /* process frames[i].contentDocument */
       }
    }

You may want to use Firefox's DOM Inspector in conjunction with Solvent.

In the future, we'll try to make Solvent deal with frames.

Cheers,

David
Received on Wed Feb 01 2006 - 12:55:39 EST