Re: Piggy Bank and Semantic Bank: scalability and performance

From: Rickard Öberg <rickard.oberg_at_senselogic.se>
Date: Sun, 23 Oct 2005 22:43:45 +0200

David Huynh wrote:
>> Well, that's just it... if you think about it 30.000 is NOTHING. Tiny.
>> Miniscule. Insignificant. I mean, if you are really really serious
>> about building semantic banks and semantic webs then you should add a
>> couple of zeroes to that number...
>
> Oh no, we do have datasets in the order of hundreds of thousands of
> items...

In a MySQL database? Does that scale better?

>>> I think heavy-duty scraping should not be supported inside Piggy
>>> Bank. We can have an entirely stand-alone application for that. You
>>> probably want sophisticated status monitoring UI, etc., etc., too.
>>
>>
>> Indeed. Any ideas for how to do that? For heavy-duty scraping I will
>> probably want to do it on a timer as well, and revisit websites
>> reasonably often to get new data.
>
> Man, I'm never gonna finish my thesis...

Oh, go ahead and finish the thesis. And start a company selling services
around the above. Think about it. Doesn't get much more Web 2.0, now
does it? So getting funding should be a breeze ;-)

> It'd be great if you can send the URL to the N3 file describing that
> scraper... Then it's easy to install it into Piggy Bank.

No can do. Only have it in my file system, so it's not available from
any website. I'll email it.

> Yes, we have a lot of technologies at hands (applet, canvas, svg, ajax)
> to build more sophisticated visualizations. It's a bright future.

Indeed. I've been looking at using OpenMap to do visualizations in Java.
Should be reasonably easy to do.

> You want to hook in Virtual Earth or Yahoo Maps and see? :-)

Sure, if you'll write the code for it ;-)

/Rickard
Received on Sun Oct 23 2005 - 20:38:20 EDT