Re: Sesame Native Store OPS indexing (was Re: [update] piggybank performance profiling)

From: Arjohn Kampman <arjohn.kampman_at_aduna.biz>
Date: Tue, 13 Sep 2005 21:50:55 +0200

Vineet Sinha wrote:
> The summary
> -----------
> OPS indexing for the Sesame Native Store has been implemented. And it
> works beautifully in Relo. Stefano should be looking at it in Piggy Bank.
>
> Details
> -------
> Single Modified File:
> http://simile.mit.edu/repository/relo/branches/sesame/src/org/openrdf/sesame/sailimpl/nativerdf/TripleStore.java
>
>
> The working jar file
> http://simile.mit.edu/repository/relo/trunk/edu.mit.csail.relo.store/lib/sesame-1.2.1-ops.jar
>
>
> Implementation was as Arjohn had suggested. I did not see any bugs.
> Adding statements seems to have added a 25% cost.

Hey, that's less than I expected! If it doesn't add more than 25%
overhead then I think we should include the change in future Sesame
releases. Can you give an indication of the amount of data that you
tested this with?

> Beyond adding the second comparator, I also renamed previous
> btree/file/filename variables to include 'spo' before them and made a
> copy for 'ops'. triples.dat is now triples-1.dat and triples-2.dat.

May I suggest to call these files "triples-spo.dat" and
"triples-ops.dat"?

> The other issue is a Jeen mentioned, the large file size for
> transmission. The best solution could be to not require it for
> transmission and build the second index automatically (in fact this
> should also increase the add performance, relying on the spo index until
> the ops index is ready).

I'm not sure if I understand this correctly. The file size of the
indexes is problematic because you are sending these files over the
internet or something?

> There were no unit tests, but tests on Relo should have been good.
>
> I can send patch, after we fix any issues. Let me know if you have
> questions.
>
> Arjohn, thanks for your help!

My pleasure.
Received on Tue Sep 13 2005 - 19:47:02 EDT

This archive was generated by hypermail 2.3.0 : Thu Aug 09 2012 - 16:39:18 EDT