Phil Dawes' Stuff - Latest Comments in importing statements at speed

Re: importing statements at speed

danja — Thu, 23 Sep 2004 03:28:06 -0000

Wow, those figures are looking good.
If you're looking at big data it may be worth keeping one eye on developments in RFC3229 and feeds as a possible means to sync'ing big stores, see:
http://bobwyman.pubsub.com/

Re: importing statements at speed

Phil Dawes — Thu, 23 Sep 2004 02:59:06 -0000

That's it - many thanks.

Have been thinking about chunked imports too - makes sense with the bulk-import approach since most of the time is spent in the parsing/preparing rather than the actual dumping to db. Chunking it would enable parallelization of the time consuming preparation stage. Would work best with something like 3store, which stores md5 hashes for URIs in its main triples table - no centralization required to agree IDs for URIs.

Unfortunately (from this perspective), my store uses generated IDs for URIs to enable a 1:many logical-resource -> URI mapping for smushing. Would probably need to import in chunks and then reconcile IDs in a subsequent sweep.

Re: importing statements at speed

Dan Brickley — Wed, 22 Sep 2004 19:58:26 -0000

Hi Phil! Nice work. Re data, you might be thinking of http://rdfdata.org/

Re imports, I'd wondered about doing this sort of think chunked into, say, 10000 triple blocks, for large imports. But yes, having a benchmarking framework would take out some of the guesswork...