DISQUS

DISQUS Hello! Phil Dawes' Stuff is using DISQUS, a powerful comment system, to manage its comments. Learn more.

Community Page

Phil Dawes' Stuff

Jump to original thread »
Author

importing statements at speed

Started by phildawes · Acum 9 luni

Spent the evening working on bulk assert speed for my ifp smushing store. Managed to get a 120000 statement import down from 410 seconds to 109 seconds - that’s over 1000 triples per second. Pretty good considering the rdflib rdf/xml parsing takes 59 seconds on its own - I wonder how f ... Continue reading »

3 comments

  • Dan Brickley

    Acum 4 ani

    Hi Phil! Nice work. Re data, you might be thinking of http://rdfdata.org/

    Re imports, I'd wondered about doing this sort of think chunked into, say, 10000 triple blocks, for large imports. But yes, having a benchmarking framework would take out some of the guesswork...
    reply
  • Phil Dawes

    Acum 4 ani

    That's it - many thanks.

    Have been thinking about chunked imports too - makes sense with the bulk-import approach since most of the time is spent in the parsing/preparing rather than the actual dumping to db. Chunking it would enable parallelization of the time consuming preparation stage. Would work best with something like 3store, which stores md5 hashes for URIs in its main triples table - no centralization required to agree IDs for URIs.

    Unfortunately (from this perspective), my store uses generated IDs for URIs to enable a 1:many logical-resource -> URI mapping for smushing. Would probably need to import in chunks and then reconcile IDs in a subsequent sweep.
    reply
  • danja

    Acum 4 ani

    Wow, those figures are looking good.
    If you're looking at big data it may be worth keeping one eye on developments in RFC3229 and feeds as a possible means to sync'ing big stores, see:
    http://bobwyman.pubsub.com/
    reply

Add New Comment

Returning? Login