Re imports, I'd wondered about doing this sort of think chunked into, say, 10000 triple blocks, for large imports. But yes, having a benchmarking framework would take out some of the guesswork...
Phil Dawes
· 5 years ago
That's it - many thanks.
Have been thinking about chunked imports too - makes sense with the bulk-import approach since most of the time is spent in the parsing/preparing rather than the actual dumping to db. Chunking it would enable parallelization of the time consuming preparation stage. Would work best with something like 3store, which stores md5 hashes for URIs in its main triples table - no centralization required to agree IDs for URIs.
Unfortunately (from this perspective), my store uses generated IDs for URIs to enable a 1:many logical-resource -> URI mapping for smushing. Would probably need to import in chunks and then reconcile IDs in a subsequent sweep.
danja
· 5 years ago
Wow, those figures are looking good. If you're looking at big data it may be worth keeping one eye on developments in RFC3229 and feeds as a possible means to sync'ing big stores, see: http://bobwyman.pubsub.com/
All Comments
Re imports, I'd wondered about doing this sort of think chunked into, say, 10000 triple blocks, for large imports. But yes, having a benchmarking framework would take out some of the guesswork...
Have been thinking about chunked imports too - makes sense with the bulk-import approach since most of the time is spent in the parsing/preparing rather than the actual dumping to db. Chunking it would enable parallelization of the time consuming preparation stage. Would work best with something like 3store, which stores md5 hashes for URIs in its main triples table - no centralization required to agree IDs for URIs.
Unfortunately (from this perspective), my store uses generated IDs for URIs to enable a 1:many logical-resource -> URI mapping for smushing. Would probably need to import in chunks and then reconcile IDs in a subsequent sweep.
If you're looking at big data it may be worth keeping one eye on developments in RFC3229 and feeds as a possible means to sync'ing big stores, see:
http://bobwyman.pubsub.com/