Phil Dawes' Stuff - Latest Comments in Suffix array performance problems

Re: Suffix array performance problems

HoangThanh — Fri, 13 Oct 2006 21:58:36 -0000

i'm HMT

Re: Suffix array performance problems

Ed — Thu, 10 Feb 2005 16:38:12 -0000

Hi !

Looking for a suffix index that would tell me what programs open files with what suffix. Such that it would tell me what program created on what programs would open files say with .jpg suffix.

Probably totally in the wrong place, but could you help?

Ed.

Re: Suffix array performance problems

Phil Dawes — Sat, 01 Jan 2005 16:32:59 -0000

Have implemented the changes in veudastore - released version 0.3.
To use this with veudas, replace the cgi-bin/veudas-0.6/veudastore directory with this release

Re: Suffix array performance problems

Phil Dawes — Sat, 01 Jan 2005 07:37:18 -0000

Cool - think I've cracked it. The solution was to remove the original suffix and links tables, and replace them with a single suffixes table that just maps char(4) suffixes to literals.
This reduces the accuracy of the suffix match (meaning that substring searches > 4 chars need to join and filter with the nodes table (containing the literals). The width of the suffix field could be increased if this becomes a bottleneck, trading off more space for speed.

Have got the query down to 30ms on my laptop (down from 82.61 seconds on the original scheme). Other queries are looking good as well (mostly in the 10-30ms mark). Should fly on the 10GB 8 proc box at work :-)

The reason this is so much faster is that the mysql query optimiser gets a better view of the number of literals each substring will match, and so correctly identifies the best match order. Also, by indexing the new suffix table both ways (suffix-literal and literal-suffix), the query analyser can chose to impose other constraints in between substring filtering
(e.g. reducing the literal matches to those that are rdfs:labels).

Now I just need to update the veudastore code to use the new scheme...