updated hash functions for postgresql v1
|
View:
New views
20 Messages
—
Rating Filter:
Alert me
|
| < Prev | 1 - 2 | Next > |
 |
updated hash functions for postgresql v1
by Kenneth Marshall-3
::
Rate this Message:
Dear PostgreSQL Developers,
This patch is a "diff -c" against the hashfunc.c from postgresql-8.3beta1. It implements the 2006 version of the hash function by Bob Jenkins. Its features include a better and faster hash function. I have included the versions supporting big-endian and little-endian machines that will be selected based on the machine configuration. Currently, I have hash_any() just a stub calling hashlittle and hashbig. In order to allow the hash index to support large indexes (>10^9 entries), the hash function needs to be able to provide 64-bit hashes. The functions hashbig2/hashlittle2 produce 2 32-bit hashes that can be used as a 64-bit hash value. I would like some feedback as to how best to include 64-bit hashes within our current 32-bit hash infrastructure. The hash-merge can simple use one of the 2 32-bit pieces to provide the current 32-bit hash values needed. Then they could be pulled directly from the hash index and not need to be recalculated at run time. What would be the best way to implement this in a way that will work on machines without support for 64-bit integers? The current patch passes all the regression tests, but has a few warnings for the different variations of the new hash function. Until the design has crystalized, I am not going to worry about them and I want testers to have access to the different functions. I am doing the initial patches to the hash index code based on a 32-bit hash, but I would like to add the 64-bit hash support pretty early in the development cycle in order to allow for better testing. Any thoughts would be welcome. Regards, Ken *** hashfunc.c pgsql/src/backend/access/hash/hashfunc.c,v 1.53 2007/09/21 22:52:52 tgl --- hashfunc.c_NEW32 Wed Oct 17 13:58:10 2007 *************** *** 197,230 **** * This hash function was written by Bob Jenkins * (bob_jenkins@...), and superficially adapted * for PostgreSQL by Neil Conway. For more information on this ! * hash function, see http://burtleburtle.net/bob/hash/doobs.html, ! * or Bob's article in Dr. Dobb's Journal, Sept. 1997. */ /*---------- * mix -- mix 3 32-bit values reversibly. ! * For every delta with one or two bits set, and the deltas of all three ! * high bits or all three low bits, whether the original value of a,b,c ! * is almost all zero or is uniformly distributed, ! * - If mix() is run forward or backward, at least 32 bits in a,b,c ! * have at least 1/4 probability of changing. ! * - If mix() is run forward, every bit of c will change between 1/3 and ! * 2/3 of the time. (Well, 22/100 and 78/100 for some 2-bit deltas.) *---------- */ #define mix(a,b,c) \ { \ ! a -= b; a -= c; a ^= ((c)>>13); \ ! b -= c; b -= a; b ^= ((a)<<8); \ ! c -= a; c -= b; c ^= ((b)>>13); \ ! a -= b; a -= c; a ^= ((c)>>12); \ ! b -= c; b -= a; b ^= ((a)<<16); \ ! c -= a; c -= b; c ^= ((b)>>5); \ ! a -= b; a -= c; a ^= ((c)>>3); \ ! b -= c; b -= a; b ^= ((a)<<10); \ ! c -= a; c -= b; c ^= ((b)>>15); \ } /* * hash_any() -- hash a variable-length key into a 32-bit value * k : the key (the unaligned variable-length array of bytes) --- 197,1060 ---- * This hash function was written by Bob Jenkins * (bob_jenkins@...), and superficially adapted * for PostgreSQL by Neil Conway. For more information on this ! * hash function, see http://burtleburtle.net/bob/hash/#lookup ! * and http://burtleburtle.net/bob/hash/lookup3.txt. Further ! * information on the original version of the hash function can ! * be found in Bob's article in Dr. Dobb's Journal, Sept. 1997. */ + #define rot(x,k) (((x)<<(k)) | ((x)>>(32-(k)))) + + #ifndef WORS_BIGENDIAN + #define HASH_LITTLE_ENDIAN 1 + #define HASH_BIG_ENDIAN 0 + #else + #define HASH_LITTLE_ENDIAN 0 + #define HASH_BIG_ENDIAN 1 + #endif + /*---------- * mix -- mix 3 32-bit values reversibly. ! * ! * This is reversible, so any information in (a,b,c) before mix() is ! * still in (a,b,c) after mix(). ! * ! * If four pairs of (a,b,c) inputs are run through mix(), or through ! * mix() in reverse, there are at least 32 bits of the output that ! * are sometimes the same for one pair and different for another pair. ! * This was tested for: ! * * pairs that differed by one bit, by two bits, in any combination ! * of top bits of (a,b,c), or in any combination of bottom bits of ! * (a,b,c). ! * * "differ" is defined as +, -, ^, or ~^. For + and -, I transformed ! * the output delta to a Gray code (a^(a>>1)) so a string of 1's (as ! * is commonly produced by subtraction) look like a single 1-bit ! * difference. ! * * the base values were pseudorandom, all zero but one bit set, or ! * all zero plus a counter that starts at zero. ! * ! * Some k values for my "a-=c; a^=rot(c,k); c+=b;" arrangement that ! * satisfy this are ! * 4 6 8 16 19 4 ! * 9 15 3 18 27 15 ! * 14 9 3 7 17 3 ! * Well, "9 15 3 18 27 15" didn't quite get 32 bits diffing ! * for "differ" defined as + with a one-bit base and a two-bit delta. I ! * used http://burtleburtle.net/bob/hash/avalanche.html to choose ! * the operations, constants, and arrangements of the variables. ! * ! * This does not achieve avalanche. There are input bits of (a,b,c) ! * that fail to affect some output bits of (a,b,c), especially of a. The ! * most thoroughly mixed value is c, but it doesn't really even achieve ! * avalanche in c. ! * ! * This allows some parallelism. Read-after-writes are good at doubling ! * the number of bits affected, so the goal of mixing pulls in the opposite ! * direction as the goal of parallelism. I did what I could. Rotates ! * seem to cost as much as shifts on every machine I could lay my hands ! * on, and rotates are much kinder to the top and bottom bits, so I used ! * rotates. *---------- */ #define mix(a,b,c) \ { \ ! a -= c; a ^= rot(c, 4); c += b; \ ! b -= a; b ^= rot(a, 6); a += c; \ ! c -= b; c ^= rot(b, 8); b += a; \ ! a -= c; a ^= rot(c,16); c += b; \ ! b -= a; b ^= rot(a,19); a += c; \ ! c -= b; c ^= rot(b, 4); b += a; \ } + /*---------- + * final -- final mixing of 3 32-bit values (a,b,c) into c + * + * Pairs of (a,b,c) values differing in only a few bits will usually + * produce values of c that look totally different. This was tested for + * - pairs that differed by one bit, by two bits, in any combination + * of top bits of (a,b,c), or in any combination of bottom bits of + * (a,b,c). + * - "differ" is defined as +, -, ^, or ~^. For + and -, I transformed + * the output delta to a Gray code (a^(a>>1)) so a string of 1's (as + * is commonly produced by subtraction) look like a single 1-bit + * difference. + * - the base values were pseudorandom, all zero but one bit set, or + * all zero plus a counter that starts at zero. + * + * These constants passed: + * 14 11 25 16 4 14 24 + * 12 14 25 16 4 14 24 + * and these came close: + * 4 8 15 26 3 22 24 + * 10 8 15 26 3 22 24 + * 11 8 15 26 3 22 24 + *---------- + */ + #define final(a,b,c) \ + { \ + c ^= b; c -= rot(b,14); \ + a ^= c; a -= rot(c,11); \ + b ^= a; b -= rot(a,25); \ + c ^= b; c -= rot(b,16); \ + a ^= c; a -= rot(c,4); \ + b ^= a; b -= rot(a,14); \ + c ^= b; c -= rot(b,24); \ + } + + /*---------- + * This works on all machines. To be useful, it requires + * -- that the key be an array of uint32's, and + * -- that the length be the number of uint32's in the key + * + * The function hashword() is identical to hashlittle() on little-endian + * machines, and identical to hashbig() on big-endian machines, + * except that the length has to be measured in uint32s rather than in + * bytes. hashlittle() is more complicated than hashword() only because + * hashlittle() has to dance around fitting the key bytes into registers. + *---------- + */ + Datum + hashword( + const uint32 *k, /* the key, an array of uint32 values */ + size_t length, /* the length of the key, in uint32s */ + uint32 initval) /* the previous hash, or an arbitrary value */ + { + uint32 a,b,c; + + /* Set up the internal state */ + a = b = c = 0xdeadbeef + (((uint32)length)<<2) + initval; + + /*------------------------------------------------- handle most of the key */ + while (length > 3) + { + a += k[0]; + b += k[1]; + c += k[2]; + mix(a,b,c); + length -= 3; + k += 3; + } + + /*------------------------------------------- handle the last 3 uint32's */ + switch(length) /* all the case statements fall through */ + { + case 3 : c+=k[2]; + case 2 : b+=k[1]; + case 1 : a+=k[0]; + final(a,b,c); + case 0: /* case 0: nothing left to add */ + break; + } + /*------------------------------------------------------ report the result */ + return UInt32GetDatum(c); + } + + + /*---------- + * hashword2() -- same as hashword(), but take two seeds and return two + * 32-bit values. pc and pb must both be nonnull, and *pc and *pb must + * both be initialized with seeds. If you pass in (*pb)==0, the output + * (*pc) will be the same as the return value from hashword(). + *---------- + */ + void hashword2 ( + const uint32 *k, /* the key, an array of uint32 values */ + size_t length, /* the length of the key, in uint32s */ + uint32 *pc, /* IN: seed OUT: primary hash value */ + uint32 *pb) /* IN: more seed OUT: secondary hash value */ + { + uint32 a,b,c; + + /* Set up the internal state */ + a = b = c = 0xdeadbeef + ((uint32)(length<<2)) + *pc; + c += *pb; + + /*------------------------------------------------- handle most of the key */ + while (length > 3) + { + a += k[0]; + b += k[1]; + c += k[2]; + mix(a,b,c); + length -= 3; + k += 3; + } + + /*------------------------------------------- handle the last 3 uint32's */ + switch(length) /* all the case statements fall through */ + { + case 3 : c+=k[2]; + case 2 : b+=k[1]; + case 1 : a+=k[0]; + final(a,b,c); + case 0: /* case 0: nothing left to add */ + break; + } + /*------------------------------------------------------ report the result */ + *pc=c; *pb=b; + } + + + /*---------- + * hashlittle() -- hash a variable-length key into a 32-bit value + * k : the key (the unaligned variable-length array of bytes) + * length : the length of the key, counting by bytes + * initval : can be any 4-byte value + * Returns a 32-bit value. Every bit of the key affects every bit of + * the return value. Two keys differing by one or two bits will have + * totally different hash values. + * + * The best hash table sizes are powers of 2. There is no need to do + * mod a prime (mod is sooo slow!). If you need less than 32 bits, + * use a bitmask. For example, if you need only 10 bits, do + * h = (h & hashmask(10)); + * In which case, the hash table should have hashsize(10) elements. + * + * If you are hashing n strings (uint8 **)k, do it like this: + * for (i=0, h=0; i<n; ++i) h = hashlittle( k[i], len[i], h); + * + * By Bob Jenkins, 2006. bob_jenkins@.... You may use this + * code any way you wish, private, educational, or commercial. It's free. + * + * Use for hash table lookup, or anything where one collision in 2^^32 is + * acceptable. Do NOT use for cryptographic purposes. + *---------- + */ + + Datum + hashlittle( const void *key, size_t length, uint32 initval) + { + uint32 a,b,c; /* internal state */ + union { const void *ptr; size_t i; } u; /* needed for Mac Powerbook G4 */ + + /* Set up the internal state */ + a = b = c = 0xdeadbeef + ((uint32)length) + initval; + + u.ptr = key; + if (HASH_LITTLE_ENDIAN && ((u.i & 0x3) == 0)) { + const uint32 *k = (const uint32 *)key; /* read 32-bit chunks */ + #ifdef VALGRIND + const uint8 *k8; + #endif + + /*------ all but last block: aligned reads and affect 32 bits of (a,b,c) */ + while (length > 12) + { + a += k[0]; + b += k[1]; + c += k[2]; + mix(a,b,c); + length -= 12; + k += 3; + } + + /*----------------------------- handle the last (probably partial) block */ + /* + * "k[2]&0xffffff" actually reads beyond the end of the string, but + * then masks off the part it's not allowed to read. Because the + * string is aligned, the masked-off tail is in the same word as the + * rest of the string. Every machine with memory protection I've seen + * does it on word boundaries, so is OK with this. But VALGRIND will + * still catch it and complain. The masking trick does make the hash + * noticably faster for short strings (like English words). + */ + #ifndef VALGRIND + + switch(length) + { + case 12: c+=k[2]; b+=k[1]; a+=k[0]; break; + case 11: c+=k[2]&0xffffff; b+=k[1]; a+=k[0]; break; + case 10: c+=k[2]&0xffff; b+=k[1]; a+=k[0]; break; + case 9 : c+=k[2]&0xff; b+=k[1]; a+=k[0]; break; + case 8 : b+=k[1]; a+=k[0]; break; + case 7 : b+=k[1]&0xffffff; a+=k[0]; break; + case 6 : b+=k[1]&0xffff; a+=k[0]; break; + case 5 : b+=k[1]&0xff; a+=k[0]; break; + case 4 : a+=k[0]; break; + case 3 : a+=k[0]&0xffffff; break; + case 2 : a+=k[0]&0xffff; break; + case 1 : a+=k[0]&0xff; break; + case 0 : return UInt32GetDatum(c); /* zero length requires no mixing */ + } + + #else /* make valgrind happy */ + + k8 = (const uint8 *)k; + switch(length) + { + case 12: c+=k[2]; b+=k[1]; a+=k[0]; break; + case 11: c+=((uint32)k8[10])<<16; /* fall through */ + case 10: c+=((uint32)k8[9])<<8; /* fall through */ + case 9 : c+=k8[8]; /* fall through */ + case 8 : b+=k[1]; a+=k[0]; break; + case 7 : b+=((uint32)k8[6])<<16; /* fall through */ + case 6 : b+=((uint32)k8[5])<<8; /* fall through */ + case 5 : b+=k8[4]; /* fall through */ + case 4 : a+=k[0]; break; + case 3 : a+=((uint32)k8[2])<<16; /* fall through */ + case 2 : a+=((uint32)k8[1])<<8; /* fall through */ + case 1 : a+=k8[0]; break; + case 0 : return UInt32GetDatum(c); + } + + #endif /* !valgrind */ + + } else if (HASH_LITTLE_ENDIAN && ((u.i & 0x1) == 0)) { + const uint16 *k = (const uint16 *)key; /* read 16-bit chunks */ + const uint8 *k8; + + /*--------------- all but last block: aligned reads and different mixing */ + while (length > 12) + { + a += k[0] + (((uint32)k[1])<<16); + b += k[2] + (((uint32)k[3])<<16); + c += k[4] + (((uint32)k[5])<<16); + mix(a,b,c); + length -= 12; + k += 6; + } + + /*----------------------------- handle the last (probably partial) block */ + k8 = (const uint8 *)k; + switch(length) + { + case 12: c+=k[4]+(((uint32)k[5])<<16); + b+=k[2]+(((uint32)k[3])<<16); + a+=k[0]+(((uint32)k[1])<<16); + break; + case 11: c+=((uint32)k8[10])<<16; /* fall through */ + case 10: c+=k[4]; + b+=k[2]+(((uint32)k[3])<<16); + a+=k[0]+(((uint32)k[1])<<16); + break; + case 9 : c+=k8[8]; /* fall through */ + case 8 : b+=k[2]+(((uint32)k[3])<<16); + a+=k[0]+(((uint32)k[1])<<16); + break; + case 7 : b+=((uint32)k8[6])<<16; /* fall through */ + case 6 : b+=k[2]; + a+=k[0]+(((uint32)k[1])<<16); + break; + case 5 : b+=k8[4]; /* fall through */ + case 4 : a+=k[0]+(((uint32)k[1])<<16); + break; + case 3 : a+=((uint32)k8[2])<<16; /* fall through */ + case 2 : a+=k[0]; + break; + case 1 : a+=k8[0]; + break; + case 0 : return UInt32GetDatum(c); /* zero length requires no mixing */ + } + + } else { /* need to read the key one byte at a time */ + const uint8 *k = (const uint8 *)key; + + /*--------------- all but the last block: affect some 32 bits of (a,b,c) */ + while (length > 12) + { + a += k[0]; + a += ((uint32)k[1])<<8; + a += ((uint32)k[2])<<16; + a += ((uint32)k[3])<<24; + b += k[4]; + b += ((uint32)k[5])<<8; + b += ((uint32)k[6])<<16; + b += ((uint32)k[7])<<24; + c += k[8]; + c += ((uint32)k[9])<<8; + c += ((uint32)k[10])<<16; + c += ((uint32)k[11])<<24; + mix(a,b,c); + length -= 12; + k += 12; + } + + /*-------------------------------- last block: affect all 32 bits of (c) */ + switch(length) /* all the case statements fall through */ + { + case 12: c+=((uint32)k[11])<<24; + case 11: c+=((uint32)k[10])<<16; + case 10: c+=((uint32)k[9])<<8; + case 9 : c+=k[8]; + case 8 : b+=((uint32)k[7])<<24; + case 7 : b+=((uint32)k[6])<<16; + case 6 : b+=((uint32)k[5])<<8; + case 5 : b+=k[4]; + case 4 : a+=((uint32)k[3])<<24; + case 3 : a+=((uint32)k[2])<<16; + case 2 : a+=((uint32)k[1])<<8; + case 1 : a+=k[0]; + break; + case 0 : return UInt32GetDatum(c); + } + } + + final(a,b,c); + return UInt32GetDatum(c); + } + + + /*---------- + * hashlittle2: return 2 32-bit hash values + * + * This is identical to hashlittle(), except it returns two 32-bit hash + * values instead of just one. This is good enough for hash table + * lookup with 2^^64 buckets, or if you want a second hash if you're not + * happy with the first, or if you want a probably-unique 64-bit ID for + * the key. *pc is better mixed than *pb, so use *pc first. If you want + * a 64-bit value do something like "*pc + (((uint64_t)*pb)<<32)". + *---------- + */ + void hashlittle2( + const void *key, /* the key to hash */ + size_t length, /* length of the key */ + uint32 *pc, /* IN: primary initval, OUT: primary hash */ + uint32 *pb) /* IN: secondary initval, OUT: secondary hash */ + { + uint32 a,b,c; /* internal state */ + union { const void *ptr; size_t i; } u; /* needed for Mac Powerbook G4 */ + + /* Set up the internal state */ + a = b = c = 0xdeadbeef + ((uint32)length) + *pc; + c += *pb; + + u.ptr = key; + if (HASH_LITTLE_ENDIAN && ((u.i & 0x3) == 0)) { + const uint32 *k = (const uint32 *)key; /* read 32-bit chunks */ + #ifdef VALGRIND + const uint8 *k8; + #endif + + /*------ all but last block: aligned reads and affect 32 bits of (a,b,c) */ + while (length > 12) + { + a += k[0]; + b += k[1]; + c += k[2]; + mix(a,b,c); + length -= 12; + k += 3; + } + + /*----------------------------- handle the last (probably partial) block */ + /* + * "k[2]&0xffffff" actually reads beyond the end of the string, but + * then masks off the part it's not allowed to read. Because the + * string is aligned, the masked-off tail is in the same word as the + * rest of the string. Every machine with memory protection I've seen + * does it on word boundaries, so is OK with this. But VALGRIND will + * still catch it and complain. The masking trick does make the hash + * noticably faster for short strings (like English words). + */ + #ifndef VALGRIND + + switch(length) + { + case 12: c+=k[2]; b+=k[1]; a+=k[0]; break; + case 11: c+=k[2]&0xffffff; b+=k[1]; a+=k[0]; break; + case 10: c+=k[2]&0xffff; b+=k[1]; a+=k[0]; break; + case 9 : c+=k[2]&0xff; b+=k[1]; a+=k[0]; break; + case 8 : b+=k[1]; a+=k[0]; break; + case 7 : b+=k[1]&0xffffff; a+=k[0]; break; + case 6 : b+=k[1]&0xffff; a+=k[0]; break; + case 5 : b+=k[1]&0xff; a+=k[0]; break; + case 4 : a+=k[0]; break; + case 3 : a+=k[0]&0xffffff; break; + case 2 : a+=k[0]&0xffff; break; + case 1 : a+=k[0]&0xff; break; + case 0 : *pc=c; *pb=b; return; /* zero length strings require no mixing */ + } + + #else /* make valgrind happy */ + + k8 = (const uint8 *)k; + switch(length) + { + case 12: c+=k[2]; b+=k[1]; a+=k[0]; break; + case 11: c+=((uint32)k8[10])<<16; /* fall through */ + case 10: c+=((uint32)k8[9])<<8; /* fall through */ + case 9 : c+=k8[8]; /* fall through */ + case 8 : b+=k[1]; a+=k[0]; break; + case 7 : b+=((uint32)k8[6])<<16; /* fall through */ + case 6 : b+=((uint32)k8[5])<<8; /* fall through */ + case 5 : b+=k8[4]; /* fall through */ + case 4 : a+=k[0]; break; + case 3 : a+=((uint32)k8[2])<<16; /* fall through */ + case 2 : a+=((uint32)k8[1])<<8; /* fall through */ + case 1 : a+=k8[0]; break; + case 0 : *pc=c; *pb=b; return; /* zero length strings require no mixing */ + } + + #endif /* !valgrind */ + + } else if (HASH_LITTLE_ENDIAN && ((u.i & 0x1) == 0)) { + const uint16 *k = (const uint16 *)key; /* read 16-bit chunks */ + const uint8 *k8; + + /*--------------- all but last block: aligned reads and different mixing */ + while (length > 12) + { + a += k[0] + (((uint32)k[1])<<16); + b += k[2] + (((uint32)k[3])<<16); + c += k[4] + (((uint32)k[5])<<16); + mix(a,b,c); + length -= 12; + k += 6; + } + + /*----------------------------- handle the last (probably partial) block */ + k8 = (const uint8 *)k; + switch(length) + { + case 12: c+=k[4]+(((uint32)k[5])<<16); + b+=k[2]+(((uint32)k[3])<<16); + a+=k[0]+(((uint32)k[1])<<16); + break; + case 11: c+=((uint32)k8[10])<<16; /* fall through */ + case 10: c+=k[4]; + b+=k[2]+(((uint32)k[3])<<16); + a+=k[0]+(((uint32)k[1])<<16); + break; + case 9 : c+=k8[8]; /* fall through */ + case 8 : b+=k[2]+(((uint32)k[3])<<16); + a+=k[0]+(((uint32)k[1])<<16); + break; + case 7 : b+=((uint32)k8[6])<<16; /* fall through */ + case 6 : b+=k[2]; + a+=k[0]+(((uint32)k[1])<<16); + break; + case 5 : b+=k8[4]; /* fall through */ + case 4 : a+=k[0]+(((uint32)k[1])<<16); + break; + case 3 : a+=((uint32)k8[2])<<16; /* fall through */ + case 2 : a+=k[0]; + break; + case 1 : a+=k8[0]; + break; + case 0 : *pc=c; *pb=b; return; /* zero length strings require no mixing */ + } + + } else { /* need to read the key one byte at a time */ + const uint8 *k = (const uint8 *)key; + + /*--------------- all but the last block: affect some 32 bits of (a,b,c) */ + while (length > 12) + { + a += k[0]; + a += ((uint32)k[1])<<8; + a += ((uint32)k[2])<<16; + a += ((uint32)k[3])<<24; + b += k[4]; + b += ((uint32)k[5])<<8; + b += ((uint32)k[6])<<16; + b += ((uint32)k[7])<<24; + c += k[8]; + c += ((uint32)k[9])<<8; + c += ((uint32)k[10])<<16; + c += ((uint32)k[11])<<24; + mix(a,b,c); + length -= 12; + k += 12; + } + + /*-------------------------------- last block: affect all 32 bits of (c) */ + switch(length) /* all the case statements fall through */ + { + case 12: c+=((uint32)k[11])<<24; + case 11: c+=((uint32)k[10])<<16; + case 10: c+=((uint32)k[9])<<8; + case 9 : c+=k[8]; + case 8 : b+=((uint32)k[7])<<24; + case 7 : b+=((uint32)k[6])<<16; + case 6 : b+=((uint32)k[5])<<8; + case 5 : b+=k[4]; + case 4 : a+=((uint32)k[3])<<24; + case 3 : a+=((uint32)k[2])<<16; + case 2 : a+=((uint32)k[1])<<8; + case 1 : a+=k[0]; + break; + case 0 : *pc=c; *pb=b; return; /* zero length strings require no mixing */ + } + } + + final(a,b,c); + *pc=c; *pb=b; + } + + + + /*---------- + * hashbig(): + * This is the same as hashword() on big-endian machines. It is different + * from hashlittle() on all machines. hashbig() takes advantage of + * big-endian byte ordering. + *---------- + */ + Datum + hashbig( const void *key, size_t length, uint32 initval) + { + uint32 a,b,c; + union { const void *ptr; size_t i; } u; /* to cast key to (size_t) happily */ + + /* Set up the internal state */ + a = b = c = 0xdeadbeef + ((uint32)length) + initval; + + u.ptr = key; + if (HASH_BIG_ENDIAN && ((u.i & 0x3) == 0)) { + const uint32 *k = (const uint32 *)key; /* read 32-bit chunks */ + #ifdef VALGRIND + const uint8 *k8; + #endif + + /*------ all but last block: aligned reads and affect 32 bits of (a,b,c) */ + while (length > 12) + { + a += k[0]; + b += k[1]; + c += k[2]; + mix(a,b,c); + length -= 12; + k += 3; + } + + /*----------------------------- handle the last (probably partial) block */ + /* + * "k[2]<<8" actually reads beyond the end of the string, but + * then shifts out the part it's not allowed to read. Because the + * string is aligned, the illegal read is in the same word as the + * rest of the string. Every machine with memory protection I've seen + * does it on word boundaries, so is OK with this. But VALGRIND will + * still catch it and complain. The masking trick does make the hash + * noticably faster for short strings (like English words). + */ + #ifndef VALGRIND + + switch(length) + { + case 12: c+=k[2]; b+=k[1]; a+=k[0]; break; + case 11: c+=k[2]&0xffffff00; b+=k[1]; a+=k[0]; break; + case 10: c+=k[2]&0xffff0000; b+=k[1]; a+=k[0]; break; + case 9 : c+=k[2]&0xff000000; b+=k[1]; a+=k[0]; break; + case 8 : b+=k[1]; a+=k[0]; break; + case 7 : b+=k[1]&0xffffff00; a+=k[0]; break; + case 6 : b+=k[1]&0xffff0000; a+=k[0]; break; + case 5 : b+=k[1]&0xff000000; a+=k[0]; break; + case 4 : a+=k[0]; break; + case 3 : a+=k[0]&0xffffff00; break; + case 2 : a+=k[0]&0xffff0000; break; + case 1 : a+=k[0]&0xff000000; break; + case 0 : return UInt32GetDatum(c); /* zero length requires no mixing */ + } + + #else /* make valgrind happy */ + + k8 = (const uint8 *)k; + switch(length) /* all the case statements fall through */ + { + case 12: c+=k[2]; b+=k[1]; a+=k[0]; break; + case 11: c+=((uint32)k8[10])<<8; /* fall through */ + case 10: c+=((uint32)k8[9])<<16; /* fall through */ + case 9 : c+=((uint32)k8[8])<<24; /* fall through */ + case 8 : b+=k[1]; a+=k[0]; break; + case 7 : b+=((uint32)k8[6])<<8; /* fall through */ + case 6 : b+=((uint32)k8[5])<<16; /* fall through */ + case 5 : b+=((uint32)k8[4])<<24; /* fall through */ + case 4 : a+=k[0]; break; + case 3 : a+=((uint32)k8[2])<<8; /* fall through */ + case 2 : a+=((uint32)k8[1])<<16; /* fall through */ + case 1 : a+=((uint32)k8[0])<<24; break; + case 0 : return UInt32GetDatum(c); + } + + #endif /* !VALGRIND */ + + } else { /* need to read the key one byte at a time */ + const uint8 *k = (const uint8 *)key; + + /*--------------- all but the last block: affect some 32 bits of (a,b,c) */ + while (length > 12) + { + a += ((uint32)k[0])<<24; + a += ((uint32)k[1])<<16; + a += ((uint32)k[2])<<8; + a += ((uint32)k[3]); + b += ((uint32)k[4])<<24; + b += ((uint32)k[5])<<16; + b += ((uint32)k[6])<<8; + b += ((uint32)k[7]); + c += ((uint32)k[8])<<24; + c += ((uint32)k[9])<<16; + c += ((uint32)k[10])<<8; + c += ((uint32)k[11]); + mix(a,b,c); + length -= 12; + k += 12; + } + + /*-------------------------------- last block: affect all 32 bits of (c) */ + switch(length) /* all the case statements fall through */ + { + case 12: c+=k[11]; + case 11: c+=((uint32)k[10])<<8; + case 10: c+=((uint32)k[9])<<16; + case 9 : c+=((uint32)k[8])<<24; + case 8 : b+=k[7]; + case 7 : b+=((uint32)k[6])<<8; + case 6 : b+=((uint32)k[5])<<16; + case 5 : b+=((uint32)k[4])<<24; + case 4 : a+=k[3]; + case 3 : a+=((uint32)k[2])<<8; + case 2 : a+=((uint32)k[1])<<16; + case 1 : a+=((uint32)k[0])<<24; + break; + case 0 : return UInt32GetDatum(c); + } + } + + final(a,b,c); + return UInt32GetDatum(c); + } + + /*---------- + * hashbig2: return 2 32-bit hash values + * + * This is identical to hashbig(), except it returns two 32-bit hash + * values instead of just one. This is good enough for hash table + * lookup with 2^^64 buckets, or if you want a second hash if you're not + * happy with the first, or if you want a probably-unique 64-bit ID for + * the key. *pc is better mixed than *pb, so use *pc first. If you want + * a 64-bit value do something like "*pc + (((uint64_t)*pb)<<32)". + *---------- + */ + void hashbig2( + const void *key, /* the key to hash */ + size_t length, /* length of the key */ + uint32 *pc, /* IN: primary initval, OUT: primary hash */ + uint32 *pb) /* IN: secondary initval, OUT: secondary hash */ + { + uint32 a,b,c; + union { const void *ptr; size_t i; } u; /* to cast key to (size_t) happily */ + + /* Set up the internal state */ + a = b = c = 0xdeadbeef + ((uint32)length) + *pc; + c += *pb; + + u.ptr = key; + if (HASH_BIG_ENDIAN && ((u.i & 0x3) == 0)) { + const uint32 *k = (const uint32 *)key; /* read 32-bit chunks */ + #ifdef VALGRIND + const uint8 *k8; + #endif + + /*------ all but last block: aligned reads and affect 32 bits of (a,b,c) */ + while (length > 12) + { + a += k[0]; + b += k[1]; + c += k[2]; + mix(a,b,c); + length -= 12; + k += 3; + } + + /*----------------------------- handle the last (probably partial) block */ + /* + * "k[2]<<8" actually reads beyond the end of the string, but + * then shifts out the part it's not allowed to read. Because the + * string is aligned, the illegal read is in the same word as the + * rest of the string. Every machine with memory protection I've seen + * does it on word boundaries, so is OK with this. But VALGRIND will + * still catch it and complain. The masking trick does make the hash + * noticably faster for short strings (like English words). + */ + #ifndef VALGRIND + + switch(length) + { + case 12: c+=k[2]; b+=k[1]; a+=k[0]; break; + case 11: c+=k[2]&0xffffff00; b+=k[1]; a+=k[0]; break; + case 10: c+=k[2]&0xffff0000; b+=k[1]; a+=k[0]; break; + case 9 : c+=k[2]&0xff000000; b+=k[1]; a+=k[0]; break; + case 8 : b+=k[1]; a+=k[0]; break; + case 7 : b+=k[1]&0xffffff00; a+=k[0]; break; + case 6 : b+=k[1]&0xffff0000; a+=k[0]; break; + case 5 : b+=k[1]&0xff000000; a+=k[0]; break; + case 4 : a+=k[0]; break; + case 3 : a+=k[0]&0xffffff00; break; + case 2 : a+=k[0]&0xffff0000; break; + case 1 : a+=k[0]&0xff000000; break; + case 0 : *pc=c; *pb=b; return; /* zero length strings require no mixing */ + } + + #else /* make valgrind happy */ + + k8 = (const uint8 *)k; + switch(length) /* all the case statements fall through */ + { + case 12: c+=k[2]; b+=k[1]; a+=k[0]; break; + case 11: c+=((uint32)k8[10])<<8; /* fall through */ + case 10: c+=((uint32)k8[9])<<16; /* fall through */ + case 9 : c+=((uint32)k8[8])<<24; /* fall through */ + case 8 : b+=k[1]; a+=k[0]; break; + case 7 : b+=((uint32)k8[6])<<8; /* fall through */ + case 6 : b+=((uint32)k8[5])<<16; /* fall through */ + case 5 : b+=((uint32)k8[4])<<24; /* fall through */ + case 4 : a+=k[0]; break; + case 3 : a+=((uint32)k8[2])<<8; /* fall through */ + case 2 : a+=((uint32)k8[1])<<16; /* fall through */ + case 1 : a+=((uint32)k8[0])<<24; break; + case 0 : *pc=c; *pb=b; return; /* zero length strings require no mixing */ + } + + #endif /* !VALGRIND */ + + } else { /* need to read the key one byte at a time */ + const uint8 *k = (const uint8 *)key; + + /*--------------- all but the last block: affect some 32 bits of (a,b,c) */ + while (length > 12) + { + a += ((uint32)k[0])<<24; + a += ((uint32)k[1])<<16; + a += ((uint32)k[2])<<8; + a += ((uint32)k[3]); + b += ((uint32)k[4])<<24; + b += ((uint32)k[5])<<16; + b += ((uint32)k[6])<<8; + b += ((uint32)k[7]); + c += ((uint32)k[8])<<24; + c += ((uint32)k[9])<<16; + c += ((uint32)k[10])<<8; + c += ((uint32)k[11]); + mix(a,b,c); + length -= 12; + k += 12; + } + + /*-------------------------------- last block: affect all 32 bits of (c) */ + switch(length) /* all the case statements fall through */ + { + case 12: c+=k[11]; + case 11: c+=((uint32)k[10])<<8; + case 10: c+=((uint32)k[9])<<16; + case 9 : c+=((uint32)k[8])<<24; + case 8 : b+=k[7]; + case 7 : b+=((uint32)k[6])<<8; + case 6 : b+=((uint32)k[5])<<16; + case 5 : b+=((uint32)k[4])<<24; + case 4 : a+=k[3]; + case 3 : a+=((uint32)k[2])<<8; + case 2 : a+=((uint32)k[1])<<16; + case 1 : a+=((uint32)k[0])<<24; + break; + case 0 : *pc=c; *pb=b; return; /* zero length strings require no mixing */ + } + } + + final(a,b,c); + *pc=c; *pb=b; + } + /* * hash_any() -- hash a variable-length key into a 32-bit value * k : the key (the unaligned variable-length array of bytes) *************** *** 239,298 **** Datum hash_any(register const unsigned char *k, register int keylen) { ! register uint32 a, ! b, ! c, ! len; ! ! /* Set up the internal state */ ! len = keylen; ! a = b = 0x9e3779b9; /* the golden ratio; an arbitrary value */ ! c = 3923095; /* initialize with an arbitrary value */ ! ! /* handle most of the key */ ! while (len >= 12) ! { ! a += (k[0] + ((uint32) k[1] << 8) + ((uint32) k[2] << 16) + ((uint32) k[3] << 24)); ! b += (k[4] + ((uint32) k[5] << 8) + ((uint32) k[6] << 16) + ((uint32) k[7] << 24)); ! c += (k[8] + ((uint32) k[9] << 8) + ((uint32) k[10] << 16) + ((uint32) k[11] << 24)); ! mix(a, b, c); ! k += 12; ! len -= 12; ! } ! ! /* handle the last 11 bytes */ ! c += keylen; ! switch (len) /* all the case statements fall through */ ! { ! case 11: ! c += ((uint32) k[10] << 24); ! case 10: ! c += ((uint32) k[9] << 16); ! case 9: ! c += ((uint32) k[8] << 8); ! /* the first byte of c is reserved for the length */ ! case 8: ! b += ((uint32) k[7] << 24); ! case 7: ! b += ((uint32) k[6] << 16); ! case 6: ! b += ((uint32) k[5] << 8); ! case 5: ! b += k[4]; ! case 4: ! a += ((uint32) k[3] << 24); ! case 3: ! a += ((uint32) k[2] << 16); ! case 2: ! a += ((uint32) k[1] << 8); ! case 1: ! a += k[0]; ! /* case 0: nothing left to add */ ! } ! mix(a, b, c); ! ! /* report the result */ ! return UInt32GetDatum(c); } /* --- 1069,1079 ---- Datum hash_any(register const unsigned char *k, register int keylen) { ! #ifndef WORDS_BIGENDIAN ! return hashlittle(k, keylen, 3923095); ! #else ! return hashbig(k, keylen, 3923095); ! #endif } /* *************** *** 305,316 **** Datum hash_uint32(uint32 k) { ! register uint32 a, ! b, ! c; ! a = 0x9e3779b9 + k; ! b = 0x9e3779b9; c = 3923095 + (uint32) sizeof(uint32); mix(a, b, c); --- 1086,1095 ---- Datum hash_uint32(uint32 k) { ! register uint32 a, b, c; ! a = 0xdeadbeef + k; ! b = 0xdeadbeef; c = 3923095 + (uint32) sizeof(uint32); mix(a, b, c); ---------------------------(end of broadcast)--------------------------- TIP 4: Have you searched our list archives? http://archives.postgresql.org |
|
|
Re: updated hash functions for postgresql v1
by Simon Riggs
::
Rate this Message:
On Sat, 2007-10-27 at 15:15 -0500, Kenneth Marshall wrote:
> Its features include a better and faster hash function. Looks very promising. Do you have any performance test results to show it really is faster, when compiled into Postgres? Better probably needs some definition also; in what way are the hash functions better? -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com ---------------------------(end of broadcast)--------------------------- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to majordomo@... so that your message can get through to the mailing list cleanly |
|
|
Re: updated hash functions for postgresql v1
by Kenneth Marshall-3
::
Rate this Message:
On Sun, Oct 28, 2007 at 05:27:38PM +0000, Simon Riggs wrote:
> On Sat, 2007-10-27 at 15:15 -0500, Kenneth Marshall wrote: > > Its features include a better and faster hash function. > > Looks very promising. Do you have any performance test results to show > it really is faster, when compiled into Postgres? Better probably needs > some definition also; in what way are the hash functions better? > > -- > Simon Riggs > 2ndQuadrant http://www.2ndQuadrant.com > terms of straight CPU time. It uses the same design as the current hash but provides code paths for aligned and unaligned access as well as separate mixing functions for different blocks in the hash run instead of having one general purpose block. I think the speed will not be an obvious win with smaller items, but will be very important when hashing larger items (up to 32kb). Better in this case means that the new hash mixes more thoroughly which results in less collisions and more even bucket distribution. There is also a 64-bit varient which is still faster since it can take advantage of the 64-bit processor instruction set. Ken ---------------------------(end of broadcast)--------------------------- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate |
|
|
Re: updated hash functions for postgresql v1
by Simon Riggs
::
Rate this Message:
On Sun, 2007-10-28 at 13:05 -0500, Kenneth Marshall wrote:
> On Sun, Oct 28, 2007 at 05:27:38PM +0000, Simon Riggs wrote: > > On Sat, 2007-10-27 at 15:15 -0500, Kenneth Marshall wrote: > > > Its features include a better and faster hash function. > > > > Looks very promising. Do you have any performance test results to show > > it really is faster, when compiled into Postgres? Better probably needs > > some definition also; in what way are the hash functions better? > > > > -- > > Simon Riggs > > 2ndQuadrant http://www.2ndQuadrant.com > > > The new hash function is roughly twice as fast as the old function in > terms of straight CPU time. It uses the same design as the current > hash but provides code paths for aligned and unaligned access as well > as separate mixing functions for different blocks in the hash run > instead of having one general purpose block. I think the speed will > not be an obvious win with smaller items, but will be very important > when hashing larger items (up to 32kb). > > Better in this case means that the new hash mixes more thoroughly > which results in less collisions and more even bucket distribution. > There is also a 64-bit varient which is still faster since it can > take advantage of the 64-bit processor instruction set. Ken, I was really looking for some tests that show both of the above were true. We've had some trouble proving the claims of other algorithms before, so I'm less inclined to take those things at face value. I'd suggest tests with Integers, BigInts, UUID, CHAR(20) and CHAR(100). Others may have different concerns. -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com ---------------------------(end of broadcast)--------------------------- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate |
|
|
Re: updated hash functions for postgresql v1
by Luke Lonergan
::
Rate this Message:
We just applied this and saw a 5 percent speedup on a hash aggregation query with four colums in a 'group by' clause run against a single TPC-H table (lineitem). |
|
|
Re: updated hash functions for postgresql v1
by CK Tan
::
Rate this Message:
Hi, this query on TPCH 1G data gets about 5% improvement.
select count (*) from (select l_orderkey, l_partkey, l_comment, count(l_tax) from lineitem group by 1, 2, 3) tmpt; Regards, -cktan On Oct 28, 2007, at 1:17 PM, Luke Lonergan wrote:
|
|
|
Re: updated hash functions for postgresql v1
by Simon Riggs
::
Rate this Message:
On Sun, 2007-10-28 at 13:19 -0700, CK Tan wrote:
> Hi, this query on TPCH 1G data gets about 5% improvement. > select count (*) from (select l_orderkey, l_partkey, l_comment, > count(l_tax) from lineitem group by 1, 2, 3) tmpt; > On Oct 28, 2007, at 1:17 PM, Luke Lonergan wrote: > > > We just applied this and saw a 5 percent speedup on a hash > > aggregation query with four colums in a 'group by' clause run > > against a single TPC-H table (lineitem). > > > > CK - can you post the query? Is this on Postgres or Greenplum? That looks like quite a wide set of columns. Sounds good though. Can we get any more measurements in? -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com ---------------------------(end of broadcast)--------------------------- TIP 6: explain analyze is your friend |
|
|
Re: updated hash functions for postgresql v1
by Luke Lonergan
::
Rate this Message:
That's on Greenplum latest. |
|
|
Re: updated hash functions for postgresql v1
by Kenneth Marshall-3
::
Rate this Message:
On Sun, Oct 28, 2007 at 08:06:58PM +0000, Simon Riggs wrote:
> On Sun, 2007-10-28 at 13:05 -0500, Kenneth Marshall wrote: > > On Sun, Oct 28, 2007 at 05:27:38PM +0000, Simon Riggs wrote: > > > On Sat, 2007-10-27 at 15:15 -0500, Kenneth Marshall wrote: > > > > Its features include a better and faster hash function. > > > > > > Looks very promising. Do you have any performance test results to show > > > it really is faster, when compiled into Postgres? Better probably needs > > > some definition also; in what way are the hash functions better? > > > > > > -- > > > Simon Riggs > > > 2ndQuadrant http://www.2ndQuadrant.com > > > > > The new hash function is roughly twice as fast as the old function in > > terms of straight CPU time. It uses the same design as the current > > hash but provides code paths for aligned and unaligned access as well > > as separate mixing functions for different blocks in the hash run > > instead of having one general purpose block. I think the speed will > > not be an obvious win with smaller items, but will be very important > > when hashing larger items (up to 32kb). > > > > Better in this case means that the new hash mixes more thoroughly > > which results in less collisions and more even bucket distribution. > > There is also a 64-bit varient which is still faster since it can > > take advantage of the 64-bit processor instruction set. > > Ken, I was really looking for some tests that show both of the above > were true. We've had some trouble proving the claims of other algorithms > before, so I'm less inclined to take those things at face value. > > I'd suggest tests with Integers, BigInts, UUID, CHAR(20) and CHAR(100). > Others may have different concerns. > Simon, I agree, that we should not take claims withoug testing them ourselves. My main motivation for posting the patch was to get feedback on how to add support for 64-bit hashes that will work with all of our supported platforms. I am trying to avoid the "work on a feature in isolation... and submit a giant patch with many problems" problem. I intend to do more extensive testing, but I am trying to reach a basic implementation level before I start the testing. I am pretty good with theory, but my coding skills are out of practice. It will take me longer to generate the tests now and without any clear benefit to the hash index implementation. I am willing to test further, but I would like to have my testing benefit the hash index implementation and not just the effectiveness and efficiency of the hashing algorithm. Regards, Ken > -- > Simon Riggs > 2ndQuadrant http://www.2ndQuadrant.com > > ---------------------------(end of broadcast)--------------------------- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq |
|
|
Re: updated hash functions for postgresql v1
by bob_jenkins
::
Rate this Message:
On Oct 28, 11:05 am, k...@... (Kenneth Marshall) wrote:
> On Sun, Oct 28, 2007 at 05:27:38PM +0000, Simon Riggs wrote: > > On Sat, 2007-10-27 at 15:15 -0500, Kenneth Marshall wrote: > > > Its features include a better and faster hash function. > > > Looks very promising. Do you have any performance test results to show > > it really is faster, when compiled into Postgres? Better probably needs > > some definition also; in what way are the hash functions better? > > > -- > > Simon Riggs > > 2ndQuadrant http://www.2ndQuadrant.com > > The new hash function is roughly twice as fast as the old function in > terms of straight CPU time. It uses the same design as the current > hash but provides code paths for aligned and unaligned access as well > as separate mixing functions for different blocks in the hash run > instead of having one general purpose block. I think the speed will > not be an obvious win with smaller items, but will be very important > when hashing larger items (up to 32kb). > > Better in this case means that the new hash mixes more thoroughly > which results in less collisions and more even bucket distribution. > There is also a 64-bit varient which is still faster since it can > take advantage of the 64-bit processor instruction set. > > Ken > > ---------------------------(end of broadcast)--------------------------- > TIP 7: You can help support the PostgreSQL project by donating at > > http://www.postgresql.org/about/donate I don't make use of 64-bit arithmetic when producing the 64-bit result in hashlittle2(). Wish I did. The routine internally produces 3 32- bit results a b c, the returned 64-bit result is roughly c | (b<<32). ---------------------------(end of broadcast)--------------------------- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to majordomo@... so that your message can get through to the mailing list cleanly |
|
|
hashlittle(), hashbig(), hashword() and endianness
by Alex Vinokur
::
Rate this Message:
On Oct 27, 10:15 pm, k...@... (Kenneth Marshall) wrote:
> Dear PostgreSQL Developers, > > This patch is a "diff -c" against the hashfunc.c from postgresql-8.3beta1. > It implements the 2006 version of the hash function by Bob Jenkins. Its > features include a better and faster hash function. I have included the > versions supporting big-endian and little-endian machines that will be > selected based on the machine configuration. [snip] I have some question concerning Bob Jenkins' functions hashword(uint32_t*, size_t), hashlittle(uint8_t*, size_t) and hashbig(uint8_t*, size_t) in lookup3.c. Let k1 by a key: uint8_t* k1; strlen(k1)%sizeof(uint32_t) == 0. 1. hashlittle(k1) produces the same value on Little-Endian and Big- Endian machines. Let hashlittle(k1) be == L1. 2. hashbig(k1) produces the same value on Little-Endian and Big-Endian machines. Let hashbig(k1) be == B1. L1 != B1 3. hashword((uint32_t*)k1) produces * L1 on LittleEndian machine and * B1 on BigEndian machine. --------------------- The question is: is it possible to change hashword() to get * L1 on Little-Endian machine and * B1 on Big-Endian machine ? Thanks. Alex Vinokur email: alex DOT vinokur AT gmail DOT com http://mathforum.org/library/view/10978.html http://sourceforge.net/users/alexvn ---------------------------(end of broadcast)--------------------------- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq |
|
|
Re: hashlittle(), hashbig(), hashword() and endianness
by Alex Vinokur
::
Rate this Message:
On Nov 15, 10:40 am, Alex Vinokur <ale...@...>
wrote: [snip] > I have some question concerning Bob Jenkins' functions > hashword(uint32_t*, size_t), hashlittle(uint8_t*, size_t) and > hashbig(uint8_t*, size_t) in lookup3.c. > > Let k1 by a key: uint8_t* k1; strlen(k1)%sizeof(uint32_t) == 0. > > 1. hashlittle(k1) produces the same value on Little-Endian and Big- > Endian machines. > Let hashlittle(k1) be == L1. > > 2. hashbig(k1) produces the same value on Little-Endian and Big-Endian > machines. > Let hashbig(k1) be == B1. > > L1 != B1 > > 3. hashword((uint32_t*)k1) produces > * L1 on LittleEndian machine and > * B1 on BigEndian machine. > > --------------------- > The question is: is it possible to change hashword() to get > * L1 on Little-Endian machine and > * B1 on Big-Endian machine > ? Sorry, it should be as follows: Is it possible to create two new hash functions on basis of hashword(): i) hashword_little () that produces L1 on Little-Endian and Big- Endian machines; ii) hashword_big () that produces B1 on Little-Endian and Big- Endian machines ? ==================================== Thanks. Alex Vinokur email: alex DOT vinokur AT gmail DOT com http://mathforum.org/library/view/10978.html http://sourceforge.net/users/alexvn ---------------------------(end of broadcast)--------------------------- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate |
|
|
Re: hashlittle(), hashbig(), hashword() and endianness
by Heikki Linnakangas-2
::
Rate this Message:
Alex Vinokur wrote:
> On Nov 15, 10:40 am, Alex Vinokur <ale...@...> > wrote: > [snip] >> I have some question concerning Bob Jenkins' functions >> hashword(uint32_t*, size_t), hashlittle(uint8_t*, size_t) and >> hashbig(uint8_t*, size_t) in lookup3.c. >> >> Let k1 by a key: uint8_t* k1; strlen(k1)%sizeof(uint32_t) == 0. >> >> 1. hashlittle(k1) produces the same value on Little-Endian and Big- >> Endian machines. >> Let hashlittle(k1) be == L1. >> >> 2. hashbig(k1) produces the same value on Little-Endian and Big-Endian >> machines. >> Let hashbig(k1) be == B1. >> >> L1 != B1 >> >> 3. hashword((uint32_t*)k1) produces >> * L1 on LittleEndian machine and >> * B1 on BigEndian machine. >> > =================================== >> --------------------- >> The question is: is it possible to change hashword() to get >> * L1 on Little-Endian machine and >> * B1 on Big-Endian machine >> ? > > Sorry, it should be as follows: > > Is it possible to create two new hash functions on basis of > hashword(): > i) hashword_little () that produces L1 on Little-Endian and Big- > Endian machines; > ii) hashword_big () that produces B1 on Little-Endian and Big- > Endian machines > ? Why? -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com ---------------------------(end of broadcast)--------------------------- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq |
|
|
Re: hashlittle(), hashbig(), hashword() and endianness
by Alex Vinokur
::
Rate this Message:
On Nov 15, 1:23 pm, hei...@... (Heikki Linnakangas)
wrote: > Alex Vinokurwrote: > > On Nov 15, 10:40 am,Alex Vinokur<ale...@...> > > wrote: > > [snip] > >> I have some question concerning Bob Jenkins' functions > >> hashword(uint32_t*, size_t), hashlittle(uint8_t*, size_t) and > >> hashbig(uint8_t*, size_t) in lookup3.c. > > >> Let k1 by a key: uint8_t* k1; strlen(k1)%sizeof(uint32_t) == 0. > > >> 1. hashlittle(k1) produces the same value on Little-Endian and Big- > >> Endian machines. > >> Let hashlittle(k1) be == L1. > > >> 2. hashbig(k1) produces the same value on Little-Endian and Big-Endian > >> machines. > >> Let hashbig(k1) be == B1. > > >> L1 != B1 > > >> 3. hashword((uint32_t*)k1) produces > >> * L1 on LittleEndian machine and > >> * B1 on BigEndian machine. > > > =================================== > >> --------------------- > >> The question is: is it possible to change hashword() to get > >> * L1 on Little-Endian machine and > >> * B1 on Big-Endian machine > >> ? > > > Sorry, it should be as follows: > > > Is it possible to create two new hash functions on basis of > > hashword(): > > i) hashword_little () that produces L1 on Little-Endian and Big- > > Endian machines; > > ii) hashword_big () that produces B1 on Little-Endian and Big- > > Endian machines > > ? > > Why? > Suppose: uint8_t chBuf[SIZE32 * 4]; // ((size_t)&chBuf[0] & 3) == 0 Function hashlittle(chBuf, SIZE32 * 4, 0) produces the same hashValue (let this value be L1) on little-endian and big-endian machines. So, hashlittle() is endianness-indepent. On other hand, function hashword ((uint32_t)chBuf, SIZE32, 0) produces hashValue == L1 on little-endian machine and hashValue != L1 on big-endian machine. So, hashword() is endianness-dependent. I would like to use both hashlittle() and hashword() (or hashword_little) on little-endian and big-endian machine and to get identical hashValues. Alex Vinokur email: alex DOT vinokur AT gmail DOT com http://mathforum.org/library/view/10978.html http://sourceforge.net/users/alexvn ---------------------------(end of broadcast)--------------------------- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to majordomo@... so that your message can get through to the mailing list cleanly |
|
|
Re: hashlittle(), hashbig(), hashword() and endianness
by Kenneth Marshall-3
::
Rate this Message:
On Fri, Nov 16, 2007 at 01:19:13AM -0800, Alex Vinokur wrote:
> On Nov 15, 1:23 pm, hei...@... (Heikki Linnakangas) > wrote: > > Alex Vinokurwrote: > > > On Nov 15, 10:40 am,Alex Vinokur<ale...@...> > > > wrote: > > > [snip] > > >> I have some question concerning Bob Jenkins' functions > > >> hashword(uint32_t*, size_t), hashlittle(uint8_t*, size_t) and > > >> hashbig(uint8_t*, size_t) in lookup3.c. > > > > >> Let k1 by a key: uint8_t* k1; strlen(k1)%sizeof(uint32_t) == 0. > > > > >> 1. hashlittle(k1) produces the same value on Little-Endian and Big- > > >> Endian machines. > > >> Let hashlittle(k1) be == L1. > > > > >> 2. hashbig(k1) produces the same value on Little-Endian and Big-Endian > > >> machines. > > >> Let hashbig(k1) be == B1. > > > > >> L1 != B1 > > > > >> 3. hashword((uint32_t*)k1) produces > > >> * L1 on LittleEndian machine and > > >> * B1 on BigEndian machine. > > > > > =================================== > > >> --------------------- > > >> The question is: is it possible to change hashword() to get > > >> * L1 on Little-Endian machine and > > >> * B1 on Big-Endian machine > > >> ? > > > > > Sorry, it should be as follows: > > > > > Is it possible to create two new hash functions on basis of > > > hashword(): > > > i) hashword_little () that produces L1 on Little-Endian and Big- > > > Endian machines; > > > ii) hashword_big () that produces B1 on Little-Endian and Big- > > > Endian machines > > > ? > > > > Why? > > > [snip] > > Suppose: > uint8_t chBuf[SIZE32 * 4]; // ((size_t)&chBuf[0] & 3) == 0 > > Function > hashlittle(chBuf, SIZE32 * 4, 0) > produces the same hashValue (let this value be L1) on little-endian > and big-endian machines. So, hashlittle() is endianness-indepent. > > On other hand, function > hashword ((uint32_t)chBuf, SIZE32, 0) > produces hashValue == L1 on little-endian machine and hashValue != L1 > on big-endian machine. So, hashword() is endianness-dependent. > > I would like to use both hashlittle() and hashword() (or > hashword_little) on little-endian and big-endian machine and to get > identical hashValues. > > > Alex Vinokur > email: alex DOT vinokur AT gmail DOT com > http://mathforum.org/library/view/10978.html > http://sourceforge.net/users/alexvn > > As I suspected, you want a hash function that is independent of the machine endian-ness. You will need to design, develop, and test such a function yourself. As you start to look at how overflow, rot's, and shifts are handled at the boundaries you may find it difficult to get a fast hash function with those properties. Good luck. Regards, Ken ---------------------------(end of broadcast)--------------------------- TIP 2: Don't 'kill -9' the postmaster |
|
|
Re: hashlittle(), hashbig(), hashword() and endianness
by Marko Kreen-2
::
Rate this Message:
On 11/16/07, Alex Vinokur <alexvn@...> wrote:
> I would like to use both hashlittle() and hashword() (or > hashword_little) on little-endian and big-endian machine and to get > identical hashValues. Whats wrong with hashlittle()? It does use the same optimized reading on LE platform that hashword() does. Or you could wrap the read values with some int2le() macro that is NOP on LE cpu. Although I suspect the performance wont be better than using hashlittle() directly. -- marko ---------------------------(end of broadcast)--------------------------- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to majordomo@... so that your message can get through to the mailing list cleanly |
|
|
Re: updated hash functions for postgresql v1
by Tom Lane-2
::
Rate this Message:
Kenneth Marshall <ktm@...> writes:
> Dear PostgreSQL Developers, > This patch is a "diff -c" against the hashfunc.c from postgresql-8.3beta1. It's pretty obvious that this patch hasn't even been tested on a big-endian machine: > + #ifndef WORS_BIGENDIAN However, why do we need two code paths anyway? I don't think there's any requirement for the hash values to come out the same on little- and big-endian machines. In common cases the byte-array data being presented to the hash function would be different to start with, so you could hardly expect identical hash results even if you had separate code paths. I don't find anything very compelling about 64-bit hashing, either. We couldn't move to that without breaking API for hash functions of user-defined types. Given all the other problems with hash indexes, the issue of whether it's useful to have more than 2^32 hash buckets seems very far off indeed. regards, tom lane -- Sent via pgsql-patches mailing list (pgsql-patches@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-patches |
|
|
Re: updated hash functions for postgresql v1
by Kenneth Marshall-3
::
Rate this Message:
On Sun, Mar 16, 2008 at 10:53:02PM -0400, Tom Lane wrote:
> Kenneth Marshall <ktm@...> writes: > > Dear PostgreSQL Developers, > > This patch is a "diff -c" against the hashfunc.c from postgresql-8.3beta1. > > It's pretty obvious that this patch hasn't even been tested on a > big-endian machine: > > > + #ifndef WORS_BIGENDIAN > > However, why do we need two code paths anyway? I don't think there's > any requirement for the hash values to come out the same on little- > and big-endian machines. In common cases the byte-array data being > presented to the hash function would be different to start with, so > you could hardly expect identical hash results even if you had separate > code paths. > > I don't find anything very compelling about 64-bit hashing, either. > We couldn't move to that without breaking API for hash functions > of user-defined types. Given all the other problems with hash > indexes, the issue of whether it's useful to have more than 2^32 > hash buckets seems very far off indeed. > > regards, tom lane > Yes, there is that typo but it has, in fact, been tested on big and little-endian machines. Since, it was a simple update to replace the current hash function used by PostgreSQL with the new version from Bob Jenkins. The test for the endian-ness of the system allows for the code paths to be optimized for the particular CPU. The 64-bit hashing was included for use during my work on on the hash index. Part of that will entail testing the performance of various permutations of previously submitted suggestions. Regards, Ken Marshall -- Sent via pgsql-patches mailing list (pgsql-patches@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-patches |
|
|
Re: updated hash functions for postgresql v1
by Tom Lane-2
::
Rate this Message:
Simon Riggs <simon@...> writes:
> On Sun, 2007-10-28 at 13:05 -0500, Kenneth Marshall wrote: >> The new hash function is roughly twice as fast as the old function in >> terms of straight CPU time. It uses the same design as the current >> hash but provides code paths for aligned and unaligned access as well >> as separate mixing functions for different blocks in the hash run >> instead of having one general purpose block. I think the speed will >> not be an obvious win with smaller items, but will be very important >> when hashing larger items (up to 32kb). >> >> Better in this case means that the new hash mixes more thoroughly >> which results in less collisions and more even bucket distribution. >> There is also a 64-bit varient which is still faster since it can >> take advantage of the 64-bit processor instruction set. > Ken, I was really looking for some tests that show both of the above > were true. We've had some trouble proving the claims of other algorithms > before, so I'm less inclined to take those things at face value. I spent some time today looking at this code more closely and running some simple speed tests. It is faster than what we have, although 2X is the upper limit of the speedups I saw on four different machines. There are several things going on in comparison to our existing hash_any: * If the source data is word-aligned, the new code fetches it a word at a time instead of a byte at a time; that is a += (k[0] + ((uint32) k[1] << 8) + ((uint32) k[2] << 16) + ((uint32) k[3] << 24)); b += (k[4] + ((uint32) k[5] << 8) + ((uint32) k[6] << 16) + ((uint32) k[7] << 24)); c += (k[8] + ((uint32) k[9] << 8) + ((uint32) k[10] << 16) + ((uint32) k[11] << 24)); becomes a += k[0]; b += k[1]; c += k[2]; where k is now pointer to uint32 instead of uchar. This accounts for most of the speed improvement. However, the results now vary between big-endian and little-endian machines. That's fine for PG's purposes. But it means that we need two sets of code for the unaligned-input code path, since it clearly won't do for the same bytestring to get two different hashes depending on whether it happens to be presented aligned or not. The presented patch actually offers *four* code paths, so that you can compute either little-endian-ish or big-endian-ish hashes on either type of machine. That's nothing but bloat for our purposes, and should be reduced to the minimum. * Given a word-aligned source pointer and a length that isn't a multiple of 4, the new code fetches the last partial word as a full word fetch and masks it off, as per the code comment: * "k[2]&0xffffff" actually reads beyond the end of the string, but * then masks off the part it's not allowed to read. Because the * string is aligned, the masked-off tail is in the same word as the * rest of the string. Every machine with memory protection I've seen * does it on word boundaries, so is OK with this. But VALGRIND will * still catch it and complain. The masking trick does make the hash * noticably faster for short strings (like English words). This I think is well beyond the bounds of sanity, especially since we have no configure support for setting #ifdef VALGRIND. I'd lose the "non valgrind clean" paths (which again are contributing to the patch's impression of bloat/redundancy). * Independently of the above changes, the actual hash calculation (the mix() and final() macros) has been changed. Ken claims that this made the hash "better", but I'm deeply suspicious of that. The comments in the code make it look like Jenkins actually sacrificed hash quality in order to get a little more speed. I don't think we should adopt those changes unless some actual evidence is presented that the hash is better and not merely faster. In short: I think we should adopt the changes to use aligned word fetches where possible, but not adopt the mix/final changes unless more evidence is presented. Lastly, the patch adds yet more code to provide the option of computing a 64-bit hash rather than 32. (AFAICS, the claim that this part is optimized for 64-bit machines is mere fantasy. It's simply Yet Another duplicate of the identical code, but it gives you back two of its three words of internal state at the end, instead of only one.) As I said before, this is just bloat for us. I've got zero interest in pursuing 64-bit hashing when we still don't have a hash index implementation that anyone would consider using in anger. Let's see if we can make the cake edible before worrying about putting a better grade of icing on it. regards, tom lane -- Sent via pgsql-patches mailing list (pgsql-patches@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-patches |
|
|
Re: updated hash functions for postgresql v1
by Kenneth Marshall-3
::
Rate this Message:
On Sat, Apr 05, 2008 at 03:40:35PM -0400, Tom Lane wrote:
> Simon Riggs <simon@...> writes: > > On Sun, 2007-10-28 at 13:05 -0500, Kenneth Marshall wrote: > >> The new hash function is roughly twice as fast as the old function in > >> terms of straight CPU time. It uses the same design as the current > >> hash but provides code paths for aligned and unaligned access as well > >> as separate mixing functions for different blocks in the hash run > >> instead of having one general purpose block. I think the speed will > >> not be an obvious win with smaller items, but will be very important > >> when hashing larger items (up to 32kb). > >> > >> Better in this case means that the new hash mixes more thoroughly > >> which results in less collisions and more even bucket distribution. > >> There is also a 64-bit varient which is still faster since it can > >> take advantage of the 64-bit processor instruction set. > > > Ken, I was really looking for some tests that show both of the above > > were true. We've had some trouble proving the claims of other algorithms > > before, so I'm less inclined to take those things at face value. > > I spent some time today looking at this code more closely and running > some simple speed tests. It is faster than what we have, although 2X > is the upper limit of the speedups I saw on four different machines. > There are several things going on in comparison to our existing > hash_any: > > * If the source data is word-aligned, the new code fetches it a word at > a time instead of a byte at a time; that is > > a += (k[0] + ((uint32) k[1] << 8) + ((uint32) k[2] << 16) + ((uint32) k[3] << 24)); > b += (k[4] + ((uint32) k[5] << 8) + ((uint32) k[6] << 16) + ((uint32) k[7] << 24)); > c += (k[8] + ((uint32) k[9] << 8) + ((uint32) k[10] << 16) + ((uint32) k[11] << 24)); > > becomes > > a += k[0]; > b += k[1]; > c += k[2]; > > where k is now pointer to uint32 instead of uchar. This accounts for > most of the speed improvement. However, the results now vary between > big-endian and little-endian machines. That's fine for PG's purposes. > But it means that we need two sets of code for the unaligned-input code > path, since it clearly won't do for the same bytestring to get two > different hashes depending on whether it happens to be presented aligned > or not. The presented patch actually offers *four* code paths, so that > you can compute either little-endian-ish or big-endian-ish hashes on > either type of machine. That's nothing but bloat for our purposes, and > should be reduced to the minimum. > I agree that a good portion of the speed up is due to the full word processing. The original code from Bob Jenkins had all of these code paths and I just dropped them in with a minimum of changes. > * Given a word-aligned source pointer and a length that isn't a multiple > of 4, the new code fetches the last partial word as a full word fetch > and masks it off, as per the code comment: > > * "k[2]&0xffffff" actually reads beyond the end of the string, but > * then masks off the part it's not allowed to read. Because the > * string is aligned, the masked-off tail is in the same word as the > * rest of the string. Every machine with memory protection I've seen > * does it on word boundaries, so is OK with this. But VALGRIND will > * still catch it and complain. The masking trick does make the hash > * noticably faster for short strings (like English words). > > This I think is well beyond the bounds of sanity, especially since we > have no configure support for setting #ifdef VALGRIND. I'd lose the > "non valgrind clean" paths (which again are contributing to the patch's > impression of bloat/redundancy). > Okay, I will strip the VALGRIND paths. I did not see a real need for them either. > * Independently of the above changes, the actual hash calculation > (the mix() and final() macros) has been changed. Ken claims that > this made the hash "better", but I'm deeply suspicious of that. > The comments in the code make it look like Jenkins actually sacrificed > hash quality in order to get a little more speed. I don't think we > should adopt those changes unless some actual evidence is presented > that the hash is better and not merely faster. > I was repeating the claims made by the functions author after his own testing. His analysis and tests were reasonable, but I do agree that we need some testing of our own. I will start pulling some test cases together like what was discussed earlier with Simon. > > In short: I think we should adopt the changes to use aligned word > fetches where possible, but not adopt the mix/final changes unless > more evidence is presented. > Okay, I agree and will work on producing evidence either way. > Lastly, the patch adds yet more code to provide the option of computing > a 64-bit hash rather than 32. (AFAICS, the claim that this part is > optimized for 64-bit machines is mere fantasy. It's simply Yet Another > duplicate of the identical code, but it gives you back two of its three > words of internal state at the end, instead of only one.) As I said > before, this is just bloat for us. I've got zero interest in pursuing > 64-bit hashing when we still don't have a hash index implementation that > anyone would consider using in anger. Let's see if we can make the cake > edible before worrying about putting a better grade of icing on it. > by the author. He sent in a correction to the mailing list, personally. Regards, Ken Marshall -- Sent via pgsql-patches mailing list (pgsql-patches@...) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-patches |
| < Prev | 1 - 2 | Next > |
| Free embeddable forum powered by Nabble | Forum Help |
