[PATCH] D132455: [ADT] add ConcurrentHashtable class.

[Alexey Lapshin via Phabricator via llvm-commits]

avl added a comment.

In D132455#4019525 <https://reviews.llvm.org/D132455#4019525>, @dexonsmith wrote:

> In D132455#4018472 <https://reviews.llvm.org/D132455#4018472>, @dblaikie wrote:
>
>> @dexonsmith & co working on the CAS have also proposed a thread safe hash table of sorts ( https://reviews.llvm.org/D133715 )- it's a bit more esoteric/specialized, but I wonder if the use cases overlap enough to be able to unify them?
>
> I won’t have time to take a look myself for a couple of weeks, but adding other interested parties.
>
> Certainly sounds like there’s crossover! The data structure in the other patch supports concurrent insertion and look-up and uses atomics rather than locks. It does not support iteration, although that could be implemented. It does not directly support arbitrary keys, but could be used to implement a more general map; the client is expected to do hashing and decide what to do with collisions. Likewise, it does not support erase, but the client could use a tombstone. Not sure if your use case requires those operations, or if the overhead would be worth it.

This hashtable(D132455 <https://reviews.llvm.org/D132455>) is implemented for https://reviews.llvm.org/D96035 patch.
The main requirement is to have a possibility to store aggregate key/data pairs
(i.e. key is separated from the data) in parallel. Another requirement is to have information
whether data inserted immidiately or by previous call. It should use memory pool
(like BumpPtrAllocator).

So far, I created a table comparing patches:

  ------------------------------------------------------------------------
                      |   HashMappedTrie     |    ConcurrentHashtable    |
  ------------------------------------------------------------------------
      thread-safe     |         yes          |           yes             |
  ------------------------------------------------------------------------
   range of resizings |     not limited      |          x2^32            |
                      |                      |     can be increased      |
  ------------------------------------------------------------------------
      key/data pairs  |         no           |           yes             |
  ------------------------------------------------------------------------
       lock-free?     |         yes          |           no              |
                      |                      | uses mutexes for locking  |
  ------------------------------------------------------------------------
      insertions      |         yes          |           yes             |
  ------------------------------------------------------------------------
        lookups       |         yes          |           no              |
                      |                      |   can be easily added     |
  ------------------------------------------------------------------------
       deletions      |          no          |           no              |
                      |                      |   can be easily added     |
  ------------------------------------------------------------------------
       iterations     |          no          |           no              |
                      |  can be easily added |  an ineffective non-thread|
                      |                      |safe solution could be done|
  ------------------------------------------------------------------------
    hash collisions   |          yes         |      no collisions        |
                      |   should be handled  |                           |
                      |      by client       |                           |                    
  ------------------------------------------------------------------------

I did some first-look performance comparisons of the patches(using
this utility https://reviews.llvm.org/D132548).
The numbers might be inaccurate if I used HashMappedTrie incorrectly.
There is a difference - I did not set any initial sizes for HashMappedTrie
while initial sizes for ConcurrentHashtable were set. Also, only the key
was stored for HashMappedTrie while the key and data pair were stored for 
ConcurrentHashtable. All runs insert 100000000 strings converted from
the corresponding integer.

  ------------------------------------------------------------------------------
                            |   HashMappedTrie     |    ConcurrentHashtable    |
  ------------------------------------------------------------------------------
   --num-threads 1          | time:       62sec    | time:           30sec     |                    
   --initial-size 100000000 | memory:     13.2G    | memory:         16.1G     |                    
  ------------------------------------------------------------------------------
   --num-threads 1          | time:       62sec    | time:           34sec     |                    
   --initial-size       100 | memory:     13.2G    | memory:         18.1G     |                    
  ------------------------------------------------------------------------------
   --num-threads 16         | time:       38sec    | time:          3.5sec     |                    
   --initial-size 100000000 | memory:     13.2G    | memory:         16.1G     |                    
  ------------------------------------------------------------------------------
   --num-threads 16         | time:       38sec    | time:          7.3sec     |                    
   --initial-size       100 | memory:     13.2G    | memory:         18.1G     |                    
  ------------------------------------------------------------------------------


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D132455/new/

https://reviews.llvm.org/D132455