Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
fix algorithm of spreading vectors over shards (#3374)
Summary: simple math: | **input n** | **input nshards** | shard_size | idx | i0 | ni | | -- |-- |-- |-- |-- |-- | | 19 | 6 | 4 | 5 | 20 | **-1** | | 1000 | 37 | 28 | 36 | 1008 | -8 | | 1000 | 64 | 16 | 63 | 1008 | -8 | root cause: integer cause precision loss, `idx * shard_size` overflows, because `(n + nshards - 1) / nshards` is roundup my solution: each shard takes at least `base_shard_size = n / nshards`, then `remain = n % nshards`, we know `0 <= remain < nshards`, next, assign those remain vectors to first `remain` shards, i.e. first `remain` shards take one more vector each. ```c++ auto i0 = idx * base_shard_size; if (i0 < remain) { // if current idx is one of the first `remain` shards i0 += idx; } else { i0 += remain; } ``` simplify above code: `i0 = idx * base_shard_size + std::min(size_t(idx), n % nshards);` Pull Request resolved: #3374 Reviewed By: fxdawnn Differential Revision: D57867910 Pulled By: junjieqi fbshipit-source-id: 7e72ea5cd197af4f3446fb7a3fd34ad08901dbb2
- Loading branch information