On 05/22/2014 08:40 AM, Simo Sorce wrote:
On Wed, 2014-05-21 at 15:31 -0400, Dmitri Pal wrote:
> On 05/19/2014 01:36 PM, Nikolai Kondrashov wrote:
>> On 05/19/2014 08:28 PM, Jakub Hrozek wrote:
>>> On Mon, May 19, 2014 at 05:55:45PM +0300, Nikolai Kondrashov wrote:
>>>>> "ramdisk" is 58x faster then HDD.
>>>>> "ramdisk" is 26x faster then HDD.
>>>> Out of curiosity: why is it so much faster? Does the code sync to
>>>> disc often?
>>> Every sysdb_transaction_commit or ldb_transaction_commit if we rely on
>>> ldb's autotransaction is several fsyncs (4 I think). That's the
reason
>>> we try to collect all the data first in the daemon and write larger
>>> chunks of data in a single transaction.
>> Ah, as I suspected, thanks!
>> I should have probably just run the tests under strace, instead of
>> asking.
> I wonder if tdb is really what we should use in the long run.
> May be there is a way to shard it internally so that you do not need to
> lock the whole database but only a portion of it?
tdb can lock per record, but that is not exposed directly via ldb nor
sysdb, as it would be really difficult to determine ahead of time what
is the record set to lock for complex transactions.
Changing databases will not change this problem. However ldb has some
performance issues, some structures internally use even O(n^3)
algorithms to deal with data, and also memory allocation could be
substantially improved, it's just a matter of dedicating someone to deal
with performance. The fsyncs do matter to some degree, but with today's
SSDs I don't think it is a huge issue in production, and for tests a
ramdisk can be used.
Simo.
I think people who moved to SSSD do not like that it is significantly
slower than older solutions.
Based on the comments on conferences and on the list I get the feeling
that it is accepted as inevitable evil as a price for other benefits but
this does not mean that people are happy.
If we see a way of getting a significant performance improvement I am
all for it and I would consider it in planning. I just have not seen a
reasonable alternative to what we so far have. If you say that redoing
ldb will give us a say 50% boost I will find time for us to make the change.
--
Thank you,
Dmitri Pal
Sr. Engineering Manager IdM portfolio
Red Hat, Inc.