Return-Path: Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 8A398C002D for ; Sat, 20 Aug 2022 14:17:36 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 58D7040993 for ; Sat, 20 Aug 2022 14:17:36 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 58D7040993 Authentication-Results: smtp2.osuosl.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=UQIp95Ot X-Virus-Scanned: amavisd-new at osuosl.org X-Spam-Flag: NO X-Spam-Score: -1.848 X-Spam-Level: X-Spam-Status: No, score=-1.848 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id k2ApQbx07023 for ; Sat, 20 Aug 2022 14:17:35 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 90085401ED Received: from mail-lj1-x22c.google.com (mail-lj1-x22c.google.com [IPv6:2a00:1450:4864:20::22c]) by smtp2.osuosl.org (Postfix) with ESMTPS id 90085401ED for ; Sat, 20 Aug 2022 14:17:34 +0000 (UTC) Received: by mail-lj1-x22c.google.com with SMTP id e2so1992750ljj.1 for ; Sat, 20 Aug 2022 07:17:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:mime-version:from:to:cc; bh=YpjZAnY9kaaAPKmuXrURbnr6GstmHIEwRCjcecxlE1w=; b=UQIp95OtdZzGpdDQSg8MO1BS4YkTTlPM+RddTnDYTjWlRKceDYMGQ7fmWIX6vQU4ET ZkKaVFX4g27wNGRBUaG48K1ltWx+/jZ6EeDBR5tAz8DStwmkgRxas3HquUMmIJyvyONk M94RAf6a0xREPepzxIRWz/qOq7Uxew5y00GuBcQsPetpw+E68HYKpDb3czAzoB5Lv6gl SSNDEC9tB7q6t4AdyfJbs+flBnsDAkoXmeIP644fH77iquvdnUMQUOnU2G8wpm3hLcZi 7iZIQ0ewCJPschaOVZMjBivbhkzfH2FQlwxsm8R9fSjUrh9QhCaipSG9Ghv6sQWFus0U zQqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc; bh=YpjZAnY9kaaAPKmuXrURbnr6GstmHIEwRCjcecxlE1w=; b=zE4G+4joymuSsTOwCrAe0c3feun0G6T8b/8voHU5J3Sl2Q0cCWMQawWr/5bj8nDoBX 7C+g0RhEfX5J3GqbOrkWo25cmtgD1NKcBea0iwxnr5fA3N77iiyv29KJLx3Mg0GHGOpG BmjSIvGcqaSXhfpyoRQHSEQ9qhtDc0faBBJyAaW0H0nSlBYZl2Pz4b2J9L+X1ZSV76WW t+4OMUsuyg5om1XCuSHobNLjQA+03IHE/pKTq6zSsA0BSKz01WUuEj+tu0wB/LQe2+9O e8djy1zec7ecXPSvTl/pxCprBEtKLZlS1+7BY3DJBlCH1DLwcEc/+3WCd7cwhY3M7y7U aLmg== X-Gm-Message-State: ACgBeo1R0qBAgZo1vMsrYvN7AHtm07vkFITIAtfuIIVteAWoPMDt0pIk B8Zc13iTd3Ot5UfZIaFp1LcZx25ws3Vx8WUsibMBHP0nYphhXhnQ1P8= X-Google-Smtp-Source: AA6agR4HVeUk7PPpyCop1OjeMJyOYK9kEh3Htr6kJFLS8Kr+SXunCK/oEcp6ABoL+e58KbuW7V6rmUiu5jfgFJchVYA= X-Received: by 2002:a05:651c:2112:b0:261:b9c1:509 with SMTP id a18-20020a05651c211200b00261b9c10509mr2938860ljq.39.1661005051429; Sat, 20 Aug 2022 07:17:31 -0700 (PDT) MIME-Version: 1.0 From: micaroni@gmail.com Date: Sat, 20 Aug 2022 11:16:53 -0300 Message-ID: To: Bitcoin Protocol Discussion Content-Type: multipart/alternative; boundary="0000000000000f879a05e6ace0a4" X-Mailman-Approved-At: Sat, 20 Aug 2022 14:19:29 +0000 Cc: Bruno Garcia Subject: [bitcoin-dev] Huge wallets make Bitcoin Core unusable (Daemon+CLI & Qt) X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Aug 2022 14:17:36 -0000 --0000000000000f879a05e6ace0a4 Content-Type: text/plain; charset="UTF-8" Hi dear devs, 1. THE ISSUE - DAEMON+CLI ======================== I had a wallet in a server production since 2017 (5 years old) and when it reached about 273 MB, 2.079.337 transactions and 446.503 generated addresses, the performance started to degrade exponentially. Most of the commands, e.g. "getbalance", "walletpassphrase" and "getreceivedbyaddress" started to timeout (more than 15 minutes delay - default timeout). The CPU was 100% used (all 32 cores - with 150 load avg) and the machine became almost unusable breaking everything else, with the default config of 16 RPC threads and 15 min timeout and some attempt calls per mi Increasing the timeout and/or the RPC threads in the config file turns things even worse. Putting the wallet.dat in a very fast SSD disk and increasing the size of the cache (I tried with 8GB) have improved but I'm not sure if it is enough. 2. TEST ON BITCOIN QT ==================== If you try to load the wallet in the "bitcoin-qt" everything gets stuck, even the system (OS/UI) doesn't respond anymore. You click on a button and receive the message "window doesn't respond, wait or terminate?" - if you wait it releases after a while but it is slow and hard to use the wallet anyway. 3. WHY IS THIS SO BAD? ===================== This is bad because the standard client becomes almost useless for the wallet feature: 3.1) the wallet Qt already is not so popular among end users. It doesn't look modern, slow to first sync and hard to use. That's why people prefer to use Electrum or Wasabi - I personally don't care but it's the sad truth; 3.2) it becomes useless now also for servers in production, forcing them to use third party solutions for huge wallets. Even if you split in 10 wallets it will just delay 10 times more each to degrade, postponing the problem but not eliminating it. Not to mention the slow and daily degradation. 4. SHOULD WE GIVE UP THE WALLET FEATURE? ======================================== Then, Bitcoin Core becomes just a reference implementation and blocks relayers, but as an application wallet itself turns into a really bad choice. --- It leads me to the following question: if we won't invest time on improving this, shouldn't we remove the wallet feature at all? Why keep a wallet feature that is not useful for the end user nor the production server? Is it useful for what then? 5. THE CURRENT "SOLUTION" IS BAD =============================== Currently, the only "solution" for huge wallets is shameful: create a new one and send the funds there from time to time. But when is the right time exactly? The performance degrades suddenly or gets worse slowly for each new address and/or tx?. And besides not being an elegant solution and "not in the handbook", it also can break a lot of things like monitoring old addresses and also can lead to privacy concerns unifying lots of inputs in a big and expensive tx. 6. OTHER USER CASES? ==================== I think this could also become an issue if we have LN nodes that use the Bitcoin Core wallet infrastructure behind to open / close many channels for a long time. 7. FINAL THOUGHTS ================= If moving the wallet from a HDD to a SSD improved a lot, maybe just caching the entire wallet in memory could improve even more, but I'm afraid some code optimization is also necessary. 8. SOME QUESTIONS ================== 8.1) Can we "optimize" a huge wallet without moving the funds to a new one? Like a "fsck" or eqv? 8.2) Can we improve the cache usage somehow? Putting the entire wallet in memory, for example? 8.3) Is it possible to reduce the wallet size (273 MB is too much for a HD wallet)? 8.4) Can we tell the CLI to ignore old addresses? What if I need to watch only the last 30 days? 8.5) How to improve the I/O treatment and/or CPU usage in the main thread on Bitcoin-Qt to avoid window freezing on big and huge wallets? 8.6) In the last case (if it was not possible to optimize the wallet or the CLI & Qt), can the CLI just warn the user like: "the wallet is becoming too big and slow, execute the command 'archive'". And then, the command "archive" could rename the current wallet to something like "wallet.dat.archive_until_20220818", create a new "wallet.dat" and move the funds automatically? Also, would it be nice to have an "autoarchivehugewallets=1" in the file config? 9. POSSIBLE RELATED AND TESTS ============================= [1] https://github.com/bitcoin/bitcoin/issues/15015 [2] https://github.com/bitcoin/bitcoin/issues/15148 [3] https://github.com/bitcoin/bitcoin/issues/16874 [4] https://github.com/bitcoin/bitcoin/pull/17135 [5] https://github.com/bitcoin/bitcoin/pull/18160 [6] https://github.com/bitpay/bitcore-node/issues/463 [7] https://github.com/RavenProject/Ravencoin/issues/499 [8] https://github.com/sugarchain-project/sugarchain/issues/106 [9] https://bitcoin.stackexchange.com/questions/111844/loadwallet-takes-too-much-and-times-out [10] https://bitcoin.stackexchange.com/a/45713/1761 ANOTHER POSSIBLE BUG ====================== Even if my node is 100% sync: 2022-08-20T13:11:43Z UpdateTip: new best=00000000000000000005bba0593c2be0f1d322223501591d2b31b544e3af3d0b height=750300 version=0x2fffe000 log2_work=93.687081 tx=758181489 date='2022-08-20T13:11:16Z' progress=1.000000 cache=4.6MiB(34964txo) After a "loadwallet" command I am getting an old / wrong balance. The wallet is already empty because I had moved the funds to a new one 3 or 4 days ago but it is still showing the old balance. I didn't receive any warning message saying the need for a rescan or something like that. I am trying the "rescanblockchain" command but it is running and it taking a looooooooooooooong time. Best regards, Felipe. --0000000000000f879a05e6ace0a4 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi dear devs,

<= br>
1. THE ISSUE - DAEMON+CLI
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
I had a wallet in a server production since 2017 (5 years old) and when it= reached about 273 MB, 2.079.337 transactions and 446.503 generated address= es, the performance started to degrade exponentially.

Most of the com= mands, e.g. "getbalance", "walletpassphrase" and "= getreceivedbyaddress" started to timeout (more than 15 minutes delay -= default timeout). The CPU was 100% used (all 32 cores - with 150 load avg)= and the machine became almost unusable breaking everything else, with the = default config of 16 RPC threads and 15 min timeout and some attempt calls = per mi

Increasing the timeout and/or the RPC threads in the confi= g file turns things even worse.

<= /div>
Putting the wallet.dat in a very fast= SSD disk and increasing the size of the cache (I tried with 8GB) have impr= oved but I'm not sure if it is enough.


2. TEST ON BITCOIN QT
=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
I= f you try to load the wallet in the "bitcoin-qt" everything gets = stuck, even the system (OS/UI) doesn't respond anymore. You click on a = button and receive the message "window doesn't respond, wait or te= rminate?" - if you wait it releases after a while but it is slow and h= ard to use the wallet anyway.
<= br>

3. WHY IS THIS SO BAD?
= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
This is bad because the standard client beco= mes almost useless for the wallet feature:

3.1) the wallet Qt already= is not so popular among end users. It doesn't look modern, slow to fir= st sync and hard to use. That's why people prefer to use Electrum or Wa= sabi - I personally don't care but it's the sad truth;

3.2) i= t becomes useless now also for servers in production, forcing them to use t= hird party solutions for huge wallets. Even if you split in 10 wallets it w= ill just delay 10 times more each to degrade, postponing the problem but no= t eliminating it. Not to mention the slow and daily degradation.


=
4. SHOULD WE GIVE UP THE WALLET FEAT= URE?
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Then, Bitcoin Core becomes just a reference implementatio= n and blocks relayers, but as an application wallet itself turns into a rea= lly bad choice. --- It leads me to the following question: if we won't = invest time on improving this, shouldn't we remove the wallet feature a= t all? Why keep a wallet feature that is not useful for the end user nor th= e production server? Is it useful for what then?


5. THE CURRENT "SOLUTION" IS BAD
= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D
Currently, the onl= y "solution" for huge wallets is shameful: create a new one and s= end the funds there from time to time. But when is the right time exactly? = The performance degrades suddenly or gets worse slowly for each new address= and/or tx?. And besides not being an elegant solution and "not in the= handbook", it also can break a lot of things like monitoring old addr= esses and also can lead to privacy concerns unifying lots of inputs in a bi= g and expensive tx.


6. OTH= ER USER CASES?
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D
I think this could also become an= issue if we have LN nodes that use the Bitcoin Core wallet infrastructure = behind to open / close many channels for a long time.


7. FINAL THOUGHTS
=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
If moving the wallet from a HDD to a= SSD improved a lot, maybe just caching the entire wallet in memory could i= mprove even more, but I'm afraid some code optimization is also necessa= ry.


8. SOME QUESTIONS
<= div class=3D"gmail_default" style=3D"font-family:arial,helvetica,sans-serif= ;font-size:small;color:#000000">=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D
8.1) Can we "op= timize" a huge wallet without moving the funds to a new one? Like a &q= uot;fsck" or eqv?

8.2) Can we improve the cache usage somehow? P= utting the entire wallet in memory, for example?

8.3) Is it possi= ble to reduce the wallet size (273 MB is too much for a HD wallet)?

8.4) Can we tell the CLI to ignore old addresses? What if I need to watc= h only the last 30 days?

8.5) How to improve the I/O treatment and/or= CPU usage in the main thread on Bitcoin-Qt to avoid window freezing on big= and huge wallets?

8.6) In the last case (if it was not possible to o= ptimize the wallet or the CLI & Qt), can the CLI just warn the user lik= e: "the wallet is becoming too big and slow, execute the command '= archive'". And then, the command "archive" could rename = the current wallet to something like "wallet.dat.archive_until_2022081= 8", create a new "wallet.dat" and move the funds automatical= ly? Also, would it be nice to have an "autoarchivehugewallets=3D1"= ; in the file config?


9. P= OSSIBLE RELATED AND TESTS
=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D

[1] h= ttps://github.com/bitcoin/bitcoin/issues/15015
[2] https://github.com/bitcoin/bitcoin/issues/15148
[3] https://github.com/bitcoin/bitcoin/issues/16874
[4] https://github.com/bitcoin/bitcoin/pull/17135
[5] https://github.com/bitcoin/bitcoin/pull/18160
[6] https://github.com/bitpay/bitcore-node/issues/463
[7] https://github.com/RavenProject/Ravencoin/issues/4= 99
[8] https://github.com/sugarchain-project/sugarchain/issues/106<= /div>
[9] https= ://bitcoin.stackexchange.com/questions/111844/loadwallet-takes-too-much-and= -times-out
[10] https://bitcoin.stackexchange.com= /a/45713/1761


ANOTHER = POSSIBLE BUG
=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

Even if my node is 100% sy= nc:
2022-08-20T13:11:43Z UpdateTip: n= ew best=3D00000000000000000005bba0593c2be0f1d322223501591d2b31b544e3af3d0b = height=3D750300 version=3D0x2fffe000 log2_work=3D93.687081 tx=3D758181489 d= ate=3D'2022-08-20T13:11:16Z' progress=3D1.000000 cache=3D4.6MiB(349= 64txo)

After a "loadwallet" command I am getting an old / w= rong balance. The wallet is already empty because I had moved the funds to = a new one 3 or 4 days ago but it is still showing the old=C2=A0 balance. I = didn't receive any warning message saying the need for a rescan or some= thing like that.

I am trying the "rescanblockchain" com= mand but it is running and it taking a looooooooooooooong time.

<= br>


Best regards,

F= elipe.
--0000000000000f879a05e6ace0a4--