Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 8221F891 for ; Tue, 19 Dec 2017 07:48:43 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from APC01-HK2-obe.outbound.protection.outlook.com (mail-oln040092255036.outbound.protection.outlook.com [40.92.255.36]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 179F687 for ; Tue, 19 Dec 2017 07:48:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=live.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=aXP4AqRRXFoCVuJKVH9DF/5btqfDE+B+3Urjx1ouGzg=; b=F36gDlS39lwfM496JqJStrErpXWyh66VKYuezM9zRPmjSvomTGAGJiGt+lfgi+9oUajYhyT9C2NhQpRmtZpgey6nbUh0ql6DpK+QzICIdBKmbQQiK6O6PUCPE/mhKBL1u15B4BzE77P0uEQwDwvAnIuTYcI25ypCq9MmjWaDgJnJJ9tbkA9BBthb1oivbPU8AMFTZbOWymbkzQQTx/Wu4ozQUUESAY4CBXKCOMSs+7CACnQlPRZhZcxS138PW452A4PenLuMtXyIUMk8KKqr1+Dt0HRR/24EwlE7XEXkYckksMRNZmCh10MwFt7tSy5OGs0k3xRmV3ueGeRUvpaOng== Received: from HK2APC01FT062.eop-APC01.prod.protection.outlook.com (10.152.248.57) by HK2APC01HT055.eop-APC01.prod.protection.outlook.com (10.152.249.220) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.20.302.6; Tue, 19 Dec 2017 07:48:37 +0000 Received: from PS2P216MB0179.KORP216.PROD.OUTLOOK.COM (10.152.248.58) by HK2APC01FT062.mail.protection.outlook.com (10.152.249.193) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.20.302.6 via Frontend Transport; Tue, 19 Dec 2017 07:48:37 +0000 Received: from PS2P216MB0179.KORP216.PROD.OUTLOOK.COM ([10.171.225.19]) by PS2P216MB0179.KORP216.PROD.OUTLOOK.COM ([10.171.225.19]) with mapi id 15.20.0323.018; Tue, 19 Dec 2017 07:48:37 +0000 From: Damian Williamson To: Chris Riley Thread-Topic: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks Thread-Index: AQHTb5z9LKaNJrfjBkOvllhZtrqBGqNEL2DvgAB3wACAAD9wXoAELFgAgAFHmoQ= Date: Tue, 19 Dec 2017 07:48:37 +0000 Message-ID: References: <5upGmF0IhXUWhcikhdV-e9Pqg-kXfUuXe0kOpGxumie_TO2jLvCgTZ5c6vgBRtaqkL6DmOJb1YftK0osufB5RkhW7YhAhhCI0zBTH3YcORY=@protonmail.com> , In-Reply-To: Accept-Language: en-AU, en-US Content-Language: en-AU X-MS-Has-Attach: X-MS-TNEF-Correlator: x-incomingtopheadermarker: OriginalChecksum:0FC04575AB0D0CA13C20F42802FE00228D1AC7D78E09900430598140F2BB06A0; UpperCasedChecksum:A50C7C4F3261FE1770B33BEC745A0C48726284BF36672ED9D866255A5DA7F516; SizeAsReceived:7641; Count:47 x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [CDd+p1u6QTQtmmJfBPlegzm3NW+nZgE6] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; HK2APC01HT055; 6:0GPa72gnOMdEApk9guZA43aq7wtEvO+Lr8ObjJHTonIhuwoZkMVDyVAf47/arK4N3CiitnPNLIsS380pcE/lx4KnZIUeY/D5Vl+3U84XvtTBKINRSkbQx0AiV6qE8GT72IUGwcanyZJgXKL5lP+3ISvUqdYIN5scUE+i8YKJ89exUBCOzjydkwD86FaxPtDKyLKWFIKe6LVRgr5/Vm5iShA/1RC/O+qc64OGnYzFMR0Mjd2IrZ4k/sLbE7AXaVjTRt7kBJSLZmw1JGFN7vBZDzA1Js0j4zO9muu0ZVyS3NvahvGqTKnopPLqhuR4fJOpj+77osaOPklN5Se7U3Kz0gsryeT/w8I0OtrcRPgPTyw=; 5:4PzqCZCMBwQoXjSOo7z5yFiQ9pCjt6zSRbeY6rxuOsPB9xIhrr9g6nFK0PPaQWYYdHgBNSLxp7VOgJ0fq90onXaPzlWJ01TF7UWcV8mUQD+p5p50JgDlTpZ+I8+1L/fu0C5ZCtrbqtmPW4VZoulCUP3AxmkCIvrD/R+q1qSFNd4=; 24:NU5LJuV7EyIKo6N9uRgANjXs41NWdkHWOj/YT02l/IkRPuOo66boO8gjsFPMTCbqF1vnOLvP2NvxG7td9BHrxW8mtb7xcW32H1VIn+p8rp0=; 7:KuarIZn84zFpqBqflmOf06jgmwHte3A7ZiJkzMqEghRtGoUZpTjBXUrZqhT4nmELBQidcB+oKoGHtgKzm1JuHEsjBbx+Srq8Wmoo7xUgLPIoFfoBIEycQSS0iRxDD2biCDKPZp4K0/wxDQ0OkUZnc1uE9JYufKoBMQqWGvxUa43xQBGVwNJ1hKHF3KhRvLbJwI2/Ii/l1WhN2jtty9KMZFrsEndeNuapV+oLqmtGjGJZICAMtZWlS3F4TXauzaSn x-incomingheadercount: 47 x-eopattributedmessage: 0 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(201702061074)(5061506573)(5061507331)(1603103135)(2017031320274)(2017031324274)(2017031323274)(2017031322404)(1601125374)(1603101448)(1701031045); SRVR:HK2APC01HT055; x-ms-traffictypediagnostic: HK2APC01HT055: x-ms-office365-filtering-correlation-id: 5fb8ff63-6adb-452c-b186-08d546b4e9ee x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(444000031); SRVR:HK2APC01HT055; BCL:0; PCL:0; RULEID:(100000803101)(100110400095); SRVR:HK2APC01HT055; x-forefront-prvs: 052670E5A4 x-forefront-antispam-report: SFV:NSPM; SFS:(7070007)(98901004); DIR:OUT; SFP:1901; SCL:1; SRVR:HK2APC01HT055; H:PS2P216MB0179.KORP216.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: multipart/alternative; boundary="_000_PS2P216MB01798757147723AF5B7B7CA49D0F0PS2P216MB0179KORP_" MIME-Version: 1.0 X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5fb8ff63-6adb-452c-b186-08d546b4e9ee X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Dec 2017 07:48:37.2460 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Internet X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-Transport-CrossTenantHeadersStamped: HK2APC01HT055 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, HTML_MESSAGE, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org X-Mailman-Approved-At: Tue, 19 Dec 2017 12:49:21 +0000 Cc: "bitcoin-dev@lists.linuxfoundation.org" Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Dec 2017 07:48:43 -0000 --_000_PS2P216MB01798757147723AF5B7B7CA49D0F0PS2P216MB0179KORP_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Thank you for your constructive feedback. I now see that the proposal intro= duces a potential issue. It is difficult to define then, what is a valid transaction? Clearly, my de= finition was insufficient. Regards, Damian Williamson ________________________________ From: Chris Riley Sent: Monday, 18 December 2017 11:09 PM To: Damian Williamson; Bitcoin Protocol Discussion Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transactio= n Priority For Ordering Transactions In Blocks Regarding "problem" #2 where you say "How do we ensure that all valid trans= actions are eventually included in the blockchain?": I do not believe that= all people would (a) agree this is a problem or (b) that we do want to *EN= SURE* that *ALL* valid transactions are eventually included in the blockcha= in. There are many *valid* transactions that oftentimes miners do not (and= should not) wish to require be confirmed and included in the blockchain. = Spam transactions for example can be valid, but used to attack bitcoin by u= sing no or low fee. Any valid transaction MAY be included by a miner, but = requiring it in some fashion at this point would open the network to other = attack vectors. Perhaps you meant it a different way. On Fri, Dec 15, 2017 at 3:59 PM, Damian Williamson via bitcoin-dev > wrote: > > There are really two separate problems to solve. > > > How does Bitcoin scale with fixed block size? > How do we ensure that all valid transactions are eventually included in t= he blockchain? > > > Those are the two issues that the proposal attempts to address. It makes = sense to resolve these two problems together. Using the proposed system for= variable block sizes would solve the first problem but there would still b= e a whole bunch of never confirming transactions. I am not sure how to reli= ably solve the second problem at scale without first solving the first. > > > >* Every node has a (potentially) different mempool, you can't use it to = decide consensus values like the max block size. > > > I do not suggest a consensus. Depending on which node solves a block the = value for next block size will be different. The consensus would be that bl= ocks will adhere to the next block size value transmitted with the current = block. It is easy to verify that the consensus is being adhered to once in = place. > > >* Increasing the entropy in a block to make it more unpredictable doesn'= t really make sense. > > Not a necessary function, just an effect of using a probability-based dis= tribution. > > >* Bitcoin should be roughly incentive compatible. Your proposal explicit= s asks miners to ignore their best interests, and confirm transactions by "= priority". What are you going to do if a "malicious" miner decides to go a= fter their profits and order by what makes them the most money. Add "ordere= d by priority" as a consensus requirement? And even if you miners can still= sort their mempool by fee, and then order the top 1MB by priority. > > I entirely agree with your sentiment that Bitcoin must be incentive compa= tible. It is necessary. > > It is in only miners immediate interest to make the most profitable block= from the available transaction pool. As with so many other things, it is n= ecessary to partially ignore short-term gain for long-term benefit. It is i= n miners and everybody's long-term interest to have a reliable transaction = service. A busy transaction service that confirms lots of transactions per = hour will become more profitable as demand increases and more users are pre= pared to pay for priority. As it is there is currently no way to fully scal= e because of the transaction bandwidth limit and that is problematic. If al= l valid transactions must eventually confirm then there must be a way to re= solve that problem. > > Bitcoin deliberately removes traditional scale by ensuring blocks take te= n minutes on average to solve, an ingenious idea and, incentive compatible = but, fixed block sizes leaves us with a problem to solve when we want to sc= ale. > > >If you could find a good solution that would allow you to know if miners= were following your rule or not (and thus ignore it if it doesn't) then yo= u wouldn't even need bitcoin in the first place. > > I am confident that the math to verify blocks based on the proposal can b= e developed (and I think it will not be too complex for a mathematician wit= h the relevant experience), however, I am nowhere near experienced enough w= ith probability and statistical analysis to do it. Yes, if Bitcoin doesn't = then it might make another great opportunity for an altcoin but I am not ev= en nearly interested in promoting any altcoins. > > > If not the proposal that I have put forward, then, hopefully, someone can= come up with a better solution. The important thing is that the issues are= resolved. > > > Regards, > > Damian Williamson > > > > ________________________________ > From: Rhavar > > Sent: Saturday, 16 December 2017 3:38 AM > To: Damian Williamson > Cc: Bitcoin Protocol Discussion > Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transact= ion Priority For Ordering Transactions In Blocks > > > I understand that there would be technical issues to resolve in impleme= ntation, but, are there no fundamental errors? > > Unfortunately your proposal is really fundamentally broken, on a few leve= ls. I think you might need to do a bit more research into how bitcoin works= before coming up with such improvements =3D) > > But just some quick notes: > > * Every node has a (potentially) different mempool, you can't use it to d= ecide consensus values like the max block size. > > * Increasing the entropy in a block to make it more unpredictable doesn't= really make sense. > > * Bitcoin should be roughly incentive compatible. Your proposal explicits= asks miners to ignore their best interests, and confirm transactions by "p= riority". What are you going to do if a "malicious" miner decides to go af= ter their profits and order by what makes them the most money. Add "ordered= by priority" as a consensus requirement? And even if you miners can still = sort their mempool by fee, and then order the top 1MB by priority. > > If you could find a good solution that would allow you to know if miners = were following your rule or not (and thus ignore it if it doesn't) then you= wouldn't even need bitcoin in the first place. > > > > > -Ryan > > > -------- Original Message -------- > Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction = Priority For Ordering Transactions In Blocks > Local Time: December 15, 2017 3:42 AM > UTC Time: December 15, 2017 9:42 AM > From: bitcoin-dev@lists.linuxfoundation.org > To: Bitcoin Protocol Discussion > > > > > I should not take it that the lack of critical feedback to this revised p= roposal is a glowing endorsement. I understand that there would be technica= l issues to resolve in implementation, but, are there no fundamental errors= ? > > I suppose that it if is difficult to determine how long a transaction has= been waiting in the pool then, each node could simply keep track of when a= transaction was first seen. This may have implications for a verify routin= e, however, for example, if a node was offline, how should it differentiate= how long each transaction was waiting in that case? If a node was restarte= d daily would it always think that all transactions had been waiting in the= pool less than one day If each node keeps the current transaction pool in = a file and updates it, as transactions are included in blocks and, as new t= ransactions appear in the pool, then that would go some way to alleviate th= e issue, apart from entirely new nodes. There should be no reason the conte= nts of a transaction pool files cannot be shared without agreement as to th= e transaction pool between nodes, just as nodes transmit new transactions f= reely. > > It has been questioned why miners could not cheat. For the question of ho= w many transactions to include in a block, I say it is a standoff and miner= s will conform to the proposal, not wanting to leave transactions with vali= d fees standing, and, not wanting to shrink the transaction pool. In any ca= se, if miners shrink the transaction pool then I am not immediately concern= ed since it provides a more efficient service. For the question of includin= g transactions according to the proposal, I say if it is possible to keep t= rack of how long transactions are waiting in the pool so that they can be i= ncluded on a probability curve then it is possible to verify that blocks co= nform to the proposal, since the input is a probability, the output should = conform to a probability curve. > > > If someone has the necessary skill, would anyone be willing to develop th= e math necessary for the proposal? > > Regards, > Damian Williamson > > > ________________________________ > > From: bitcoin-dev-bounces@lists.linuxfoundation.org > on behalf of Da= mian Williamson via bitcoin-dev > > Sent: Friday, 8 December 2017 8:01 AM > To: bitcoin-dev@lists.linuxfoundation.org > Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction = Priority For Ordering Transactions In Blocks > > > > Good afternoon, > > The need for this proposal: > > We all must learn to admit that transaction bandwidth is still lurking as= a serious issue for the operation, reliability, safety, consumer acceptanc= e, uptake and, for the value of Bitcoin. > > I recently sent a payment which was not urgent so; I chose three-day targ= et confirmation from the fee recommendation. That transaction has still not= confirmed after now more than six days - even waiting twice as long seems = quite reasonable to me. That transaction is a valid transaction; it is not = rubbish, junk or, spam. Under the current model with transaction bandwidth = limitation, the longer a transaction waits, the less likely it is ever to c= onfirm due to rising transaction numbers and being pushed back by transacti= ons with rising fees. > > I argue that no transactions are rubbish or junk, only some zero fee tran= sactions might be spam. Having an ever-increasing number of valid transacti= ons that do not confirm as more new transactions with higher fees are creat= ed is the opposite of operating a robust, reliable transaction system. > > Business cannot operate with a model where transactions may or may not co= nfirm. Even a business choosing a modest fee has no guarantee that their va= lid transaction will not be shuffled down by new transactions to the realm = of never confirming after it is created. Consumers also will not accept thi= s model as Bitcoin expands. If Bitcoin cannot be a reliable payment system = for confirmed transactions then consumers, by and large, will simply not ac= cept the model once they understand. Bitcoin will be a dirty payment system= , and this will kill the value of Bitcoin. > > Under the current system, a minority of transactions will eventually be t= he lucky few who have fees high enough to escape being pushed down the list= . > > Once there are more than x transactions (transaction bandwidth limit) eve= ry ten minutes, only those choosing twenty-minute confirmation (2 blocks) w= ill have initially at most a fifty percent chance of ever having their paym= ent confirm. Presently, not even using fee recommendations can ensure a suf= ficiently high fee is paid to ensure transaction confirmation. > > I also argue that the current auction model for limited transaction bandw= idth is wrong, is not suitable for a reliable transaction system and, is wr= ong for Bitcoin. All transactions must confirm in due time. Currently, Bitc= oin is not a safe way to send payments. > > I do not believe that consumers and business are against paying fees, eve= n high fees. What is required is operational reliability. > > This great issue needs to be resolved for the safety and reliability of B= itcoin. The time to resolve issues in commerce is before they become great = big issues. The time to resolve this issue is now. We must have the foresig= ht to identify and resolve problems before they trip us over. Simply doubl= ing block sizes every so often is reactionary and is not a reliable permane= nt solution. I have written a BIP proposal for a technical solution but, ne= ed your help to write it up to an acceptable standard to be a full BIP. > > I have formatted the following with markdown which is human readable so, = I hope nobody minds. I have done as much with this proposal as I feel that = I am able so far but continue to take your feedback. > > # BIP Proposal: UTPFOTIB - Use Transaction Priority For Ordering Transact= ions In Blocks > > ## The problem: > Everybody wants value. Miners want to maximize revenue from fees (and we = presume, to minimize block size). Consumers need transaction reliability an= d, (we presume) want low fees. > > The current transaction bandwidth limit is a limiting factor for both. As= the operational safety of transactions is limited, so is consumer confiden= ce as they realize the issue and, accordingly, uptake is limited. Fees are = artificially inflated due to bandwidth limitations while failing to provide= a full confirmation service for all transactions. > > Current fee recommendations provide no satisfaction for transaction relia= bility and, as Bitcoin scales, this will worsen. > > Bitcoin must be a fully scalable and reliable service, providing full tra= nsaction confirmation for every valid transaction. > > The possibility to send a transaction with a fee lower than one that is a= cceptable to allow eventual transaction confirmation should be removed from= the protocol and also from the user interface. > > ## Solution summary: > Provide each transaction with an individual transaction priority each tim= e before choosing transactions to include in the current block, the priorit= y being a function of the fee paid (on a curve), and the time waiting in th= e transaction pool (also on a curve) out to n days (n=3D60 ?). The transact= ion priority to serve as the likelihood of a transaction being included in = the current block, and for determining the order in which transactions are = tried to see if they will be included. > > Use a target block size. Determine the target block size using; current t= ransaction pool size x ( 1 / (144 x n days ) ) =3D number of transactions t= o be included in the current block. Broadcast the next target block size wi= th the current block when it is solved so that nodes know the next target b= lock size for the block that they are building on. > > The curves used for the priority of transactions would have to be appropr= iate. Perhaps a mathematician with experience in probability can develop th= e right formulae. My thinking is a steep curve. I suppose that the probabil= ity of all transactions should probably account for a sufficient number of = inclusions that the target block size is met although, it may not always be= . As a suggestion, consider including some zero fee transactions to pad, hi= ghest BTC value first? > > **Explanation of the operation of priority:** > > If transaction priority is, for example, a number between one (low) and= one-hundred (high) it can be directly understood as the percentage chance = in one-hundred of a transaction being included in the block. Using probabil= ity or likelihood infers that there is some function of random. If random (= 100) < transaction priority then the transaction is included. > > >To break it down further, if both the fee on a curve value and the time = waiting on a curve value are each a number between one and one-hundred, a r= udimentary method may be to simply multiply those two numbers, to find the = priority number. For example, a middle fee transaction waiting thirty days = (if n =3D 60 days) may have a value of five for each part (yes, just five,= the values are on a curve). When multiplied that will give a priority valu= e of twenty-five, or, a twenty-five percent chance at that moment of being= included in the block; it will likely be included in one of the next four = blocks, getting more likely each chance. If it is still not included then t= he value of time waiting will be higher, making for more probability. A ver= y low fee transaction would have a value for the fee of one. It would not b= e until near sixty-days that the particular low fee transaction has a high = likelihood of being included in the block. > > I am not concerned with low (or high) transaction fees, the primary reaso= n for addressing the issue is to ensure transactional reliability and scala= bility while having each transaction confirm in due time. > > ## Pros: > * Maximizes transaction reliability. > * Fully scalable. > * Maximizes possibility for consumer and business uptake. > * Maximizes total fees paid per block without reducing reliability; becau= se of reliability, in time confidence and overall uptake are greater; there= fore, more transactions. > * Market determines fee paid for transaction priority. > * Fee recommendations work all the way out to 30 days or greater. > * Provides additional block entropy; greater security since there is less= probability of predicting the next block. > > ## Cons: > * Could initially lower total transaction fees per block. > * Must be first be programmed. > > ## Solution operation: > This is a simplistic view of the operation. The actual operation will nee= d to be determined in a spec for the programmer. > > 1. Determine the target block size for the current block. > 2. Assign a transaction priority to each transaction in the pool. > 3. Select transactions to include in the current block using probability = in transaction priority order until the target block size is met. > 5. Solve block. > 6. Broadcast the next target block size with the current block when it is= solved. > 7. Block is received. > 8. Block verification process. > 9. Accept/reject block based on verification result. > 10. Repeat. > > ## Closing comments: > It may be possible to verify blocks conform to the proposal by showing th= at the probability for all transactions included in the block statistically= conforms to a probability distribution curve, *if* the individual transact= ion priority can be recreated. I am not that deep into the mathematics; how= ever, it may also be possible to use a similar method to do this just based= on the fee, that statistically, the blocks conform to a fee distribution. = Any zero fee transactions would have to be ignored. This solution needs a c= lever mathematician. > > I implore, at the very least, that we use some method that validates full= transaction reliability and enables scalability of block sizes. If not thi= s proposal, an alternative. > > Regards, > Damian Williamson > > > > > _______________________________________________ > bitcoin-dev mailing list > bitcoin-dev@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev > --_000_PS2P216MB01798757147723AF5B7B7CA49D0F0PS2P216MB0179KORP_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

Thank you for your constructive f= eedback. I now see that the proposal introduces a potential issue.


It is difficult to define then, w= hat is a valid transaction? Clearly, my definition was insufficient.


Regards,

Damian Williamson



From: Chris Riley <cri= ley@gmail.com>
Sent: Monday, 18 December 2017 11:09 PM
To: Damian Williamson; Bitcoin Protocol Discussion
Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Tra= nsaction Priority For Ordering Transactions In Blocks
 
Regarding "problem" #2 where you say "How d= o we ensure that all valid transactions are eventually included in the bloc= kchain?":  I do not believe that all people would (a) agree this = is a problem or (b) that we do want to *ENSURE* that *ALL* valid transactions are eventually included in the blockchain.  There = are many *valid* transactions that oftentimes miners do not (and should not= ) wish to require be confirmed and included in the blockchain.  Spam t= ransactions for example can be valid, but used to attack bitcoin by using no or low fee.  Any valid transaction= MAY be included by a miner, but requiring it in some fashion at this point= would open the network to other attack vectors.  Perhaps you meant it= a different way.


On Fri, Dec 15, 2017 at 3:59 PM, Damian Williamson via bitcoin-dev <bitcoin-dev@lists.linu= xfoundation.org> wrote:
>
> There are really two separate problems to solve.
>
>
> How does Bitcoin scale with fixed block size?
> How do we ensure that all valid transactions are eventually included i= n the blockchain?
>
>
> Those are the two issues that the proposal attempts to address. It mak= es sense to resolve these two problems together. Using the proposed system = for variable block sizes would solve the first problem but there would stil= l be a whole bunch of never confirming transactions. I am not sure how to reliably solve the second problem at sc= ale without first solving the first.
>
>
> >* Every node has a (potentially) different mempool, you can't use = it to decide consensus values like the max block size.
>
>
> I do not suggest a consensus. Depending on which node solves a block t= he value for next block size will be different. The consensus would be that= blocks will adhere to the next block size value transmitted with the curre= nt block. It is easy to verify that the consensus is being adhered to once in place.
>  
> >* Increasing the entropy in a block to make it more unpredictable = doesn't really make sense.
>
> Not a necessary function, just an effect of using a probability-based = distribution.
>
> >* Bitcoin should be roughly incentive compatible. Your proposal ex= plicits asks miners to ignore their best interests, and confirm transaction= s by "priority".  What are you going to do if a "malici= ous" miner decides to go after their profits and order by what makes them the most money. Add "ordered by priority" as a c= onsensus requirement? And even if you miners can still sort their mempool b= y fee, and then order the top 1MB by priority.
>
> I entirely agree with your sentiment that Bitcoin must be incentive co= mpatible. It is necessary.
>
> It is in only miners immediate interest to make the most profitable bl= ock from the available transaction pool. As with so many other things, it i= s necessary to partially ignore short-term gain for long-term benefit. It i= s in miners and everybody's long-term interest to have a reliable transaction service. A busy transaction servic= e that confirms lots of transactions per hour will become more profitable a= s demand increases and more users are prepared to pay for priority. As it i= s there is currently no way to fully scale because of the transaction bandwidth limit and that is problematic. = If all valid transactions must eventually confirm then there must be a way = to resolve that problem.
>
> Bitcoin deliberately removes traditional scale by ensuring blocks take= ten minutes on average to solve, an ingenious idea and, incentive compatib= le but, fixed block sizes leaves us with a problem to solve when we want to= scale.
>
> >If you could find a good solution that would allow you to know if = miners were following your rule or not (and thus ignore it if it doesn't) t= hen you wouldn't even need bitcoin in the first place.
>
> I am confident that the math to verify blocks based on the proposal ca= n be developed (and I think it will not be too complex for a mathematician = with the relevant experience), however, I am nowhere near experienced enoug= h with probability and statistical analysis to do it. Yes, if Bitcoin doesn't then it might make another grea= t opportunity for an altcoin but I am not even nearly interested in promoti= ng any altcoins.
>
>
> If not the proposal that I have put forward, then, hopefully, someone = can come up with a better solution. The important thing is that the issues = are resolved.
>
>
> Regards,
>
> Damian Williamson
>
>
>
> ________________________________
> From: Rhavar <rhavar@proto= nmail.com>
> Sent: Saturday, 16 December 2017 3:38 AM
> To: Damian Williamson
> Cc: Bitcoin Protocol Discussion
> Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Trans= action Priority For Ordering Transactions In Blocks
>  
> > I understand that there would be technical issues to resolve in i= mplementation, but, are there no fundamental errors?
>
> Unfortunately your proposal is really fundamentally broken, on a few l= evels. I think you might need to do a bit more research into how bitcoin wo= rks before coming up with such improvements =3D)
>
> But just some quick notes:
>
> * Every node has a (potentially) different mempool, you can't use it t= o decide consensus values like the max block size.
>
> * Increasing the entropy in a block to make it more unpredictable does= n't really make sense.
>
> * Bitcoin should be roughly incentive compatible. Your proposal explic= its asks miners to ignore their best interests, and confirm transactions by= "priority".  What are you going to do if a "malicious&= quot; miner decides to go after their profits and order by what makes them the most money. Add "ordered by priority" as a consen= sus requirement? And even if you miners can still sort their mempool by fee= , and then order the top 1MB by priority.
>
> If you could find a good solution that would allow you to know if mine= rs were following your rule or not (and thus ignore it if it doesn't) then = you wouldn't even need bitcoin in the first place.
>
>
>
>
> -Ryan
>
>
> -------- Original Message --------
> Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transacti= on Priority For Ordering Transactions In Blocks
> Local Time: December 15, 2017 3:42 AM
> UTC Time: December 15, 2017 9:42 AM
> From: bitcoin= -dev@lists.linuxfoundation.org
> To: Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>
>
>
>
> I should not take it that the lack of critical feedback to this revise= d proposal is a glowing endorsement. I understand that there would be techn= ical issues to resolve in implementation, but, are there no fundamental err= ors?
>
> I suppose that it if is difficult to determine how long a transaction = has been waiting in the pool then, each node could simply keep track of whe= n a transaction was first seen. This may have implications for a verify rou= tine, however, for example, if a node was offline, how should it differentiate how long each transaction was wai= ting in that case? If a node was restarted daily would it always think that= all transactions had been waiting in the pool less than one day If each no= de keeps the current transaction pool in a file and updates it, as transactions are included in blocks and,= as new transactions appear in the pool, then that would go some way to all= eviate the issue, apart from entirely new nodes. There should be no reason = the contents of a transaction pool files cannot be shared without agreement as to the transaction pool betwee= n nodes, just as nodes transmit new transactions freely.
>
> It has been questioned why miners could not cheat. For the question of= how many transactions to include in a block, I say it is a standoff and mi= ners will conform to the proposal, not wanting to leave transactions with v= alid fees standing, and, not wanting to shrink the transaction pool. In any case, if miners shrink the transact= ion pool then I am not immediately concerned since it provides a more effic= ient service. For the question of including transactions according to the p= roposal, I say if it is possible to keep track of how long transactions are waiting in the pool so that the= y can be included on a probability curve then it is possible to verify that= blocks conform to the proposal, since the input is a probability, the outp= ut should conform to a probability curve.
>
>
> If someone has the necessary skill, would anyone be willing to develop= the math necessary for the proposal?
>
> Regards,
> Damian Williamson
>
>
> ________________________________
>
> From: bitcoin-dev-bounces@lists.linuxfoundation.org <bitcoin-dev-bounces@lists.linu= xfoundation.org> on behalf of Damian Williamson via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org>
> Sent: Friday, 8 December 2017 8:01 AM
> To: bitcoin-d= ev@lists.linuxfoundation.org
> Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transacti= on Priority For Ordering Transactions In Blocks
>  
>
>
> Good afternoon,
>
> The need for this proposal:
>
> We all must learn to admit that transaction bandwidth is still lurking= as a serious issue for the operation, reliability, safety, consumer accept= ance, uptake and, for the value of Bitcoin.
>
> I recently sent a payment which was not urgent so; I chose three-day t= arget confirmation from the fee recommendation. That transaction has still = not confirmed after now more than six days - even waiting twice as long see= ms quite reasonable to me. That transaction is a valid transaction; it is not rubbish, junk or, spam. Under the curren= t model with transaction bandwidth limitation, the longer a transaction wai= ts, the less likely it is ever to confirm due to rising transaction numbers= and being pushed back by transactions with rising fees.
>
> I argue that no transactions are rubbish or junk, only some zero fee t= ransactions might be spam. Having an ever-increasing number of valid transa= ctions that do not confirm as more new transactions with higher fees are cr= eated is the opposite of operating a robust, reliable transaction system.
>
> Business cannot operate with a model where transactions may or may not= confirm. Even a business choosing a modest fee has no guarantee that their= valid transaction will not be shuffled down by new transactions to the rea= lm of never confirming after it is created. Consumers also will not accept this model as Bitcoin expands. If = Bitcoin cannot be a reliable payment system for confirmed transactions then= consumers, by and large, will simply not accept the model once they unders= tand. Bitcoin will be a dirty payment system, and this will kill the value of Bitcoin.
>
> Under the current system, a minority of transactions will eventually b= e the lucky few who have fees high enough to escape being pushed down the l= ist.
>
> Once there are more than x transactions (transaction bandwidth limit) = every ten minutes, only those choosing twenty-minute confirmation (2 blocks= ) will have initially at most a fifty percent chance of ever having their p= ayment confirm. Presently, not even using fee recommendations can ensure a sufficiently high fee is paid to en= sure transaction confirmation.
>
> I also argue that the current auction model for limited transaction ba= ndwidth is wrong, is not suitable for a reliable transaction system and, is= wrong for Bitcoin. All transactions must confirm in due time. Currently, B= itcoin is not a safe way to send payments.
>
> I do not believe that consumers and business are against paying fees, = even high fees. What is required is operational reliability.
>
> This great issue needs to be resolved for the safety and reliability o= f Bitcoin. The time to resolve issues in commerce is before they become gre= at big issues. The time to resolve this issue is now. We must have the fore= sight to identify and resolve problems before they trip us over.  Simply doubling block sizes every so often= is reactionary and is not a reliable permanent solution. I have written a = BIP proposal for a technical solution but, need your help to write it up to= an acceptable standard to be a full BIP.
>
> I have formatted the following with markdown which is human readable s= o, I hope nobody minds. I have done as much with this proposal as I feel th= at I am able so far but continue to take your feedback.
>
> # BIP Proposal: UTPFOTIB - Use Transaction Priority For Ordering Trans= actions In Blocks
>
> ## The problem:
> Everybody wants value. Miners want to maximize revenue from fees (and = we presume, to minimize block size). Consumers need transaction reliability= and, (we presume) want low fees.
>
> The current transaction bandwidth limit is a limiting factor for both.= As the operational safety of transactions is limited, so is consumer confi= dence as they realize the issue and, accordingly, uptake is limited. Fees a= re artificially inflated due to bandwidth limitations while failing to provide a full confirmation service for all t= ransactions.
>
> Current fee recommendations provide no satisfaction for transaction re= liability and, as Bitcoin scales, this will worsen.
>
> Bitcoin must be a fully scalable and reliable service, providing full = transaction confirmation for every valid transaction.
>
> The possibility to send a transaction with a fee lower than one that i= s acceptable to allow eventual transaction confirmation should be removed f= rom the protocol and also from the user interface.
>
> ## Solution summary:
> Provide each transaction with an individual transaction priority each = time before choosing transactions to include in the current block, the prio= rity being a function of the fee paid (on a curve), and the time waiting in= the transaction pool (also on a curve) out to n days (n=3D60 ?). The transaction priority to serve as the likelih= ood of a transaction being included in the current block, and for determini= ng the order in which transactions are tried to see if they will be include= d.
>
> Use a target block size. Determine the target block size using; curren= t transaction pool size x ( 1 / (144 x n days ) ) =3D number of transaction= s to be included in the current block. Broadcast the next target block size= with the current block when it is solved so that nodes know the next target block size for the block that they are = building on.
>
> The curves used for the priority of transactions would have to be appr= opriate. Perhaps a mathematician with experience in probability can develop= the right formulae. My thinking is a steep curve. I suppose that the proba= bility of all transactions should probably account for a sufficient number of inclusions that the target block size i= s met although, it may not always be. As a suggestion, consider including s= ome zero fee transactions to pad, highest BTC value first?
>
> **Explanation of the operation of priority:**
> > If transaction priority is, for example, a number between one (lo= w) and one-hundred (high) it can be directly understood as the percentage c= hance in one-hundred of a transaction being included in the block. Using pr= obability or likelihood infers that there is some function of random. If random (100) < transaction priority then= the transaction is included.
>
> >To break it down further, if both the fee on a curve value and the= time waiting on a curve value are each a number between one and one-hundre= d, a rudimentary method may be to simply multiply those two numbers, to fin= d the priority number. For example, a middle fee transaction waiting thirty days (if n =3D 60 days) may have a v= alue of five for each part  (yes, just five, the values are on a curve= ). When multiplied that will give a priority value of twenty-five, or, &nbs= p;a twenty-five percent chance at that moment of being included in the block; it will likely be included in one of the n= ext four blocks, getting more likely each chance. If it is still not includ= ed then the value of time waiting will be higher, making for more probabili= ty. A very low fee transaction would have a value for the fee of one. It would not be until near sixty-days tha= t the particular low fee transaction has a high likelihood of being include= d in the block.
>
> I am not concerned with low (or high) transaction fees, the primary re= ason for addressing the issue is to ensure transactional reliability and sc= alability while having each transaction confirm in due time.
>
> ## Pros:
> * Maximizes transaction reliability.
> * Fully scalable.
> * Maximizes possibility for consumer and business uptake.
> * Maximizes total fees paid per block without reducing reliability; be= cause of reliability, in time confidence and overall uptake are greater; th= erefore, more transactions.
> * Market determines fee paid for transaction priority.
> * Fee recommendations work all the way out to 30 days or greater.
> * Provides additional block entropy; greater security since there is l= ess probability of predicting the next block.
>
> ## Cons:
> * Could initially lower total transaction fees per block.
> * Must be first be programmed.
>
> ## Solution operation:
> This is a simplistic view of the operation. The actual operation will = need to be determined in a spec for the programmer.
>
> 1. Determine the target block size for the current block.
> 2. Assign a transaction priority to each transaction in the pool.
> 3. Select transactions to include in the current block using probabili= ty in transaction priority order until the target block size is met.
> 5. Solve block.
> 6. Broadcast the next target block size with the current block when it= is solved.
> 7. Block is received.
> 8. Block verification process.
> 9. Accept/reject block based on verification result.
> 10. Repeat.
>
> ## Closing comments:
> It may be possible to verify blocks conform to the proposal by showing= that the probability for all transactions included in the block statistica= lly conforms to a probability distribution curve, *if* the individual trans= action priority can be recreated. I am not that deep into the mathematics; however, it may also be possible to= use a similar method to do this just based on the fee, that statistically,= the blocks conform to a fee distribution. Any zero fee transactions would = have to be ignored. This solution needs a clever mathematician.
>
> I implore, at the very least, that we use some method that validates f= ull transaction reliability and enables scalability of block sizes. If not = this proposal, an alternative.
>
> Regards,
> Damian Williamson
>
>
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@l= ists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
--_000_PS2P216MB01798757147723AF5B7B7CA49D0F0PS2P216MB0179KORP_--