Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id BDFC340C for ; Fri, 15 Dec 2017 20:59:59 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from APC01-SG2-obe.outbound.protection.outlook.com (mail-oln040092253027.outbound.protection.outlook.com [40.92.253.27]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 4158D1F2 for ; Fri, 15 Dec 2017 20:59:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=live.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=/P4s1g9u4i71tPFCezGIiuctTzYyt+KHkYOUx1YAFBM=; b=kTFBMwIgI3OvGmcNQdlN23Jk7V1ACtaBJjKkwnQJMk4oWpPwOet9i6zr0iI5qHk1hkhgoTNjehvjs+93KDkN+B5MMBtenMR7wYpTgp2WbFrA2Tt8dLqXKWMZC8FpI8QJO36DctwrXxVzd08yDkBHzCuMLP3DWj6wygxjpMX7d8q8xqUqRc8NDnddLB7Oz6HMKu9hyygP7iQ4hCpTZ8PraIGXrgV0boXOidnRMQM/K+j2GeF8hK/4suHkARMhTa1k78qPc9cQxrOJaBjFxZdze9Wpfl2ZlTccDSkPmUKnGmbRZ3O77lzIHDNIC9VjoHsr8kvvWs0RcX2M4179ircBaA== Received: from PU1APC01FT024.eop-APC01.prod.protection.outlook.com (10.152.252.57) by PU1APC01HT108.eop-APC01.prod.protection.outlook.com (10.152.253.86) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.20.302.6; Fri, 15 Dec 2017 20:59:52 +0000 Received: from PS2P216MB0179.KORP216.PROD.OUTLOOK.COM (10.152.252.60) by PU1APC01FT024.mail.protection.outlook.com (10.152.252.233) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.20.282.5 via Frontend Transport; Fri, 15 Dec 2017 20:59:52 +0000 Received: from PS2P216MB0179.KORP216.PROD.OUTLOOK.COM ([10.171.225.19]) by PS2P216MB0179.KORP216.PROD.OUTLOOK.COM ([10.171.225.19]) with mapi id 15.20.0302.014; Fri, 15 Dec 2017 20:59:52 +0000 From: Damian Williamson To: Rhavar Thread-Topic: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks Thread-Index: AQHTb5z9LKaNJrfjBkOvllhZtrqBGqNEL2DvgAB3wACAAD9wXg== Date: Fri, 15 Dec 2017 20:59:51 +0000 Message-ID: References: , <5upGmF0IhXUWhcikhdV-e9Pqg-kXfUuXe0kOpGxumie_TO2jLvCgTZ5c6vgBRtaqkL6DmOJb1YftK0osufB5RkhW7YhAhhCI0zBTH3YcORY=@protonmail.com> In-Reply-To: <5upGmF0IhXUWhcikhdV-e9Pqg-kXfUuXe0kOpGxumie_TO2jLvCgTZ5c6vgBRtaqkL6DmOJb1YftK0osufB5RkhW7YhAhhCI0zBTH3YcORY=@protonmail.com> Accept-Language: en-AU, en-US Content-Language: en-AU X-MS-Has-Attach: X-MS-TNEF-Correlator: x-incomingtopheadermarker: OriginalChecksum:7CFD2FBBC7403A01AB4352A95B66CE40A2CAFD156D6B9FC8D0C89F3DE6F6917A; UpperCasedChecksum:F612D53F64FCDF2B57496F5EFC125B7AD89AECF5E4CE26B535C619DAA11793D1; SizeAsReceived:7518; Count:47 x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [GIvVXoRwqTGfEwpdkm79Y441j9uRwm8N] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; PU1APC01HT108; 6:DfGn4PVd/2da39MH0qaEbsNU4XmFVEO8F2VwVa1APoXyz3ogApD9aZjCQqjBzk0hmiJiHXLOx91g8bFLa2xKKIyHQ2aaGKaKZSMAulKpyOUdf8fu2MKbw/qUrooiKxCC7munJa2qd+maZWSJCyflMlIrfSenAkb4ASlWdFGHVcC7l0hcm8g3rJI4yZRZ9KFSLY9y/WKMHR7Kz1ZRKjK2Xx6a38N8zpE+jjaUj71kFUPLh+1qgqIak0PrHqW5Q3jUduJw00Ij1vwwmQI7haxg4aEydNn0721MU5X8IxfYQd8gqChClHt+g7MSNVXlCFKmWEaSLPGlkySlXxio8m4ketGXat8fm3/W1ZBpG6voMQs=; 5:bf46my0NGWc7eFLuD8p3ZC4r7S37eqmhjRzjMipkrXU7plnQp3acJMS1XQEWQG5b1cszpMdnfTv9GYA+qdqjz9QmSlLB3UXszAxzWXgqq0wgSrm84M4/cjQ2SIP16Xa+WYap1UrJmlexHGBOFOvREJKMva16ft03HDpuurV/rVM=; 24:WE33ivVdBVOPZnaux1piMP9pIVx1xKUrAUh6crzivH0eCano52ATIiawdo/UHzfV5+pIaJAi4vlG/xUr3gudrEZIvBkpQSwCeBmMPU16Qag=; 7:1dFkUsMhwsGZTteV7H11XEGza78pWQ4jm+eFOA7Ribuu84reAhe+hZKNXGE2q9n1ttzGpcjdf2QkA97uerUoMBox+Wd2WA1X0SeKIarBoHG4LISnknI42p2Y7KjspCPIDCv3mUrarndxScPyAML69MJ+cF9Jf6Dvad3qudZZiX/6E94d5cPUN9Lk1nixmo7/ucu1YBq8dW0gYywLDDz+xn68BSErkNP7qvlmjZtI9PEZirzQ179LSr2AJ97NEDKf x-incomingheadercount: 47 x-eopattributedmessage: 0 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(201702061074)(5061506573)(5061507331)(1603103135)(2017031320274)(2017031324274)(2017031323274)(2017031322404)(1601125374)(1603101448)(1701031045); SRVR:PU1APC01HT108; x-ms-traffictypediagnostic: PU1APC01HT108: x-ms-office365-filtering-correlation-id: c8d6b87c-899f-4b31-97ff-08d543fec9ab x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(444000031); SRVR:PU1APC01HT108; BCL:0; PCL:0; RULEID:(100000803101)(100110400095); SRVR:PU1APC01HT108; x-forefront-prvs: 05220145DE x-forefront-antispam-report: SFV:NSPM; SFS:(7070007)(98901004); DIR:OUT; SFP:1901; SCL:1; SRVR:PU1APC01HT108; H:PS2P216MB0179.KORP216.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: multipart/alternative; boundary="_000_PS2P216MB0179544A6503C2992190CEB69D0B0PS2P216MB0179KORP_" MIME-Version: 1.0 X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: c8d6b87c-899f-4b31-97ff-08d543fec9ab X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Dec 2017 20:59:51.9513 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Internet X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-Transport-CrossTenantHeadersStamped: PU1APC01HT108 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, HTML_MESSAGE, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org X-Mailman-Approved-At: Mon, 18 Dec 2017 11:55:36 +0000 Cc: Bitcoin Protocol Discussion Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Dec 2017 20:59:59 -0000 --_000_PS2P216MB0179544A6503C2992190CEB69D0B0PS2P216MB0179KORP_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable There are really two separate problems to solve. 1. How does Bitcoin scale with fixed block size? 2. How do we ensure that all valid transactions are eventually included = in the blockchain? Those are the two issues that the proposal attempts to address. It makes se= nse to resolve these two problems together. Using the proposed system for v= ariable block sizes would solve the first problem but there would still be = a whole bunch of never confirming transactions. I am not sure how to reliab= ly solve the second problem at scale without first solving the first. >* Every node has a (potentially) different mempool, you can't use it to de= cide consensus values like the max block size. I do not suggest a consensus. Depending on which node solves a block the va= lue for next block size will be different. The consensus would be that bloc= ks will adhere to the next block size value transmitted with the current bl= ock. It is easy to verify that the consensus is being adhered to once in pl= ace. >* Increasing the entropy in a block to make it more unpredictable doesn't = really make sense. Not a necessary function, just an effect of using a probability-based distr= ibution. >* Bitcoin should be roughly incentive compatible. Your proposal explicits = asks miners to ignore their best interests, and confirm transactions by "pr= iority". What are you going to do if a "malicious" miner decides to go aft= er their profits and order by what makes them the most money. Add "ordered = by priority" as a consensus requirement? And even if you miners can still s= ort their mempool by fee, and then order the top 1MB by priority. I entirely agree with your sentiment that Bitcoin must be incentive compati= ble. It is necessary. It is in only miners immediate interest to make the most profitable block f= rom the available transaction pool. As with so many other things, it is nec= essary to partially ignore short-term gain for long-term benefit. It is in = miners and everybody's long-term interest to have a reliable transaction se= rvice. A busy transaction service that confirms lots of transactions per ho= ur will become more profitable as demand increases and more users are prepa= red to pay for priority. As it is there is currently no way to fully scale = because of the transaction bandwidth limit and that is problematic. If all = valid transactions must eventually confirm then there must be a way to reso= lve that problem. Bitcoin deliberately removes traditional scale by ensuring blocks take ten = minutes on average to solve, an ingenious idea and, incentive compatible bu= t, fixed block sizes leaves us with a problem to solve when we want to scal= e. >If you could find a good solution that would allow you to know if miners w= ere following your rule or not (and thus ignore it if it doesn't) then you = wouldn't even need bitcoin in the first place. I am confident that the math to verify blocks based on the proposal can be = developed (and I think it will not be too complex for a mathematician with = the relevant experience), however, I am nowhere near experienced enough wit= h probability and statistical analysis to do it. Yes, if Bitcoin doesn't th= en it might make another great opportunity for an altcoin but I am not even= nearly interested in promoting any altcoins. If not the proposal that I have put forward, then, hopefully, someone can c= ome up with a better solution. The important thing is that the issues are r= esolved. Regards, Damian Williamson ________________________________ From: Rhavar Sent: Saturday, 16 December 2017 3:38 AM To: Damian Williamson Cc: Bitcoin Protocol Discussion Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transactio= n Priority For Ordering Transactions In Blocks > I understand that there would be technical issues to resolve in implement= ation, but, are there no fundamental errors? Unfortunately your proposal is really fundamentally broken, on a few levels= . I think you might need to do a bit more research into how bitcoin works b= efore coming up with such improvements =3D) But just some quick notes: * Every node has a (potentially) different mempool, you can't use it to dec= ide consensus values like the max block size. * Increasing the entropy in a block to make it more unpredictable doesn't r= eally make sense. * Bitcoin should be roughly incentive compatible. Your proposal explicits a= sks miners to ignore their best interests, and confirm transactions by "pri= ority". What are you going to do if a "malicious" miner decides to go afte= r their profits and order by what makes them the most money. Add "ordered b= y priority" as a consensus requirement? And even if you miners can still so= rt their mempool by fee, and then order the top 1MB by priority. If you could find a good solution that would allow you to know if miners we= re following your rule or not (and thus ignore it if it doesn't) then you w= ouldn't even need bitcoin in the first place. -Ryan -------- Original Message -------- Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Pr= iority For Ordering Transactions In Blocks Local Time: December 15, 2017 3:42 AM UTC Time: December 15, 2017 9:42 AM From: bitcoin-dev@lists.linuxfoundation.org To: Bitcoin Protocol Discussion I should not take it that the lack of critical feedback to this revised pro= posal is a glowing endorsement. I understand that there would be technical = issues to resolve in implementation, but, are there no fundamental errors? I suppose that it if is difficult to determine how long a transaction has b= een waiting in the pool then, each node could simply keep track of when a t= ransaction was first seen. This may have implications for a verify routine,= however, for example, if a node was offline, how should it differentiate h= ow long each transaction was waiting in that case? If a node was restarted = daily would it always think that all transactions had been waiting in the p= ool less than one day If each node keeps the current transaction pool in a = file and updates it, as transactions are included in blocks and, as new tra= nsactions appear in the pool, then that would go some way to alleviate the = issue, apart from entirely new nodes. There should be no reason the content= s of a transaction pool files cannot be shared without agreement as to the = transaction pool between nodes, just as nodes transmit new transactions fre= ely. It has been questioned why miners could not cheat. For the question of how = many transactions to include in a block, I say it is a standoff and miners = will conform to the proposal, not wanting to leave transactions with valid = fees standing, and, not wanting to shrink the transaction pool. In any case= , if miners shrink the transaction pool then I am not immediately concerned= since it provides a more efficient service. For the question of including = transactions according to the proposal, I say if it is possible to keep tra= ck of how long transactions are waiting in the pool so that they can be inc= luded on a probability curve then it is possible to verify that blocks conf= orm to the proposal, since the input is a probability, the output should co= nform to a probability curve. If someone has the necessary skill, would anyone be willing to develop the = math necessary for the proposal? Regards, Damian Williamson ________________________________ From: bitcoin-dev-bounces@lists.linuxfoundation.org on behalf of Damian Williamson via bitcoin-dev Sent: Friday, 8 December 2017 8:01 AM To: bitcoin-dev@lists.linuxfoundation.org Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Pr= iority For Ordering Transactions In Blocks Good afternoon, The need for this proposal: We all must learn to admit that transaction bandwidth is still lurking as a= serious issue for the operation, reliability, safety, consumer acceptance,= uptake and, for the value of Bitcoin. I recently sent a payment which was not urgent so; I chose three-day target= confirmation from the fee recommendation. That transaction has still not c= onfirmed after now more than six days - even waiting twice as long seems qu= ite reasonable to me. That transaction is a valid transaction; it is not ru= bbish, junk or, spam. Under the current model with transaction bandwidth li= mitation, the longer a transaction waits, the less likely it is ever to con= firm due to rising transaction numbers and being pushed back by transaction= s with rising fees. I argue that no transactions are rubbish or junk, only some zero fee transa= ctions might be spam. Having an ever-increasing number of valid transaction= s that do not confirm as more new transactions with higher fees are created= is the opposite of operating a robust, reliable transaction system. Business cannot operate with a model where transactions may or may not conf= irm. Even a business choosing a modest fee has no guarantee that their vali= d transaction will not be shuffled down by new transactions to the realm of= never confirming after it is created. Consumers also will not accept this = model as Bitcoin expands. If Bitcoin cannot be a reliable payment system fo= r confirmed transactions then consumers, by and large, will simply not acce= pt the model once they understand. Bitcoin will be a dirty payment system, = and this will kill the value of Bitcoin. Under the current system, a minority of transactions will eventually be the= lucky few who have fees high enough to escape being pushed down the list. Once there are more than x transactions (transaction bandwidth limit) every= ten minutes, only those choosing twenty-minute confirmation (2 blocks) wil= l have initially at most a fifty percent chance of ever having their paymen= t confirm. Presently, not even using fee recommendations can ensure a suffi= ciently high fee is paid to ensure transaction confirmation. I also argue that the current auction model for limited transaction bandwid= th is wrong, is not suitable for a reliable transaction system and, is wron= g for Bitcoin. All transactions must confirm in due time. Currently, Bitcoi= n is not a safe way to send payments. I do not believe that consumers and business are against paying fees, even = high fees. What is required is operational reliability. This great issue needs to be resolved for the safety and reliability of Bit= coin. The time to resolve issues in commerce is before they become great bi= g issues. The time to resolve this issue is now. We must have the foresight= to identify and resolve problems before they trip us over. Simply doublin= g block sizes every so often is reactionary and is not a reliable permanent= solution. I have written a BIP proposal for a technical solution but, need= your help to write it up to an acceptable standard to be a full BIP. I have formatted the following with markdown which is human readable so, I = hope nobody minds. I have done as much with this proposal as I feel that I = am able so far but continue to take your feedback. # BIP Proposal: UTPFOTIB - Use Transaction Priority For Ordering Transactio= ns In Blocks ## The problem: Everybody wants value. Miners want to maximize revenue from fees (and we pr= esume, to minimize block size). Consumers need transaction reliability and,= (we presume) want low fees. The current transaction bandwidth limit is a limiting factor for both. As t= he operational safety of transactions is limited, so is consumer confidence= as they realize the issue and, accordingly, uptake is limited. Fees are ar= tificially inflated due to bandwidth limitations while failing to provide a= full confirmation service for all transactions. Current fee recommendations provide no satisfaction for transaction reliabi= lity and, as Bitcoin scales, this will worsen. Bitcoin must be a fully scalable and reliable service, providing full trans= action confirmation for every valid transaction. The possibility to send a transaction with a fee lower than one that is acc= eptable to allow eventual transaction confirmation should be removed from t= he protocol and also from the user interface. ## Solution summary: Provide each transaction with an individual transaction priority each time = before choosing transactions to include in the current block, the priority = being a function of the fee paid (on a curve), and the time waiting in the = transaction pool (also on a curve) out to n days (n=3D60 ?). The transactio= n priority to serve as the likelihood of a transaction being included in th= e current block, and for determining the order in which transactions are tr= ied to see if they will be included. Use a target block size. Determine the target block size using; current tra= nsaction pool size x ( 1 / (144 x n days ) ) =3D number of transactions to = be included in the current block. Broadcast the next target block size with= the current block when it is solved so that nodes know the next target blo= ck size for the block that they are building on. The curves used for the priority of transactions would have to be appropria= te. Perhaps a mathematician with experience in probability can develop the = right formulae. My thinking is a steep curve. I suppose that the probabilit= y of all transactions should probably account for a sufficient number of in= clusions that the target block size is met although, it may not always be. = As a suggestion, consider including some zero fee transactions to pad, high= est BTC value first? **Explanation of the operation of priority:** > If transaction priority is, for example, a number between one (low) and o= ne-hundred (high) it can be directly understood as the percentage chance in= one-hundred of a transaction being included in the block. Using probabilit= y or likelihood infers that there is some function of random. If random (10= 0) < transaction priority then the transaction is included. >To break it down further, if both the fee on a curve value and the time wa= iting on a curve value are each a number between one and one-hundred, a rud= imentary method may be to simply multiply those two numbers, to find the pr= iority number. For example, a middle fee transaction waiting thirty days (i= f n =3D 60 days) may have a value of five for each part (yes, just five, t= he values are on a curve). When multiplied that will give a priority value = of twenty-five, or, a twenty-five percent chance at that moment of being i= ncluded in the block; it will likely be included in one of the next four bl= ocks, getting more likely each chance. If it is still not included then the= value of time waiting will be higher, making for more probability. A very = low fee transaction would have a value for the fee of one. It would not be = until near sixty-days that the particular low fee transaction has a high li= kelihood of being included in the block. I am not concerned with low (or high) transaction fees, the primary reason = for addressing the issue is to ensure transactional reliability and scalabi= lity while having each transaction confirm in due time. ## Pros: * Maximizes transaction reliability. * Fully scalable. * Maximizes possibility for consumer and business uptake. * Maximizes total fees paid per block without reducing reliability; because= of reliability, in time confidence and overall uptake are greater; therefo= re, more transactions. * Market determines fee paid for transaction priority. * Fee recommendations work all the way out to 30 days or greater. * Provides additional block entropy; greater security since there is less p= robability of predicting the next block. ## Cons: * Could initially lower total transaction fees per block. * Must be first be programmed. ## Solution operation: This is a simplistic view of the operation. The actual operation will need = to be determined in a spec for the programmer. 1. Determine the target block size for the current block. 2. Assign a transaction priority to each transaction in the pool. 3. Select transactions to include in the current block using probability in= transaction priority order until the target block size is met. 5. Solve block. 6. Broadcast the next target block size with the current block when it is s= olved. 7. Block is received. 8. Block verification process. 9. Accept/reject block based on verification result. 10. Repeat. ## Closing comments: It may be possible to verify blocks conform to the proposal by showing that= the probability for all transactions included in the block statistically c= onforms to a probability distribution curve, *if* the individual transactio= n priority can be recreated. I am not that deep into the mathematics; howev= er, it may also be possible to use a similar method to do this just based o= n the fee, that statistically, the blocks conform to a fee distribution. An= y zero fee transactions would have to be ignored. This solution needs a cle= ver mathematician. I implore, at the very least, that we use some method that validates full t= ransaction reliability and enables scalability of block sizes. If not this = proposal, an alternative. Regards, Damian Williamson --_000_PS2P216MB0179544A6503C2992190CEB69D0B0PS2P216MB0179KORP_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

There are really two separate pro= blems to solve.


  1. How does Bitcoin scale with fixed block size?
  2. How do we ensure = that all valid transactions are eventually included in the blockchain?
    3. =

Those are the two issues that the= proposal attempts to address. It makes sense to resolve these two problems= together. Using the proposed system for variable block sizes would solve t= he first problem but there would still be a whole bunch of never confirming transactions. I am not sure how to re= liably solve the second problem at scale without first solving the first.


>* Every node has a (potential= ly) different mempool, you can't use it to decide consensus values like the= max block size. 


I do not suggest a consensus. Depending on which node solves a block the va= lue for next block size will be different. The consensus would be that bloc= ks will adhere to the next block size value transmitted with the current bl= ock. It is easy to verify that the consensus is being adhered to once in place.
 
>* Increasing the entropy in a block to make it more unpredictable = doesn't really make sense. 

Not a necessary function, just an effect of using a probability-based distr= ibution.

>* Bitcoin should be roughly incentive compatible. Your proposal ex= plicits asks miners to ignore their best interests, and confirm transaction= s by "priority".  What are you going to do if a "malici= ous" miner decides to go after their profits and order by what makes them the most money. Add "ordered by priority" as = a consensus requirement? And even if you miners can still sort their mempoo= l by fee, and then order the top 1MB by priority.

I entirely agree with your sentiment that Bitcoin must be incentive compati= ble. It is necessary.

It is in only miners immediate interest to make the most profitable block f= rom the available transaction pool. As with so many other things, it is nec= essary to partially ignore short-term gain for long-term benefit. It is in = miners and everybody's long-term interest to have a reliable transaction service. A busy transaction servic= e that confirms lots of transactions per hour will become more profitable a= s demand increases and more users are prepared to pay for priority. As it i= s there is currently no way to fully scale because of the transaction bandwidth limit and that is problematic. = If all valid transactions must eventually confirm then there must be a way = to resolve that problem.

Bitcoin deliberately removes traditional scale by ensuring blocks take ten = minutes on average to solve, an ingenious idea and, incentive compatible bu= t, fixed block sizes leaves us with a problem to solve when we want to scal= e.

>If you could find a good solution that would allow you to know if = miners were following your rule or not (and thus ignore it if it doesn't) t= hen you wouldn't even need bitcoin in the first place.

I am confident that the math to verify blocks based on the proposal can be = developed (and I think it will not be too complex for a mathematician with = the relevant experience), however, I am nowhere near experienced enough wit= h probability and statistical analysis to do it. Yes, if Bitcoin doesn't then it might make another great opportu= nity for an altcoin but I am not even nearly interested in promoting any al= tcoins.


If not the proposal that I have p= ut forward, then, hopefully, someone can come up with a better solution. Th= e important thing is that the issues are resolved.


Regards,

Damian Williamson



From: Rhavar <rhavar@p= rotonmail.com>
Sent: Saturday, 16 December 2017 3:38 AM
To: Damian Williamson
Cc: Bitcoin Protocol Discussion
Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Tra= nsaction Priority For Ordering Transactions In Blocks
 
> I understand that there would be technical issues to resolve= in implementation, but, are there no fundamental errors?

Unfortunately your proposal is really fundamentally broken, on a few l= evels. I think you might need to do a bit more research into how bitcoin wo= rks before coming up with such improvements =3D)

But just some quick notes:

* Every node has a (potentially) different mempool, you can't use it t= o decide consensus values like the max block size. 

* Increasing the entropy in a block to make it more unpredictable does= n't really make sense. 

* Bitcoin should be roughly incentive compatible. Your proposal explic= its asks miners to ignore their best interests, and confirm transactions by= "priority".  What are you going to do if a "malicious&= quot; miner decides to go after their profits and order by what makes them the most money. Add "ordered by priority" as a c= onsensus requirement? And even if you miners can still sort their mempool b= y fee, and then order the top 1MB by priority.

If you could find a good solution that would allow you to know if mine= rs were following your rule or not (and thus ignore it if it doesn't) then = you wouldn't even need bitcoin in the first place.




-Ryan


-------- Original Message --------
Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transacti= on Priority For Ordering Transactions In Blocks
Local Time: December 15, 2017 3:42 AM
UTC Time: December 15, 2017 9:42 AM
From: bitcoin-dev@lists.linuxfoundation.org
To: Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.= org>



I should not take it that the lack of critical feedback to this revise= d proposal is a glowing endorsement. I understand that there would be techn= ical issues to resolve in implementation, but, are there no fundamental err= ors?

I suppose that it if is difficult to determine how long a transaction = has been waiting in the pool then, each node could simply keep track of whe= n a transaction was first seen. This may have implications for a verify rou= tine, however, for example, if a node was offline, how should it differentiate how long each transaction wa= s waiting in that case? If a node was restarted daily would it always think= that all transactions had been waiting in the pool less than one day If ea= ch node keeps the current transaction pool in a file and updates it, as transactions are included in blocks and,= as new transactions appear in the pool, then that would go some way to all= eviate the issue, apart from entirely new nodes. There should be no reason = the contents of a transaction pool files cannot be shared without agreement as to the transaction pool = between nodes, just as nodes transmit new transactions freely.

It has been questioned why miners could not cheat. For the question of= how many transactions to include in a block, I say it is a standoff and mi= ners will conform to the proposal, not wanting to leave transactions with v= alid fees standing, and, not wanting to shrink the transaction pool. In any case, if miners shrink the transact= ion pool then I am not immediately concerned since it provides a more effic= ient service. For the question of including transactions according to the p= roposal, I say if it is possible to keep track of how long transactions are waiting in the pool so that the= y can be included on a probability curve then it is possible to verify that= blocks conform to the proposal, since the input is a probability, the outp= ut should conform to a probability curve.


If someone has the necessary skill, would anyone be willing to develop= the math necessary for the proposal?

Regards,
Damian Williamson



From: bitcoin-dev-boun= ces@lists.linuxfoundation.org <bitcoin-dev-bounces@lists.linuxfoundation= .org> on behalf of Damian Williamson via bitcoin-dev <bitcoin-dev@lists.linuxfoundation.org>
Sent: Friday, 8 December 2017 8:01 AM
To: bitcoin-dev@lists.linuxfoundation.org
Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transac= tion Priority For Ordering Transactions In Blocks
 


Good afternoon,

The need for this proposal:

We all must learn to admit that transaction bandwidth is still lurking= as a serious issue for the operation, reliability, safety, consumer accept= ance, uptake and, for the value of Bitcoin.

I recently sent a payment which was not urgent so; I chose three-day t= arget confirmation from the fee recommendation. That transaction has still = not confirmed after now more than six days - even waiting twice as long see= ms quite reasonable to me. That transaction is a valid transaction; it is not rubbish, junk or, spam. Unde= r the current model with transaction bandwidth limitation, the longer a tra= nsaction waits, the less likely it is ever to confirm due to rising transac= tion numbers and being pushed back by transactions with rising fees.

I argue that no transactions are rubbish or junk, only some zero fee t= ransactions might be spam. Having an ever-increasing number of valid transa= ctions that do not confirm as more new transactions with higher fees are cr= eated is the opposite of operating a robust, reliable transaction system.

Business cannot operate with a model where transactions may or may not= confirm. Even a business choosing a modest fee has no guarantee that their= valid transaction will not be shuffled down by new transactions to the rea= lm of never confirming after it is created. Consumers also will not accept this model as Bitcoin expands. = If Bitcoin cannot be a reliable payment system for confirmed transactions t= hen consumers, by and large, will simply not accept the model once they und= erstand. Bitcoin will be a dirty payment system, and this will kill the value of Bitcoin.

Under the current system, a minority of transactions will eventually b= e the lucky few who have fees high enough to escape being pushed down the l= ist.

Once there are more than x transactions (transaction bandwidth limit) = every ten minutes, only those choosing twenty-minute confirmation (2 blocks= ) will have initially at most a fifty percent chance of ever having their p= ayment confirm. Presently, not even using fee recommendations can ensure a sufficiently high fee is paid to en= sure transaction confirmation.

I also argue that the current auction model for limited transaction ba= ndwidth is wrong, is not suitable for a reliable transaction system and, is= wrong for Bitcoin. All transactions must confirm in due time. Currently, B= itcoin is not a safe way to send payments.

I do not believe that consumers and business are against paying fees, = even high fees. What is required is operational reliability.

This great issue needs to be resolved for the safety and reliability o= f Bitcoin. The time to resolve issues in commerce is before they become gre= at big issues. The time to resolve this issue is now. We must have the fore= sight to identify and resolve problems before they trip us over.  Simply doubling block sizes every so often= is reactionary and is not a reliable permanent solution. I have written a = BIP proposal for a technical solution but, need your help to write it up to= an acceptable standard to be a full BIP.

I have formatted the following with markdown which is human readable s= o, I hope nobody minds. I have done as much with this proposal as I feel th= at I am able so far but continue to take your feedback.

# BIP Proposal: UTPFOTIB - Use Transaction Priority For Ordering Trans= actions In Blocks

## The problem:
Everybody wants value. Miners want to maximize revenue from fees (and = we presume, to minimize block size). Consumers need transaction reliability= and, (we presume) want low fees.

The current transaction bandwidth limit is a limiting factor for both.= As the operational safety of transactions is limited, so is consumer confi= dence as they realize the issue and, accordingly, uptake is limited. Fees a= re artificially inflated due to bandwidth limitations while failing to provide a full confirmation service= for all transactions.

Current fee recommendations provide no satisfaction for transaction re= liability and, as Bitcoin scales, this will worsen.

Bitcoin must be a fully scalable and reliable service, providing full = transaction confirmation for every valid transaction.

The possibility to send a transaction with a fee lower than one that i= s acceptable to allow eventual transaction confirmation should be removed f= rom the protocol and also from the user interface.

## Solution summary:
Provide each transaction with an individual transaction priority each = time before choosing transactions to include in the current block, the prio= rity being a function of the fee paid (on a curve), and the time waiting in= the transaction pool (also on a curve) out to n days (n=3D60 ?). The transaction priority to serve as the = likelihood of a transaction being included in the current block, and for de= termining the order in which transactions are tried to see if they will be = included.

Use a target block size. Determine the target block size using; curren= t transaction pool size x ( 1 / (144 x n days ) ) =3D number of transaction= s to be included in the current block. Broadcast the next target block size= with the current block when it is solved so that nodes know the next target block size for the block that th= ey are building on.

The curves used for the priority of transactions would have to be appr= opriate. Perhaps a mathematician with experience in probability can develop= the right formulae. My thinking is a steep curve. I suppose that the proba= bility of all transactions should probably account for a sufficient number of inclusions that the target blo= ck size is met although, it may not always be. As a suggestion, consider in= cluding some zero fee transactions to pad, highest BTC value first?

**Explanation of the operation of priority:**
> If transaction priority is, for example, a number between one (lo= w) and one-hundred (high) it can be directly understood as the percentage c= hance in one-hundred of a transaction being included in the block. Using pr= obability or likelihood infers that there is some function of random. If random (100) < transaction priorit= y then the transaction is included.

>To break it down further, if both the fee on a curve value and the= time waiting on a curve value are each a number between one and one-hundre= d, a rudimentary method may be to simply multiply those two numbers, to fin= d the priority number. For example, a middle fee transaction waiting thirty days (if n =3D 60 days) may have a= value of five for each part  (yes, just five, the values are on a cur= ve). When multiplied that will give a priority value of twenty-five, or,&nb= sp; a twenty-five percent chance at that moment of being included in the block; it will likely be included in one of the n= ext four blocks, getting more likely each chance. If it is still not includ= ed then the value of time waiting will be higher, making for more probabili= ty. A very low fee transaction would have a value for the fee of one. It would not be until near sixty-days tha= t the particular low fee transaction has a high likelihood of being include= d in the block.

I am not concerned with low (or high) transaction fees, the primary re= ason for addressing the issue is to ensure transactional reliability and sc= alability while having each transaction confirm in due time.

## Pros:
* Maximizes transaction reliability.
* Fully scalable.
* Maximizes possibility for consumer and business uptake.
* Maximizes total fees paid per block without reducing reliability; be= cause of reliability, in time confidence and overall uptake are greater; th= erefore, more transactions.
* Market determines fee paid for transaction priority.
* Fee recommendations work all the way out to 30 days or greater.
* Provides additional block entropy; greater security since there is l= ess probability of predicting the next block.

## Cons:
* Could initially lower total transaction fees per block.
* Must be first be programmed.

## Solution operation:
This is a simplistic view of the operation. The actual operation will = need to be determined in a spec for the programmer.

1. Determine the target block size for the current block.
2. Assign a transaction priority to each transaction in the pool.
3. Select transactions to include in the current block using probabili= ty in transaction priority order until the target block size is met.
5. Solve block.
6. Broadcast the next target block size with the current block when it= is solved.
7. Block is received.
8. Block verification process.
9. Accept/reject block based on verification result.
10. Repeat.

## Closing comments:
It may be possible to verify blocks conform to the proposal by showing= that the probability for all transactions included in the block statistica= lly conforms to a probability distribution curve, *if* the individual trans= action priority can be recreated. I am not that deep into the mathematics; however, it may also be possible = to use a similar method to do this just based on the fee, that statisticall= y, the blocks conform to a fee distribution. Any zero fee transactions woul= d have to be ignored. This solution needs a clever mathematician.

I implore, at the very least, that we use some method that validates f= ull transaction reliability and enables scalability of block sizes. If not = this proposal, an alternative.

Regards,
Damian Williamson



--_000_PS2P216MB0179544A6503C2992190CEB69D0B0PS2P216MB0179KORP_--