From: Damian Williamson <willtech@live.com.au>
To: Rhavar <rhavar@protonmail.com>
Thread-Topic: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction
	Priority For Ordering Transactions In Blocks
Thread-Index: AQHTb5z9LKaNJrfjBkOvllhZtrqBGqNEL2DvgAB3wACAAD9wXg==
Date: Fri, 15 Dec 2017 20:59:51 +0000
Message-ID: <PS2P216MB0179544A6503C2992190CEB69D0B0@PS2P216MB0179.KORP216.PROD.OUTLOOK.COM>
References: <PS2P216MB01794ABD544248B27BF0DFD99D330@PS2P216MB0179.KORP216.PROD.OUTLOOK.COM>
	<PS2P216MB01795BFC05612E021CCEDD7C9D0B0@PS2P216MB0179.KORP216.PROD.OUTLOOK.COM>,
	<5upGmF0IhXUWhcikhdV-e9Pqg-kXfUuXe0kOpGxumie_TO2jLvCgTZ5c6vgBRtaqkL6DmOJb1YftK0osufB5RkhW7YhAhhCI0zBTH3YcORY=@protonmail.com>
In-Reply-To: <5upGmF0IhXUWhcikhdV-e9Pqg-kXfUuXe0kOpGxumie_TO2jLvCgTZ5c6vgBRtaqkL6DmOJb1YftK0osufB5RkhW7YhAhhCI0zBTH3YcORY=@protonmail.com>
Accept-Language: en-AU, en-US
Content-Language: en-AU
x-ms-exchange-messagesentrepresentingtype: 1
x-tmn: [GIvVXoRwqTGfEwpdkm79Y441j9uRwm8N]
x-ms-publictraffictype: Email
x-microsoft-exchange-diagnostics: 1; PU1APC01HT108;
	6:DfGn4PVd/2da39MH0qaEbsNU4XmFVEO8F2VwVa1APoXyz3ogApD9aZjCQqjBzk0hmiJiHXLOx91g8bFLa2xKKIyHQ2aaGKaKZSMAulKpyOUdf8fu2MKbw/qUrooiKxCC7munJa2qd+maZWSJCyflMlIrfSenAkb4ASlWdFGHVcC7l0hcm8g3rJI4yZRZ9KFSLY9y/WKMHR7Kz1ZRKjK2Xx6a38N8zpE+jjaUj71kFUPLh+1qgqIak0PrHqW5Q3jUduJw00Ij1vwwmQI7haxg4aEydNn0721MU5X8IxfYQd8gqChClHt+g7MSNVXlCFKmWEaSLPGlkySlXxio8m4ketGXat8fm3/W1ZBpG6voMQs=;
	5:bf46my0NGWc7eFLuD8p3ZC4r7S37eqmhjRzjMipkrXU7plnQp3acJMS1XQEWQG5b1cszpMdnfTv9GYA+qdqjz9QmSlLB3UXszAxzWXgqq0wgSrm84M4/cjQ2SIP16Xa+WYap1UrJmlexHGBOFOvREJKMva16ft03HDpuurV/rVM=;
	24:WE33ivVdBVOPZnaux1piMP9pIVx1xKUrAUh6crzivH0eCano52ATIiawdo/UHzfV5+pIaJAi4vlG/xUr3gudrEZIvBkpQSwCeBmMPU16Qag=;
	7:1dFkUsMhwsGZTteV7H11XEGza78pWQ4jm+eFOA7Ribuu84reAhe+hZKNXGE2q9n1ttzGpcjdf2QkA97uerUoMBox+Wd2WA1X0SeKIarBoHG4LISnknI42p2Y7KjspCPIDCv3mUrarndxScPyAML69MJ+cF9Jf6Dvad3qudZZiX/6E94d5cPUN9Lk1nixmo7/ucu1YBq8dW0gYywLDDz+xn68BSErkNP7qvlmjZtI9PEZirzQ179LSr2AJ97NEDKf
x-incomingheadercount: 47
x-eopattributedmessage: 0
x-microsoft-antispam: UriScan:; BCL:0; PCL:0;
	RULEID:(201702061074)(5061506573)(5061507331)(1603103135)(2017031320274)(2017031324274)(2017031323274)(2017031322404)(1601125374)(1603101448)(1701031045);
	SRVR:PU1APC01HT108; 
x-ms-traffictypediagnostic: PU1APC01HT108:
x-ms-office365-filtering-correlation-id: c8d6b87c-899f-4b31-97ff-08d543fec9ab
x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(444000031);
	SRVR:PU1APC01HT108; BCL:0; PCL:0;
	RULEID:(100000803101)(100110400095); SRVR:PU1APC01HT108; 
x-forefront-prvs: 05220145DE
x-forefront-antispam-report: SFV:NSPM; SFS:(7070007)(98901004); DIR:OUT;
	SFP:1901; SCL:1; SRVR:PU1APC01HT108;
	H:PS2P216MB0179.KORP216.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:;
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: multipart/alternative;
	boundary="_000_PS2P216MB0179544A6503C2992190CEB69D0B0PS2P216MB0179KORP_"
MIME-Version: 1.0
X-OriginatorOrg: outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c8d6b87c-899f-4b31-97ff-08d543fec9ab
X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Dec 2017 20:59:51.9513 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Internet
X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PU1APC01HT108
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID, HTML_MESSAGE, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
	smtp1.linux-foundation.org
X-Mailman-Approved-At: Mon, 18 Dec 2017 11:55:36 +0000
Cc: Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>
Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction
 Priority For Ordering Transactions In Blocks
X-BeenThere: bitcoin-dev@lists.linuxfoundation.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Bitcoin Protocol Discussion <bitcoin-dev.lists.linuxfoundation.org>
List-Unsubscribe: <https://lists.linuxfoundation.org/mailman/options/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=unsubscribe>
List-Archive: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/>
List-Post: <mailto:bitcoin-dev@lists.linuxfoundation.org>
List-Help: <mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=help>
List-Subscribe: <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>,
	<mailto:bitcoin-dev-request@lists.linuxfoundation.org?subject=subscribe>
X-List-Received-Date: Fri, 15 Dec 2017 20:59:59 -0000

--_000_PS2P216MB0179544A6503C2992190CEB69D0B0PS2P216MB0179KORP_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

There are really two separate problems to solve.


  1.  How does Bitcoin scale with fixed block size?
  2.  How do we ensure that all valid transactions are eventually included =
in the blockchain?


Those are the two issues that the proposal attempts to address. It makes se=
nse to resolve these two problems together. Using the proposed system for v=
ariable block sizes would solve the first problem but there would still be =
a whole bunch of never confirming transactions. I am not sure how to reliab=
ly solve the second problem at scale without first solving the first.


>* Every node has a (potentially) different mempool, you can't use it to de=
cide consensus values like the max block size.

I do not suggest a consensus. Depending on which node solves a block the va=
lue for next block size will be different. The consensus would be that bloc=
ks will adhere to the next block size value transmitted with the current bl=
ock. It is easy to verify that the consensus is being adhered to once in pl=
ace.

>* Increasing the entropy in a block to make it more unpredictable doesn't =
really make sense.

Not a necessary function, just an effect of using a probability-based distr=
ibution.

>* Bitcoin should be roughly incentive compatible. Your proposal explicits =
asks miners to ignore their best interests, and confirm transactions by "pr=
iority".  What are you going to do if a "malicious" miner decides to go aft=
er their profits and order by what makes them the most money. Add "ordered =
by priority" as a consensus requirement? And even if you miners can still s=
ort their mempool by fee, and then order the top 1MB by priority.

I entirely agree with your sentiment that Bitcoin must be incentive compati=
ble. It is necessary.

It is in only miners immediate interest to make the most profitable block f=
rom the available transaction pool. As with so many other things, it is nec=
essary to partially ignore short-term gain for long-term benefit. It is in =
miners and everybody's long-term interest to have a reliable transaction se=
rvice. A busy transaction service that confirms lots of transactions per ho=
ur will become more profitable as demand increases and more users are prepa=
red to pay for priority. As it is there is currently no way to fully scale =
because of the transaction bandwidth limit and that is problematic. If all =
valid transactions must eventually confirm then there must be a way to reso=
lve that problem.

Bitcoin deliberately removes traditional scale by ensuring blocks take ten =
minutes on average to solve, an ingenious idea and, incentive compatible bu=
t, fixed block sizes leaves us with a problem to solve when we want to scal=
e.

>If you could find a good solution that would allow you to know if miners w=
ere following your rule or not (and thus ignore it if it doesn't) then you =
wouldn't even need bitcoin in the first place.

I am confident that the math to verify blocks based on the proposal can be =
developed (and I think it will not be too complex for a mathematician with =
the relevant experience), however, I am nowhere near experienced enough wit=
h probability and statistical analysis to do it. Yes, if Bitcoin doesn't th=
en it might make another great opportunity for an altcoin but I am not even=
 nearly interested in promoting any altcoins.


If not the proposal that I have put forward, then, hopefully, someone can c=
ome up with a better solution. The important thing is that the issues are r=
esolved.


Regards,

Damian Williamson


________________________________
From: Rhavar <rhavar@protonmail.com>
Sent: Saturday, 16 December 2017 3:38 AM
To: Damian Williamson
Cc: Bitcoin Protocol Discussion
Subject: Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transactio=
n Priority For Ordering Transactions In Blocks

> I understand that there would be technical issues to resolve in implement=
ation, but, are there no fundamental errors?

Unfortunately your proposal is really fundamentally broken, on a few levels=
. I think you might need to do a bit more research into how bitcoin works b=
efore coming up with such improvements =3D)

But just some quick notes:

* Every node has a (potentially) different mempool, you can't use it to dec=
ide consensus values like the max block size.

* Increasing the entropy in a block to make it more unpredictable doesn't r=
eally make sense.

* Bitcoin should be roughly incentive compatible. Your proposal explicits a=
sks miners to ignore their best interests, and confirm transactions by "pri=
ority".  What are you going to do if a "malicious" miner decides to go afte=
r their profits and order by what makes them the most money. Add "ordered b=
y priority" as a consensus requirement? And even if you miners can still so=
rt their mempool by fee, and then order the top 1MB by priority.

If you could find a good solution that would allow you to know if miners we=
re following your rule or not (and thus ignore it if it doesn't) then you w=
ouldn't even need bitcoin in the first place.




-Ryan


-------- Original Message --------
Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Pr=
iority For Ordering Transactions In Blocks
Local Time: December 15, 2017 3:42 AM
UTC Time: December 15, 2017 9:42 AM
From: bitcoin-dev@lists.linuxfoundation.org
To: Bitcoin Protocol Discussion <bitcoin-dev@lists.linuxfoundation.org>




I should not take it that the lack of critical feedback to this revised pro=
posal is a glowing endorsement. I understand that there would be technical =
issues to resolve in implementation, but, are there no fundamental errors?

I suppose that it if is difficult to determine how long a transaction has b=
een waiting in the pool then, each node could simply keep track of when a t=
ransaction was first seen. This may have implications for a verify routine,=
 however, for example, if a node was offline, how should it differentiate h=
ow long each transaction was waiting in that case? If a node was restarted =
daily would it always think that all transactions had been waiting in the p=
ool less than one day If each node keeps the current transaction pool in a =
file and updates it, as transactions are included in blocks and, as new tra=
nsactions appear in the pool, then that would go some way to alleviate the =
issue, apart from entirely new nodes. There should be no reason the content=
s of a transaction pool files cannot be shared without agreement as to the =
transaction pool between nodes, just as nodes transmit new transactions fre=
ely.

It has been questioned why miners could not cheat. For the question of how =
many transactions to include in a block, I say it is a standoff and miners =
will conform to the proposal, not wanting to leave transactions with valid =
fees standing, and, not wanting to shrink the transaction pool. In any case=
, if miners shrink the transaction pool then I am not immediately concerned=
 since it provides a more efficient service. For the question of including =
transactions according to the proposal, I say if it is possible to keep tra=
ck of how long transactions are waiting in the pool so that they can be inc=
luded on a probability curve then it is possible to verify that blocks conf=
orm to the proposal, since the input is a probability, the output should co=
nform to a probability curve.



If someone has the necessary skill, would anyone be willing to develop the =
math necessary for the proposal?

Regards,
Damian Williamson


________________________________

From: bitcoin-dev-bounces@lists.linuxfoundation.org <bitcoin-dev-bounces@li=
sts.linuxfoundation.org> on behalf of Damian Williamson via bitcoin-dev <bi=
tcoin-dev@lists.linuxfoundation.org>
Sent: Friday, 8 December 2017 8:01 AM
To: bitcoin-dev@lists.linuxfoundation.org
Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Pr=
iority For Ordering Transactions In Blocks



Good afternoon,

The need for this proposal:

We all must learn to admit that transaction bandwidth is still lurking as a=
 serious issue for the operation, reliability, safety, consumer acceptance,=
 uptake and, for the value of Bitcoin.

I recently sent a payment which was not urgent so; I chose three-day target=
 confirmation from the fee recommendation. That transaction has still not c=
onfirmed after now more than six days - even waiting twice as long seems qu=
ite reasonable to me. That transaction is a valid transaction; it is not ru=
bbish, junk or, spam. Under the current model with transaction bandwidth li=
mitation, the longer a transaction waits, the less likely it is ever to con=
firm due to rising transaction numbers and being pushed back by transaction=
s with rising fees.

I argue that no transactions are rubbish or junk, only some zero fee transa=
ctions might be spam. Having an ever-increasing number of valid transaction=
s that do not confirm as more new transactions with higher fees are created=
 is the opposite of operating a robust, reliable transaction system.

Business cannot operate with a model where transactions may or may not conf=
irm. Even a business choosing a modest fee has no guarantee that their vali=
d transaction will not be shuffled down by new transactions to the realm of=
 never confirming after it is created. Consumers also will not accept this =
model as Bitcoin expands. If Bitcoin cannot be a reliable payment system fo=
r confirmed transactions then consumers, by and large, will simply not acce=
pt the model once they understand. Bitcoin will be a dirty payment system, =
and this will kill the value of Bitcoin.

Under the current system, a minority of transactions will eventually be the=
 lucky few who have fees high enough to escape being pushed down the list.

Once there are more than x transactions (transaction bandwidth limit) every=
 ten minutes, only those choosing twenty-minute confirmation (2 blocks) wil=
l have initially at most a fifty percent chance of ever having their paymen=
t confirm. Presently, not even using fee recommendations can ensure a suffi=
ciently high fee is paid to ensure transaction confirmation.

I also argue that the current auction model for limited transaction bandwid=
th is wrong, is not suitable for a reliable transaction system and, is wron=
g for Bitcoin. All transactions must confirm in due time. Currently, Bitcoi=
n is not a safe way to send payments.

I do not believe that consumers and business are against paying fees, even =
high fees. What is required is operational reliability.

This great issue needs to be resolved for the safety and reliability of Bit=
coin. The time to resolve issues in commerce is before they become great bi=
g issues. The time to resolve this issue is now. We must have the foresight=
 to identify and resolve problems before they trip us over.  Simply doublin=
g block sizes every so often is reactionary and is not a reliable permanent=
 solution. I have written a BIP proposal for a technical solution but, need=
 your help to write it up to an acceptable standard to be a full BIP.

I have formatted the following with markdown which is human readable so, I =
hope nobody minds. I have done as much with this proposal as I feel that I =
am able so far but continue to take your feedback.

# BIP Proposal: UTPFOTIB - Use Transaction Priority For Ordering Transactio=
ns In Blocks

## The problem:
Everybody wants value. Miners want to maximize revenue from fees (and we pr=
esume, to minimize block size). Consumers need transaction reliability and,=
 (we presume) want low fees.

The current transaction bandwidth limit is a limiting factor for both. As t=
he operational safety of transactions is limited, so is consumer confidence=
 as they realize the issue and, accordingly, uptake is limited. Fees are ar=
tificially inflated due to bandwidth limitations while failing to provide a=
 full confirmation service for all transactions.

Current fee recommendations provide no satisfaction for transaction reliabi=
lity and, as Bitcoin scales, this will worsen.

Bitcoin must be a fully scalable and reliable service, providing full trans=
action confirmation for every valid transaction.

The possibility to send a transaction with a fee lower than one that is acc=
eptable to allow eventual transaction confirmation should be removed from t=
he protocol and also from the user interface.

## Solution summary:
Provide each transaction with an individual transaction priority each time =
before choosing transactions to include in the current block, the priority =
being a function of the fee paid (on a curve), and the time waiting in the =
transaction pool (also on a curve) out to n days (n=3D60 ?). The transactio=
n priority to serve as the likelihood of a transaction being included in th=
e current block, and for determining the order in which transactions are tr=
ied to see if they will be included.

Use a target block size. Determine the target block size using; current tra=
nsaction pool size x ( 1 / (144 x n days ) ) =3D number of transactions to =
be included in the current block. Broadcast the next target block size with=
 the current block when it is solved so that nodes know the next target blo=
ck size for the block that they are building on.

The curves used for the priority of transactions would have to be appropria=
te. Perhaps a mathematician with experience in probability can develop the =
right formulae. My thinking is a steep curve. I suppose that the probabilit=
y of all transactions should probably account for a sufficient number of in=
clusions that the target block size is met although, it may not always be. =
As a suggestion, consider including some zero fee transactions to pad, high=
est BTC value first?

**Explanation of the operation of priority:**
> If transaction priority is, for example, a number between one (low) and o=
ne-hundred (high) it can be directly understood as the percentage chance in=
 one-hundred of a transaction being included in the block. Using probabilit=
y or likelihood infers that there is some function of random. If random (10=
0) < transaction priority then the transaction is included.

>To break it down further, if both the fee on a curve value and the time wa=
iting on a curve value are each a number between one and one-hundred, a rud=
imentary method may be to simply multiply those two numbers, to find the pr=
iority number. For example, a middle fee transaction waiting thirty days (i=
f n =3D 60 days) may have a value of five for each part  (yes, just five, t=
he values are on a curve). When multiplied that will give a priority value =
of twenty-five, or,  a twenty-five percent chance at that moment of being i=
ncluded in the block; it will likely be included in one of the next four bl=
ocks, getting more likely each chance. If it is still not included then the=
 value of time waiting will be higher, making for more probability. A very =
low fee transaction would have a value for the fee of one. It would not be =
until near sixty-days that the particular low fee transaction has a high li=
kelihood of being included in the block.

I am not concerned with low (or high) transaction fees, the primary reason =
for addressing the issue is to ensure transactional reliability and scalabi=
lity while having each transaction confirm in due time.

## Pros:
* Maximizes transaction reliability.
* Fully scalable.
* Maximizes possibility for consumer and business uptake.
* Maximizes total fees paid per block without reducing reliability; because=
 of reliability, in time confidence and overall uptake are greater; therefo=
re, more transactions.
* Market determines fee paid for transaction priority.
* Fee recommendations work all the way out to 30 days or greater.
* Provides additional block entropy; greater security since there is less p=
robability of predicting the next block.

## Cons:
* Could initially lower total transaction fees per block.
* Must be first be programmed.

## Solution operation:
This is a simplistic view of the operation. The actual operation will need =
to be determined in a spec for the programmer.

1. Determine the target block size for the current block.
2. Assign a transaction priority to each transaction in the pool.
3. Select transactions to include in the current block using probability in=
 transaction priority order until the target block size is met.
5. Solve block.
6. Broadcast the next target block size with the current block when it is s=
olved.
7. Block is received.
8. Block verification process.
9. Accept/reject block based on verification result.
10. Repeat.

## Closing comments:
It may be possible to verify blocks conform to the proposal by showing that=
 the probability for all transactions included in the block statistically c=
onforms to a probability distribution curve, *if* the individual transactio=
n priority can be recreated. I am not that deep into the mathematics; howev=
er, it may also be possible to use a similar method to do this just based o=
n the fee, that statistically, the blocks conform to a fee distribution. An=
y zero fee transactions would have to be ignored. This solution needs a cle=
ver mathematician.

I implore, at the very least, that we use some method that validates full t=
ransaction reliability and enables scalability of block sizes. If not this =
proposal, an alternative.

Regards,
Damian Williamson



--_000_PS2P216MB0179544A6503C2992190CEB69D0B0PS2P216MB0179KORP_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-=
1">
<style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi=
n-bottom:0;} --></style>
</head>
<body dir=3D"ltr">
<div id=3D"divtagdefaultwrapper" style=3D"font-size: 12pt; color: rgb(0, 0,=
 0); font-family: Calibri,Helvetica,sans-serif,&quot;EmojiFont&quot;,&quot;=
Apple Color Emoji&quot;,&quot;Segoe UI Emoji&quot;,NotoColorEmoji,&quot;Seg=
oe UI Symbol&quot;,&quot;Android Emoji&quot;,EmojiSymbols;" dir=3D"ltr">
<p style=3D"margin-top:0;margin-bottom:0">There are really two separate pro=
blems to solve.</p>
<p style=3D"margin-top:0;margin-bottom:0"><br>
</p>
<ol style=3D"margin-bottom: 0px; margin-top: 0px;">
<li>How does Bitcoin scale with fixed block size?</li><li>How do we ensure =
that all valid transactions are eventually included in the blockchain?</li>=
</ol>
<br>
<p style=3D"margin-top:0;margin-bottom:0">Those are the two issues that the=
 proposal attempts to address. It makes sense to resolve these two problems=
 together. Using the proposed system for variable block sizes would solve t=
he first problem but there would still
 be a whole bunch of never confirming transactions. I am not sure how to re=
liably solve the second problem at scale without first solving the first.<b=
r>
</p>
<p style=3D"margin-top:0;margin-bottom:0"><br>
</p>
<p style=3D"margin-top:0;margin-bottom:0">&gt;* Every node has a (potential=
ly) different mempool, you can't use it to decide consensus values like the=
 max block size.&nbsp;<br>
</p>
<div><br>
I do not suggest a consensus. Depending on which node solves a block the va=
lue for next block size will be different. The consensus would be that bloc=
ks will adhere to the next block size value transmitted with the current bl=
ock. It is easy to verify that the
 consensus is being adhered to once in place.<br>
&nbsp;<br>
</div>
<div>&gt;* Increasing the entropy in a block to make it more unpredictable =
doesn't really make sense.&nbsp;</div>
<div><br>
Not a necessary function, just an effect of using a probability-based distr=
ibution.
<br>
<br>
</div>
<div>&gt;* Bitcoin should be roughly incentive compatible. Your proposal ex=
plicits asks miners to ignore their best interests, and confirm transaction=
s by &quot;priority&quot;.&nbsp; What are you going to do if a &quot;malici=
ous&quot; miner decides to go after their profits and order
 by what makes them the most money. Add &quot;ordered by priority&quot; as =
a consensus requirement? And even if you miners can still sort their mempoo=
l by fee, and then order the top 1MB by priority.<br>
<br>
I entirely agree with your sentiment that Bitcoin must be incentive compati=
ble. It is necessary.<br>
<br>
It is in only miners immediate interest to make the most profitable block f=
rom the available transaction pool. As with so many other things, it is nec=
essary to partially ignore short-term gain for long-term benefit. It is in =
miners and everybody's long-term
 interest to have a reliable transaction service. A busy transaction servic=
e that confirms lots of transactions per hour will become more profitable a=
s demand increases and more users are prepared to pay for priority. As it i=
s there is currently no way to fully
 scale because of the transaction bandwidth limit and that is problematic. =
If all valid transactions must eventually confirm then there must be a way =
to resolve that problem.<br>
<br>
Bitcoin deliberately removes traditional scale by ensuring blocks take ten =
minutes on average to solve, an ingenious idea and, incentive compatible bu=
t, fixed block sizes leaves us with a problem to solve when we want to scal=
e.<br>
<br>
</div>
<div>&gt;If you could find a good solution that would allow you to know if =
miners were following your rule or not (and thus ignore it if it doesn't) t=
hen you wouldn't even need bitcoin in the first place.<br>
<br>
I am confident that the math to verify blocks based on the proposal can be =
developed (and I think it will not be too complex for a mathematician with =
the relevant experience), however, I am nowhere near experienced enough wit=
h probability and statistical analysis
 to do it. Yes, if Bitcoin doesn't then it might make another great opportu=
nity for an altcoin but I am not even nearly interested in promoting any al=
tcoins.<br>
</div>
<p></p>
<p style=3D"margin-top:0;margin-bottom:0"><br>
</p>
<p style=3D"margin-top:0;margin-bottom:0">If not the proposal that I have p=
ut forward, then, hopefully, someone can come up with a better solution. Th=
e important thing is that the issues are resolved.<br>
</p>
<p style=3D"margin-top:0;margin-bottom:0"><br>
</p>
<p style=3D"margin-top:0;margin-bottom:0">Regards,</p>
<p style=3D"margin-top:0;margin-bottom:0">Damian Williamson<br>
</p>
<br>
<br>
<div style=3D"color: rgb(0, 0, 0);">
<hr style=3D"display:inline-block;width:98%" tabindex=3D"-1">
<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font style=3D"font-size:11pt" face=
=3D"Calibri, sans-serif" color=3D"#000000"><b>From:</b> Rhavar &lt;rhavar@p=
rotonmail.com&gt;<br>
<b>Sent:</b> Saturday, 16 December 2017 3:38 AM<br>
<b>To:</b> Damian Williamson<br>
<b>Cc:</b> Bitcoin Protocol Discussion<br>
<b>Subject:</b> Re: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Tra=
nsaction Priority For Ordering Transactions In Blocks</font>
<div>&nbsp;</div>
</div>
<div>
<div>&gt;&nbsp;I understand that there would be technical issues to resolve=
 in implementation, but, are there no fundamental errors?<br>
</div>
<div><br>
</div>
<div>Unfortunately your proposal is really fundamentally broken, on a few l=
evels. I think you might need to do a bit more research into how bitcoin wo=
rks before coming up with such improvements =3D)<br>
</div>
<div><br>
</div>
<div>But just some quick notes:<br>
</div>
<div><br>
</div>
<div>* Every node has a (potentially) different mempool, you can't use it t=
o decide consensus values like the max block size.&nbsp;<br>
</div>
<div><br>
</div>
<div>* Increasing the entropy in a block to make it more unpredictable does=
n't really make sense.&nbsp;</div>
<div><br>
</div>
<div>* Bitcoin should be roughly incentive compatible. Your proposal explic=
its asks miners to ignore their best interests, and confirm transactions by=
 &quot;priority&quot;.&nbsp; What are you going to do if a &quot;malicious&=
quot; miner decides to go after their profits and order by
 what makes them the most money. Add &quot;ordered by priority&quot; as a c=
onsensus requirement? And even if you miners can still sort their mempool b=
y fee, and then order the top 1MB by priority.<br>
</div>
<div><br>
</div>
<div>If you could find a good solution that would allow you to know if mine=
rs were following your rule or not (and thus ignore it if it doesn't) then =
you wouldn't even need bitcoin in the first place.</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div class=3D"x_protonmail_signature_block">
<div class=3D"x_protonmail_signature_block-user">
<div>-Ryan<br>
</div>
</div>
<div class=3D"x_protonmail_signature_block-proton x_protonmail_signature_bl=
ock-empty">
<br>
</div>
</div>
<div><br>
</div>
<blockquote class=3D"x_protonmail_quote" type=3D"cite">
<div>-------- Original Message --------<br>
</div>
<div>Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transacti=
on Priority For Ordering Transactions In Blocks<br>
</div>
<div>Local Time: December 15, 2017 3:42 AM<br>
</div>
<div>UTC Time: December 15, 2017 9:42 AM<br>
</div>
<div>From: bitcoin-dev@lists.linuxfoundation.org<br>
</div>
<div>To: Bitcoin Protocol Discussion &lt;bitcoin-dev@lists.linuxfoundation.=
org&gt;<br>
</div>
<div><br>
</div>
<div><br>
</div>
<div dir=3D"ltr" style=3D"font-size:12pt; color:rgb(0,0,0); font-family:Cal=
ibri,Helvetica,sans-serif,&quot;EmojiFont&quot;,&quot;Apple Color Emoji&quo=
t;,&quot;Segoe UI Emoji&quot;,NotoColorEmoji,&quot;Segoe UI Symbol&quot;,&q=
uot;Android Emoji&quot;,EmojiSymbols">
<p style=3D"margin-top:0; margin-bottom:0"><br>
</p>
<div>
<div>I should not take it that the lack of critical feedback to this revise=
d proposal is a glowing endorsement. I understand that there would be techn=
ical issues to resolve in implementation, but, are there no fundamental err=
ors?<br>
</div>
<div><br>
</div>
<div>I suppose that it if is difficult to determine how long a transaction =
has been waiting in the pool then, each node could simply keep track of whe=
n a transaction was first seen. This may have implications for a verify rou=
tine, however, for example, if a
 node was offline, how should it differentiate how long each transaction wa=
s waiting in that case? If a node was restarted daily would it always think=
 that all transactions had been waiting in the pool less than one day If ea=
ch node keeps the current transaction
 pool in a file and updates it, as transactions are included in blocks and,=
 as new transactions appear in the pool, then that would go some way to all=
eviate the issue, apart from entirely new nodes. There should be no reason =
the contents of a transaction pool
 files cannot be shared without agreement as to the transaction pool <span>=
between nodes</span>, just as nodes transmit new transactions freely.<br>
</div>
<div><br>
</div>
<div>It has been questioned why miners could not cheat. For the question of=
 how many transactions to include in a block, I say it is a standoff and mi=
ners will conform to the proposal, not wanting to leave transactions with v=
alid fees standing, and, not wanting
 to shrink the transaction pool. In any case, if miners shrink the transact=
ion pool then I am not immediately concerned since it provides a more effic=
ient service. For the question of including transactions according to the p=
roposal, I say if it is possible
 to keep track of how long transactions are waiting in the pool so that the=
y can be included on a probability curve then it is possible to verify that=
 blocks conform to the proposal, since the input is a probability, the outp=
ut should conform to a probability
 curve.<br>
</div>
</div>
<div><br>
</div>
<p><br>
</p>
<div>If someone has the necessary skill, would anyone be willing to develop=
 the math necessary for the proposal?<br>
</div>
<div><br>
</div>
<div>Regards,<br>
</div>
<div>Damian Williamson<br>
</div>
<div><br>
</div>
<div><br>
</div>
<div style=3D"color:rgb(0,0,0)">
<div>
<hr tabindex=3D"-1" style=3D"display:inline-block; width:98%">
<br>
</div>
<div dir=3D"ltr">
<div><span class=3D"x_font" style=3D"font-family:Calibri,sans-serif"><span =
class=3D"x_colour" style=3D"color:rgb(0,0,0)"><b>From:</b> bitcoin-dev-boun=
ces@lists.linuxfoundation.org &lt;bitcoin-dev-bounces@lists.linuxfoundation=
.org&gt; on behalf of Damian Williamson via bitcoin-dev
 &lt;bitcoin-dev@lists.linuxfoundation.org&gt;<br>
<b>Sent:</b> Friday, 8 December 2017 8:01 AM<br>
<b>To:</b> bitcoin-dev@lists.linuxfoundation.org<br>
<b>Subject:</b> [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transac=
tion Priority For Ordering Transactions In Blocks</span></span></div>
<div>&nbsp;<br>
</div>
</div>
<div dir=3D"ltr">
<div dir=3D"ltr" style=3D"font-size:12pt; color:rgb(0,0,0); font-family:Cal=
ibri,Helvetica,sans-serif,&quot;EmojiFont&quot;,&quot;Apple Color Emoji&quo=
t;,&quot;Segoe UI Emoji&quot;,NotoColorEmoji,&quot;Segoe UI Symbol&quot;,&q=
uot;Android Emoji&quot;,EmojiSymbols">
<p style=3D"margin-top:0; margin-bottom:0"><br>
</p>
<div>
<div>Good afternoon,<br>
</div>
<div><br>
</div>
<div>The need for this proposal:<br>
</div>
<div><br>
</div>
<div>We all must learn to admit that transaction bandwidth is still lurking=
 as a serious issue for the operation, reliability, safety, consumer accept=
ance, uptake and, for the value of Bitcoin.<br>
</div>
<div><br>
</div>
<div>I recently sent a payment which was not urgent so; I chose three-day t=
arget confirmation from the fee recommendation. That transaction has still =
not confirmed after now more than six days - even waiting twice as long see=
ms quite reasonable to me. That
 transaction is a valid transaction; it is not rubbish, junk or, spam. Unde=
r the current model with transaction bandwidth limitation, the longer a tra=
nsaction waits, the less likely it is ever to confirm due to rising transac=
tion numbers and being pushed back
 by transactions with rising fees.<br>
</div>
<div><br>
</div>
<div>I argue that no transactions are rubbish or junk, only some zero fee t=
ransactions might be spam. Having an ever-increasing number of valid transa=
ctions that do not confirm as more new transactions with higher fees are cr=
eated is the opposite of operating
 a robust, reliable transaction system.<br>
</div>
<div><br>
</div>
<div>Business cannot operate with a model where transactions may or may not=
 confirm. Even a business choosing a modest fee has no guarantee that their=
 valid transaction will not be shuffled down by new transactions to the rea=
lm of never confirming after it
 is created. Consumers also will not accept this model as Bitcoin expands. =
If Bitcoin cannot be a reliable payment system for confirmed transactions t=
hen consumers, by and large, will simply not accept the model once they und=
erstand. Bitcoin will be a dirty
 payment system, and this will kill the value of Bitcoin.<br>
</div>
<div><br>
</div>
<div>Under the current system, a minority of transactions will eventually b=
e the lucky few who have fees high enough to escape being pushed down the l=
ist.<br>
</div>
<div><br>
</div>
<div>Once there are more than x transactions (transaction bandwidth limit) =
every ten minutes, only those choosing twenty-minute confirmation (2 blocks=
) will have initially at most a fifty percent chance of ever having their p=
ayment confirm. Presently, not even
 using fee recommendations can ensure a sufficiently high fee is paid to en=
sure transaction confirmation.<br>
</div>
<div><br>
</div>
<div>I also argue that the current auction model for limited transaction ba=
ndwidth is wrong, is not suitable for a reliable transaction system and, is=
 wrong for Bitcoin. All transactions must confirm in due time. Currently, B=
itcoin is not a safe way to send
 payments.<br>
</div>
<div><br>
</div>
<div>I do not believe that consumers and business are against paying fees, =
even high fees. What is required is operational reliability.<br>
</div>
<div><br>
</div>
<div>This great issue needs to be resolved for the safety and reliability o=
f Bitcoin. The time to resolve issues in commerce is before they become gre=
at big issues. The time to resolve this issue is now. We must have the fore=
sight to identify and resolve problems
 before they trip us over.&nbsp; Simply doubling block sizes every so often=
 is reactionary and is not a reliable permanent solution. I have written a =
BIP proposal for a technical solution but, need your help to write it up to=
 an acceptable standard to be a full
 BIP.<br>
</div>
<div><br>
</div>
<div>I have formatted the following with markdown which is human readable s=
o, I hope nobody minds. I have done as much with this proposal as I feel th=
at I am able so far but continue to take your feedback.<br>
</div>
<div><br>
</div>
<div># BIP Proposal: UTPFOTIB - Use Transaction Priority For Ordering Trans=
actions In Blocks<br>
</div>
<div><br>
</div>
<div>## The problem:<br>
</div>
<div>Everybody wants value. Miners want to maximize revenue from fees (and =
we presume, to minimize block size). Consumers need transaction reliability=
 and, (we presume) want low fees.<br>
</div>
<div><br>
</div>
<div>The current transaction bandwidth limit is a limiting factor for both.=
 As the operational safety of transactions is limited, so is consumer confi=
dence as they realize the issue and, accordingly, uptake is limited. Fees a=
re artificially inflated due to
 bandwidth limitations while failing to provide a full confirmation service=
 for all transactions.<br>
</div>
<div><br>
</div>
<div>Current fee recommendations provide no satisfaction for transaction re=
liability and, as Bitcoin scales, this will worsen.<br>
</div>
<div><br>
</div>
<div>Bitcoin must be a fully scalable and reliable service, providing full =
transaction confirmation for every valid transaction.<br>
</div>
<div><br>
</div>
<div>The possibility to send a transaction with a fee lower than one that i=
s acceptable to allow eventual transaction confirmation should be removed f=
rom the protocol and also from the user interface.<br>
</div>
<div><br>
</div>
<div>## Solution summary:<br>
</div>
<div>Provide each transaction with an individual transaction priority each =
time before choosing transactions to include in the current block, the prio=
rity being a function of the fee paid (on a curve), and the time waiting in=
 the transaction pool (also on a
 curve) out to n days (n=3D60 ?). The transaction priority to serve as the =
likelihood of a transaction being included in the current block, and for de=
termining the order in which transactions are tried to see if they will be =
included.
<br>
</div>
<div><br>
</div>
<div>Use a target block size. Determine the target block size using; curren=
t transaction pool size x ( 1 / (144 x n days ) ) =3D number of transaction=
s to be included in the current block. Broadcast the next target block size=
 with the current block when it is
 solved so that nodes know the next target block size for the block that th=
ey are building on.<br>
</div>
<div><br>
</div>
<div>The curves used for the priority of transactions would have to be appr=
opriate. Perhaps a mathematician with experience in probability can develop=
 the right formulae. My thinking is a steep curve. I suppose that the proba=
bility of all transactions should
 probably account for a sufficient number of inclusions that the target blo=
ck size is met although, it may not always be. As a suggestion, consider in=
cluding some zero fee transactions to pad, highest BTC value first?<br>
</div>
<div><br>
</div>
<div>**Explanation of the operation of priority:**<br>
</div>
<div>&gt; If transaction priority is, for example, a number between one (lo=
w) and one-hundred (high) it can be directly understood as the percentage c=
hance in one-hundred of a transaction being included in the block. Using pr=
obability or likelihood infers that
 there is some function of random. If random (100) &lt; transaction priorit=
y then the transaction is included.<br>
</div>
<div><br>
</div>
<div>&gt;To break it down further, if both the fee on a curve value and the=
 time waiting on a curve value are each a number between one and one-hundre=
d, a rudimentary method may be to simply multiply those two numbers, to fin=
d the priority number. For example,
 a middle fee transaction waiting thirty days (if n =3D 60 days) may have a=
 value of five for each part&nbsp; (yes, just five, the values are on a cur=
ve). When multiplied that will give a priority value of twenty-five, or,&nb=
sp; a twenty-five percent chance at that moment
 of being included in the block; it will likely be included in one of the n=
ext four blocks, getting more likely each chance. If it is still not includ=
ed then the value of time waiting will be higher, making for more probabili=
ty. A very low fee transaction would
 have a value for the fee of one. It would not be until near sixty-days tha=
t the particular low fee transaction has a high likelihood of being include=
d in the block.<br>
</div>
<div><br>
</div>
<div>I am not concerned with low (or high) transaction fees, the primary re=
ason for addressing the issue is to ensure transactional reliability and sc=
alability while having each transaction confirm in due time.<br>
</div>
<div><br>
</div>
<div>## Pros:<br>
</div>
<div>* Maximizes transaction reliability.<br>
</div>
<div>* Fully scalable.<br>
</div>
<div>* Maximizes possibility for consumer and business uptake.<br>
</div>
<div>* Maximizes total fees paid per block without reducing reliability; be=
cause of reliability, in time confidence and overall uptake are greater; th=
erefore, more transactions.<br>
</div>
<div>* Market determines fee paid for transaction priority.<br>
</div>
<div>* Fee recommendations work all the way out to 30 days or greater.<br>
</div>
<div>* Provides additional block entropy; greater security since there is l=
ess probability of predicting the next block.<br>
</div>
<div><br>
</div>
<div>## Cons:<br>
</div>
<div>* Could initially lower total transaction fees per block.<br>
</div>
<div>* Must be first be programmed.<br>
</div>
<div><br>
</div>
<div>## Solution operation:<br>
</div>
<div>This is a simplistic view of the operation. The actual operation will =
need to be determined in a spec for the programmer.<br>
</div>
<div><br>
</div>
<div>1. Determine the target block size for the current block.<br>
</div>
<div>2. Assign a transaction priority to each transaction in the pool.<br>
</div>
<div>3. Select transactions to include in the current block using probabili=
ty in transaction priority order until the target block size is met.<br>
</div>
<div>5. Solve block.<br>
</div>
<div>6. Broadcast the next target block size with the current block when it=
 is solved.<br>
</div>
<div>7. Block is received.<br>
</div>
<div>8. Block verification process.<br>
</div>
<div>9. Accept/reject block based on verification result.<br>
</div>
<div>10. Repeat.<br>
</div>
<div><br>
</div>
<div>## Closing comments:<br>
</div>
<div>It may be possible to verify blocks conform to the proposal by showing=
 that the probability for all transactions included in the block statistica=
lly conforms to a probability distribution curve, *if* the individual trans=
action priority can be recreated.
 I am not that deep into the mathematics; however, it may also be possible =
to use a similar method to do this just based on the fee, that statisticall=
y, the blocks conform to a fee distribution. Any zero fee transactions woul=
d have to be ignored. This solution
 needs a clever mathematician.<br>
</div>
<div><br>
</div>
<div>I implore, at the very least, that we use some method that validates f=
ull transaction reliability and enables scalability of block sizes. If not =
this proposal, an alternative.<br>
</div>
<div><br>
</div>
<div>Regards,<br>
</div>
<div>Damian Williamson<br>
</div>
</div>
<p><br>
</p>
</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
</div>
</div>
</div>
</body>
</html>

--_000_PS2P216MB0179544A6503C2992190CEB69D0B0PS2P216MB0179KORP_--