Let’s talk about SegWit. While I never was a fan of SegWit, the Hong Kong agreement seemed like a reasonable enough compromise to me to go along with it. Now that Bitcoin Core decided to betray the community by not abiding by its agreement, it is time for me to express why SegWit is not great.
What is SegWit ?
SegWit is short for segregated witness. It is an upgrade to the Bitcoin network to fix transaction malleability by separating the transaction data – description of the transaction itself, and the witness data – cryptographic proof that the rightful owner of the funds is doing the transaction. While this is the main raison d’être of SegWit, it also comes with a series of improvements such as a solution to the quadratic hashing problem, the possibility to discard witness data for old transactions to speedup new node synchronization, a capacity improvement, and a few other things.
And these changes are improvements. They actually do make Bitcoin better.
But SegWit chose to do these changes as a soft fork. It means that the changes are done in such a way that software that is not updated will simply go along with it. In order to achieve this, SegWit uses a technique known as extension block.
SegWit continues to make use of a block that has the same structure as current Bitcoin blocks, and that old software will accept as valid. In addition, it creates a block extension in which the protocol is updated. Updated software, that knows where to look will be able to understand this block extension and validate it. Older software will be unaware of the existence of this extension. In this extension, the rules can be changed in various ways to provide new features.
Currently, when Alice wants to send money to Bob, Alice creates a transaction transferring coins she owns to one of Bob’s addresses. In order to prove Alice is the owner of the coins, she will have to provide a signature that proves she is indeed the owner of the coins.
The transaction in green in the picture above transfers coins from Alice to Bob. The transaction in orange was some previous transaction that transferred coins to Alice, as Alice needs coin to be able to spend them.
As we mentioned before, SegWit creates an extension block and moves parts of the data there so it can be subjected to new rules. The same transaction between Alice and Bob using SegWit would looks fairly different:
To be able to use SegWit, Alice needs to use a new kind of address. This new kind of address allows her to move her signature into the extension block. However, the data in the regular block needs to be accepted by software that is not upgraded for SegWit to be a soft fork, which means Alice still needs to put an empty signature in the regular block. It is also necessary that the format of the new address looks like it can be spent by anyone. As far as this software is concerned, no signature will be provided.
The old software see transactions that are always valid, stripped of all their security elements and do not understand SegWit addresses. As a result, they are essentially “zombies” on the network.
Soft fork or hard fork ?
When making any change to the Bitcoin protocol, it is important to keep in mind the disruption it creates for the ecosystem. For simple changes, a soft fork is often preferable. For instance, BIP146 is a good fit for a soft fork. It adds an extra validity rule, one that is very easy to implement by most software, but that also wouldn’t completely disable software that isn’t updated yet.
On the other hand, SegWit is essentially a hard fork disguised as a soft fork. It strips the regular block out of most meaningful information and moves it to the extension block. While software that isn’t updated to support SegWit will still accept the blockchain, it has lost all ability to actually understand and validate it.
An old wallet won’t understand if its owner is being sent money. It won’t be able able to spend it. A node is unable to validate the transactions in the blockchain as they all look valid no matter what. Overall, while SegWit can be technically qualified as a soft fork, it puts anyone who does not upgrade at risk.
While SegWit provide various benefits, the most urgent one is probably a capacity upgrade. So let’s looks at what SegWit delivers on that front.
Because SegWit is a soft fork, the size of the regular block cannot be increased, only the extension block can be extended. Therefore, the capacity can only be increased to the extent that transactions without witness data will fit in the 1Mb regular block size. An analysis of current usage patterns of users reveals that SegWit will deliver to them the equivalent of a 1.7Mb block size. Transactions contain only so much witness data that can be stripped out.
On the other hand, it is possible to create transactions that contain an extraordinary amount of witness data. Most users do not care about it, as they do not wish to do this kind of transactions, but a node operator or a miner needs to be ready to absorb them. In other words, miners and nodes needs to be ready to absorb blocks that are up to 4Mb in size.
If we compare to a block size increase to 2Mb, it becomes apparent that the value delivered to users is inferior at only 1.7Mb. On the other hand, the cost for miners and nodes is higher as they need to be able to process 4Mb blocks. But it doesn’t stop there. As we discussed, SegWit has a significant impact on the ecosystem. However, a large portion of the ecosystem would not be affected by a change in the block size. In fact, only miners and nodes are affected, and only in a minor way.
The primary reason SegWit was created is to solve transaction malleability. Currently, it is possible to modify a transaction in such a way that its id changes, but the transaction remains valid. This is an issue for various reasons that are fairly technical, but it is indeed worth fixing.
To do so, SegWit effectively creates a new transaction format which doesn’t suffer from the problem. However, the constraint is that the new format must fit in the old one. For this reason, anyone can spend transaction are created, and all transaction have an empty signature where none is needed. Overall, this waste space and makes the transaction format bigger and more complex than it needs to be.
Once again, it is possible to introduce a new format that does not suffer from malleability but does not introduce unnecessary complexity like a new kind of address. Software that needs to understand these transaction needs to be updated either way so the disruption is no greater doing it as a hard fork. However, it avoids creating technical debt. Both formats can coexist for a while until the old format is phased out. Such a proposal exists in the name of Flexible Transaction and, while its current implementation suffers from various flaws, it’s very promising.
The new transaction format could be introduced with a variation of BIP143 which is a solution to the quadratic hashing problem. Not only would this solve the problem for the new transaction format like SegWit does, but it would also solve it when spending existing outputs as they use the same kind of addresses.
Ironically, the solution introduced in SegWit is probably the best one, however, the potential of this specific aspect of SegWit is limited by other aspects of it.
SegWit is overly complex
We know established that SegWit solves useful problems, but in an inferior way in order to be a soft fork. Yet another problem is that it does it all at once. A very dishonest strategy employed by SegWit supporters is to point out quadratic hashing when shown that SegWit is an inferior scaling solution. They will then point at scaling once you start discussing quadratic hashing. If you nail both they’ll divert to malleability, essentially shifting the goalpost every time.
Ironically, this is a symptom of a deeper problem in SegWit: it tries to solve all problems at once. There are a few best practices in software engineering principles known as the Single Responsibility Principle, or SRP and Separation of Concerns. In short, when a piece of code does many things at once, it harder to understand and work with, it is more likely to contain bugs, and generally more risky – which is not exactly desirable in a software that runs a $10B market.
The reason SegWit does everything at once is, once again, soft fork. The extension block technique used in SegWit essentially allows to disguise a hard fork into a soft fork. But once the extension block is out there and and being validated, it becomes impossible to change it in some way without a hard fork, or … a block extension extension. Inception is a great movie, but isn’t a good inspiration for software engineering. Because one doesn’t want to nest extension block ad infinitum, the temptation is there to put as much as possible in each. And this is how you get the kitchen sink that is SegWit.
SegWit’s complexity already claimed a victim, in the name of Compact Block. The first release had to be shipped without support for SegWit because of the extra complexity involved, even though both have been promoted by the same developers. They have hopefully been able to fix it since then, but that’s good sign of how disruptive it is.
There is a very well known problem in economics called the Economic Calculation Problem. It is essentially the economic equivalent of Heisenberg’s uncertainty principle. It state that when setting a value in stone, not only you are very likely to chose a value that isn’t optimal because you don’t have all the information required to do so, but worse, you’ll destroy part of this information in the process.
This is valid for the block size. Now that blocks are full, the ecosystem is adapting to work with that constraint. Because it does so, it becomes harder and harder to figure out what the right block size actually is.
In the same vein, SegWit introduce an economic incentive to produce more witness data than non witness data. It is argued by SegWit supporters that it is an incentive to reduce the UTXO set growth. It may well be, but just as the block size limit does, it is going to destroy the economic information required to price the UTXO appropriately.
This is an economic disaster in the making.
The worst part being, this central planning measure is not enough to create the incentive required to not bloat the UTXO. See this exchange at Scaling Bitcoin 2016 in Milan.
All in all, SegWit is not great. It solves useful problems, but doesn’t do it in the best possible way and there is no good engineering or economic reason to pursue that road. I can only assume the motivation are political in nature.
It was a piece of code I was willing to accept as a compromise as stipulated in the Hong Kong agreement but if its promoters are not willing live up to their side of the bargain, I am forced now to reject it.