In partnership with Zeppelin Solutions, the Coinbase and Coinbase Custody security teams found a critical vulnerability in MKR voting contracts. In the piece below, we discuss the steps we took to identify the issue and how we worked closely with our partners and the MKR team to remediate the vulnerability—leading to no loss of customer funds.
This is a companion piece to Zeppelin Solutions’ own technical audit.
Coinbase regularly conducts security audits of smart contracts we write, or assets we list or are evaluating for listing. Most of the time we come away with minor, if any, concerns. Every now and then, we run across something we feel needs to be disclosed directly to the project but is of a relatively low severity. Very, very rarely we’ll discover an issue we believe poses a critical risk to an asset. Our recent review of MakerDAO voting contracts is one such example.
Below, we share the process we followed to identify and remediate the issue and ensure that all participants in this ecosystem were protected. We’ve also extracted a few specific things we think are worth learning from for anyone building a blockchain-based asset. It’s worth highlighting right up front that we’re not the only organization that takes on this kind of effort. There are a number of examples of asset audits and responsible disclosure across the industry, with varying results.
This story starts with smart contracts. We’ve historically stayed away from smart contracts as part of our infrastructure as we see the smart contract ecosystem as still fairly young. While there are some amazing innovations going on in the realm of formal verification and security tooling for smart contracts, we don’t see the same level of toolchain maturity as we’d expect to see in a traditional programming language. With Coinbase Custody’s push to provide governance services to its clients, however, along with our increased pace of listing smart contract-based assets, smart contract security expertise became something we had to develop, and we’ve done some through a blend of external partnerships and internal expertise.
In both our asset listing process and before any internal use of smart contracts, we put a number of safety measures in place, one of which is requiring a security review of all production smart contracts. As part of our project to integrate MakerDAO voting directly with our cold storage system, we built a custom VoteProxy smart contract. Once we had what we thought was a good version, we submitted it to one of our external audit partners, Zeppelin, and made sure the review scope included the inter-contract interactions in the MakerDAO voting ecosystem (mainly, the interactions between our contract and the DS-Chief contract). We expected to get a few suggestions back and to work with the Zeppelin team to implement fixes and move forward. This time around, however, we got a touch more than we’d expected. A key takeaway here is that we strongly encourage anyone writing production smart contract code to make comprehensive third-party security review a regular part of your development lifecycle. If you’re building an asset, we’d encourage you to go one step farther and make those reviews public. One of the key inputs we look for when evaluating assets for listing on Coinbase is a regular cadence of 3rd party audits from credible firms.
We knew something unusual was happening when Zeppelin scheduled an unplanned check-in. At this point, they briefly let us know they’d found a critical bug in MakerDAO voting. We reached out to the MakerDAO team and we all got on a call together within hours of the initial findings. Of note here, and a suggestion for other companies in similar situations, Zeppelin didn’t disclose vulnerability details to us until we’d coordinated a call and had the MakerDAO team on the line. This ensured that all parties had equal access to information and that the interests of all parties were protected.
On a video conference with both the Coinbase and MakerDao security teams, Zeppelin went over the vulnerability details, shared example exploit code, clarified a number of assumptions and proposed some mitigations. The MakerDAO team had the right people on the line to evaluate the report and start to dive into the technical details. We established a joint communications channel and set a next check in time, and the MakerDAO team went off to explore the report in detail. Once the MakerDAO team was ready to propose a had a path forward, we convened again and provided feedback. This is a good example of a positive, engaged response to a vulnerability disclosure. If you are writing code that touches money in an environment as new as a smart contract language, you should fully expect to be on the receiving end of critical vulnerability reports. Build a policy and plan. Conduct tabletops. Publish your plan. Make sure you have an easy, secure vulnerability reporting mechanism. Conduct more tabletops.
With a shared game plan in place, the teams set out to develop a fix. On the Coinbase Custody side, that included making sure we had communications ready to go for all of our clients, that we’d added detection for this activity in our blockchain monitoring tool (the same tool that detected the ETC 51% attack) and that we actively prevented customer harm by blacklisting the old contract addresses and delaying the rollout of our MakerDAO voting feature.
There were a couple of catches, however.
First, as soon as MakerDAO announced that a critical vulnerability had been found and pointed folks at the new DS-Chief contract, it would only be a matter of time until folks had de-compiled the new contract and reverse-engineered the vulnerability. At the same time, MakerDAO couldn’t force existing network participants to withdraw their MKR from the old vulnerable smart contract. This put network participants at risk of loss if an attacker had started to attack the old contract before all network participants had withdrawn their MKR. However, the MakerDAO team was able to come up with a suite of mitigations that would significantly reduce the impact of any active exploitation. We think this is a fairly interesting corner case in vulnerability management in this kind of environment. On the blockchain, all code (at least, all bytecode) is public and unlike the traditional software development world (e.g. when Microsoft ships patches) smart contract patches are going to tend to be smaller and less obfuscatable. We need to assume that the gap between patch and general vulnerability discovery is going to be very short and build our vulnerability response plans with that in mind.
Second, MakerDAO had released the vulnerable smart contract as open source. If other projects had picked up and were using that smart contract, they could also be vulnerable to the same issue. This, unfortunately, is a common problem in open source software development without a great general solution. We do see point solutions in specific ecosystems, for example, Ruby’s bundler-audit, where there are more robust library systems. On the other hand, open source software can be safely leveraged when integrating extensively used and battle-tested libraries such as OpenZeppelin. It is our hope that, as the smart contract ecosystem matures, we see more mature library systems evolve that will enable effective dependency monitoring and security alerting.
In the end, MakerDAO was able to ship a new contract, get network participants moved over and avoid any loss. This was only possible because of the outstanding work in discovering the vulnerability by Zeppelin and the rapid, collaborative involvement of all three parties in reviewing and addressing the issue. This should serve as an example of how to handle a critical vulnerability disclosure in any industry. If you’re involved with writing production smart contract code, I’d encourage you to think through the scenario above in the context of your project and ask yourself how your team would respond? Explore the corner cases, the edge scenarios and see where your plan could use some tuning. Don’t have a plan? No better time to start writing one than right now.