The State of the Ethereum Blockchain with Andrew Koller

0:00Ethereum is a decentralized blockchain platform that was created by Vitalikbuterin and Gavin Wood in 2015. It uses a cryptocurrency called Ether as its native token to power transactions and operations on the Ethereum network. Ethereum's proponents envision a future, where the network forms the foundation for a second platform layer called L2, where decentralized applications are run. As we approach the 10th anniversary of Ethereum's creation, we wanted to understand the state of the technology, so we spoke with Andrew Collar, who is an engineer at Kraken, which is a software company and popular cryptocurrency exchange.

0:38In this conversation, Andrew talks about Kraken, security considerations at an exchange, the history of Ethereum, L2, and the future of Ethereum. This episode is hosted by Sean Falconer. Check the show notes for more information on Sean's work and where to find him.

1:06Andrew, welcome to the show. Hey, Sean. Thanks for having me. Yeah, absolutely. Great to meet you. Thanks for being here. Excited to talk about this. So Kraken's one of the world's largest leading cryptocurrency exchanges. But I think for those that are in that market and steeped in the world of blockchain, they might not know that much about it. So to start off, can you tell me a little bit about Kraken? What is it? How is it unique? Yeah, absolutely. So, you know, cryptocurrency has gone through many, many cycles, right? You see it in the news when the price is going up. You also see in the news when the price is going down and everybody has these articles that come out that are, hey, I told you so, it's not going to work.

1:46And then, you know, two years later, it starts going up again. So I think with any kind of emerging technology, AI included, there's all these kinds of hype cycles around it of like, okay, we've pushed the limits of the tech. Everybody's been going nuts on improving efficiency. And then all of a sudden, we kind of crash back to earth of like, okay, what's the use case? Think of the machine learning in the 2010 time where it's like the hot thing. And then there's kind of like a cool off period of, okay, actually, where do we apply all of this? And now with Gen AI and all the recent resurgence, it's clear that there's applications.

2:21Maybe this will be another bubble. I don't know. So just a little precursor to, you know, cryptocurrency and kind of like the cycles of it. But Kraken, we were founded back, I believe it was around 2013, 2014. And it was after this exchange that had not really great security practices in Japan called Mt. Gox collapsed and they kind of became insolvent. So our co-founders, one of them being Jesse Powell, started Kraken as kind of a response to that of like, let's bring some trust into the cryptocurrency space and make a very secure exchange where people can bring the enthusiasm that they have for this emerging kind of like distributed ledger technology and be able to use a trusted platform that they can conduct their trades on.

3:03Over time, we've evolved that to getting into basically doing everything that you would do with your finances in a daily life and have very ambitious goals to be kind of like a one-stop shop platform for finance, but with a heavy emphasis on crypto. So yeah, that's kind of cracking in a nutshell in its place. We're second in the US, we're leading in Europe among kind of like the big regulated exchanges. And it's a wild ride. Yeah, I would think that being part of any sort of startup and technology company, especially going through growth phases, has a lot of ups and downs. And I think crypto, at least like as an outsider, I would think that those things are even more amplified just because the swings are even more wild than even sort of like a conventional startup.

3:44Yeah. In the crypto space, we always say that bear markets are for building. And right now we're kind of in a bull market, like Bitcoin is hovering around$100 ,000. And we are still building, but it's difficult as in the crypto space, especially when you're kind of like a small startup to not pay attention to the hype that's going on when you're just trying to build like cool technology and good products. So there's a little bit of distraction that happens, but But typically, like, yeah, when there's bear markets, you just kind of, you huddle up, you shelter, and you don't care about price. You just push the technology to its limits because people that work in this space get very fanatical about it because we believe in it.

4:21And then you mentioned with the Japanese exchange that there were some challenges around, like, safety and trust. And that's been a big area of focus for Kraken. But given that, like, at least my understanding with blockchain, like, you know, the advantage of sort of decentralization, immutability, you know, crypto security, all these things is sort of like the focus has been sort of safety and trust. So why was that a problem? Like, what was sort of the disconnect there? For a centralized exchange, you are still taking deposits on behalf of clients, right? So even though they are using a decentralized ledger like Bitcoin or Ethereum, where you don't have to worry about your funds being compromised because you fully own and self-custody your private key.

5:02If you want to go do an on and off ramp, you have to take that to usually a centralized entity. Because no matter what, even though cryptocurrencies allow you to kind of do your entire kind of family's wealth with, you know, just at your desk without a middleman. If you are plugging back into the traditional banking system, you do still have to do, you know, typical AML, KYC laws and make sure that you're adhering to it. All the jurisdictions that have their all kind of, you know, little flavors of how they handle AML and KYC. So that's like kind of where the trust comes in. If I had five Bitcoin and I want to get some USD for it because I have to go pay my mortgage or something, then I'm going to go to a trusted on-ramp, in my case, Kraken, and make sure I'm all KYC'd.

5:44And then when Kraken takes my Bitcoin, it's custody under our own secure kind of private key storage and infrastructure. So that's like where some early exchanges may have gotten that wrong. And I don't necessarily blame them because cryptocurrency was in an infancy, right? It was, you know, you could generate a private key with just running a command on your terminal and you see the output of the private key and you think, OK, cool, this is secure. But, you know, that's why we've evolved to have some kind of so many offline private key generation, a lot more advancements in cryptography with multi-party computation.

6:19And the space is a lot more mature. So, you know, a lot of the big exchanges out there, you can do a decent amount of trust that they're going to be custodianing the funds in the way they should. Otherwise, you get into all kinds of insolvency things. If there's a hack and your funds leak, you're going to be hit with all kinds of penalties and it's not good for the space. So that's why Kraken spends so much money and so much time investing in security and making sure that those private keys that hold deposits are 100 % sound. Got it. Yeah. So it's less to do with sort of the underlying sort of model of the blockchain, you know, what things are on the chain is secure, but how do I, you know, offload that essentially in a secure way and exchange it for some other, you know, financial currency like a USD or something like that.

7:03And that's where sort of conventional security and trust is going to come into play. Exactly. Yeah. I mean, if I talk about the top, you know, whatever, 10 currencies on coin market cap, you can very safely say that most of those are what we would consider decentralized So if I had a private key on my desk and I have, I don't know, X amount of Bitcoin associated with it, barring any kind of quantum changes in that technology that could crack a private key, nobody's ever going to get that. The Bitcoin consensus is decentralized enough where that's never going to be compromised. There's not going to be a cabal of miners that come together to try to censor transactions or anything like that.

7:40There's so many people mining or in Ethereum's case, validating the blockchain that that censorship just isn't there. Okay. And then you've been at Kraken for six years or so. And we talked a little bit about the change that's happened in the general crypto market and the ups and downs. But what about from an engineering and technology perspective? How have things changed for you and your team there over the six-year period that you've been working on it? I think the biggest change is even around what we were just talking about. So you create these kind of like offline air-gapped places where you generate private keys and hold client funds.

8:17And maybe in the early days, it's more of like, you know, you're just kind of using an offline laptop or something. But as the space has evolved, the people that provide like HSMs, which are hardware security modules, they get more advanced and they start adopting blockchain technology or the not really blockchain technology on the HSMs, more of like what elliptic curves do these cryptocurrencies need? And the HSMs will then start supporting those out of the box because they know exchanges like us need good tamper-proof hardware. So I think one of the biggest changes is that private key storage that, you know, I led the engineering efforts building out our qualified custody product started about three years ago.

8:52And we launched that last year. And it took a lot of engineering effort to make a qualified custody product that can be audited and can be shown to all kinds of jurisdictions and prospective clients so they can kind of trace exactly how we're doing private key generation offline. They can see the whole technology stack where if one person has access to it, they can't compromise it. So it's a lot of time in front of the whiteboard, kind of just threat gaming your technology to make sure that truly nobody can ever access this. And that is a constant evolution at Kraken. I think five years from now, there will be a whole nother way that we're doing private key storage.

9:29We just always have to stay on top of it. Okay. Can you walk me through a little bit about like, how does that private key generation offline system work? Yeah, not to get too into like the secret sauce, you know, for prospective qualified custody clients, they can definitely jump into it. But if you have like a hardware security module that has a tamper proof processor in it, and you want to have sufficient entropy to be able to make sure that you're having good randomness to your private key, then typically, it's just sourcing some entropy from that. And, you know, generating your 32 bytes of randomness that goes into being the seed of the private key.

10:07Because at the end of the day, on all these curves, the private key and public key are just big integers. And you just got to make sure that, you know, as you're generating these, that you're not doing it in any kind of predictable way. So using like cryptographically secure algorithms or cryptographically secure entropy to be able to create these is what's needed. Because there has been over the decade plus in crypto, there have been some libraries that come out that claim to be a safe generation of private keys, But say the entropy they're using is from the CPU clock. And people have been able to kind of reverse engineer and see that if you're generating a private key with these libraries, then you can kind of like predict what's going to happen.

10:49Like if you know somebody generated a private key at like midnight on some date and there's these kinds of processes happening, you might be able to kind of tell what the CPU is doing and then get close to that private key. So as long as we're using those hardware security modules and doing this with sufficient entropy, you can be very confident that it's cryptographically secure. And in terms of CPU security, are you talking about enclaves in this situation? Yeah, like that's another thing is we were huge fans of Intel SGX before. And that's like more on the kind of like encrypted memory part.

11:22But we were huge fans of being able to have like a hot wallet, which is like a private key that's not in cold storage. Because cold storage is where you truly don't have it connected to the Internet anywhere. And it's like you have to physically go there to like perform an operation. But if we had wallets in the hot setting, then utilizing something like Intel SGX is great because at least we would have confidence that if something was compromised, an intruder is not going to be able to decrypt the contents of the process that's happening. Got it. And then, you know, it's really difficult to get encryption right.

11:55Right. I have a little bit of experience with this working previously at a privacy security company. And even based on some of the things that you're talking about here, choosing the right libraries, choosing the right algorithms, encryption key rotation, like there's all these sort of tripwires that can really creep up that someone could eventually exploit. How did you kind of learn this space and know that you're following the best practices? Good question. Personally, I had no formal cryptography training, But I think that's one of the huge benefits of getting enthusiastic about cryptocurrencies and kind of like importing your first few libraries where maybe you're just making a wallet and a private key on a terminal.

12:33You naturally have to start to do your own education and understanding of like, okay, Ethereum uses ECDSA on the SECP256K1 curve, right? Or Bitcoin has Schnorr signatures that allows the Edwards curve and EDDSA signatures. So when you see the libraries having to perform these things, it naturally makes the crypto enthusiasts just go explore. And we never at Kraken, we would never release a new currency for people to deposit, withdraw and trade with. we would never really put deadlines on them because we 100 % wanted to make sure it was secure before it went out. So we would have dozens of eyes looking at how we're doing the signing on the hot wallets and everything and inspecting the libraries that these currencies might have as a wrapper of a high-level SDK and getting deeper into what library are they actually using for the private key generation for the signing.

13:26And most of them use the audited ones out of the box. like, you know, something with SECP256K1 is going to be very popular among all the SDKs. So at least we have some confidence that there's not just dozens of Kraken eyes, there's also thousands of open source eyes looking at this. Unfortunately, though, we have to be really on top of it, because some of those popular ones can still get vulnerabilities. And knock on wood, we've never had that at Kraken. But, you know, if there was a major one that a vulnerability came out, then us and probably all exchanges would need to have like a 24-hour war room where we're rotating some keys and upgrading the processes there.

14:03Even outside of the, you know, whatever sort of encryption libraries are you using, what kind of care do you have to take when working in, you know, financial space when you're thinking about incorporating like a new library, a new service? Like how do you sort of protect against the risk of some sort of supply chain attack that happens like way down through a variety of different library dependencies? Yeah, first, like, just on the initial kind of like build out, like, say you haven't written a line of code, and you're going to look at how we're going to do this new currency or this new storage system, most of the lower level dependencies, we're going to be looking to see if they're NIST compliant, they're audited, we might even fund our own third party auditing firm to go look at it.

14:43We've done that a few times, even though they have some certifications, we just want our own kind of set of eyes on it. And so, as I said before, we have a massive security team at Kraken, I think more than any other exchange, because we will rotate people around to just go do our own independent audits, even outside of a third party. So every single library, lower level, we have our own list that we've performed audits on, but we'll look at anything under the sun to make sure it's up to our standards. And then care on these, it's like the actual kind of architecture and planning these, I think can, like for the custody build out of that new cold storage system, I want to say it took maybe four months of just constant iteration, being in front of a whiteboard, filling it out, wiping it and trying again, because there's so many other considerations outside of just like libraries and generation where you want to make sure that these private keys don't have access by one individual.

15:36So that's where you get into threshold cryptography so you can split these things up and make sure that there's an M of N, like three or five people have to come together. I would think that we're not unique there. Everybody has to do some of that because we also have to show to regulators and everyone that no individual, no one person can go to this private key. You must have certain directors with secret material to come together to do something with it. Okay. I want to talk about some of the work that you're doing with respect to layer two protocol. So first of all, can you give a little bit of background on what layer one protocol is, what layer two protocol is, how do they kind of compare and what are the advantages of this layer two protocol over layer one?

16:21Yeah. So, you know, earlier when we were mentioning like the security of a centralized side and then, you know, the security of a decentralized side, you know, in order to sufficiently maintain a level of decentralization on a blockchain, you really want it to be as accessible to all people. So if you want good decentralization, you want it so that anybody can go open up like a laptop or spin up a kind of cheap AWS server and go run the binary of Bitcoin or Ethereum or something and be able to start validating and mining blocks. Independent of anybody telling you what to do, it should be like very cheap to do so.

16:58Therefore, you can have maximum world participation in that protocol. Some protocols take a different route and they push the technology to its limits and they might do some decentralization trade-offs and say, you know what, okay, you have to spend$20 ,000 a month on this AWS server to run and participate in the blockchain. That definitely limits the amount of people that can participate. And so you kind of run into these little bit of centralization risks. And us getting involved in an L2 protocol and wanting to take Kraken more on chain is really latching on to the decentralization ethos that Ethereum maintains.

17:34They really want to make it so that it's super cheap to run something. And to do that, you can't really push the scaling limits as much as you want, again, because the cost will go so high if you're pushing that throughput to like... 100 milliseconds a block. You have to have some beefy infrastructure to do that. So Ethereum is taking this different route, decentralization above all else. And that's where L2s come in. Ethereum has a scaling roadmap so that on the layer one, which is the base blockchain of Ethereum, you don't push the limits of that. You allow layer twos to have their own blockchains that actually settle back to the L1 for security.

18:08And the L2s can do all the experimentation they want. So if we really wanted to push it and say, our L2 by the end of 25 is going to have 200 millisecond block time. It might be more costly to run it. And that can be a trade-off that the user then takes is like, I'm going to go participate in Kraken's L2 with really fast block times. But at the end of the day, I know that the L2 that I participate in, and I say I have a million dollars or something on there, it's still secured by the L1. So even if Kraken went defunct tomorrow, and people had a ton of money on our L2, they still can cryptographically submit something to the L1 to get their funds back in control at the Ethereum kind of base layer.

18:44How does that work? The part where essentially of, you know, getting the funds back from the layer one? Yeah. Like, you know, there's this concept of bridging in the scaling roadmap of Ethereum. So if I have funds on Ethereum right now, and I want to go to Kraken's L2 Inc., then there's a contract that you actually send the funds to. And a contract is like a smart contract, basically a program that you can deploy on chain. And the validators are all participating in the computation of what you want to do on chain. So if there's a contract deployed, it's usually called a bridge contract. And, you know, I would send funds to it and the contract will lock my funds and then actually mint you the equivalent on the L2 because the L2 is posting data back to the L1.

19:26So there's a bi-directional communication there. And the L2 will understand, okay, Andrew sent funds to this contract. It's locked there. Now I'm going to credit him on the L2 for the equivalent amount. And then you can go participate in the L2 with those funds. And then when you go back to the L1, it does the same thing. The Ethereum on the L2 goes away, your balance goes to zero, and then you get released the funds back on the L1. Yeah. So there's essentially sort of built in backwards compatibility. Yep. And again, if you went away tomorrow, you could interact with that contract on the L1 and you can do a proof that the blockchain is not running and posting data to the L1.

20:04Then that will allow you to actually unlock your funds on the L1. So it's a beautiful kind of like trustless system where you really don't have to trust Kraken here to maintain ink. And you can, you know, you can get your funds off even if we turn off everything tomorrow. Are there new problems that get introduced with this new protocol? Fragmentation, I think, is the biggest right now. And that's what we're all thinking about pretty much every day in the blockchain space. So you want to maintain decentralization. Great. That means you need to kind of scale horizontally in all these L2s. But fragmentation comes into place.

20:40If we, you know, I think globally, there's like 100 million people interacting with decentralized ledgers. And so that's kind of a small fraction of the world still. And if you have a bunch of people using Coinbase's L2, you have a bunch using ours. And then there's like 50 other L2s that people are dabbling in. The liquidity of funds kind of gets fragmented all these places. And you might have the same, say, lending protocol deployed in multiple places. Well, you can't really tap into the liquidity if you're on Inc over to Coinbase's L2 base. And this is a huge scaling problem that the blockchain researchers and the whole community is thinking about is how do we make communication channels between the L2s such that if I'm on Inc, I could use liquidity on base and vice versa without the user even knowing.

21:26And I think a lot of those interoperable research projects are going to start coming online in about mid-25. Was the learning curve for L2 a challenge above what you would experience with L1? L1s are, that's just like you launch a blockchain and it's there, right? And you have whatever block times and centralization or descent trade-offs. L2s introduce all kinds of, I think, new complexities because they had to hard fork the L1 to be able to support this kind of communication channel where the L2s post data back to the L1. So the L1 can look at it and say, OK, Inc performed all these actions in the last hour.

22:07I'm going to bundle them up and post them to a block in the L1 so it's now immutable. That kind of learning and getting into the protocol changes for that was a huge feat, I think, for all of us. because we weren't protocol engineers to start. And we really had to dive into some complex, you know, Go code and Rust and all that to really understand like how this is working so that if we do any modifications of the protocol, we don't mess it up. And that's like in crypto kind of financial engineering, that's the thing that can maybe make you sprout a few gray hairs early in your life because one small change, one small mistake on the protocol level, you could be compromising people's funds.

22:51That's maybe why some people perceive cryptocurrency moves slow on the research side, because you really have to get this right. Yeah, I mean, I think that's the case, even with sort of conventional or like non-blockchain based financial services and fintech is essentially the consequences of getting it wrong are massive. Yeah. And you will always see in the news that some protocol was hacked or, and it's not, you know i kind of scoff a little bit at the headlines because it's not really a hack like somebody has breached a system and you know like stolen traditional like database data and those database leaks but it's really just looking at the assembly code on the ethereum virtual machine and noticing that somebody doesn't have a lock correctly coded on a liquidity pool or something so they find some roundabout way to be able to drain these funds you know it has a lot of bad press.

23:42But that's why the auditing firms in the blockchain space make so much money, because they have to have that hardcore expertise to review these things. From an engineering perspective, what do you have to do in order to do everything within your power to reduce the risk of, oh, we misinterpreted this particular part of the protocol, and we ended up with a situation where we did risk somebody's money? I think this is just not even trusting yourselves because we can engineer everything and build out this beautiful like change to a protocol or making our own smart contracts but then then it's like a multi-month process of getting just as many eyes as you can on it and so that's why like things like uniswap which is one of the biggest decentralized exchanges on the ethereum virtual machine space they build entirely in the open and you know they have dozens of audits on on their their contracts and the hope is that, yeah, more open source work in this space is going to catch these things earlier.

24:43There's some protocols that like the build closed for a while. And I think a lot of the hacks or a lot of the kind of exploits have been from places like that. So kicking off that process of like, after we're done with something to just kind of give it out to the world for a while and just see what comes back, that's key to this process. And I think, you know, hundreds of other startups that engage in this. So outside of some of the stuff we're talking about in terms of the attention to detail that you need to have in order to do this right, what were some of the other technical or conceptual challenges with this build out?

25:19So we use something called the OP stack, which is this optimistic roll-up stack. And so you can Google just OP stack, OP hyphen stack, and you can see that this is one technology stack that people have created to be able to launch an L2. There's other things like ZK Sync, which uses zero-knowledge proofs. There's Arbitrum, which is also using optimistic proofs like Optimism. And they all kind of have their own flavor of how they're allowing this kind of rolled-up data be posted back to the L1. And so some of the challenges were like thinking, how do we differentiate in the space? If everybody's deploying these roll-up stacks that are very similar to each other at the end of the day, then how do we push the limits of the protocol to make sure we're differentiating without violating any kind of consensus?

26:06So that's where, you know, if you run the OP stack, you are locking yourself into all the other people that run the OP stack and you all have to agree on any protocol changes together through like a decentralized governance. And so a big challenge is like, you know, if we want to push the limits of the block time and maybe make this go down to 250 milliseconds, which is a target for us by sometime in 25, then we have to develop all kinds of sidecars around this binary. And so if the OP stack is sequencing blocks on the L2, we now kind of have to make a little bit of a Frankenstein to say, transactions come to our sequencer.