Harmony ONE Protocol: Scaling without Sacrificing Decentralization

Hey guys, welcome back to the channel. Today we have someone really cool, Nick White, co-founder of Harmony, and they're talking about a really next generation blockchain platform. And something that we've been talking a lot about is scaling. And these guys are on the forefront of scaling with sharding solutions and a new consensus algorithm. So Nick, welcome to the channel. So the first question I want to say was, what is your passion? What's your drive? Why do you want to make this blockchain in the first place? Well, first of all, thanks for the introduction. Lovely to be on the channel. I'm really excited to do this interview and get to know everyone on the Box Money channel. So at Harmony, our motivation is to scale trust of billions of people. And that's kind of embedded in the name, right? Harmony. Something that enables the humankind to sort of reach a new level of collaboration and prosperity. That's our big, big vision. And the way that we get there is by providing a more scalable blockchain that can power the next generation of de-applications and all the things that we haven't even dreamed of yet. It's coming at a good time too. We were at a time of distrust. We just saw Huawei being kind of almost banned in China. And now we're talking about how do we even just make technology without states, without countries. So that's kind of the objective. Then how open is this consensus? Because we've seen a lot of consensus mechanisms where it relies on a small number of nodes that were pre-selected. So can anyone become a proof of stakeholder? Yeah. So that's really important to us is maintaining decentralization and making sure that the network is permissionless. Because that's really the core fundamental value of blockchain is that it's decentralized. So we see in a lot of other competing blockchains that are trying to scale, they really sacrifice the decentralization aspect. And I think in the long term, that's a really good thing to make. Exactly. Because the whole idea of blockchain was supposed to be that we solve open consensus. That's what Satoshi Nakamoto did. He solved anyone can join the network. And then even if they turn out to be dishonest, the network still has a way to deal with it. But the more you make it trust-based, then where's that trust coming from? It's more centralized. What's the difference between this and the giant database? Right? So how do you guys solve this problem? Yeah. So anyone who has Harmony tokens can join the network and stake them to become a node in the network. There's sort of a threshold. So you need a minimum amount of tokens. But we've really optimized our network to provide for more nodes and lower resource nodes. So it's the opposite of an EOS delegated people stake where you need to have some mega machine to run the protocol. We've built it in a way where you could run on your laptop. It should be very accessible for everyone. And the one thing that you need, and this is where all the securities come from, is you need to have the token to stake to prove that that's our civil resistance mechanism. If there were no token to stake, then anyone can join and then some hacker can just make a thousand nodes and take over the whole network. So the only thing you need is a pretty reasonable computer and a certain amount of tokens. And you are welcome to participate and validate and get block rewards. Right. And how does the selection process work? There's a random selection for all the committees. And if you made the threshold, you should be given at least one voting share. Okay. That voting share will be assigned to a given shard. And so anyone with the voting share gets the right to vote, they may not be the leader of that shark. So they may not be the one proposing the blocks, but they are signing the blocks and collecting fees and block rewards. Got it. Yeah. So as part of this network, you get to stake, you get to start vote in the consensus. You actually get to be part of the network. And then as a result, you get to also get rewards for the block creation as well. Exactly. What makes you different from Ethereum? That's a great question. So most people know Ethereum for now is proof of work. They're working on a new solution, Ethereum 2.0, which people may be familiar with. We are building a proof of stake sharded blockchain. And our specific approach to sharding is called deep sharding. And it differs to Ethereum 2.0 and a lot of the other sharding protocols on a number of different dimensions, the randomness generation. And but most importantly, the deep sharding aspect, which is that we think of sharding as one way to parallelize the blockchain. But then you have to think a layer deeper to the networking layer itself and integrate the layer one consensus with the networking layer, layer zero to achieve the best results and the most scalability, the fastest finality. Okay, so so how does that kind of work? Because that's the kind of the first time I actually heard about this kind of end to end idea. So what do you include as part of this end to end solution? So for now, we're really focused on the networking layer, in addition to the consensus layer. Right, because the issue, of course, is the network latency. We've always been talking if it takes a while for this whole network to even communicate with each other. I mean, it feels like it's hard. It's it feels like the internet's really fast. I mean, right now, I'm calling you from Hong Kong, and you're out of the way to the States, it feels like we can communicate. But it's harder when we add other countries like China into the mix, where they do have great firewalls, and it kind of interrupts that communication. Is there any particular tricks that you guys deploy to decrease that latency of the entire network? Yeah, for sure. So blockchain, at the end of the day, is a coordination problem. The only way to come to consensus and everyone to be on the same page is for us to send a lot of messages to each other and all be, you know, aware of what's happening. So one of the really common problems is block synchronization. Block synchronization can actually take a long time because usually there's one leader that proposes the block. In Bitcoin, is the person who mines the next block. In our algorithm, which is PBFT-based, it's a leader that proposes the block himself. And his problem is he needs to send this, let's say, one megabyte or two megabyte block to all of his peers in his shard. And that can take a long time. If he has to send it one at a time, it'll take him maybe a minute to send it to everyone. But what we do is called adaptive IDA, which is using the ratio coding mechanism to split the block into a ton of different chunks and then spread all those chunks around and let them sort of let each peer then share that piece with everyone else. And that allows us to utilize all the bandwidth in the network. Because you have to think about bandwidth as a scarce resource. As fast as my phone is, or as fast as my internet is, I have a limited amount. So I can only send so much data for every unit of time. And by leveraging all the resources in the network as best as we can through adaptive IDA, we can synchronize a block much, much faster. So it's almost like BitTorrent, right? Like BitTorrent is like splitting a file into smaller bits. You're splitting a block into smaller bits. Yeah, very similar. Actually, that's a very good point. In fact, there's a lot of similarities between these torrenting and these decentralized networks for blockchain. So that's a really good point. But yeah, the point is that if you speed up block synchronization, that's one really easy low-hanging fruit where all of a sudden the time between blocks is that much shorter. And that means if each block has however many thousand transactions, that means you get that much faster transactions per second. Awesome. Okay, so we talked about the network and then we now we need to move on to sharding. So I briefly explained what sharding is in the channel, but can you really explain to us what is sharding? Yeah, so sharding is a technique that actually comes from traditional databases. And databases, they have this problem where, you know, let's say you Google, you have to store the whole internet. You can't actually store it all on one machine, right? So you have to split it up into separate chunks and store them separately. And the problem with that is then who keeps track of what and then how do you synchronize information between them? So it's a very similar thing in blockchain. We're trying to, if the whole world had to run on one single blockchain, then the whole world would have to be in sync and come to consensus about every single thing. So the idea of sharding is that we we want to split the network into smaller bits and each little shard runs its own blockchain and they run in parallel. But these blockchains have a way to talk to each other and share data. So the idea of sharding essentially is to parallelize the network. No factor, right? So this, I saw this online and this is one of the theories of why the name came about because sharding is like, it sounds very magical, right? So the origin of this, one of the the possible origins is from Ultima Online. So it's a big, the first one of the first MMOs ever built like, and they had a huge world, but their world was too big. So they invented this mythology around it that there was a crystal representing the world and this wizard broke it into many different shards. And this is how the terminology may have came about. It's like World of Warcraft where you're like different servers. Yes, exactly. The world split into multiple pieces. So we're splitting Harmony into multiple pieces. It's all called different shards. And they process, they kind of work together within the team to work together. That's kind of how it works, right? But the issue here, of course, is that we've been talking about sharding for a long time. Ethereum has been talking about sharding for a long time. And there were issues about achieving sharding. So how did you overcome some of the issues when it came to achieving a real sharded environment? Yeah. So one of the biggest problems with sharding is that you can probably imagine that when you split the blockchain into smaller pieces, each piece is more vulnerable to be attacked. So it's sort of similar to Bitcoin. If you took Bitcoin and then you split it into a bunch of smaller chains running side by side, the proof of work sort of security would get split between them. It's similar. We're proof of stake, but it's a similar concept. And so the way that you can, you have to think carefully about how to keep it secure. And the way to do it is to split the shards randomly so that no one can, if I could choose where to vote, where to put my stake or do my mining, then we could all coordinatedly attack a particular shard, right? But if you choose the shards randomly, then no malicious agent is able to to all collude and join one shard and take it over. So that's one of the hardest parts is how do you do this randomness? And our random approach is what we call distributed random generation. It's a combination of VRF, a verifiable random function, and a VDF, a verifiable delay function. You can generate randomness in a way that's unliasable and it will be faster. So I don't have to get too deep into the details there, but that's one of the advantages that we have over the blockchains. So it's kind of using very, very complicated mathematics to ensure that the selection process for these shards is completely random. It can't be fooled. No one attacker can gain full control over the shard and then create malicious transactions. All right. So in that sense, something that I've always wondered is how do these shards communicate with each other? They need to communicate across these shards. So then that way, if a transaction is spent in a single shard, you don't get a double spend. How does that work? Okay. So this is a really fantastic question. There's two aspects that I want to touch on. One is, so if you really want to parallelize the network, you have to make sure that when I send a transaction from one shard to another, that the shard, the originating shard and the destination shard don't have to, it doesn't have to touch every shard. It should, the message should get there as efficient as possible. If you have to gossip the message to every shard, then you're, you're not actually scaling, right? The more shards you add and the more communication overhead you have between the shards. So what we use specifically for cross shard routing is called Kademlio routing, which is, it's actually similar to distributed hash tables, which is in torrent. So it, it, it basically each shard has a, a table of peers that it knows the hierarchical structure so that it can talk to the peer that is nearest to the end origin. And each step of this process will take it logarithmically closer to the destination. So it doesn't scale, you know, as N squared as the number of shards actually scales logarithmically, which is a huge advantage. But then back to your question of how do you actually make sure that the communication between shards is secure? When a shard is trying to send transaction from its own network to another shard, they generate basically a proof where they, they lock up this, the funds or this contract, this, this, this state, and then they send it to the other shard. And the other shard can verify that it was signed and validated and basically look up the proof and validate the proof before creating the funds in the new, the new, uh, shard. So that's how you keep it secure. Obviously, if a shard were malicious, that's, that's a problem because they can create all these proofs and send them. And then the proofs don't have actually any validity because they can, um, you know, the, the, the whole committee is corrupted. So, right. Hopefully that answers the question. Right. So, so I think the biggest analogy here, um, is actually with gaming. So when the first video games came out and they were sharded, each world would be on its own shard. So people found ways of throwing down items in one world, jump to the next world, and the item will still be in their bag because that communication has a delay. It hasn't communicated to the other shard yet. So what you're talking about here is that you're locking it up. So if, if something does happen, it gets locked up in one particular shard first, and then you're making sure that communication happens as fast as possible between this, these two shards as well. Yeah. Yeah. And, and, and, you know, the whole power of launching is this concept of, you know, digital scarcity. So if, if you were able to create something or duplicate it right across shards, and then that would be a real problem. I think everything would break. Um, and so yeah, having that, that locked up mechanism, it creates an atomic cross shard transaction where either it goes through and it's transferred or it's not transferred. Right. So you're not stuck in this in between where either it's destroyed and doesn't appear here or the worst thing is it's here and here at the same time. Yeah, that's true. The, the other issue is that you're also proof of stake and something that we saw, um, which is a problem in proof of stake is the problem of what happens when nodes just randomly shut down. And that was one of the biggest fear if someone, you know, you can allow anyone to stake, but what if their servers or they, um, a bunch of servers get attacked by an attacker and maybe 60% of the network goes down at the same time. Okay. Is there any way to deal with that? Yeah. So, um, that's the issue of liveness, right? So liveness is this concept of, um, if validators go down and a certain amount of them go down, then the network loses liveness, meaning that it can't, it won't continue to function. It will just hang. It doesn't mean that something bad necessarily happens. Like, you know, it's not like the same as being corrupted, but it is, um, it does mean that, that the, that shard would stop operating, but there's, there are a lot of ways to, to avoid this. And that's where really the, the validating software and sort of the structure that, um, you know, staking as a service companies or people who are running nodes, um, can, can make many sort of, uh, can, can program in redundancy. So if one node goes down, another node comes in to, to, uh, help it. So, uh, uh, liveness is really important, um, part of any, uh, PBFT based network. Whereas, you know, for, for Bitcoin, if, if half the miners go down, um, in theory, it still keeps going just at half, half the speed. Uh, whereas in PBFT, if you lose a certain fraction of the, the, the voters in the shard, uh, the, the shard will hang. The other nice thing about sharding is that, um, you know, it will be isolated to that one distinct shard. It wouldn't be across the network as a whole. What happens if one particular shard hangs for a while? Is there a way, if there is a mechanism to deal with that? I think the, the, the best way that it would be dealt with is that by the time the epoch, uh, transition comes around, those nodes will get shuffled out and we'll have a new batch of nodes coming in as validators. So, um, in the longest it would last is, is one epoch. And then we should be able to sort of get everything back, um, in line. Got it. So, okay. So we've been, this week, this year, we've pretty much been bombarded by scaling solutions. Everyone saw this as the, the first problem to try to solve. How does Harmony differ from other scaling solutions? Interesting. Yeah. So I think, you know, you, you're mentioning, um, layer two scaling, right? And there's also a whole, you know, many different flavors of layer two scaling. It has different security properties. So it's, it's kind of different trade-offs along the, uh, sort of the trilemma, if you will. Um, but I think that layer two scaling is actually very important and, and is complementary to layer one scaling. So I think we'll, um, we'll see ways in which they pay together and we're actively engaging with, uh, layer two, uh, projects that actually want to build on a more scalable blockchain. Right. They go together. And I think one of the advantages of layer two is that they can do faster finality. There's a lot of ways in which, you know, for certain applications where you don't need the utmost security that layer two makes a lot of sense can be cheaper and faster. Um, and when you combine that with an already more scalable layer one, it's just better. So I don't, I don't see them necessarily as competitors so much as complementary approaches that, um, will, will, you know, be combined to suit different needs. Right. Um, in the future, you're processing an immense amount of data. Layer two will say, okay, let's just discard this. Um, it's no longer important. We'll report back to the blockchain and record only the, the outcome of a bunch of transactions taking place. But here you're recording every single transaction. How do you deal with that immense amount of data? Yeah. So, um, part of the way that we've designed the protocol is specifically to try to reduce the data bloat as much as possible. Um, and where that comes in is that, um, you don't, we don't require the validators to store the entire history of the blockchain, because if you're doing, you know, a thousand transactions per second, pretty quickly that gets out of hand. We're already seeing how Ethereum is, you know, really going exponential, um, with its own blockchain state to store. So, um, we enable the validators to actually, um, trim the history and just maintain the actual state of everyone's balances, but not all the records of every transaction. So that's, um, that's really important. But yeah, you raise a really good question, um, about, you know, this data blockchains. I think one of the, one of the things that we have to remember too, is that, um, not all the data gets stored on chain. So there also will be lots of decentralized storage solutions like Myopoint. Um, we have some partners in that realm that are, will be integrating with us. So, um, as a D app developer, you don't have to store everything, um, on chain. So I think, I think we'll see, um, uh, you know, the ability, you only, you only interact with the layer one for data that is really essential to the trust of your application. Right. And one distinction that you made with your sharding is that your state-based sharding rather than just transactional sharding. Can you, um, tell me what the differences are? Yeah, sure. So, um, transaction sharding means that, um, all the transactions are, are verified by, you know, shards. So that means that not everyone has to validate that every transaction is legitimate. Um, but then, uh, so like Zillik is a good example of this, where they shard out the transaction verification, but at the end of the day, the beacon chain has to keep track of everything that's happening in all the state of everyone's accounts and whatnot. Um, and the problem with that is that, you know, as you add more shards, the beacon chain all of a sudden has to, you know, store more and more states. So it's not really scalable in that sense. Um, you can see a future where there's so many shards that it's, it's overwhelming the beacon chain and it's just too heavy. So what we do is called state sharding, where each shard keeps track of its own state of, of the balances of accounts and, uh, smart contracts and whatnot. There is a global view. We do have a beacon chain, but it's not in the same, the same way. So each, as you add more shards, it doesn't increase the load as much. If you don't, if you don't do state sharding, there's some, some bottlenecks that you haven't removed. Right. Okay. So, so we talked about all the technicals, um, now onto adoption, onto, um, what you see in the future for all this as well. Um, so we have done four different iterations of our test net, um, each one rolling out new functionality, um, and, you know, achieving different levels of state stability. Uh, and now we're, we're approaching the soft launch, uh, what we call a, uh, throttle launch of our mainnet, uh, coming in June. So our blockchain is, uh, fully operational. It's open source. Um, we have EVM cloud compatibility and smart contract support. We're ready to launch in some ways. Uh, but again, it's going to be a soft launch because there's probably still a lot of testing that we need to do. Um, but we want to start bringing people in and allowing them to mine and incentivize them to, to run nodes. And in that way, we'll have more of a realistic view of what our network is going to be like. Um, and we, so we can, we can do more optimization. We can do more testing around that and make sure that the protocol is really secure and smooth. Got it. So, so I see that you're upcoming on a Binance, uh, launch pad as well. So when you launch in a bunch pad and you buy those tokens, so, um, are those tokens mainnet tokens? How would those tokens be used on a network? Yeah. So, uh, at the, at the outset, we, we've, we've been running in the foundational node, uh, program, uh, it's an investment program and we had quite a bit of interest. Um, and the whole, the whole point of that program was to allow people that want to be uh, participants in the network. So for now, the, this, this mainnet is going to be mostly just for those, uh, limited participants, um, and not part of the broader, um, IEO pool. But at some point, there will be, uh, like the more full mainnet launch in which all tokens are transferred over and converted into the, the native token of the protocol. Okay. So, so there will be, so there will be a token transfer process. So initially, are you doing that ERC 20 based token or are you doing, doing a, um, main, like a mainnet based token? So, uh, how would that transfer? Yeah. So it'll be, it'll be a token first, um, likely a VEP two token, which is the native token, uh, standard for the Binance chain. Right. Right. Got it. And then there'll be a token transfer process once the mainnet, uh, and the test launch process is fully live. Right. Got it. Got it. Okay. So, so next, next. Okay. So let's assume that everything is live now. Let's assume that everything would, um, go as well. Then, uh, who, who do you think are the main, first few users? What's the vision that you see going forward in this space? I see a lot of adoption happening, uh, early adoption happening really in the centralized finance and the gaming space. I think the centralized finance has a tremendous use cases, um, especially in the developing world. Um, so one of the things that we've talked to Linance about is actually trying to target, you know, South America and, uh, in Africa as, uh, you know, places where people really have need or organic need of, of cryptocurrency. And decentralized finance. Um, but on the other hand, gaming is also really interesting, um, because, you know, gamers have always been early adopters of technology, sort of like how you're talking about how sharding came about, you know, they're probably breaking databases, you know, way back then. And it's, it's similar today. You know, gamers are always at the forefront of new technology trends. Um, and, and there's a lot of exciting stuff happening there. Um, we're partnering with some gaming companies and gaming, uh, platforms. Um, but then in the long term, uh, what we're really excited about is data sharing, right? You know, our, our vision is to create a open infrastructure for the world's data. And a lot of us come from an AI background and we see blockchain as a way to decentralize data and, and make it, make data, uh, break the data monopolies and allow, um, people to not only have privacy, but also have ownership, sovereignty of of their data. Um, and then they can even themselves monetize and receive benefit if they're, if they choose to, uh, give their data over to some advertiser or to some machine learning engineer. That's really the long-term, um, is, you know, decentralizing these companies and these data monopolies like Facebook and Google. Yeah. Um, and like I said, I think what we need to, we actually start need to head towards those areas. I mean, especially when we see articles, like people trying to break apart Facebook and, um, all these, uh, state monopolies, the more data they have, the more, uh, powerful they are. And also the more centralized it is. And that's going to become a very big problem. You know, the, the information cartel is actually becoming, um, pretty real actually. So, um, so I think Nick, um, I definitely would like to thank you so much for teaching us a lot about sharding and about how this consensus mechanism can be built and really moving this kind of needle forward in the cryptocurrency space. And especially in terms of making blockchains that can be really useful. So thank you so much for your time. I hope to catch up with you, um, very soon about everything that's happening. And I hope that mainnet launch goes well and you can teach me how to stake on harmony. Absolutely. Well, well, thank you. Um, Michael has been great, uh, doing this interview and I hope everyone learned something. Um, check us out. Our website is harmony.one. Um, join our discord and our telegram. Uh, we'd love to chat with you. Don't be shy. And, uh, I look forward to everyone helping to create this, this future that we envision. Um, and, uh, I hope that you resonate with Harmony's mission to scale trust to millions of people. Awesome. Yeah, dude. Thank you so much, man.