Matt Odell | Citadel Dispatch #90: Spiral Initiatives with Steve Lee
Link to the YouTube (the timestamps are based on this): https://www.youtube.com/live/JwuLe-PF4pI?feature=share
Table of Contents:
I. Approaches to Open Source Funding [7:53]
II. Stratum v2 [23:59]
III. Fedimint [40:03]
IV. WalletScrutiny [47:12]
V. Separating the Core Out of Bitcoin Core [59:19]
VI. LDK/LDK Node [1:02:43]
VII. Rapid Gossip Sync [1:10:28]
VIII. VLS [1:28:18]
IX. Unified QR Codes/BIP21 [1:35:45]
X. Preparing Lightning For Mainstream Adoption [1:40:36]
XI. BOLT12 [1:47:22]
Intro:
Matt Odell [4:31]: We have Steve Lee in the studio. I recorded with Steve last year — pretty much almost exactly a year: it’s been 14 months. Steve leads Spiral at Block (formerly Square), and we will be talking about a variety of things including building secure mobile wallets, but also the various Spiral initiatives. How’s it going, Steve? Good morning.
Steve Lee [4:57]: It’s going great! Great to be back. Nice to do it in person as well.
Matt Odell: It’s always better in person.
Steve Lee [5:01]: Better in person — also it’s my first time to Nashville ever. First time to Bitcoin Park. The whole Spiral team is here, and all but one of us: first time in Nashville. We’ve had a great week.
Matt Odell: It’s an honor and privilege to host you.
Steve Lee [5:18]: The hype was real about Bitcoin Park, but it’s exceeded even the hype.
Matt Odell: We try not to hype it too much — we just love when people come and visit us.
Steve Lee: I definitely recommend Bitcoiners come visit Nashville and Bitcoin Park.
Matt Odell: And it’s the ideal setup for an off-site, right?
Steve Lee [5:33]: Yeah, I was talking about that with our team — the Spiral team tries to get together around three times a year in person since we’re distributed all over the place. And in the past we’ve piggybacked off of conferences, which works well. We met in Atlanta for TABConf — TABConf also is a great conference.
Matt Odell: It’s amazing.
Steve Lee [5:56]: Really really good conference, but it’s also pretty energy-draining to both do a pretty intense 3-day conference and a 3-day team off-site. So this week we had a couple nice events: the Nashville BitDevs, and there were a couple night events which gave a good community feel, met new people, but it’s not as intense as a 3-day full-on conference. So it’s been a really good week. And I didn’t know that Nashville was renowned for curling!
Matt Odell: Yeah you guys all went curling last night, right?
Steve Lee [6:24]: We all went curling, and all but one of us it was our first time — there were five or six of us who had never even seen curling before, so those folks were definitely in for a surprise for what an unusual sport it is. But it was a ton of fun. One person is actually Canadian and plays twice a week.
Matt Odell: Wow.
Steve Lee [6:47]: Thunderbiscuit.
Matt Odell: Did he tell you that before or after you started playing?
Steve Lee: We got hints before — a ringer. I highly recommend it as a team team event because it’s approachable for everyone, and it was a lot of fun.
Matt Odell [7:01]: Well I love to hear it and I love these physical spaces — I want to see them pop up all around the world — and for me, Rod, and Josh at Bitcoin Park, like I said: it’s an honor and a privilege to host you guys. You guys are doing great work and to me it’s the least we can do. So it’s been just a great week. So obviously I’ve had the BDK team on this week, I’ve had Val and Jeff from LDK, and now we’re capping you off with a Spiral trifecta of the week — so where do you want to start, Steve?
Steve Lee [7:37]: So a year ago when we did this, I gave an update of the first two and a half years of Spiral. How about we just start with what Spiral’s done in the past year?
Matt Odell: That’s great. So what has Spiral done in the last year?
I. Approaches to Open Source Funding
Steve Lee [7:53]: One thing — we have a really robust grant program: we’ve given out over 60 grants to people in 20 different countries. We’re really excited about what we’ve done there, and I believe every single one of those grantees it was their first time being funded for open source Bitcoin work.
Matt Odell: That’s amazing.
Steve Lee [8:16]: So, definitely proud of bringing really talented people into open source Bitcoin. What we’ve observed the past year, though — now that we’re three years into it — we have a couple successful outcomes: one is projects that are on their way to be sustainable, and we can go through those in a minute, but the second one is that we now have an alumni base of Spiral grantees, and they’re going on to do even bigger and better things. We have a couple folks that have joined Block full-time, we have someone who joined Voltage full-time — Stephen DeLorme just joined Voltage.
Matt Odell: I love Stephen.
Steve Lee [8:59]: He’s amazing. He’s been here this week too — it’s been great. And then Justin Moon is an alum, and he’s gone on to co-found Fedi.
Matt Odell: One of the hottest startups in the space.
Steve Lee [9:13]: Yeah. And Johns Beharry has done really awesome stuff with BOLT.FUN. So it’s really great to see people who went to the program — aren’t part of the program anymore, but — they’re doing even better things. People are staying in Bitcoin.
Matt Odell: BOLT.FUN is amazing. Do you guys fund BOLT.FUN as well or no?
Steve Lee [9:35]: We haven’t yet but we’re definitely bands and cheerleaders.
Matt Odell: Yeah with OpenSats I’ve dropped the ball a little bit on BOLT.FUN — I promised them that I was gonna help support that. So John if you’re listening, it’s a priority.
Steve Lee [9:51]: Yeah last fall I thought what they did there was phenomenal. We can talk about some of the challenges of development in Bitcoin, but it’s hard to get a lot of groups of developers doing an extended hackathon, and he did a great job. A lot of great projects.
Matt Odell: It was deliberate and he pulled it off.
Steve Lee [10:12]: So we’re happy with this new revelation that our folks that went through the grant program are doing great.
Matt Odell: Which is what you want: you want a pipeline, basically. You want to teach a man how to fish, not feed them fish every day.
Steve Lee [10:26]: Yep, exactly. But there’s also folks that are doing great work and they want to continue working on their projects. So we’re renewing them, and that’s a good segue into: Spiral had a post maybe nine months ago about our view on creating a framework for sustainable open source Bitcoin projects, because we’ve funded now on the order of 8–10 different projects. Sometimes we’re funding the project directly, sometimes we’re funding individuals to start projects, but a common problem or challenge in open source is that maybe a developer starts a project — great idea, makes good headway — and then it just doesn’t see adoption. And it could be due to just that developer doesn’t spend their time marketing it, or maybe that developer isn’t going out doing the PM work of understanding customer requirements to really get a good fit, or maybe they move on to something else and then it just dies and doesn’t go anywhere. We think every project we’re funding is going to be valuable to Bitcoin — we’d like to see it succeed. So we wrote this post. It’s not that long, but it just walks through four different phases: (1) first phase is the seed stage. It’s often one developer, maybe two — they have a great idea — fund them to start coding and seed that project, get it going, get it to a prototype stage or a proof of concept stage. (2) And then the next phase would be getting additional contributors to turn this from a one-person show to a community project — attracting other developers. And that shows a sign of a maturity for that project, too. Usually you have to make it a little more approachable for other developers to understand the code, build tools, documentation, etc.
Matt Odell: It’s a big leap from one developer to actually many contributors, right?
Steve Lee [12:42]: Yeah, and again some developers naturally write a bunch of tests and docs, and others often don’t, so that’s a big milestone. (3) Then going to the next stage would be — in the case that Spiral is funding it — trying to find other funders as well, where the Spiral team is highly attentive to: we don’t want to be the only one funding these projects. That’s not a sustainable approach — that’s not a healthy approach for Bitcoin. We think we’re good actors in the space and doing good things, but it’s just not healthy to depend on one source or one point of view.
Matt Odell: Everything in Bitcoin we want to be distributed, including funding.
Steve Lee [13:21]: Yeah, absolutely. So that next stage is getting others to fund a project. (4) And then what we described as the last stage would be: the project is so sustainable that Spiral could just back off, or whoever the original funder was — disappears completely — and it doesn’t skip a beat. Now one thing we talked about this week, Matt, is maybe there’s another level beyond that: some new type of funding mechanism.
Matt Odell: Business model.
Steve Lee [13:57]: Opening a new business model or something — exploring all of those as experiments is super important, too. Anyway, I think it’s a good framework for everyone to look at and see and test. It’s what we’re following at Spiral. If others like it too, they can copy it. And then if you look at the projects we funded, you’ll find them at all the different stages. So BTCPay is the first grant we did — that was September 2019. We renewed them in 2020, 2021, 2022, so four years. Huge supporters of BTCPay — love the project, the mission, the people, so that was an easy decision then and it continued to be just an easy to decision to support them. Spiral is the first funder of the project, but they also set up a foundation that others could fund the project, and they have a governance model of who gets to decide how those funds are dispersed.
Matt Odell: We support them through OpenSats now, too.
Steve Lee [15:03]: Oh that’s great. And I know it works well — they’ve attracted ten or so funders over time, so it’s great that it’s not just Spiral. It’s ten different funders. Now it’s not been perfect either — not all ten of those funders renew every year, but still: achieving ten funders is really great, and some do renew. That is what most projects would want to aspire to, unless they figure out a business model.
Matt Odell: Right, BTCPay is the gold standard of a non-Bitcoin Core open source project in the space, I would say.
Steve Lee [15:41]: Another thing I’ve observed in the space is that a lot of other funders who are doing great things by funding open source Bitcoin, they often are only comfortable supporting Bitcoin Core, because other projects — especially if they’re nascent or brand new — if they don’t have someone full-time or a technical Bitcoin person who has somewhat of a vision for where things should go or can go, it feels a little risky funding a new project or funding an unknown person. Like I said earlier: Spiral’s funded new people, bringing new talent to the space. It’s definitely harder! You have to spend more time evaluating.
Matt Odell: Especially if you’re a large company, you’re putting yourself out there.
Steve Lee [16:33]: Yeah, and we’ll walk through some of the folks we’ve funded in projects, and one actually has been controversial — we can talk about that one. And so because other companies might only be comfortable at least starting to fund Bitcoin Core, that’s also influenced how — Spiral has definitely funded 5–6 developers on Core, but we don’t overweight there for a couple reasons: (1) one, the reason I just said.
Matt Odell: There’s already a lot of funding.
Steve Lee: (2) Secondly, we don’t want Spiral funding the majority of Bitcoin Core. That’s also an unhealthy balance.
Matt Odell: It’s a bit of a threat to the network, is the idea.
Steve Lee [17:15]: Yeah. I mean in the past, other organizations — Chaincode, Blockstream —
Matt Odell: Even if there’s no influence, you don’t want to give the appearance of influence to Bitcoin Core, right?
Steve Lee [17:26]: Exactly. And even in our organization, individuals who are 100% good ethos and good intentions, there’s always bias. People have applied for grants at Spiral that we’ve said No to who are phenomenal people and do great things, and some of them come back to me and rib me because they’ve gone on and done great things and it’s like, Why didn’t you accept me?
Matt Odell: It’s always difficult saying No.
Steve Lee [17:54]: We’re also not perfect at picking folks. Pretty much everyone we have selected we’re really happy with, but there’s definitely been really good people who have applied that we didn’t say Yes to. So that’s another reason it’s great to have other organizations who have other views on what’s important to fund in Bitcoin.
Matt Odell: You have a pretty good track record! So we have Satimoto in the live chat. He’s wondering: Does Spiral only fund open source projects?
Steve Lee [18:24]: Yeah only open source.
Matt Odell: Do you think about the licenses at all? Does the license matter?
Steve Lee [18:33]: We do — it’s never really been an issue. There’s only one project that we had a discussion about licenses.
Matt Odell: Before you funded [them]?
Steve Lee [18:45]: Yeah, just because we tend to like MIT/Apache. We don’t exclusively fund Rust projects, but most are Rust, and the Rust default is to do dual license: both MIT and Apache.
Matt Odell: Got it, which is an extremely permissive license — you can do anything you want with the code.
Steve Lee [19:04]: Yeah, which is aligned with our philosophy.
Matt Odell: Right. But I know specifically you have funded some GPL licenses, which are a little bit more restricted.
Steve Lee: Which one?
Matt Odell: Mempool, right? Is that the only one?
Steve Lee [19:19]: Yeah, and I think one reason it’s a little different is almost everything we fund is protocol development, infrastructure, dev roles —
Matt Odell: So you want to be as least restrictive as possible?
Steve Lee [19:31]: Yeah, and Mempool is more [an] application — there is an API — it was a little outside of what we usually fund. So I think because of that they have a little bit different considerations for licensing.
Matt Odell: So, you know Mempool streams during all of my dispatches?
Steve Lee: Oh I didn’t know that.
Matt Odell: Mempool.space streams during all the dispatches — it’s literally on the screen right now. But yeah, it’s an awesome project.
Steve Lee [19:55]: And then also about our thinking on funding, we just feel so fortunate that Block gives Spiral a really healthy budget for doing what we think is amazing stuff for the space.
Matt Odell: Specifically, no strings attached?
Steve Lee [20:13]: No strings attached.
Matt Odell: It’s a $50 billion dollar company or something like that, right? It’s impressive.
Steve Lee [20:19]: Yeah. There’s a lot of really impactful projects that don’t have a natural business model, and that’s where we want to direct our dollars. So, some people have applied or spoken to us about — they have a for-profit business. We’re absolutely supportive of people creating startups for profit — we’d like to see that grow too, it’s just like: that’s not where Spiral dollars go. There’s several VCs that are targeting Lightning development.
Matt Odell: Including us at Ten31.
Steve Lee [20:54]: Exactly. We just feel like the entrepreneurship and venture capital model is proven to satisfy that. So ours goes towards free and open source.
Matt Odell: That’s great.
Steve Lee [21:11]: And then underlying all this, a point I wanted to make is that there’s more than just Bitcoin Core as a project in the space. Bitcoin Core is an extremely important project — it is the core of Bitcoin, especially the parts of Bitcoin Core that are consensus, peer-to-peer — it’s the foundation for everything else. So it is the most important project, but it’s not the only project. And I think a lot of developers who are new to this or interested in contributing are maybe only aware of Bitcoin Core. So I’d love to spread the word that there’s a lot of other important projects: there’s a lot of layer 2 protocols — Lightning and Fedimint and others — that you can contribute to, there’s multiple Lightning implementations, there’s also other Lightning infrastructure projects that we funded that are important like a watchtower project and the Validating Lightning Signer (VLS) project, which is a way to isolate signing Lightning transactions to its own separate computer and process.
Matt Odell: And the watchtower is Eye of Satoshi, right?
Steve Lee [22:18]: Eye of Satoshi, yeah. Those projects are agnostic to which implementation they work with, but they’re critical infrastructure projects for building robust trust-minimized Lightning applications.
Matt Odell: And on that front, too, I got to meet the person behind the Twitter account and the blog post, and you guys do a really really good job of just covering the open source ecosystem around Bitcoin and getting more exposure to not only the projects that you support but also the tangential projects.
Steve Lee [22:53]: We don’t even know if it’s human, or black cats. But whoever or whatever it is, they might be in Nashville and you met them.
Matt Odell: That was fantastic, but yes.
Steve Lee [23:09]: We talked about the LDK film last podcast, and that person was awesome.
Matt Odell: Right, Puppet Jack!
Steve Lee [23:16]: I actually revealed that person’s name in that podcast, for someone who wants to go sleuth. But yeah, there are a lot of projects that at least we believe are valuable, and I think a lot of developers in the space that have been contributing for a while appreciate that there’s more than just Bitcoin Core. So anyone out there looking to get involved with open source as an engineer or even as a non-engineer — designer, PM, this dev-type person — Bitcoin Core is not the only game in town in terms of contributing, and also funding. I strongly encourage funding entities to look at other projects that are important to Bitcoin.
Matt Odell: Great.
II. Stratum v2
Steve Lee [23:59]: Let’s get back to what’s happened in the past year with Spiral — Spiral-funded projects. Let’s talk about Stratum V2. So Stratum V2, people might have been hearing about it for years — Matt Corallo proposed BetterHash four or five years ago.
Matt Odell: And then that morphed into Stratum V2.
Steve Lee [24:24]: Yeah, Jan and Pavel at Braiins were working on a protocol proposal for Stratum V2. For folks that don’t know what this is, there’s a protocol called Stratum which is between miners and mining pools — it’s been around 10 years — it’s a core protocol for how mining works today. And as you might guess with a protocol that’s 10 years old, there’s areas of improvement — ways to make it better. That’s what Stratum V2 is. Matt’s primary focus with BetterHash was around decentralizing mining. And specifically, today: the only entities that can do transaction selection are pools — miners actually don’t do transaction selection.
Matt Odell: [That is,] which transactions go in the block.
Steve Lee [25:12]: Yep. A core property of Bitcoin is censorship resistance — arguably the most important property. And there’s many layers of censorship resistance, but a very important one rests with a few pools today. It’s not super brittle, because if pools did start censoring, it’s very low switching costs for miners.
Matt Odell: Right, you have a soft check on them: if they start acting malicious, if they start not including transactions, then you leave and go to a different pool.
Steve Lee [25:50]: Yeah. But it’s a stronger network, a healthier network, if we have better decentralization of transactions.
Matt Odell: And you’re essentially giving regulators a person, an entity that they can go pressure and say, Don’t include these transactions in a block.
Steve Lee [26:12]: Especially as the long time concern was concentration in China, which is a valid concern — now we don’t have that anymore. But now we have a new concern — which is arguably a lesser concern, but it’s still a concern — which is too much in the United States, North America, public companies, more entities. There’s more hashrate that’s closer to the US Treasury than before, and if that grows too much that gets to be a little scary.
Matt Odell: You want it to be distributed as much as possible.
Steve Lee [26:40]: So BetterHash, which has now been bundled into Stratum V2 — Matt, Jan and Pavel did a great job working together to merge their ideas and their proposals into one proposal with the Stratum V2 — that happened in September 2019 in Prague, and now we’re in 2023. So what’s happened since then? Braiins has gone ahead and they’ve implemented —
Matt Odell: Everything except the Job Negotiations.
Steve Lee [27:11]: Yes, so Stratum V2 has several sub-protocols — Braiins has implemented all but the Job Negotiation.
Matt Odell: Which is the part that allows the individual miners to construct the blocks of the transactions.
Steve Lee [27:27]: Which, all of the improvements are important, but it’s definitely the most important for decentralization. They have not done that yet, but they’ve done the other stuff — they’ve actually deployed that.
Matt Odell: The other improvements are good for efficiency and they’re good for security.
Steve Lee [27:40]: Yeah Stratum V2 adds authentication and encryption.
Matt Odell: Right, we’re just plaintext communicating with the Stratum servers.
Steve Lee [27:48]: Matt Corallo demonstrated before: hijacking hashrate.
Matt Odell: There’s just a man in the middle between the miner and the pool.
Steve Lee [27:57]: Yeah so all of this is just decidedly better — it’s just a matter of building it, testing it, making sure it’s high quality.
Matt Odell: And arguably the Job Negotiation part is the most difficult part.
Steve Lee [28:11]: Yeah it is, both to build and to deploy.
Matt Odell: Yeah because individual miners have to actually run their own nodes now if they want to participate.
Steve Lee [28:17]: So Braiins has done some of the work. There’s some miners that are using Braiins firmware and Braiins Pool — formerly known as SLUSH Pool — does support Stratum V2. So part of it is live, but all of us including Braiins wants to see much broader adoption: we want to see the network go to 100% and we want to see Job Negotiation being adopted.
Matt Odell: And if I’m correct with Braiins’s current implementation, there’s two main manufacturers: one is BITMAIN, the other is MicroBT, that makes the Whatsminers — you need to run Braiins’s custom firmware to do Stratum V2, because those two manufacturers don’t ship it in their stock firmware yet. And that only is available on Antminers.
Steve Lee [29:07]: That’s my understanding, too — at least as of last Fall that was true.
Matt Odell: So not only do you have to be using Braiins Pool but you also need to be running an Antminer that has Braiins OS on it.
Steve Lee [29:18]: It’s good that it’s not zero — some of it’s out there, real world usage. That’s always a good proof point, but we’re still very early phases. So a little over two years ago, Spiral funded a developer who goes by Fi3 to work on an independent implementation of Stratum V2, because there was a little bit of concern both from the folks at Braiins and from Matt Corallo, me, and others, that the industry perceived Braiins’s implementation as being — like, they’re a for-profit company and a mining pool, and they have their own vested interest. Those guys I think have super strong Bitcoin ethos, but still. We funded Fi3 to work on an independent implementation, and he’s been working on that for two years. Then along the way, a year and a half ago, Rachel joined that project — I don’t know if folks know Rachel but she’s at Galaxy. And she started contributing and did a lot of great work for the project. And now the project has more than half a dozen people contributing.
Matt Odell: And Foundry’s gotten involved, which is the largest US pool.
Steve Lee [30:38]: Yeah Foundry got involved, they funded another developer to work on Job Negotiator, and there are several companies that have a lot of developers involved.
Matt Odell: Through OpenSats we’re about to fund Stratum V2 as well.
Steve Lee [30:54]: That’s great. Yeah so that’s another six. It’s still not all the way there, but it’s a success story in terms of attracting multiple developers, multiple funders, and the project has good momentum — Pavlenex also. A lot of people probably know Pavlenex in the space — Spiral funds him, and he’s a non-developer! Another thing I’d give a shout out for is: if you’re not a developer, you can absolutely contribute and do really positive things for open source.
Matt Odell: He’s one of the reasons BTCPay Server was so successful, and now he’s taking Stratum V2 under his wing.
Steve Lee [31:28]: And in between, the Bitcoin Design community got involved.
Matt Odell: And he’s launching his version of Bitcoin Park in Croatia, which is great.
Steve Lee [31:36]: Anyone who’s not a coder but wants to help out and it’s nebulous or you can’t really figure out how to, he’s a good example to model after. You don’t have to do exactly what he’s done, but he’s found ways to add a lot of value.
Matt Odell: Let him inspire you, and also he’s always happy to give feedback and help you along your way, too.
Steve Lee [31:55]: In fact, quick tangent there: he started a new Discord and a new community — the Bitcoin product community — doing exactly what you just said. He wanted to help other folks. Connor and Haley from my team have joined, I’ve joined, Adam Jonas from Chaincode and Mike Schmidt — a bunch of people who have done a lot of great things in the space — either we’re not engineers or we’re not actively coding, but we find other ways to help.
Matt Odell: Yeah I already knew Connor — absolute legend — but Haley is amazing. She’s really amazing. First time meeting her.
Steve Lee [32:31]: Anyone interested in contributing in a non-coding way but has no idea how to start, join that Discord.
Matt Odell: How do you find the Discord?
Steve Lee: People can DM me on Twitter @Moneyball.
Matt Odell: It’s called Bitcoin Product Community? [I’ll] put it in the show notes.
Steve Lee [33:03]: Anyway, Pavlenex is involved in Stratum V2 and he’s helping steer the ship, like a good PM does. And a good PM is not the authority of a project — a good PM respects all the contributors in the project, knows that they’re not the boss, but they just find ways to help organize. They find gaps, holes, weaknesses, plug them, find other talented people to fill in, and that’s exactly what he does really well. And people people love Pavlenex. So Stratum V2: where is it at now? It’s now matured tremendously in terms of the code, the quality. It’s at a point now where people can start using it. I wouldn’t recommend a miner or pool put a lot of machines on it yet, but it’s absolutely ready for running on a machine, getting familiar with it. And I think with a period of time to do that testing, work out remaining kinks for production, it’s conceivable that later this year you could actually move some some material hashrate onto it. Now you mentioned earlier that the only firmware that exists for Stratum V2 is Braiins and it only works on certain machines, but one thing that this open source project has done — and it’s called SRI: Stratum Reference Implementation is the name of the open source project — SRI has created a proxy. So mining machines that are running Stratum V1 firmware, which is pretty much every machine, can still do Stratum V2 by running this proxy at the mining site. So it can talk to a Stratum V2 pool and you get the benefits.
Matt Odell: Oh so you don’t have to run the firmware or anything? That’s huge.
Steve Lee [34:59]: All existing equipment can get all the benefits of Stratum V2: you can get the security benefits. You can also do your own transaction selection. Also interesting is the way that architecture works for transaction selection: a third party can run Bitcoin Core and do transaction selection, so it doesn’t even have to be the pool or the miner.
Matt Odell: All they could do is just do transaction selection — offer it as a service, maybe?
Steve Lee [35:30]: I don’t know what the future holds, but I just know that with Stratum V1 — architecturally — the only option is the pool. And now anyone can do transaction selection, so hopefully that is good for decentralization. It’s not a silver bullet but it at least provides options.
Matt Odell: The way I think about in my head is: even if it’s not fully rolled out, if there’s a situation where censorship starts, the fact that it exists will mean that people can quickly move to it if they need to.
Steve Lee [36:07]: Yeah. Also you mentioned earlier: it means a miner would have to run a full node or connect to a third party that is. That’s going to be an operational hurdle, but I also think there’ll be products coming out on the market that make it easier for miners to do that.
Matt Odell: I mean we have .eths running full nodes now so that they can inscribe NFTs on the Bitcoin blockchain — I think miners can figure it out if they need to.
Steve Lee [36:30]: I know Block itself has a new mining initiative where they’re designing their own custom ASICs. They’re designing rigs — it’s all going to be open. I’m super excited about that — there’s tremendous progress on that, separate from Spiral. That’s a commercial effort at the company, but the great thing about the commercial efforts at Block is strong Bitcoin ethos. And generally to the extent the company can make it open —
Matt Odell: All the way from the top — Dorsey really believes in Bitcoin.
Steve Lee [37:07]: Jack definitely does.
Matt Odell: I have to call him Dorsey because we have Mallers too, otherwise it gets very confusing.
Steve Lee [37:15]: Yeah and jumping into that really quickly, I’ve been at the company now going on four years, and I’ve seen a lot of Bitcoin ethos grow within the company, too. The way Jack runs the company — just because he’s a big believer in Bitcoin doesn’t mean thousands of people at the company all are as strong Bitcoiners as you, me, him, or the folks on Spiral, but when I joined an internal Bitcoin Slack channel, there were 30 people on it when I joined, and now there’s 1,500–2,000 people — it’s grown tremendously. The interest is very high and I think we’ve also seen a shift. The interest is not all positive: there’s the curious and there’s skeptics, which you’re going to find in any big tech company, and it’s healthy. But we’ve seen a shift: curious people have turned into Bitcoin advocates. And also there’s more and more commercial projects and products within the company now, and those teams generally have a good balance of hiring some external people with Bitcoin development experience, but also others that are new, and that’s a really healthy mix as well — it creates really good questions during the development of products. But the Bitcoin mining initiative at Block, they certainly have Stratum V2 in mind — they’re going to support Stratum V2, and it’s certainly in their interest to make it easy for miners to run a full node and improve the user experience for miners when they’re buying rigs. So Stratum V2 — I’m super bullish on it. Three years ago I loved the proposal and the tech side of it and the promise, but I was worried about adoption. I’m less worried about adoption now. It’s still a challenge: miners have to run a full node — that’s a challenge. What is the incentive? I guess a new insight for pools — the only incentive I thought would be if miners demand it, pools will emerge and support competition, but I think even existing pools will have an incentive, especially if their compliance team might actually want them.
Matt Odell: You see that with Foundry: they want to reduce that regulatory risk, because if they are forced to censor then all of a sudden their business model dies because a lot of people will move to a different pool.
III. Fedimint
Steve Lee [40:03]: Right, so if you’re a US pool like them, you might have an interest in not selecting transactions. So I think there will be natural incentives to distribute that. In terms of the last year, Fedimint’s really come onto the scene and it’s a new project, it’s an exciting project. I’m a huge supporter of it. Spiral has funded three developers now — two have been announced, one will be announced soon for that. The three we funded all have fantastic backgrounds — they’re all super enthusiastic about Fedimint. At least a couple of them left high-paying big tech company jobs because they’re so mission-driven. All these folks are amazing. But it’s important for everyone to keep in mind: Fedimint is an open source protocol, and the three developers that we’re funding — and Eric who started the project — they are focused on open source Fedimint, and hopefully there’s lots of applications built on that, lots of clients, lots of companies built. Separately, there’s a company, Fedi, which Obi and Justin Muun co-founded with Eric. And Justin’s definitely involved with the open source work, but Justin and Obi — they’re trying to build a business. It’s important to keep those two separate in people’s minds, but I think it’s great.
Matt Odell: Eric’s a co-founder but right now he’s mostly focused on the open source protocol work.
Steve Lee [41:58]: Yeah, and I’m super excited for what Fedi will do. I’m an investor as well in it.
Matt Odell: Same.
Steve Lee [42:04]: I’m a believer. I also hope there’s many competitors to Fedi. For a healthy protocol, it’s not going to be just one company, otherwise it’s not really an open protocol.
Matt Odell: Just to be clear: the way the audience can conceptualize where Fedi comes into the Fedimint open source protocol is Fedi’s essentially a front-end app to the open protocol. So ideally we would see many front-end apps for that protocol.
Steve Lee [42:32]: Yep. And hopefully this year they’re gonna launch sooner than later and get some proof points, get some adoption, show some real success, and that will attract competitors.
Matt Odell: I already know of at least two teams that are working on another front end.
Steve Lee [42:51]: Oh that’s great. And the reason I’m excited about Fedimint: number one is privacy, and just e-cash the concept — I love that it’s finally being applied to Bitcoin. It’s a way for people to have really really strong privacy with a good user experience. And the second thing is scaling, because as much of a proponent of Lightning as I am, and Spiral is — and we invest a lot of resources on it and we’re going to talk about it on this show — we still don’t have a long-term scaling plan for billions of people, machine-to-machine payments, with Lightning. So hopefully we figure that out over time but, we might not. And the great thing about Fedimint is it can scale to everyone on the planet. It also interoperates really nicely with Lightning. It doesn’t compete with Lightning — it actually enhances Lightning, and Lightning enhances it. I see a lot of synergies between those.
Matt Odell: Yeah it’s very complementary.
Steve Lee [43:58]: But it’s not perfect — everything has trade-offs. It is custodial. I think we’re both huge proponents of non-custodial trust-minimized, self-sovereign usage of Bitcoin — that’s where most of Spiral’s resources and budget goes towards. And so it’s a trade-off with Fedimint.
Matt Odell: Well one of the reasons that Fedimint’s a cool project to me is that it’s very pragmatic. First of all it doesn’t need any changes to Bitcoin, second of all it is custodial, but it is reduced custodial risk from the purely custodial — like, right now we see a lot of people using Wallet of Satoshi, we see a lot of people using Binance, these fully custodial wallets. But with Fedimint you have this idea of essentially a multisig custodian where you would need multiple people, multiple entities to come together to essentially exit scam and take your money, right?
Steve Lee [44:57]: And the types of people who would be those custodians or guardians can be very different. So if you’re FTX: if you’re in the Bahamas and your coins are with someone you have no relationship to and don’t know at all, obviously you can see the risks that happened in the past year and actually throughout the history of Bitcoin — lots of exit scams. But with Fedimint, not only can you have a 5-of-7 in which you need at least five people to collude, but if you’re doing it in a community where a lot of communities around the world you have generation upon generation of people who stay in that community and their family name and reputation means something — their personal reputation means something — there’s a lot of strong incentive not to screw people over.
Matt Odell: And that’s specifically a massive priority of the Fedimint open source project, is that it’s easy — not everyone can just launch a Wallet of Satoshi, but it needs to be easy for these small communities to just spin up their own Fedimints and you’ll have many different Fedimints of different sizes and different trade-offs and whatnot.
Steve Lee [46:11]: So the ideal outcome there is that there’s thousands, tens of thousands, maybe millions of Fedimints around the world — they’re all small-scale.
Matt Odell: You could even have one that’s just your family, and maybe you use it for larger amounts. And then the community one you use for slightly less amounts. And then the big continent-wide one you use for even smaller amounts.
Steve Lee [46:36]: Yeah, I’m super excited about that future. And again, there’s no altcoin — it’s using Bitcoin. Bitcoin is money: you just send and receive Bitcoin. It’s compatible with Lightning, it’s compatible with the Bitcoin Network.
Matt Odell: That’s the key, because you can send between the different Fedimints using Bitcoin, using Lightning.
Steve Lee [46:59]: Yeah I could be a user that’s never heard of Fedimint and know nothing about it, but I can send and receive money with you.
Matt Odell: Yeah you can pay me through your full node using Zeus to my Fedimint wallet — boom, done — using Lightning.
IV. WalletScrutiny
Steve Lee [47:12]: Let’s talk a little bit about WalletScrutiny, which is the project that I mentioned was a little bit controversial, and before the show you said you had maybe some concerns about it. Let me describe the project, why Spiral’s excited about it, why I’m excited about it, why it’s important for the space, and why I think this particular project can do a lot of good for the space. For people not familiar with WalletScrutiny, it’s been around three maybe four years. The idea behind it is that even if you do self-custody — you’re using a mobile wallet — you’ve eliminated a lot of risks, but you also introduce new risks. And some risks are that even if that project is open source and you can look at the code and you might have a trusted friend who’s an engineer who’s looked at it and says it’s good, you actually don’t know if the binary from the App Store on your phone matches what you’re running. They could be a man in the middle attack, whether it’s Apple or a third party or the wallet developers themselves. Or even if the wallet developers are earnest, their computer could be hacked. So they could be super-high integrity, but if their computer’s hacked then all bets are off. The devastating scenario is a wallet that becomes very popular — millions, tens of millions, 100 million people are using it — and there’s a zero day type of attack where everyone’s wallet is swept in one moment. That would be very bad. The goal of WalletScrutiny is to help reduce that risk by really focusing on reproducible builds. Reproducible builds, for people not familiar with it, is when an independent party can take the source code that’s open source, build the software, and confirm that the binary they produce is the exact same that was digitally signed and what you’re running on your phone. It’s a technical process, it’s complicated, trying to do this across hundreds or even thousands of projects is an enormous task, but that’s what this project has set out to do. And then the project tries to categorize each wallet and communicate to users if there’s any concerns. So it’s an important problem, the intention of the project is really good, and I’ve gotten to know the folks involved in it. I met Leo three years ago, gotten to know them over time, as well as several other people that have been working with them. I think they’re very earnest and trying to do the right thing, so I’m excited about this project. Are there areas of improvement? Absolutely. And others have let me know that — I’ve seen it myself but I’ve listened to others, so I’m excited to go fix those. For one: any project like this is naturally going to be controversial in the space because it’s a little bit antagonistic to the wallets and wallet developers. Most people don’t want to be scrutinized! It’s just naturally going to be contentious for that reason. If you feel like it’s unfair scrutiny or it is unfair scrutiny, then you definitely are not gonna like it. And one challenge the project has had — I think they’ve reviewed literally over a thousand projects with a very small team of volunteers who are not paid, so it’s just been way too much work for a few people who aren’t paid. Hopefully with funding it can attract more people, but I do think there’s a quality/quantity challenge that they’ve had: too many projects, not enough resources — that’s one challenge. And I also think that the design of the website and the copy can be improved.
Matt Odell: It scares the shit out of a lot of people — the average end user when they go to the website
Steve Lee [51:36]: It scares the shit out of people. And it’s a really tricky balance though because you also do want to scrutinize. It’s a tricky balance. So you need to balance doing what’s right for the user and being transparent and communicating important information to them so they can make a good decision — you also don’t want to necessarily go out of your way to be adversarial to wallets, because most of these wallets and wallet devs are absolutely trying to do the right thing. And also, there’s no perfect wallet.
Matt Odell: And a lot of them don’t have their own business models — they’re just pure open source projects. They’re basically contributing their time and energy to develop and maintain [them].
Steve Lee [52:20]: And if we walk through every wallet, both of us could find flaws everywhere. There’s no perfect wallet. It’s easy to find flaws, point them out and highlight those, and scare people away. And then you have to ask yourself: If this mobile wallet checks 8 of 10 boxes, do you want to scare people away from that? And then they keep their coins on an exchange.
Matt Odell: Or even worse: I had Sergej on from Bitrefill, and one of the most commonly used wallets there is Trust Wallet by Binance. Reproducibility is great, but they use a reusable Bitcoin address — it’s just the one fixed address. There’s a disconnect.
Steve Lee [53:08]: So it’s a really hard problem because it’s not a simple thumbs up or thumbs down answer to any of these wallets — there’s just a lot of nuance, a lot of trade-offs. I don’t know what the right answer is, but I’m optimistic because we reached out to the Bitcoin design community to see if they would help with this project. And we got an amazing response from the design community, because they saw the importance of this project — they saw the seeds of something that could be really valuable, but they also saw ways that they can help. And so several people from the design community have been working for the past few months on the project. And also Spiral’s copywriter — who does all of our copy for our tweets and website and our creative person — he’s helping as well on the copy for that site. So I think we’re going to see a lot of improvements there. And also, the goal of the founder of the project is not to just have him be the only one behind it — he doesn’t want to be the dictator of all these reviews.
Matt Odell: I think that’s where a lot of the concerns lie.
Steve Lee [54:32]: He doesn’t want to have that responsibility — Spiral doesn’t either. And hopefully with funding now and more resources and just more people in the community knowing about it and helping out, we’re going to get more expert opinions, more contributors to the project, and it becomes less about one person’s personal review on every project, and much more of a community feel. So we’re in a good spot in that there’s awareness of some of the problems and challenges — there’s a plan to address them, so let’s see how that turns out.
Matt Odell: I’ll just say my piece really quick: first of all it’s an important goal, it’s an important mission. My concerns have mostly been the maintainer — it feels like there’s a little bit of bad faith that happens. Like we said earlier: it really scares the shit out of people from otherwise very good wallets that have very strong code bases that really care about security. You mentioned earlier that there’s no business model attached to it, but the lead maintainer is former Mycelium wallet — if you go to the website, he only says there’s five wallets that work, that are reproducible, that are good, and one of them is Mycelium. And so there’s some issues with incentive alignment there, and that’s why it’d be great to see more contributors — I completely agree with that. And then last but not least: anyone who’s using an Apple phone — this doesn’t help them at all no matter what, because you cannot tell what’s what’s running on the phone when you download it from the App Store. You can’t sideload. On Android it’s a different story: on Android you can build from source, you can then pull an APK, you can check PGP-signed APKs. And especially in the global south and whatnot, the majority of them are using Android phones: if you look in Africa, you look at Latin America, on the Android side it can be very helpful. But at least in America on the iPhone side, I’m not going to say it’s not a priority, but you’re not really accomplishing any real goals on the App Store side. If you even check on this website right now, the five that he has are all Play Store — none of the App Store ones are reproducible. But like I said: you wouldn’t even be able to tell, if it was on iOS. Anyway, that’s my concerns. But then again: first of all, no one’s perfect — there’s always going to be controversy whenever money is involved in general. You know better than most, but I have seen it firsthand as well. So yeah that’s where I come from, but I’m interested to see how it goes going forward.
Steve Lee [57:39]: Cool. I agree with those concerns. What I hope people cheer on is these improvements that I talked about, plus others, and if we see those improvements it’ll be a really positive thing for the ecosystem.
Matt Odell: Agreed.
Steve Lee [58:01]: And one thing just to hammer on more: relationships with the wallet developers is a really important relationship to have. It’s always going to be naturally a little bit contentious, but you don’t have to throw fuel on the fire. And also: ideally, wallets see the value in this and actually contribute back, because building the open source software is not always straightforward — some projects don’t have great instructions to follow, so you have to figure it out yourself, which has caused some tension. So hopefully better relationships can be built with the wallets, and some wallets hopefully see this as a great way to market their wallet as well, if they do reproducible builds.
Matt Odell: There’s a direct incentive there.
Steve Lee [58:59]: Yes, and if they do that then others will be incentivized to follow. But that’s all for WalletScrutiny. Lots of other projects we support — we don’t have to go through them in detail — Summer of Bitcoin. We’ve been a huge funder and supporter of that.
Matt Odell: That’s been impressive man — that’s been really impressive.
V. Separating the Core Out of Bitcoin Core
Steve Lee [59:19]: They’ve had a really good reach. I mentioned earlier we have funded six Bitcoin Core developers. And we just announced Sebastian, who is taking over for Carl Dong’s work separating out the core of Core, which has been various attempts and projects for longer than I’ve been in Bitcoin to do that. It’s hard but it’s very important.
Matt Odell: So right now when you run Bitcoin Core you have this full wallet, you have everything bundled together with the consensus code, but it would be nice if we just had that consensus element — arguably the most important part, the foundation of the network — separate from everything else, so we could keep that stable while other iterations are happening, right?
Steve Lee [1:00:11]: There’s numerous benefits by doing that — one is just general software engineering practice: if you isolate, that you’re less likely to introduce bugs in the really really critical stuff if you’re adding something.
Matt Odell: We don’t want accidental forks.
Steve Lee [1:00:25]: Right. There’s also the benefit for the contributors themselves. Contributing to the GUI QT wallet is a very different type of exercise than changing consensus codes or mempool code, so it’s really two types of contributor bases. And just being able to separate that is important. And also there’s been different views and camps as to: Do we need more competition for Bitcoin Core? Should there be more implementations? And the folks that think we should put all the arrows behind one project — Bitcoin Core — generally point to: it would be dangerous to have independent consensus implementations because there’s no spec, and the spec is Satoshi’s code and it’s really complicated and we don’t want inadvertent forks. I sympathize with that view, however it would be great if we can separate that code out so that all other parts of Bitcoin Core can have a lot of competition and competing projects. In my opinion that’s the right compromise or approach on that. Carl did a lot of great work on it — he’s moved on, but to his credit he found his successor. So Sebastian is a developer who’s going to take over that project and we’re funding him. We funded Vasil and John Atack and Gloria — folks that are working on nitty-gritty peer-to-peer, mempool, and privacy. Like, Vasil added TOR v3 support. We want to fund the more core part of Core and less of the wallet, not because we don’t believe wallets are valuable, but just in general there’s hundreds of wallets, so there’s no funding strategy where we can fund all of them, and we don’t really want to pick and choose. Spiral’s about building dev tools, infrastructure to make it easier for there to be more wallets, and require less resources to build a great wallet — that’s where we focus our [attention].
Matt Odell: A strong foundation to build on top of.
VI. LDK/LDK Node
Steve Lee [1:02:43]: Yeah. Let’s talk a little bit about LDK. I know that Jeff and Val just were on your show the other day so I don’t have to repeat what they already talked about, but just to highlight or re-highlight things that we’ve learned and new initiatives within LDK project the past year: one is that we heard a lot of feedback from people using LDK. There’s really two camps [of developers]. For folks not familiar with LDK, it’s a full Lightning implementation, in some ways similar to LND, CLN, Eclair full Lightning implementation, but the way it’s delivered to people is not a binary — it is an API with hundreds and hundreds of methods. It gives the developer a lot of control and flexibility and power to build awesome Lightning applications. So for developers who have maybe tried to build a Lightning product, or they have one — Lighting’s super complicated — they tend to know a lot more about Lightning and they know what they want to get out of their application. They tend to love LDK because they see exactly how it addresses their problems. So for that group, we’ve seen over a dozen projects actively developing, with at least two developers on LDK right now. We’re seeing growth there. But there’s another group of developers who it’s their first experience building Lightning applications — they’re experienced developers, but because of Lightning’s complexity they’re at the point of just trying to get LND or CLN up and running and connecting to peers and learning about rebalancing channels and funding and things like that. So LDK is both perceived as too complex, and the reality is: you’ve got to roll up your sleeves and do some coding, and they’re not ready for that yet. One project we started last Summer is called LDK Node and it’s almost ready for to be to be unveiled and people can start using it. While LDK API is not opinionated on how a Lightning node or wallet is built, LDK Node is a specific recipe, it is opinionated, and it is a full Lightning node presented in an API of six functions — very simple: initialize, connect to peer, fund channel, send payment. That level of simplicity. And it should make building a mobile wallet much much easier. For example, it’s made a decision about how is block data sourced, and it can be sourced from Esplora or Electrum, which are two popular options for mobile. It includes a BDK-based on-chain wallet, because LDK API by itself doesn’t have an on-chain wallet — you bring your own wallet. Which, if you already have your own wallet, LDK is amazing because LDK doesn’t make you switch to LDK’s idea of a wallet. But LDK Node is its own on-chain wallet.
Matt Odell: In comparison to something like LND — just to put in perspective: it’s this full integrated blob that you just install and it has the on-chain wallet, it has everything already.
Steve Lee [1:06:19]: It has everything, and that makes it much easier to get up and running and start funding channels and send receive payments, which certainly has an attraction to it. People like to get up and running. For a lot of uses, people are successfully using LND in the space and it’s 80% of the network, but for other applications that maybe never got off the ground, they want to use Lightning in a new way.
Matt Odell: It’s not as extensible as something like LDK or LDK Node.
Steve Lee [1:06:53]: And having having to fork the Go code of LND and modify that has a number of barriers: you have to understand Go, maintain that code. Any Lightning implementation is very complicated and it’s just generally not something one wants to do. But LDK Node is a full Lightning node implementation with an on-chain wallet, and we think it’s a great starter plan for anyone interested in building a mobile wallet.
Matt Odell: A lot of the focus in general on LDK is has been mobile, right?
Steve Lee [1:07:40]: The emphasis has been mobile. We’ve always thought, or at least I’ve always thought LDK can absolutely run on servers as well. That’s what Cash App’s doing. Our the primary mainnet application of LDK right now is on a server, but we want to focus on where we see the greatest need in the ecosystem. And from the start we thought mobile was underserved, non-custodial underserved, privacy underserved — so that’s where we tend to prioritize. But we can talk a little bit about some shifting views on that as well, and some need for server work. And actually I can speak to some of the users of LDK beyond Cash App that are using it on servers.
Matt Odell: I’m curious your opinion: if the goal is a library and software that’s built on top of it that’s highly efficient and can work in a mobile environment — because obviously mobile has a lot of constraints already — usually it’s a lot easier for that to be adapted to a server environment, but the reverse is not true: if something’s built for an always-on, high performance server, that thing is probably not going to work on a mobile phone. And when you talk about mobile phones in general, there’s going to be billions of people that use Bitcoin that only use it from their mobile phones — they’re never going to use a desktop computer, they’re never going to use a server, they’re never going to use a laptop — they’re going to be doing mobile-first. So for us to tackle that problem is absolutely crucial.
Steve Lee [1:09:12]: That’s correct, and that’s been our view and how LDK’s been architected and designed. There’s been attempts at running a Lightning node on the phone — it generally has produced a subpar performance experience. I think that can be boiled down into two or three things: one is that if you try to do pathfinding — when you’re finding a route for your payment on the phone — you need to know the entire network graph of all the Lightning nodes to be able to select an optimal path. That’s a lot of data. And so when there’s been attempts where all of that data is downloaded to the phone, it can cause a problem when the mobile OS won’t run the application in the background. So if it’s been a week or a month since a person opened their wallet, it might spend dozens of seconds or even minutes downloading that network graph just to be able to route a payment — that’s a showstopper user experience. People aren’t going to tolerate that.
Matt Odell: Any friction — any people will go somewhere else.
VII. Rapid Gossip Sync
Steve Lee [1:10:28]: Jeff and Val talked about this in the show a couple days ago, but a project that Spiral worked on in the past year is called Rapid Gossip Sync, which just shrinks the amount of information that needs to be sent to the phone dramatically. We’ve measured it as less than one second to download the data for the full graph and process it. So that can be done when the app starts up and it’s hidden from the user. The reason why pathfinding the phone is important is privacy, because if you outsource that to a third-party server — which is also often done today — that third-party server knows every single payment you make and has your entire [history].
Matt Odell: Right, how that transaction flows through the Lightning Network.
Steve Lee [1:11:20]: It’s a huge privacy win by running on the phone, but I think historically that’s one reason people have seen bad performance on a phone. Hopefully Rapid Gossip Sync eliminates that problem and solve that problem. A second reason: some wallets have used compact block filters, also known as BIP157, BIP158. It’s also been called Neutrino — lots of names for it. I mean I love the technology — there’s all kinds of great benefits: it allows you to connect directly to the Bitcoin peer-to-peer network, get your block data from the peer-to-peer network, you’re not connected to one trusted server for block data, it’s excellent for privacy. But the challenge on a mobile phone is that the compact block filter — the name implies that it’s not the full block — it’s a filter but it’s still 3% of the block size. So your mobile app still has to download 3% of blocks.
Matt Odell: It’s still pretty heavy for mobile.
Steve Lee [1:12:26]: Especially if we’re able to figure out how to background this, then it’s easy to do that in the background and when you start your app the user would not notice that. But that’s a big big challenge on mobile phones, the way the OS schedules things — it’s non-deterministic.
Matt Odell: They’re just killing processes left and right in the background, yeah.
Steve Lee [1:12:45]: For good reasons overall. And for battery life. Both Android and iOS have have introduced more advanced APIs over time to facilitate more backgrounding so that’s good, but it’s still not in any way guaranteed. The other problem: even if it was perfectly backgrounded then you’d have no performance problem, but for some people around the world, maybe many people, it might be cost prohibitive because it could be 100 to 200 megabytes a month of data for these compact block filters. And for some people’s data plan that’s no big deal — other people that might be meaningful, maybe several dollars or even more per month. There are trade-offs there. I’d like to see wallets that support that, but that’s been another challenge in the past. I have no grand solution to that. I’d love to see wallets do it and try it out and see if number one we can get it performant, and then at least for some users that the data will be fine. So the solution to that though is just you use Esplora or Electrum — it definitely nails the performance criteria. It sacrifices a bit of privacy if you’re using someone else’s. You can run your own Electrum whatever, but if you’re talking about mainstream, they’re not going to. I do believe in a future where someone makes a $5-$10 dollar board that can go in your home that goes in some other hardware device that you put in your home, and then 100 million people would use that. But that’s still a small percentage. I mean that would be great — that’s 100 million more people — but it’s still not billions of people.
Matt Odell: You gotta operate under the assumption the majority will not.
Steve Lee [1:14:46]: But anyway, almost everything else about a Lightning node is totally performant on a mobile phone. Absolutely Lightning nodes can run on a mobile phone in terms of performance. The other big challenge on a mobile phone though is that it’s like a leaf node on the Lightning Network because you it can’t be a forwarding node — also known as a routing node — because it’s not a server up and running 24/7. It’s only when the user’s operating with the wallet. As a leaf node, that introduces liquidity challenges. As a forwarding node, your node might rebalance itself — it depends on who you’re connected to and how you’re connected, but if you’re a leaf node it’s all about your own spending patterns. And depending on your spreading patterns and how you receive money, it could balance out nicely or not. And it just introduces really tricky problems with inbound liquidity and how do you get that inbound liquidity, who’s managing channels. People who are running Lightning routers tend to know how to operate — they’re at least enthusiasts or hobbyists who like to do it or cares about it. For mainstream users, no way. And already Breez and Phoenix and Muun Wallet have shown if you automate that and hide that from the user, it dramatically improves the user experience, so there’s just no question that’s needed.
Matt Odell: Once again, friction: I would go as far as saying if there’s any channel management or any liquidity management, it’s already lost for most users.
Steve Lee [1:16:28]: There’s no question that has to be hidden from mainstream users, but it’s really hard to do. The good news is those wallets I mentioned — they’ve done it, and they’ve done a great job at first attempts at it. I love the Phoenix wallet, I love Muun the user experience — it’s not a full Lightning wallet — it’s Lightning in terms of what you see as a user: you can you can handle Lightning invoices and pay over the Lightning Network, but the actual implementation is not Lightning.
Matt Odell: Every transaction is an on-chain transaction. All the balances are held on-chain.
Steve Lee [1:17:09]: Whereas Phoenix is a true Lightning wallet, and so if fees rise on-chain, Phoenix should outperform Muun in terms of fees.
Matt Odell: Muun has had the benefit of very low fees on the Bitcoin Network — that will probably not always be the case and it seems with these inscription NFTs, that might be getting supercharged.
Steve Lee [1:17:57]: All these wallets we’re mentioning, the people behind them have just done incredible work because they’ve had to deal with very poor development tools, a Lightning protocol that’s still a work in progress, and strive to create as good of UX as they could. They couldn’t have done better, basically.
Matt Odell: Just to go back for a second — not trying to be a dick but — if you go to walletscrutiny.com, they tell you not to use Phoenix wallet.
Steve Lee [1:18:29]: Another one is BlueWallet — they call it custodial.
Matt Odell: That’s what the bad faith feels like.
Steve Lee [1:18:35]: I agree. With BlueWallet, again it gets to this nuance, because BlueWallet Lightning is custodial and it’s important for users to understand that. BlueWallet on-chain is non-custodial.
Matt Odell: And you could use BlueWallet Lightning in a non-custodial way, but by default it’s custodial.
Steve Lee [1:18:55]: Yeah there’s just lots of nuance and it’s a difficult problem. I don’t think you’re a dick. So just back to liquidity and Lightning challenges: we have examples of wallets that did a good job of creating a user experience around that. They’ve tried different business models there, because as an LSP where you’re providing inbound liquidity to an end user wallet, you are tying up your own money in a channel that can’t get forwarding fees through other users. There’s just a cost of capital there. There needs to be some business model. We’re seeing different experiments tried — it’s healthy. We need to see more experiments. Another development that we need to see is just a healthier LSP marketplace.
Matt Odell: It seems we’re going to the same place. It’s a bit of a trusted relationship: Phoenix is self-custody but they’re the node you’re connecting to and they’re providing all the liquidity, so if the Phoenix team disappears tomorrow, you can’t use the wallet anymore. You can get your funds out if they’re not actively malicious. If they’re not actually trying to steal your money, you can still get your funds out.
Steve Lee [1:20:22]: But even if they’re trying to steal your money, as long as you’re using the wallet regularly — there’s no watchtower involved but — you have to keep it open. Theoretically they can’t just at one moment steal it. As the counterparty of the channels they could try to unilaterally close out if you’re not paying attention, then they can steal the money. It’s definitely not perfect trust-minimized — I find at this moment in time it’s acceptable, but to get to a better state would be more LSPs. LSPs that are not just tied to the wallet, but LSPs as a business and then they supply many different wallets. And then wallets that use multiple LSPs simultaneously, or easily be able to switch. So having standards between a wallet and LSP I think is important.
Matt Odell: Then we’ll have a proper marketplace and you’ll have competition and more providers. And I just want to go back real quick on Phoenix just because I feel they deserve a shout-out: pretty much all tools have trade-offs, and the Phoenix team is amazing at disclosing the trade-offs on their website — they go through everything for you.
Steve Lee [1:21:37]: Yep, they they’re very good actors and they contribute in many ways: not only a great user experience Lightning wallet, but they’re protocol developers as well on Lightning. It’s a great team there, but every project, every team, every wallet can definitely improve.
Matt Odell: So we want LSP marketplaces. We want some standards there so that there can actually be multiple providers.
Steve Lee [1:22:01]: And the good news is there’s activity on that: there’s a team at Block working on Lightning infrastructure. It hasn’t been announced in a big way yet, but Nick Slaney leads this team. He’s on Twitter — he’s talked about it, and they’ve hired some really great folks. They’re building up really cool Lightning infrastructure stuff which will be talked about by that team in more detail soon. Synonym has a Blocktank which is an LSP. There’s a couple of developers, a couple of teams, companies, that are using LDK. There’s a team that’s taking LDK — talk about running LDK in a server — they’re running it on an enterprise HSM and they’re building out an LSP model. There’s other users of LDK that are building LSPs as well. Some of them are building it just for themselves, some want it to be open to others and also be a business. So there’s definitely activity and development on that front. I expect to see more of this to be public, be real, be usable this year, and then bleeding into next year it’ll be much better. There’s been efforts for a standard spec. It’s hard because there’s a lot of options out there. Lightning Labs has some liquidity options, Blockstream’s proposed liquidity options, companies like Bitrefill and Breez and ACINQ with Phoenix — they developed their own homegrown for their own wallet with lots of different ideas — it’ll be a little messy over the next few years but it’s generally good messy. We want a lot of experiments, and hopefully over time the ecosystem will see what’s sticking, what are good models, and they can become standards that LSPs and wallets adopt. What we don’t want is one LSP to become dominant — especially if it comes from a big company and everyone’s using that and it gives the lowest fees, the best performance, and it attracts everyone — that’s not a good outcome at all for Lightning.
Matt Odell: Massive central point of failure.
Steve Lee: That’ll be a big development for Lightning.
Matt Odell: Tony in the live chat is asking: Is there an LDK/LSP group? It seems like a lot of potential overlap and duplicated work that might be going on?
Steve Lee [1:24:45]: That’s a great question. Actually about a year ago the Spiral team — and me, specifically — started looking into that, because if there was code that could be written that could be reused by others, that would be certainly within scope of something Spiral would like to do. Spiral doesn’t want to operate an LSP — we’re not a business, we don’t want to have operating capital and operating ongoing servers and stuff, but our conclusion there was: not really. Also I believe Blocktank is open source, I believe stuff Breez has done is open source — I expect stuff from Block to be open source. And then any emerging spec I would expect to be open source as well. I’m not sure if Spiral needs to really engage on that — it’ll come. But if anyone has specific ideas on what Spiral can do, we’re all ears on that.
Matt Odell: And it’d be great in general — it doesn’t have to be Spiral, but it sounds like there probably should be at least maybe a Discord group or something that people that are working on.
Steve Lee [1:26:01]: There is a Telegram group for the LSP that is public. It’s a working group. This is not a help desk, but if you’re interested in the LSP spec, absolutely people are welcome to join that. Getting back to — LDK Node, that’s been a big project the past eight months or so. It’ll be ready soon for mobile, but as we developed it we realized we can take the same code base and run it on a server as well. Really we just need to reconfigure it: instead of using Esplora or Electrum for block data we just plug in Bitcoin Core for full blocks because that’s the obvious choice if you’re on a server because you’re not resource constrained. You may or may not want a different local database — we’ll see, but we think without that much extra effort, we can configure LDK Node on a server which will make it much easier for users to get up and running with an LDK-based solution on a server. We’re also considering just going that final step with that six method API and just shipping a daemon — actually shipping a binary — because it’s not that much extra effort. Why would we do that? Well there’s a lot of features that we’re building into LDK now like BOLT12 and async payments — splicing is on the horizon — and things that wallets and LSPs are demanding and wanting. And we might be the only or the best solution out there for them, so it would be a shame if it’s not adopted by someone who wants it just because they’re intimidated by even a six function API.
Matt Odell: So in that situation would that be a binary that targets end users? An end user could just install the binary and be running it? Or is that more of a developer-focused thing still?
VIII. VLS
Steve Lee [1:28:18]: That’s a good question — is it targeted at someone who’s running a business like a Strike or OpenNode or something like that, or is this for someone with a Raspberry Pi or an Umbrel? I don’t know, I’m thinking more though the larger companies. We can get to BOLT12 and async payments and the problems that it solves — the status — but the deployment of that is really hard because of interoperability. So it’s important that companies that are server-based that want to support BOLT12 and async payments that they’re able to and have many options at their disposal. We’re looking at that. Jeff and Val spoke about the VLS project which is new on the radar — a lot of people probably haven’t heard of it because it’s relatively new and it’s not even launched yet, but it solves an important problem, which is: if you have anyone building a mobile wallet with Lightning, you have critical state that if you don’t back up you can lose your funds. If you only store it on the phone — you lose your phone — you are at the mercy of your peers. Which might be your LSP — which you can probably trust in certain situations — but you are at their mercy to get your money back. There needs to be a better solution. You should be assured that if you lose your phone — which all of us lose a phone at one point in life — that you’re not going to lose your money. There needs to be a way to back up that state on a server in the cloud securely and in a private way. You need to encrypt the data and then store it in the cloud, so whoever’s storing it in the cloud, they can’t see it, they can’t mess with it. The VSS project is all about solving two problems: one that backup, and two being able to access your wallet from multiple devices. And Jeff and Val talked about that a little bit. I don’t believe anyone supports that today.
Matt Odell: It’s something my mom would expect, if she’d lose her iPad [or iPhone].
Steve Lee [1:30:42]: Any user would expect this. A lot of users only have a mobile phone around the world, but there’s a lot of people that have a laptop and a phone, or an iPad and a phone, and they would just expect to be able to access their money. They wouldn’t and they shouldn’t have to expect that it’s tied to a particular device. Now of course Lightning gets super complicated if you have two different self-sovereign clients that have your private keys messing with your Lightning state and they’re not coordinating with each other — you’re going to lose your money.
Matt Odell: Specifically if one of your channel partners — if you broadcast an old state — it’s built into the Lightning model, it’s how Lightning is secure, is that they can then take your money if they see that. Because that’s the same way that it would look if you were cheating the partner.
Steve Lee [1:31:37]: Exactly. So it’s very important that multiple clients would synchronize access to that state, and that’s pretty complicated. VSS is intended to solve both of those problems, do it in a way that’s free open source library that anyone can use, and it doesn’t just support LDK — it’ll support the VLS project as well, and it can support other implementations, in theory at least. That’s another thing we’re working on: phase one of that — which is the backup phase — we hope to have ready first half of this year, and then the multi-device access will come after that. Another thing we’ve learned and observed — anyone who’s been in Lightning — it’s obvious Lightning’s really complicated, developing Lightning is really complicated, so we’re building these tools. I think everyone’s read Mastering Bitcoin who’s into Bitcoin, but then a lot of developers went through Jimmy Song’s programming Bitcoin workshop, and it packs a lot of punch in two days — it’s drinking from the fire hose. It’s a really intense two days — I took it about five years ago, and it was a really really good course that leveled up your knowledge and gave you a great foundation to go learn more. And that’s a missing in Lightning, so a few of us have started a project recently this year to develop a programming Lightning workshop, in the same vein as programming Bitcoin, but for Lightning. And we actually did a first test run here in Nashville at Bitcoin Park. It was a great first test. We only have 5% of the content done, but we tested it out — we use Replit as an environment, we use Rust as the language, which we’re a little unsure of — we don’t want to have students get tripped up over a language details, and Rust is a fairly sophisticated language. But if it can work, great, because a lot of great libraries in Bitcoin are inRust. But if it doesn’t work that’s fine. But also those projects acknowledge not everyone is a Rust developer, so they support language bindings — they support Java and Python etc. Maybe we have to do this project in Python — to be determined, very early stages. But if anyone is interested in being a guinea pig early student, let us know. Or if they want to contribute to the project. We can go over quite a bit more about Lightning complexity — I wanted to talk about the more challenges of developing Lightning applications and wallets, and especially on user experience. Here’s one thing that I don’t think is controversial but it might be controversial: I think Lightning should be hidden from users. Hide the Lightning — you’re just spending Bitcoin.
Matt Odell: There’s a lot of complaints about Muun architecturally, but the single biggest thing they did was you just scan a QR code and just pays it — the user doesn’t have to know how they’re paying it.
Steve Lee [1:35:17]: Muun did a great job, but an example to improve it is: if you want to receive, it’s a tabbed design. So you’re presented with on-chain Bitcoin — I forget what vocabulary they use, what copy they use.
Matt Odell: It says Bitcoin/Lightning, which is even more confusing because I do so much work in the education space and people are like, What is this — not Bitcoin?
IX. Unified QR Codes/BIP21
Steve Lee [1:35:45]: I totally get it. To anyone new — because it is a really good user [pain point] but — they get tripped up on that. They think of it as Litecoin or some other coin, and they certainly don’t understand the difference between on-chain — they just don’t understand the benefits of the the technology, nor should they. So there’s been efforts from the design community and the community at large — I’m part of that — for a unified QR code. Cash App as a commercial product made a big splash with that, and to Cash App’s credit, they launched a unified QR code. Actually to Dario’s credit, Muun launched a unified QR code two years ago, but they took it back because a lot of other wallets in the ecosystem couldn’t successfully send to it. BIP21 which has been around forever — it’s on-chain address-based, but you have extra information. You can include a Lightning invoice.
Matt Odell: Which is why it’s useful because it’s backwards compatible, so if you pay from an old on-chain wallet or just any on-chain wallet that doesn’t have Lightning, they don’t even see the Lightning component. But as long as new Lightning wallets know what to look for, they can then pull the Lightning invoice and pay with real Lightning.
Steve Lee [1:37:00]: Exactly. There’s been decent BIP21 support from exchanges and stuff, but Muun had launched it, pulled it back — it was too early. Then there was both this community effort to describe this problem, the opportunity, listed out a bunch of exchanges and wallets who supported it, who didn’t. It was crowdsourced — people could update that. Cash App made a big splash and then actually launched with it and did a bunch of testing themselves, and now we see BTCPay following and Zeus followed, and Muun added the option in the settings, and there’s a few others too.
Matt Odell: Same with BTCPay Server: it might not default to it but it might be in settings. But that’s a little bit more realistic, because if you’re operating a BTCPay or you’re a little bit more technical than a Muun user. You’d like to see it default in the actual apps.
Steve Lee [1:38:01]: So there’s good momentum for that and absolutely it’s a better user experience to just not have to think about that. All you’re thinking about is: I’m receiving Bitcoin.
Matt Odell: I will say, hat on the table, Steve: when my good friend Miles Suter told me that BIP21 support was absolutely crucial, I was a little bit dismissive. I personally thought we’ll just move to Lightning in general, so there will be one QR code but it’ll be a Lightning QR code, but I will say: now that it’s actually rolled out — and I think we were talking about specifically with the BTCPay Server point of sale — it is so clean, it is so nice. I love it. I think it’s a massive massive UX improvement.
Steve Lee [1:38:42]: That’s great to hear, because it is this simple detail, but it’s those simple details that are massive for mainstream adoption.
Matt Odell: 100%.
Steve Lee [1:38:51]: I’ve still never personally experienced paying for something in El Salvador, but I’ve seen videos where the merchant — the person behind that app — really doesn’t know what they’re doing, because it’s complicated. As you said: a clusterfuck.
Matt Odell: You see Bitcoiners, they have to take their phone and go to change it to Lightning and it just becomes a whole process.
Steve Lee [1:39:16]: So a small detail, but really needed. But just in general, Lightning is a technology layer that us developers and enthusiasts and early adopters of Bitcoin can get excited about and talk about, but I really think it should be hidden. I just think mainstream users don’t care — they just want to send and receive money, and that being Bitcoin. On the podcast with Jeff and Val, you go from being enthusiastic about Lightning and then back to on-chain, and I totally understand that psychology. There’s days where I get a little depressed about all the challenges of Lightning, but overall very optimistic.
Matt Odell: The other day, Steve — we were at Bitcoin Park — we had our open house, our monthly meet up. To the listeners: we have BitDevs and then we have open house — but we had open house and we had 200 people here. This guy comes up to me and he just started his Lightning node and he’s asking how he can get inbound liquidity. I go into this whole thing for 10 minutes and then at the end I’m like, I’ll just open a channel to you — that’s when I get lower on the ebbs and flows. And then obviously when I see Lightning out there in the wild and it’s working, I get very bullish.
X. Preparing Lightning For Mainstream Adoption
Steve Lee [1:40:36]: Well that’s why I don’t think Lightning becomes mainstream unless there’s a healthy LSP and the business models work out and it’s fair to end users and they’re getting value and LSPs have a business model, etc. But if we assume Lightning’s successful, I think most people’s Bitcoin experiences will be dominated by Lightning. The vast majority of their payments, their interactions will be. But again: it’s a technology layer underneath, just like when I’m using a web browser I’m not constantly seeing or thinking about all the different protocol layers underneath that. I just use it.
Matt Odell: Makes sense to me.
Steve Lee [1:41:23]: Whereas on-chain — you just do the math: unless we increase block size, there’s only many transactions that can occur, and if you get super broad adoption — billions of people — you can just do the math. There’s only so many you can have on one on-chain UTXO or a Lightning channel, let alone a savings wallet etc. So there’ll be a lot of users that are on Fedimint or something like that — some other technology where you’re sharing UTXOs, and they’ll just not interact with on-chain.
Matt Odell: There will be people that presumably only use Lightning and never touch the chain.
Steve Lee [1:42:08]: Another user experience problem with Lightning is around liquidity. For people not familiar with Lightning, you can think of an abacus: if Matt and I have a channel between the two of us, we might have 100,000 satoshis on each side, but even those 200,000 satoshis in that channel — I can’t receive 200,000 satoshis through that channel. Matt can only slide 100,000 over to me — and that’s just not how people think about money. That’s not how their $20 bill in their wallet works. It’s hard to conceptualize. One how do you hide that? But two what do you do technically when I actually want to receive a payment of 500,000 satoshis and I don’t have enough of this inbound liquidity? So that’s where an LSP can come into play: they can on the fly do that with a just-in-time injection of liquidity. They could open a new channel, they could use splicing — which is a technology that’s in development — but there’s still the problem of: Do we have to wait for block confirmations on-chain? How many? One? Three? Six? Even one is a really bad user experience. And for the end user it’s: Why sometimes when I receive money it’s instant, it’s amazing, and other times I have to wait a long time? And again if they’re hidden from Lightning and all this technology and all these challenges, it’s just a weird experience that most of the time it’s fast, sometimes it’s really slow. So one solution that’s been deployed by various wallets and LSPs is Zero-Conf. And I think in this instance, that can be a good choice because in this case a Zero-Conf — unlike on-chain — you as the recipient are not trusting the person sending you the money, you’re actually trusting your LSP, which is a very different trust relationship. Because all of a sudden if some random stranger comes and buys your couch or something and sends you money, you don’t know them and they leave and you never see them again. If they double spend you, you’re out your money. But in this case, over Lightning: if you’re accepting Zero-Conf, you’re not trusting them, you’re trusting that your LSP doesn’t steal your money. And I think that it’s far more reasonable in most situations that you can trust your LSP because you’re using them on an ongoing basis — they probably have a reputation in a business model to protect. They’re unlikely to double spend you then, but it is a real trade-off that you’re making. Recently there was a new proposal called Swap-in-Potentiam which was discussed at Nashville BitDevs, which is an alternative which is pretty interesting. It’s a good tool to have in the tool shed. It’s an alternative to Zero-Conf here. It can provide the speed and instantaneous benefit of Zero-Conf without the risk of Zero-Conf, but it does introduce a different type of trade-off — there’s actually two trade-offs, and one of them wasn’t discussed at BitDevs. But the first is that it encumbers the money that’s coming in with the 2-of-2, where the LSP has one of the keys for a period of time with the time lock. So during that period of time, you do not have your lateral control of your funds — you’re trusting your LSP to always co-sign for you during that period of time. If for some reason they go offline and they become untrustworthy, you still haven’t lost your funds permanently — you just have to wait for the timeout and then you have access. So that seems like a reasonable, acceptable approach, but of course the other trade-off is once that timeout is done, then you can no longer use that for instant injection. It still doesn’t fully solve that. But anyway, there’s solutions in the works to address that, which is a big problem. It improves that trust relationship of a 2-of-2 with a time lock. And those two can also be used together, too — they’re not competing. You could apply both. You could prefer Swap-in-Potentiam, and if there’s no coins available to do that then you could Zero-Conf. Let’s talk about BOLT12, LNURL and just what are the problems being solved there.
Matt Odell: BOLT11 is the normal invoice that we’re used to right now.
XI. BOLT12
Steve Lee [1:47:22]: [BOLT11] served a good purpose. It has a lot of us doing Lightning payments, but it has a number of problems: one is that it’s invoice-based. It’s pretty good for a merchant-customer relationship where a customer is paying a merchant, but for peer-to-peer money if I’m just paying you back or whatever, anyone familiar with Venmo or Cash App or anything like that — it’s just not how you interact with someone. Even if it’s not a friend — even if it’s a babysitter or something: a babysitter yes is a business, but they don’t have a back end system and they don’t think in terms of invoice. There’s lots of jobs like that where you pay people as well.
Matt Odell: The babysitter’s not running a BTCPay Server.
Steve Lee [1:48:19]: Right. So traditionally you might hand them a $20 bill or write them a check, or today you might Venmo them or use Cash App or something like that, and you can’t do that with BOLT11. You have to do this invoice stamp. So one thing BOLT12 does is it has the notion of an Offer which is reusable. Oh another thing about BOLT11 — you should only use that invoice once.
Matt Odell: Are there reusable BOLT11s?
Steve Lee [1:49:03]: Well if you try to pay it again, money can be stolen along the path. That’s not widely known I don’t think in Lightning either.
Matt Odell: I guess your wallet is just protecting you from that: if they know it’s already paid, they’re not sending it.
Steve Lee [1:49:18]: In practice I’ve not heard of it being a problem, but if you did try to pay it again it could be. It could be stolen but I don’t know if intermediate nodes are actively looking to steal.
Matt Odell: But in practice what you usually see is if you try and pay an invoice that’s already paid it says This invoice has already paid and you can’t pay it — for the end user.
Steve Lee [1:49:35]: Although I think that’s because it actually timed out. So there’s the notion of an expiration date in BOLT11 invoices as well. For multiple reasons, but that’s one. Now, BOLT12 Offers are reusable payment identifiers, so they solve a use case of tipping. They also solve a use case of where I just want to send you a payment and not do this invoice dance. Donations, tipping, and just the Venmo, Cash App use case, getting rid of this interactive dance with the person receiving the money. So, massive UX wins for these very popular use cases that people use money for and how they transmit money — it unlocks that. It has another benefit: we talked about the unified QR code. A challenge with that is the density of the QR code. The more information you put in a QR code, the more dense that QR code, and all of a sudden phones that were scanning it fine all of a sudden struggle, and you might have to spend 20 seconds zooming in perfectly and it’s not a good experience.
Matt Odell: Especially in the global South where it’s old Android bad cameras.
Steve Lee [1:50:55]: And sometimes it just won’t scan it at all. There’s a threshold there. Another beautiful thing about a BOLT12 Offer is it’s much less data than a BOLT11 invoice — it’s a smaller QR code. Also with Lightning, what you can embed in a BOLT11 invoice are routing hints.
Matt Odell: Yes, basically where my private channels are, to receive.
Steve Lee [1:51:25]: And there’s another technology called blinded paths that’s being developed, and if you include that, it’s more data. And in BOLT11 it’ll just break because it’ll be way too much data to get the privacy you want and the payment success rate you want by providing all these hints for paths — it’s too much data, the QR code gets too big. With BOLT12 it pushes that off to a second step in the process — it’s not in the Offer. There are some paths though in the BOLT12 Offer — those are onion message paths — to do this initial communication to receive an invoice.
Matt Odell: Jeff mentioned that which was a part I did not realize where, in that situation, if those paths change or those routing nodes change, then you actually need a new QR code or a new text string. So they’re reusable to a point.
Steve Lee [1:52:22]: Reusable to a point. In practice it might be that you only need to include a path to your LSP. Most users will be on a mobile phone: you’ll have one LSP and the path will lead to the LSP and that LSP will be around for a long time and it’ll be pretty stable. So from a UX perspective it’ll be fine and not a big problem. Privacy — it’s not perfect. For people who really need stronger levels of privacy, they might have to choose a particular type of wallet that is more privacy-sensitive, uses multiple LSPs, uses more sophisticated blinded paths, multiple blinded paths etc., and then maybe you’d have a little bit of a degradation — your BOLT12 offer might have to be updated every once in a while.
Matt Odell: It’s still a massive win if you’re updating until about a year or every six months or whatever.
Steve Lee [1:53:24]: But with BOLT12 with the Offer: if I’m sending you money, I somehow obtain your offer — and we can talk about that in a second, different ways I might find it but — I scan it, then my wallet would reach out to your wallet over onion messages to fetch the invoice. So there’s still an invoice involved, but it’s hidden from the user. And because it’s done as this computer-to-computer communication, it can be bigger, it can include lots of blinded paths and route hints and data, and you don’t suffer the cost of the QR code. And so reducing the data in the QR code is another benefit of BOLT12. Let me talk a little bit about async payments as well which is a orthogonal topic but it’s related and it’ll be integrated as part of BOLT12. But a huge problem with mobile phones is that again they don’t guarantee background access to the CPU. So if my phone is in my pocket — which is where people’s phones typically are, or sitting on this table right now — if someone’s sending me money right now while I’m on this podcast, it likely fails over Lightning. Because with Lightning — unlike on-chain Bitcoin — both the sender and the recipient have to sign the transaction. You have to activate the private key — you have to be online — and even though my phone’s connected to the Internet right now, the iPhone typically doesn’t wake up my wallet. It [needs to] open the wallet and sign the transaction to receive the money. It’s a big problem. Because again to the end user they don’t know any of these details, they just know they’re not getting their money and they want their money. And it’s also not what they’re accustomed to with other everything apps.
Matt Odell: Even on-chain Bitcoin.
Steve Lee [1:55:25]: Right. So it’s bad and also just not what they’re used to. So async payments — Matt Corallo proposed a solution to this Fall of 2021. He sketched out a high level solution that looked promising. We discussed it last year in Oakland when we got Lightning protocol developers together from all four major implementations, and people seem to like the concept. I’d call it a concept ACK or approach acknowledgment, but there are a lot of details left to be figured out. So starting a few months ago Val and her team took on the project — she wants to build async payments and I had been looking for a developer. There’s a lot of excitement about doing it but to find someone who has free time was hard, and finally Val signed up for it so I’m really excited about that. It was covered a little bit on the other show but high level how it works: if I’m sending you money and we’re both mobile phone users, I would notify my LSP that I’m sending an async payment, it then notifies your LSP — and your phone might be in your pocket — and basically when your phone wakes up, your LSP will know that, it will tell my LSP and then the payment will be sent. There’s been a proposal in the past from Breez to do something similar, but where it’s held at the end of the last hop instead of the first hop. It was trying to solve the same problem. But the problem with that solution is that it ties up liquidity for an unbounded amount of time — for that whole route, that liquidity cannot be used for forwarding, so it could be hours, days, weeks, because you might create a really long time-out, too. So a huge win with this proposal is that it’s held by the sender’s LSP — no one’s liquidity is being tied up.
Matt Odell: It hasn’t even gone down the path yet.
Steve Lee [1:57:44]: But it is complicated: it requires trampoline, or something like trampoline. Trampoline ends up being a good solution because when I trigger that payment to send to you, my LSP is waiting to hear from your LSP, but my phone’s gone offline. I can’t retry a payment if that first path doesn’t work and you need to retry a new path — I no longer am available. I also don’t want to trust my LSP to do this because I don’t want to lose my privacy and I don’t want to give up my keys or whatever. So trampoline is a good solution that allows it to retry while being trust minimized and privacy sensitive. And then from a UX perspective it can be incorporated into BOLT12, so that when I scan a BOLT12 Offer it will actually basically have a flag, an indicator, saying as a recipient: Do you want async payments or not? And if you do, then my wallet goes down that path and it’ll initially use keysend because your phone’s offline. It’s not in a position to generate an invoice. If I send you an onion message saying Hey I’m ready I want to pay your offer — your phone’s offline and you can’t trust your LSP to generate an invoice, because then all of a sudden you’re trusting your LSP to make sure that the invoice gets you paid versus your LSP to get paid. By using keysend you avoid that invoice dance. A drawback with keysend is that there’s no proof of payment. There’s no receipt. It’s important, but as far as I know no one’s really using receipts right now in the space. It’s an enormous win to enable people to receive money on their phone and it’s a good trade-off.
Matt Odell: To be clear: Dispatch is a Podcasting 2.0 show and I receive thousands of keysend payments a day. It works great but there’s no receipts.
Steve Lee [1:59:55]: And there’s and there’s potential for receipts. Val challenged the community: Hey can someone come up with an idea how do we get receipts — maybe using PTLCs or other technology? And AJ’s already sketched out a solution which looks promising. Now even if that solution ends up working, it depends on PTLCs — that’s going to take years to develop and deploy. But great that’s slow and steady. Anyway that’s high level how async payments works: it solves arguably the biggest UX problem for mobile. There’s definitely good experiences to be had with Lightning today, but they’re often custodial or the person’s running their own Umbrel or something, which are totally valid and totally fine. But long term if we only have custodial Lightning, then we lose most of the benefits of censorship resistance, privacy, etc. And being able to deliver great user experiences on non-custodial mobile is essential — it’s 10x harder technically. We’ve outlined a bunch of the problems. The good news is there’s generally good solutions in the works and every year we’re going to see more of those delivered. The async payments — the hope is that this year Val can come out with a demo that demonstrates that it works, and simultaneously complete a spec that can get feedback from fellow protocol developers. As long as it works as we’re currently thinking and we don’t come across any more hiccups, I don’t expect it to be controversial in terms of the spec, so it’s just a matter of finding another implementations in addition to LDK that implements it. That’ll usually go through finding some bugs and problems, and then hopefully 2024 that can go live and start to become popular. But with both BOLT12 and async payments it’s a really big challenge for deployment, because ideally every wallet, every LSP, and even every node in the network needs to get updated.
Matt Odell: It’s a massive interoperability challenge.
Steve Lee [2:02:08]: Yep, and not everything has to be updated for certain situations to work. There’s gonna be a transition period, but it’s a big challenge and it’s a UX challenge, because when you add this to your wallet you have to anticipate if you’re receiving from someone, sometimes they are going to be BOLT12 compatible, sometimes they’re not, and you have to handle that. I’ve chatted with folks in the Bitcoin design community. I think a good resource that can be developed is: put some smart design people together — who understand the tech but also design-oriented — to think through: What are best practices for a wallet to approach this? Because everyone’s going to face the same question. And I spent some quality time with Stephen DeLorme this week in Nashville at Bitcoin Park and he has some really great thinking around this. A few of us are going to put together some of those thoughts and then share them with wallets to get their feedback. Hopefully that’s a good resource for wallets. I recently started a BOLT12 Discord community that’s public. Right now I’ve invited people who develop wallets, or designers that are BOLT12 advocates, and trying to develop a rhythm there. But just a place where we can talk about all the problems ahead of us. We all want to make it make it happen and come up with the plan, and once we develop a rhythm I’ll publicize that more and get more people involved in that. Again we want every wallet in every LSP to adopt this, but this year I’m optimistic we can get half a dozen or more wallets implementing it and launching it. And at least between those wallets you’ll be able to use it, and then hopefully set ourselves up for next year really going broad. To demonstrate these benefits we’re talking about, it’s all about running code. We can talk about the benefits, understand them in theory, but let’s actually see it in practice and hopefully see that this year, and in the next year see really broad adoption.
Matt Odell: Awesome I’m excited, Steve. I’m feeling bullish. I’m on the bullish curve of my Lightning feels.
Steve Lee [2:04:35]: Oh and also a lot of people think that BOLT12 is still just an academic project by one protocol developer, Rusty, who proposed it — but there’s been tremendous progress: LDK, CLN, and Eclair all are nearly complete with onion messages, BOLT12 invoices, and blinded paths. CLN supported this for a while, but they’re updating the implementation to the latest spec — that should be available soon. Eclair will be available soon, so should LDK. Soon meaning on the order of a few months. And even LND has has a solid plan in place where some developers have gotten some PRs into LND that will allow an external daemon to run — they handle onion messages in BOLT12, and at least allow LND users to send BOLT12 payments, which is critical for interoperability. Because all the exchanges and custodians that support Lightning today and run LND — it’s important for them to be able to fund and send money from there to a mobile wallet that’s non-custodial that might not be on LND and use a different implementation. It’s really important for that use case to be unlocked — that would be unlocked with this solution. And then hopefully we’ll show how much demand there is for BOLT12 and how awesome it is and get it as a higher priority to merge it all into LND.
Matt Odell: That’s really good, Steve. I’m excited. Well it’s been an absolute pleasure. We’re over two hours I’ve enjoyed it — I hope you’ve enjoyed it. I hope everyone out there has enjoyed listening and found it helpful. Can we put it on the record that at least once a year we’ll do updates, but maybe more? I would love to have you back. It’s always a fun time. Thank you for coming to Nashville and visiting and hope you come back to Nashville as well. I usually wrap it up with final thoughts. Do you have any final thoughts to the audience?
Steve Lee: I’ll just shill Bitcoin Park again. It really has been a good experience. I’m not saying that just because I’m on your show right now, but it’s been great. I would recommend people find a way to work Nashville into your travel. Clearly people in Nashville in the local area can become members and come here a lot, but even visitors I think you have something planned. Definitely learn more about Bitcoin Park and check it out.
Matt Odell: Awesome well thank you sir.
