🚧 | Y3: Efficient subnet consensus protocols

[jsoares]

Description

Our goal is to design and implement scalable, efficient consensus protocols for subnets (i.e. anything below the top-level Filecoin consensus). This should enable secure, low-latency operation up to ~500 nodes per subnet.

These protocols will be integrated with Eudico, the research clone of the Lotus Filecoin client, supporting a hierarchy of subnets, each running its own instance of a consensus protocol.

To achieve our, we develop Mir, a framework for implementing distributed protocols. The first protocol to be implemented is ISS, a multi-leader BFT-style consensus protocol, in conjunction with Narwhal, a state-of-the-art mempool implementation enabling sale-out throughput.

The design should be, however, general enough, and its implementation sufficiently modular, to make it easy to implement various different consensus protocols and select any of them for any particular subnet deployment.

Scope

• Specify mid-tear and leaf-level consensus
• Implement a modular and scalable (up to low hundred of nodes) consensus solution
• Integrate the solution with the the (experimental) Filecoin client Eudico.

Resources

• Mir distributed protocol implementation framework
• ISS consensus protocol implementation (in progress)
• Incremental Design of Eudico's Ordering Layer (describes high-level ideas and logical abstractions)
• Implementation of Eudico’s Ordering Layer
• Testing and verification plan (By @dnkolegov , WIP)

Papers

• ISS protocol
• Narwhal and Tusk
• Jolteon and Ditto
• Bullshark

Talks

soon™️

Demos

2022-04-07, MirBFT

2022-06-02, Crash failures in Mir and its integration with Eudico

2022-08-12, Reproducible Integration Testing in Mir

2022-08-12, Taking Pseudocode To An Implementation With Mir Framework

2022-09-01, Reconfigurable SMR with Mir

2022-09-01, Dynamically adding new nodes to Eudico with Mir

[jsoares]

2022-02-21 meeting notes

• Discussion with Alfonso and Denis around implementation approach. Synchronised on the state of each part of the project. Tendermint almost done. Next upcoming chunk is on the Eudico side (~2 weeks). In parallel, we’ll look into implementing MirBFT stub.
• We’ll also look into how ISS works and identify the key parts of the ISS protocol that need to be implemented to have a working prototype.
  • Marko: We should come up with roadmap for this
    • Timeline: With testnet in Q4, what can we expect to have ready in time?
      • Having stable interface is the priority.
      • Estimating effort and timeline is next step — aiming to have someone by Monday. Let’s try for monthly milestones and zero-mean error.
• Narwhal: first focus on the framework, interface and requirements for usage. For now, plan to use default mempool. Alfonso will be supporting on Eudico side. Immediate AIs: share ISS paper with Denis and go through MirBFT.
  • Is it clear that Narwahl can be added on top of Mir-BFT? Is there a unifying architecture?
    • Narwhal and MirBFT are not interchangeable. Narwhal can use MirBFT as a blackbox ordering service. The architecture should be modular, with MirBFT and Narwhal as specific instances.
    • We may lose some specific properties/advantages in favour of generality.
• How should we do state transfer work in Eudico?
  • Have a snapshot of the state tree. Unclear how efficient it will be. This is synchronised to a specific epoch and consistent across nodes. We can also add reconfiguration messages — these can be part of a block and be part of the total order.

[jsoares]

2022-02-28

✋ Attendees

• @matejpavlovic
• @dnkolegov
• @adlrocha
• @jsoares
• @LefKok
• @vukolic

📣 Updates

• Discussing architecture and making a list of to-do items to make MirBFT a consensus protocol for Eudico
• First milestone define
  • M1: minimal demoable implementation
    • Eudico client-service interface (~ that for Tendermint) [Denis, 3 wks]
    • Implementation of PBFT subprotocol in ISS [Matej, 3 wks]
    • State garbage collection and transfer [4 wks]
    • Prepare demo [1 day]

🧵 Discussion

• Did we consider alternatives to PBFT?
  • Alternatives considered but PBFT is the most settled and general. Fastest to implement. Good performance in previous experiments.
  • We can try others in parallel but preference for the team to sart there.
• Timeline for the overall project
  • End of Q3 realistic for testnet
  • Only critical items after M1 is
    • Reconfiguration
    • Persistent request store
    • Persistent write-ahead log (trickier)
  • Let's try to organise the follow-up milestones
• Matej + Denis + Sergey sounds like full staffing; shouldn't need any more capacity. Some parts are not highly parallelisable.
• We may be able to reuse testing framework from Lotus/Eudico instead of spending a lot of dev cycles on it

🎯 Up next

• Define timeline for M1 (start and end date)
• Define high-level milestones for the rest of the project
• Organise meeting on testing tools and invite Marko

[jsoares]

2022-03-07

✋ Attendees

• @adlrocha
• @dnkolegov
• @jsoares
• @matejpavlovic
• @vukolic

📣 Updates

• Refined Project Milestones
  • Assigned tasks by priority to different milestones
  • Separated parallelizable best-effort tasks
• Added estimated timeline for Milestone 🥽 | Green: VM / Runtime #1 (P0)
• Wrote internship proposal

🧵 Discussion

• Applicability of VDFs to PBFT vs. other protocols
• Internship proposal comments?
• Robust leader election
  • Want to verify an idea/assumption that PBFT could easily be made robust in this sense.

🎯 Up next

• Continue implementing PBFT sub-protocol
• Check alignment of interests with Srivatsan
• Potentially submit Hyperledger mentorship proposal (deadline on Wed)
• @jsoares to add milestones to roadmap

[jsoares]

2022-03-14

✋ Attendees

• @adlrocha
• @dnkolegov
• @jsoares
• @matejpavlovic
• @vukolic

📣 Updates

• Good case of PBFT works.
• Basic mircat demonstrable with PBFT.
• Camera-ready for ISS paper almost ready.
• mircat as a subproject.

🧵 Discussion

• Would a demo video showing basic usage of mirbft (using a sample chat application) be useful? Another video with failures could follow when PBFT view change is implemented.
  • Let's go with short recordings for intermediate progress, and plan a demo on the April demo day.
  • Look into visualisation tooling for traces. Consider a dev shop.
    • Jorge having regrets about us not having joined the hyperledger mentorship programme

🎯 Up next

• Consolidate documentation and high-level project description for broader audiences.
• Implement PBFT view change.
• Focus on the mirbft interface
• Create a GitHub issue for mirbft Eudico interface