libp2p is becoming a fundamental piece for many more projects than just IPFS and with that, it requires to be handled with more mature open source approaches in the same way that we treat IPFS (e.g docs, examples, demos, tutorials, testing, etc). Some of which was developed alongside it, namely testing, docs and tutorials, but interoperability and compliance testing has been only done by testing IPFS directly, which in part tests libp2p. We need to improve that.
Today, there are 3 libp2p implementations:
In order to build a set of robust set of interoperability and compliance tests, I'm proposing two approaches:
- Create the libp2p service (aka daemon) which would let us interact with libp2p nodes in the same way that we IPFS nodes and in which we build interop tests -- https://github.com/ipfs/js-ipfs/tree/master/test/interop.
- Create PDD tests. Tests designed as stories that libp2p language implementers can follow and create a program that does exactly what the story describes. These stories always have more than one node interacting, so that different implementations can be used against each other.
The libp2p service (aka daemon)
We have had some discussions about building the libp2p service. I believe that in order to motivate the creation of a first iteration, we need a simple utility that uses the libp2p service and is really well scoped. The idea in mind is creating a p2pcat for libp2p.
I know that we want to avoid arriving at the same situation that IPFS and its HTTP-API are, it is hard to test, document, create compliance tests and that forces everyone to implement manually their RPC clients for every single language. There has to be a better way.
After looking into multiple RPC libraries, I also started thinking that I wanted to avoid any RPC swiss army knife and focus on the core of an RPC library, something that enables a user to do remote procedure calls and that's it.
I'll continue the thread on research for the better RPC library for libp2p in a separate thread. The design must fulfil the goal of a) not take us into the same issues we have had with the HTTP-API of IPFS, b) enable us to generate docs, tests, and clients automatically and c) be mountable over any libp2p stream so that libp2p users can also have access to a great RPC library for their applications.
All this to mean that yes, this is work we want to do and will enable us to create interoperable tests, however, in order to do it right it will require some more time and so it is better to follow the second proposal first.
Update: I've added issues on libp2p/libp2p for
PDD is a technique to test implementations of protocols through stories generated from their specification. These stories describe a test case and if the implementation is correct, a test can use two or more implementations to take roles in the story without causing any failure.
The core principle is that the truth lies in the wire and if one implementation in language Y can understand the input generated by language X and generate the same output, then the implementation is compliant.
Some of these tests can be done asynchronously, this means that the output and input can be captured and frozen for later comparison, however, some of these tests require to be synchronous, in which both implementations must run at the same time because there are some time variants which the protocol doesn't need to specify (i.e order of messages in a stream protocol or any crypto channel).
I'll be creating these stories for libp2p during this week focusing on:
- Qualitative Tests - Also known as compliance tests, ensuring that the 3 implementations have tha same functionality
- Quantitative Tests - Also known as stress tests, ensuring that more than just working as expected for one case, the implementation is stable when it has to handle multiple cases at the same time.
libp2p is becoming a fundamental piece for many more projects than just IPFS and with that, it requires to be handled with more mature open source approaches in the same way that we treat IPFS (e.g docs, examples, demos, tutorials, testing, etc). Some of which was developed alongside it, namely testing, docs and tutorials, but interoperability and compliance testing has been only done by testing IPFS directly, which in part tests libp2p. We need to improve that.
Today, there are 3 libp2p implementations:
In order to build a set of robust set of interoperability and compliance tests, I'm proposing two approaches:
The libp2p service (aka daemon)
We have had some discussions about building the libp2p service. I believe that in order to motivate the creation of a first iteration, we need a simple utility that uses the libp2p service and is really well scoped. The idea in mind is creating a p2pcat for libp2p.
I know that we want to avoid arriving at the same situation that IPFS and its HTTP-API are, it is hard to test, document, create compliance tests and that forces everyone to implement manually their RPC clients for every single language. There has to be a better way.
After looking into multiple RPC libraries, I also started thinking that I wanted to avoid any RPC swiss army knife and focus on the core of an RPC library, something that enables a user to do remote procedure calls and that's it.
I'll continue the thread on research for the better RPC library for libp2p in a separate thread. The design must fulfil the goal of a) not take us into the same issues we have had with the HTTP-API of IPFS, b) enable us to generate docs, tests, and clients automatically and c) be mountable over any libp2p stream so that libp2p users can also have access to a great RPC library for their applications.
All this to mean that yes, this is work we want to do and will enable us to create interoperable tests, however, in order to do it right it will require some more time and so it is better to follow the second proposal first.
Update: I've added issues on libp2p/libp2p for
PDD - Protocol Driven Development
PDD is a technique to test implementations of protocols through stories generated from their specification. These stories describe a test case and if the implementation is correct, a test can use two or more implementations to take roles in the story without causing any failure.
The core principle is that the truth lies in the wire and if one implementation in language Y can understand the input generated by language X and generate the same output, then the implementation is compliant.
Some of these tests can be done asynchronously, this means that the output and input can be captured and frozen for later comparison, however, some of these tests require to be synchronous, in which both implementations must run at the same time because there are some time variants which the protocol doesn't need to specify (i.e order of messages in a stream protocol or any crypto channel).
I'll be creating these stories for libp2p during this week focusing on: