SIP: Add spin plugin#682
Conversation
Signed-off-by: karthik Ganeshram <karthik.ganeshram@fermyon.com>
karthik2804
added
the
sip
kate-goldenring
left a comment
kate-goldenring
left a comment
There was a problem hiding this comment.
Thank you for putting this together! This is an exciting addition. Added some comments/questions/thoughts.
| - `install` | ||
| - `uninstall` | ||
|
|
||
| ### Types of plugins |
There was a problem hiding this comment.
Would be great to include examples of each of these categories
|
|
||
| Spin dependant plugins can be distinguished from standalone binary plugins based on the name of the plugin. The Spin Dependant Plugins must follow the additional naming constraints. | ||
|
|
||
| - The name of the plugin must be `spin-$plugin` where `$plugin` must be `<type_of_plugin>-<name_of_plugin>` |
There was a problem hiding this comment.
is <type-of-plugin> one of the names from the Plugin Types section? That seems fairly verbose. We should potentially come up with one-word descriptors and limit the total number of options
There was a problem hiding this comment.
Or it seems that "type of plugin" may be an overloaded term. Maybe clarify <type_of_dependent_plugin> here (even though it is implied by the section name) and potentially add another branch to the tree of plugin types. It also may be good to say that a trigger is a primary near future example adn it's dependent plugin "keyword" is trigger; however, some other ones could be ... Loaders etc and their keywords will be defined at the future point of use?
| Spin dependant plugins will be categorized based on the setup/functionality sharing required from the spin binary. Based on this, they will be grouped into `<type_of_plguin>`. Some of the possible types are | ||
|
|
||
| - Host components | ||
| - Loaders |
There was a problem hiding this comment.
Can you elaborate on Loaders and Config providers or give examples?
There was a problem hiding this comment.
Currently, loaders allow us to convert local configuration files or one pulled from bindle to spin configurations. In the future, we may potentially have multiple other sources from which the application configurations can be pulled depending on requirements.
There was a problem hiding this comment.
Sounds like Loaders is an established Spin concept, and bindle is the only one we currently have support for. Thanks for explaining it to me! So what is the difference between a loader and configuration provider? One hosts the config and the other loads it into spin?
There was a problem hiding this comment.
Configuration is about runtime configuration like what you might expect to get from environment variables, Vault (secrets), etc.
|
|
||
| ## Proposed implementation of `spin plugin` | ||
|
|
||
| ### The `install` subcommand |
There was a problem hiding this comment.
Can a section be added on how to do updates. With spin templates the flow is spin templates install --update. To create a similar experience, it could be spin plugin install $plugin --update which will update it if it exists otherwise install latest
There was a problem hiding this comment.
Yes, this sounds like a good idea. I will work on it.
There was a problem hiding this comment.
It would be very nice (perhaps in a second iteration) to have something equivalent to spin plugin update --all. I think the expected behavior differs from templates in that you pretty much always want the latest version of a plugin.
There was a problem hiding this comment.
Makes sense. Should an update do a check to see if an update is needed or always overwrite, even if it is bringing down the same version? I am not sure what the best way to do a version check would be, unless we save the current plugin manifest in the directory of the plugin and query the version there and upstream to see if there is a newer version.
There was a problem hiding this comment.
Yeah I think this would require storing information about anything installed via spin plugin install somewhere locally. I would probably lean towards a e.g. plugins.json in the same parent directory of plugins/
| - It will contain information about the plugins such as location of binary release for the various operating system, version and checksum along with potentially other details such as author details and a description. | ||
| - Creators of new plugins can submit PRs with the required information to add it to the plugins index repository. | ||
|
|
||
| The structure of data stored for every plugin is a json file with the following structure |
There was a problem hiding this comment.
Will we support multiple versions of a plugin? Should the file be named $plugin_name-$version.json and by default spin will pull the highest versioned (following SemVer)? and install could also take in a --version $version flag?
There was a problem hiding this comment.
It is possible to support multiple versions of the plugins but the current idea is to provide only the latest release of every major release version but a discussion around this would be needed.
|
|
||
| ## Future Considerations | ||
|
|
||
| - Updates with breaking changes. |
There was a problem hiding this comment.
Can you elaborate here? Is this addressing the need to handle how we do updates when there are breaking changes? is a prerequisite of this supporting updates?
There was a problem hiding this comment.
There has been some discussion on this although more discussion is required on this. One of the potential solutions is for the plugins to specify the compatibility with various versions of spin and use that to verify before installing.
There was a problem hiding this comment.
Can you add these thoughts to the SIP itself?
|
|
||
| ### Types of plugins | ||
|
|
||
|  |
There was a problem hiding this comment.
wonder if instead of linking an image here it would be better to represent via text since it is a short chart. Also would be easier to track changes. maybe like this:
Spin Plugins
├── Spin Dependant Plugins
│ ├── Internal Spin Plugins
│ └── Spin Dependent User Plugins
└── Standalone Binary Plugins
There was a problem hiding this comment.
This makes sense, I will make the changes.
|
|
||
|  | ||
|
|
||
| - Standalone Binary Plugins (external subcommands) - These work to add features to spin and do not need any use of the spin binary. |
There was a problem hiding this comment.
if both Internal Spin Plugins and Spin Dependent User Plugins fall under the "Spin Dependent" category, then maybe they should both be prefixed by "Spin Dependent". IE Internal Spin Plugins => Spin Dependent Internal Plugin
There was a problem hiding this comment.
Will make the changes.
|
|
||
| ### Spin dependant binary | ||
|
|
||
| For each `<type_of_plguin>`, a new spin execution path will be created sharing the required features/functionality before calling the spin plugin. |
There was a problem hiding this comment.
nit spelling: "plugin". I find this Code Spell Checker VSCode extension to be a lifesaver as i always am mixing around letters
itowlson
left a comment
itowlson
left a comment
There was a problem hiding this comment.
This is very cool - I'm excited at the possibilities you imply. However it would be really useful to have examples of the different kinds of plugin and a bit more detail - even if only illustrative - of how Spin executes them. I'd also really really like to see a bit more detail on versioning, which I know is unglamorous but is going to bite people if we don't provide some control. But a great start - thanks for working through this!
|
|
||
| #### Update of plugins | ||
|
|
||
| The system will use a rolling updates system where the latest plugin will be installed. |
There was a problem hiding this comment.
Can you clarify this - are you saying that Spin will poll for new versions of plugins and automatically update? This is potentially a bit scary unless the user has a way to pin to a 'known good' version. What do you mean by "rolling" update here?
There was a problem hiding this comment.
No, Spin does not poll for new versions of plugins. By rolling release, I mean whenever the user tries to install a plugin, they will be grabbing the latest version of the plugin.
| "name": "test", | ||
| "description": "Some description.", | ||
| "homepage": "www.example.com", | ||
| "version": "1.0", |
There was a problem hiding this comment.
Is there a document per version, or do new versions completely replace old ones?
There was a problem hiding this comment.
The idea is to replace the old ones completely with the new ones. The exception is major version changes. There will exist a document for every latest major version of the plugin. So if a plugin as has 2 versions. Then 2 documents will exist in the names $plugin.json (latest version) and $plugin@1.json.
There was a problem hiding this comment.
I may have missed it if it is already in here, but can you add this approach for major changes to the SIP?
| A plugin can be installed using the following command. | ||
|
|
||
| ```bash | ||
| $ spin plugin install <name_of_plugin> |
There was a problem hiding this comment.
There will need to be an option to install from local file (for testing) or from custom repo (for non-blessed plugins).
There was a problem hiding this comment.
Also an option to install a specific version (particularly important for plguins that may run in a CI or production environment).
There was a problem hiding this comment.
The plugins can be directly placed in the install location for testing but a flag can be added to give these options.
There was a problem hiding this comment.
"Place it directly in the install location" isn't going to be great for a production environment though. An operator is going to want to be able to script creation of the environment, and have confidence that the script is repeatable.
There was a problem hiding this comment.
That makes sense. Adding an option to install from a local file makes sense.
There was a problem hiding this comment.
This could be simply pointing to a local plugin-manifest.json rather than grabbing from remote, right? spin plugin install some-plugin -f /path/to/some-plugin.json. Installing a specific version could work similarly, where they always pass in the plugin-manifest with the version they want.
There was a problem hiding this comment.
There are two different things you could do here:
- pointing to a local manifest, which will lead to the actual binary being downloaded from one of the URLs in the manifest
- pointing to a local binary, where you just want to copy/symlink that binary into the plugins directory
Not sure which of these would be better for production install I guess you probably want the former for production installs - the URLs can always be updated to point somewhere local if that's what the operator wants.
| - **Standalone Binary Plugin** | ||
| - Once the plugin is invoked using `spin $plugin <args_to_plugin>`, the external subcommand is invoked as `spin-$plugin <args>`. | ||
| - **Spin Dependant User Plugins** | ||
| - Spin first provides all the functionality available in it required by the plugin and then proceeds to call the plugin along with the required parameters. |
There was a problem hiding this comment.
I am not quite getting the distinction here. Are you saying that Spin will have special knowledge of certain plugins and will have code to detect if one of these is being invoked and execute it in a special way?
There was a problem hiding this comment.
I will try to make it clearer with some made-up example plugins.
Standalone Binary Plugin - These are just basic binaries that can add functionality to spin (eg) spin routes just displays the HTTP routes for a given application from spin.toml.
Spin-based User Plugins - A made-up example of this would be spin deploy where the user invokes the command but as a plugin, it would require some functionality from spin. Therefore deploy is a type of command and it could have multiple plugins each of which deploys to different services.
Spin-based Internal Plugins - An example of this would be triggered, where the user does not directly call for the triggers but uses them by specifying them in the application manifest. Loaders would be another example of this.
And yes, Spin will be able to distinguish between the types of plugins from the name of the plugin.
Both Categories of Spin-based plugins will have keywords signifying types and all the types will be known ahead of time as each type of plugin follows a different execution path. The categories will be split into internal and user plugins.
An example of this would be a time trigger plugin called spin-trigger-time where the type of plugin is trigger which will be contained in a list of internal plugins.
There was a problem hiding this comment.
These are great examples - thanks!
So for both categories of "dependent plugin", the behaviour is:
- Spin looks at the name of the plugin binary (e.g.
spin-foo-barand extracts the second component. This identifies a type of plugin and must be a well known string. [Or is the type derived in some other way, e.g. from the Spin command being issued, and the name of the plugin inferred from that.] - Based on that type, Spin knows the contract for calling that type of plugin, e.g. what files must be placed where, what environment variables must be set, what command line flags must be provided, etc.
- Spin performs that type-based preparation.
- Spin invokes the plugin.
Is that right? If so, it would be super useful (to me at least) to capture that flow.
User and internal DPs, then, have the exact same flow at the technical level, but differ in how Spin decides to launch them. E.g. a hypothetical AWS deployment plugin would be chosen explicitly by the command line user issuing spin deploy aws - Spin knows how to call deploy plugins but just blindly calls them by name. But trigger plugins are called implicitly based on the content of spin.toml, and that's what makes them internal. Is that right?
There was a problem hiding this comment.
Yes, This is exactly the flow.
The type of plugin is inferred from the second component of the plugin name and it is a well-known string.
| - Spin first provides all the functionality available in it required by the plugin and then proceeds to call the plugin along with the required parameters. | ||
|
|
||
| - **Spin Dependant Internal Plugins** | ||
| - These plugins cannot be directly invoked by the user as it is used by spin internally. Therefore, only spin invokes these. |
There was a problem hiding this comment.
I don't understand what 'internal' means in this context? Are they built into the Spin binary? But then in what sense are they plugins? I'd find some clarification of the precise differences between the three execution models really useful - sorry if I'm being slow!
There was a problem hiding this comment.
The main point here for me is that we can extract e.g. the redis trigger into a separate spin-trigger-redis plugin and apart from needing to install that plugin nothing else changes.
|
|
||
| ## Future Considerations | ||
|
|
||
| - Updates with breaking changes. |
There was a problem hiding this comment.
I'm not sure deferring this to "future considerations" is a wise move - would be good to have at least some sort of plan for it.
There was a problem hiding this comment.
There has been some discussion on this although more discussion is required on this. One of the potential solutions is for the plugins to specify the compatibility with various versions of spin and use that to verify before installing.
|
|
||
| - Host components | ||
| - Loaders | ||
| - Config providers |
There was a problem hiding this comment.
These examples are "internal plugins" right? This is a super useful list - it would be good to have this up top where you define the three categories.
There was a problem hiding this comment.
Yes, these are examples of spin-based internal plugins.
Signed-off-by: karthik Ganeshram <karthik.ganeshram@fermyon.com>
Signed-off-by: karthik Ganeshram <karthik.ganeshram@fermyon.com>
|
|
||
| **Dealing with Breaking changes** | ||
|
|
||
| This is a section that still needs discussion. One of the solutions is for the plugin manifest to list the versions of spin that it is compatible with and use that to make an informed update of plugins avoiding updates to incompatible versions. |
There was a problem hiding this comment.
I worry about synchronicity issues here. When a new version of spin is released, then all plugins will need to update their compat lists. This means the plugin owner will have to stay fairly involved in the plugins repository. I wonder what other package managers do. I know SPEC files allow for requires syntax but it doesnt always have to give version and sometimes it can be an equality operator (requires: python >= 1.3): http://ftp.rpm.org/api/4.4.2.2/dependencies.html
There was a problem hiding this comment.
The semver crate supports version constraints: https://docs.rs/semver/1.0.13/semver/struct.VersionReq.html
|
|
||
| ``` | ||
| { | ||
| "name": "test", |
There was a problem hiding this comment.
according to above explaination on naming conventions, shouldnt this be prefixed with "spin" so spin-test
|
|
||
| ## Background | ||
|
|
||
| As the functionality fo spin gets extended, there will be a point where every feature will not be required/used by every user. An example of this would be different triggers for spin along with possible subcommands to add more functionality to spin. Therefore it would make sense to be able to add features as plugins allowing for addition of only the required features without modifying the spin binary. |
There was a problem hiding this comment.
Could you please capitalise Spin (and other proper nouns)?
| │ └── Spin-Dependent User Plugins | ||
| └── Standalone Binary Plugins | ||
| ``` | ||
| **Standalone Binary Plugin** - These are just basic binaries that can add functionality to spin (eg) spin routes just displays the HTTP routes for a given application from spin.toml. |
There was a problem hiding this comment.
Is the word "basic" here necessary?
|
|
||
| **Spin-based User Plugins** - A made-up example of this would be spin deploy where the user invokes the command but as a plugin, it would require some functionality from spin. Therefore deploy is a type of command and it could have multiple plugins each of which deploys to different services. | ||
|
|
||
| **Spin-based Internal Plugins** - An example of this would be triggered, where the user does not directly call for the triggers but uses them by specifying them in the application manifest. Loaders would be another example of this. |
Signed-off-by: karthik Ganeshram <karthik.ganeshram@fermyon.com>
fibonacci1729
left a comment
fibonacci1729
left a comment
There was a problem hiding this comment.
Nice work Karthik! This is shaping up nicely and i'm looking forward to this feature!
|
|
||
| For each `<type>`, a new Spin execution path will be created sharing the required features/functionality before calling the Spin plugin. | ||
|
|
||
| So once the Spin plugin is invoked using `spin $plugin`, the Spin binary does all the functions defined for that specific `<type>` before calling `spin-$plugin` with all the required environment variables set. |
There was a problem hiding this comment.
It's not quite clear to me what "the Spin binary does all the functions defined for that specific <type>" means here. I am correct to read this as: Spin will initialize a new execution environment implied by the plugin type and execute the plugin binary by invoking spin $plugin?
There was a problem hiding this comment.
Yes, That is correct.
Signed-off-by: Kate Goldenring <kate.goldenring@fermyon.com>
Add JSON schema to spin plugins SIP
|
Closing for revised version in #712 |
6 participants
SIP to add plugins to spin.
rendered
Signed-off-by: karthik Ganeshram karthik.ganeshram@fermyon.com