Skip to content

Add docs on data retrievability and pruning - #1123

Merged
jcstein merged 13 commits into
mainfrom
musalbas/retrievability
Oct 11, 2023
Merged

Add docs on data retrievability and pruning#1123
jcstein merged 13 commits into
mainfrom
musalbas/retrievability

Conversation

@musalbas

@musalbas musalbas commented Oct 9, 2023

Copy link
Copy Markdown
Member

No description provided.

@github-actions

github-actions Bot commented Oct 9, 2023

Copy link
Copy Markdown
Contributor
PR Preview Action v1.4.4
Preview removed because the pull request was closed.
2023-10-11 20:13 UTC

@liamsi liamsi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do think the params and suggested practices all make sense.

Comment thread docs/developers/retrievability.md Outdated

Rollups may need to access historic data in order to allow new rollup nodes to reconstruct the latest state by replaying historic blocks. Once data has been published on Celestia and guaranteed to have been made available, rollups and applications are responsible for storing their historical data.

While it is possible to continue do this by using the `GetAll` API method in celestia-node on historic blocks as long as archival nodes exist on the public Celestia network, rollup developers should not rely on this as the only method to access historical data, as archival nodes serving requests for historical data for free is not guaranteed. Below are some other suggested methods to access historical data.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One idea, i’ve been discussing with Mary is the ability to plug various data sources into the node. In the same way, we have multiple options to get data from p2p, we can also allow developers to plug in CDNs, cloud providers, and web3 storage providers. This is similar to marketed Espresso’s tiramisu with the difference that we have required foundation implemented. If we enable plugging various standardized Getters, then there will be less need for Rollup devs to migrate off GetAll and implement custom interfaces and data retrievals. They would be able to just use standardized data sources in our ecosystem.

@Bidon15 Bidon15 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@musalbas
musalbas marked this pull request as ready for review October 11, 2023 14:45
@jcstein
jcstein self-requested a review October 11, 2023 15:23
Comment thread docs/developers/retrievability.md Outdated
Comment thread docs/developers/retrievability.md Outdated
Comment thread docs/developers/retrievability.md Outdated
Comment thread docs/developers/retrievability.md Outdated
Comment thread docs/developers/retrievability.md Outdated

@jcstein jcstein left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - left some suggestions

Comment thread docs/developers/retrievability.md Outdated
Comment thread docs/developers/retrievability.md Outdated
Comment thread docs/developers/retrievability.md Outdated
Comment thread docs/developers/retrievability.md Outdated
Comment thread docs/developers/retrievability.md
musalbas and others added 8 commits October 11, 2023 16:55
Co-authored-by: Josh Stein <46639943+jcstein@users.noreply.github.com>
Co-authored-by: Josh Stein <46639943+jcstein@users.noreply.github.com>
Co-authored-by: Josh Stein <46639943+jcstein@users.noreply.github.com>
Co-authored-by: Josh Stein <46639943+jcstein@users.noreply.github.com>
Co-authored-by: Josh Stein <46639943+jcstein@users.noreply.github.com>
@jcstein
jcstein merged commit 4b8ff62 into main Oct 11, 2023
@jcstein
jcstein deleted the musalbas/retrievability branch October 11, 2023 20:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants