Repository navigation
Suggestion: Limit the size of the datacite.xml generated by Dataverse for datasets with many datafiles (thousands) #12638
Description
Activity
@jggautier - any DataCite or GREI recommendations on this?
Hey @qqmyers and @janvanmansum.
What the folks at DataCite say about how to include sizes and formats is at https://datacite-metadata-schema.readthedocs.io/en/4.7/properties/size/ and https://datacite-metadata-schema.readthedocs.io/en/4.7/properties/format/.
It doesn't look like they say things like if each file's size should be listed, or the sum all file sizes, or each file format or just distinct file formats.
There are no GREI recommendations about including sizes and formats.
- changed the title
[-]Suggestion:[/-][+]Suggestion: Limit the size of the datacite.xml generated by Dataverse for datasets with many datafiles (thousands)[/+]on Aug 24, 2026 OK, sounds like the suggested brief format would be fine then (and the fields are optional, so none is OK as well). I might suggest that we add a size limit rather than separate flags (i.e. add per file info up to some limit like 1K (configurable) and then switch to brief). That would keep full info for smaller datasets while avoiding the problem for large ones (and still let someone use all full or all brief if they want). This wouldn't cover the 'none' case - that could be a separate setting (either true/false or another limit size one), but - does anyone want/need it?
I agree with you @janvanmansum about how the usefulness of having a size and format element for each file is questionable, especially the way it's organized as two separate lists.
I don't know who wants or needs this information and why. Maybe folks at DataCite know or have talked about this before. There might be discussion in one of their forums.
@janvanmansum, would you be able to look through their forums or ask them? Kelly Stathis at DataCite has been very responsive in the past.
If you don't have time either, I'd say it's probably more important to resolve this sooner by limiting the size of these xml files in some way so that folks can publish their datasets.
We tell Dataverse to include the same file size and format information in the OpenAIRE export, too. But I suppose that when repository managers tell OpenAIRE's systems to harvest from them and they have datasets with lots of files, OpenAIRE is able to. Or at least I haven't heard of any problems. DataverseNO and Dataverse e-cienciaDatos are two repositories I know of whose managers have told OpenAIRE to harvest from them directly. Might be worth it sometime in the future to see if OpenAIRE's systems have issues harvesting because of big xml files.
Overview of the Suggestion
Limit the size of the datacite.xml generated by Dataverse for datasets with many datafiles (thousands) by only sending one size element (for the sum of all datafile sizes) and one unique format element for each datafile format that occurs, instead of sending a size and a format element for each datafile.
What kind of user is the suggestion intended for?
The administrator of the Dataverse installation.
What inspired this idea?
For the past few months, publication of datasets with thousands of files has been failing due to a 408 timeout error when uploading the datacite.xml to DataCite during the publication process. Also, the usefulness of having a size and format element for each datafile is questionable.
What existing behavior do you want changed?
Currently, the datacite.xml generated by Dataverse contains both a
sizeas well as aformatelement for each file. The idea is to change this to onesizeelement for the sum of all file sizes and one unique format element for each file format that occurs. This could be made configurable with a JVM optiondataverse.datacite.xml.datafile-infowith the following modes:expanded: (default) the current behavior, one size and one format element for each datafilebrief: one size element for the sum of all datafile sizes and one unique format element for each datafile format that occursnone: no size or format elements at allNote, the mode only takes effect on the next dataset version or when calling https://guides.dataverse.org/en/latest/admin/dataverses-datasets.html#update-metadata-for-a-published-dataset-at-the-pid-provider
Any brand new behavior do you want to add to Dataverse?
No, this is a modification to existing behavior.
Any open or closed issues related to this suggestion?
None that I am aware of.
Are you thinking about creating a pull request for this issue?
Yes.