Skip to content

Commit ad3d919

Browse files
qqmyersqqmyers
committed
Merge remote-tracking branch 'GDCC/scriptUpdates'
# Conflicts: # 5.2curlcommands.md # 6.1curlcommands.md # previewers/betatest/js/refiqdacore.js # previewers/betatest/js/refiqdpx.js
2 parents d93de78 + cef0263 commit ad3d919

15 files changed

Lines changed: 1143 additions & 160 deletions

‎5.2curlcommands.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1045,3 +1045,45 @@ curl -X POST -H 'Content-type: application/json' http://localhost:8080/api/admin
10451045
## Beta Versions:
10461046

10471047
As updates and additions are made, they will be added to a /betatest sub-folder which will make them accessible to users via github.io without affecting people who are using the latest numbered release (e.g. v1.5). When previewers in the betatest branch are included in the next release (e.g. v1.6), sites may wish to change their registration of the tools to use the stable version as the betatest version may continue to receive changes.
1048+
1049+
### 7zip Previewer
1050+
1051+
* Can be used with *.7z files
1052+
* Requirement: Dataverse version v5.12+
1053+
* Requirement: Local file storage or S3 with direct download configured
1054+
* When using S3 storage: Extended CORS rule settings for the S3 bucket (see also: https://guides.dataverse.org/en/latest/developers/big-data-support.html#s3-direct-upload-and-download)
1055+
```json
1056+
{
1057+
"CORSRules": [
1058+
{
1059+
"AllowedOrigins": ["*"],
1060+
"AllowedHeaders": ["*"],
1061+
"AllowedMethods": ["PUT", "GET", "HEAD"],
1062+
"ExposeHeaders": ["ETag", "Accept-Ranges", "Content-Range", "Content-Encoding"]
1063+
}
1064+
]
1065+
}
1066+
```
1067+
1068+
```bash
1069+
curl -X POST -H 'Content-type: application/json' http://localhost:8080/api/admin/externalTools -d \
1070+
'{
1071+
"displayName":"Preview 7zip file",
1072+
"description":"Preview the structure of a 7zip file.",
1073+
"toolName":"sevenZipPreviewer",
1074+
"scope":"file",
1075+
"types":["preview"],
1076+
"toolUrl":"https://gdcc.github.io/dataverse-previewers/previewers/betatest/SevenZipPreview.html",
1077+
"toolParameters": {
1078+
"queryParameters":[
1079+
{"fileid":"{fileId}"},
1080+
{"siteUrl":"{siteUrl}"},
1081+
{"key":"{apiToken}"},
1082+
{"datasetid":"{datasetId}"},
1083+
{"datasetversion":"{datasetVersion}"},
1084+
{"locale":"{localeCode}"}
1085+
]
1086+
},
1087+
"contentType":"application/x-7z-compressed"
1088+
}'
1089+
```

‎6.1curlcommands.md‎

Lines changed: 67 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1475,7 +1475,7 @@ curl -X POST -H 'Content-type: application/json' http://localhost:8080/api/admin
14751475

14761476
### ESRI Shape Previewer
14771477

1478-
This previewer includes a hard-coded file size limit of a zipped shp file of 20 MB (larger zips will not be loaded). If you want to change this limit you can change the value in "previewers/v1.5/js/mapshp.js", but then you have to host the customised previewer yourself (e.g. via github pages). Instructions on how to build external tools yourself can be found here: https://guides.dataverse.org/en/latest/api/external-tools.html.
1478+
Works with zipped shapefiles in any known projection. This previewer includes a hard-coded file size limit of a zipped shp file of 50 MB (larger zips will not be loaded). If you want to change this limit you can change the value in "previewers/betatest/js/mapshp.js", but then you have to host the customised previewer yourself (e.g. via github pages). Instructions on how to build external tools yourself can be found here: https://guides.dataverse.org/en/latest/api/external-tools.html.
14791479

14801480
```bash
14811481
curl -X POST -H 'Content-type: application/json' http://localhost:8080/api/admin/externalTools -d \
@@ -1485,7 +1485,7 @@ curl -X POST -H 'Content-type: application/json' http://localhost:8080/api/admin
14851485
"toolName":"mapShpPreviewer",
14861486
"scope":"file",
14871487
"types":["preview"],
1488-
"toolUrl":"qualitativedatarepository.github.io/dataverse-previewers/previewers/v1.5/MapShpPreview.html",
1488+
"toolUrl":"qualitativedatarepository.github.io/dataverse-previewers/previewers/betatest/MapShpPreview.html",
14891489
"toolParameters": {
14901490
"queryParameters":[
14911491
{"fileid":"{fileId}"},
@@ -1519,21 +1519,19 @@ curl -X POST -H 'Content-type: application/json' http://localhost:8080/api/admin
15191519
}'
15201520
```
15211521

1522-
### GeoTIFF Previewer
1522+
### GeoTIFF + TIFF Previewer
15231523

1524-
Please note that the Geotiff Previewer tries to display ALL image/tiff files in a map (because there is no own mimetype for Geotiffs at the moment). Therefore please only use it if you want to use Geotiffs only. As soon as there is an own mimetype for Geotiffs, this will be updated.
1525-
1526-
Limits are also defined for previewing raster files. Besides the file size limit (20 MB), a column and row limit (50,000) and a loading timeout (30 seconds) are defined. In case of an unsupported GeoTIFF (e.g. no projection specified), an error message is displayed after 30 seconds stating that the tiff image cannot be loaded. These limits can be adjusted as needed in previewers/v1.5/js/mapraster.js, although this adjusted version must then also be hosted by yourself (e.g. via github pages). Instructions on how to build external tools yourself can be found here: https://guides.dataverse.org/en/latest/api/external-tools.html.
1524+
This previewer reads GeoTIFF and "standard" TIFF files. It is recommended to save GeoTIFFs as COG (cloud-optimized GeoTIFF), for example using gdal_translate. This automatically adds overviews (pyramid layers) to GeoTIFFs and splits them into tiles (blocks), ensuring they are displayed efficiently. If color maps (color palettes) are defined in the GeoTIFF for grayscale rasters (1 band), they are rendered in color. Any projection should be supported, as long as it is correctly defined and known (for rendering, the data is automatically reprojected to epsg:3857). GeoTIFFs without overviews are only loaded if they are 50 MB or smaller. This file size limit can be adjusted as needed in previewers/betatest/js/mapraster.js, although this adjusted version must then also be hosted by yourself (e.g. via github pages). Instructions on how to build external tools yourself can be found here: https://guides.dataverse.org/en/latest/api/external-tools.html.
15271525

15281526
```bash
15291527
curl -X POST -H 'Content-type: application/json' http://localhost:8080/api/admin/externalTools -d \
15301528
'{
15311529
"displayName":"View Map",
15321530
"description":"View a map of the file.",
1533-
"toolName":"mapShpPreviewer",
1531+
"toolName":"mapTiffPreviewer",
15341532
"scope":"file",
15351533
"types":["preview"],
1536-
"toolUrl":"qualitativedatarepository.github.io/dataverse-previewers/previewers/v1.5/MapRasterPreview.html",
1534+
"toolUrl":"qualitativedatarepository.github.io/dataverse-previewers/previewers/betatest/MapRasterPreview.html",
15371535
"toolParameters": {
15381536
"queryParameters":[
15391537
{"fileid":"{fileId}"},
@@ -1958,7 +1956,6 @@ curl -X POST -H 'Content-type: application/json' http://localhost:8080/api/admin
19581956

19591957
## Beta Versions:
19601958
As updates and additions are made, they will be added to a /betatest sub-folder which will make them accessible to users via github.io without affecting people who are using the latest numbered release (e.g. v1.5). When previewers in the betatest branch are included in the next release (e.g. v1.6), sites may wish to change their registration of the tools to use the stable version as the betatest version may continue to receive changes.
1961-
19621959
### CDI Previewer
19631960

19641961
```bash
@@ -2246,4 +2243,65 @@ curl -X POST -H 'Content-type: application/json' http://localhost:8080/api/admin
22462243
}
22472244
]
22482245
}'
2246+
```
2247+
2248+
### 7zip Previewer
2249+
2250+
* Can be used with *.7z files
2251+
* Requirement: Dataverse version v5.12+
2252+
* Requirement: Local file storage or S3 with direct download configured
2253+
* When using S3 storage: Extended CORS rule settings for the S3 bucket (see also: https://guides.dataverse.org/en/latest/developers/big-data-support.html#s3-direct-upload-and-download)
2254+
```json
2255+
{
2256+
"CORSRules": [
2257+
{
2258+
"AllowedOrigins": ["*"],
2259+
"AllowedHeaders": ["*"],
2260+
"AllowedMethods": ["PUT", "GET", "HEAD"],
2261+
"ExposeHeaders": ["ETag", "Accept-Ranges", "Content-Range", "Content-Encoding"]
2262+
}
2263+
]
2264+
}
2265+
```
2266+
2267+
```bash
2268+
curl -X POST -H 'Content-type: application/json' http://localhost:8080/api/admin/externalTools -d \
2269+
'{
2270+
"displayName":"Preview 7zip file",
2271+
"description":"Preview the structure of a 7zip file.",
2272+
"toolName":"sevenZipPreviewer",
2273+
"scope":"file",
2274+
"types":["preview"],
2275+
"toolUrl":"https://gdcc.github.io/dataverse-previewers/previewers/betatest/SevenZipPreview.html",
2276+
"toolParameters": {
2277+
"queryParameters":[
2278+
{"fileid":"{fileId}"},
2279+
{"siteUrl":"{siteUrl}"},
2280+
{"datasetid":"{datasetId}"},
2281+
{"datasetversion":"{datasetVersion}"},
2282+
{"locale":"{localeCode}"}
2283+
]
2284+
},
2285+
"contentType":"application/x-7z-compressed",
2286+
"allowedApiCalls": [
2287+
{
2288+
"name": "retrieveFileContents",
2289+
"httpMethod": "GET",
2290+
"urlTemplate": "/api/v1/access/datafile/{fileId}/auxiliary/qdc/1.0?gbrecs=true",
2291+
"timeOut": 3600
2292+
},
2293+
{
2294+
"name": "downloadFile",
2295+
"httpMethod": "GET",
2296+
"urlTemplate": "/api/v1/access/datafile/{fileId}/auxiliary/qdc/1.0?gbrecs=false",
2297+
"timeOut": 3600
2298+
},
2299+
{
2300+
"name": "getDatasetVersionMetadata",
2301+
"httpMethod": "GET",
2302+
"urlTemplate": "/api/v1/datasets/{datasetId}/versions/{datasetVersion}",
2303+
"timeOut": 3600
2304+
}
2305+
]
2306+
}'
22492307
```

‎installPreviewers.py‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
#!/usr/bin/env python3
2+
3+
import argparse
4+
import subprocess
5+
from pathlib import Path
6+
7+
8+
BETA_MARKER = "## Beta Versions:"
9+
10+
11+
def find_curl_blocks(path, version):
12+
blocks = []
13+
in_beta_section = False
14+
in_code_block = False
15+
current = []
16+
17+
for line in Path(path).read_text().splitlines(keepends=True):
18+
stripped = line.strip()
19+
20+
# Everything after this marker is considered beta-only.
21+
if stripped.startswith(BETA_MARKER):
22+
in_beta_section = True
23+
continue
24+
25+
# Select either the content before or after the beta marker.
26+
include_section = (
27+
in_beta_section if version == "beta" else not in_beta_section
28+
)
29+
30+
if not in_code_block:
31+
if include_section and stripped == "```bash":
32+
in_code_block = True
33+
current = []
34+
else:
35+
if stripped == "```":
36+
command = "".join(current)
37+
38+
first_line = next(
39+
(
40+
line.strip()
41+
for line in command.splitlines()
42+
if line.strip()
43+
),
44+
"",
45+
)
46+
47+
if first_line == "curl" or first_line.startswith("curl "):
48+
blocks.append(command)
49+
50+
in_code_block = False
51+
else:
52+
current.append(line)
53+
54+
return blocks
55+
56+
57+
def main():
58+
parser = argparse.ArgumentParser(
59+
description="Find and execute curl commands in fenced bash blocks."
60+
)
61+
parser.add_argument(
62+
"file",
63+
help="Input file containing the curl commands",
64+
)
65+
parser.add_argument(
66+
"version",
67+
choices=("standard", "beta"),
68+
help=(
69+
"Select standard previewers before '## Beta Versions:' "
70+
"or beta-only previewers after it"
71+
),
72+
)
73+
parser.add_argument(
74+
"--dry-run",
75+
action="store_true",
76+
help="Print commands without executing them",
77+
)
78+
args = parser.parse_args()
79+
80+
blocks = find_curl_blocks(args.file, args.version)
81+
82+
print(
83+
f"Found {len(blocks)} {args.version} curl command block(s)."
84+
)
85+
86+
for index, command in enumerate(blocks, start=1):
87+
print(f"\n--- Command {index} ---")
88+
print(command.rstrip())
89+
90+
if args.dry_run:
91+
continue
92+
93+
print("--- Executing ---")
94+
subprocess.run(
95+
command,
96+
shell=True,
97+
executable="/bin/bash",
98+
check=True,
99+
)
100+
101+
102+
if __name__ == "__main__":
103+
main()

‎localinstall.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,7 @@ if [ ! -z "$2" ]; then
7272
cd "${dirLocal}"
7373
echo Changing example curl commands to use local URLs
7474
localurl="$2"
75-
sed -i "s,https://gdcc.github.io/dataverse-previewers/previewers/v1.[34],$localurl/$folder,g" *curlcommands.md
75+
sed -i "s,https://gdcc.github.io/dataverse-previewers/previewers/[^/]*,$localurl,g" *curlcommands.md
7676
echo Done changing example curl commands to use local URLs
7777
cd "${folder}"
7878
fi
@@ -85,4 +85,4 @@ rm replace_js.sh
8585
rm replace_css.sh
8686

8787
echo Done
88-
exit 0
88+
exit 0

‎previewers/betatest/MapRasterPreview.html‎

Lines changed: 35 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
<!DOCTYPE html>
21
<html>
32
<head>
43
<meta charset="utf-8">
@@ -15,20 +14,42 @@
1514
<!-- Optional theme -->
1615
<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/3.4.1/css/bootstrap-theme.min.css"
1716
integrity="sha384-6pzBo3FDv/PJ8r2KRkGHifhEocL+1X2rVCTTkUfGk7/0pbek5mMa1upzvWbrUbOZ" crossorigin="anonymous">
18-
<!-- Leaflet -->
19-
<link rel="stylesheet" href="https://unpkg.com/leaflet@1.7.1/dist/leaflet.css"
20-
integrity="sha512-xodZBNTC5n17Xt2atTPuE1HxjVMSvLVW9ocqUKLsCC5CXdbqCmblAshOMAS6/keqq/sMZMZ19scR4PsZChSR7A==" crossorigin="anonymous"/>
21-
<script src="https://unpkg.com/leaflet@1.7.1/dist/leaflet.js"
22-
integrity="sha512-XQoYMqMTK8LvdxXYG3nZ448hOEQiglfqkJs1NOQV44cWnUrBc8PkAOcXy20w0vlaXaVUearIOBhiXZ5V3ynxwA==" crossorigin="anonymous"></script>
23-
<link type="text/css" rel="stylesheet" href="css/preview.css" />
24-
<!--drawing raster-->
25-
<script src="https://unpkg.com/georaster@1.6.0/dist/georaster.browser.bundle.min.js"
26-
integrity="sha512-SkPYBKB00LoLZf/I0yaoIwp88e7E+X5iFliB4n84vmzhjdEdEla/vouiA0Nr6aLHoRCB54DK8CCynKNd0DfCSg==" crossorigin="anonymous"></script>
27-
<script src="https://unpkg.com/georaster-layer-for-leaflet@3.10.0/dist/v3/webpack/bundle/georaster-layer-for-leaflet.min.js"
28-
integrity="sha512-bNJ+bvpXepatbIMeaida52kR2AT5lr6a9dTfb/OB0c/3O2KzUrdUi3Gtri0zvJUbfKgmG+wN3GKTkBi/6OkVVA==" crossorigin="anonymous"></script>
17+
18+
<!-- proj4 -->
19+
<script
20+
src="https://cdn.jsdelivr.net/npm/proj4@2.19.10/dist/proj4.js"
21+
integrity="sha512-QXAXxYhXWvAc5fA3edU2rOQANjhwKAAlqz1WEhh/FXzODQoqVb9PkHqJcCG7aFoJpN0VQMNnxOBztoIQOcEO5w=="
22+
crossorigin="anonymous">
23+
</script>
24+
25+
<!-- open layers -->
26+
<link rel="stylesheet"
27+
href="https://cdn.jsdelivr.net/npm/ol@10.9.0/ol.css"
28+
integrity="sha384-Jpg1uvTciFJuHq+90Ly0/2SKRT9yS5OhbIN5FuFoFWji9fc44ok8xxO88CJi8Z5W"
29+
crossorigin="anonymous">
30+
31+
<script src="https://cdn.jsdelivr.net/npm/ol@10.9.0/dist/ol.js"
32+
integrity="sha384-H9f6d9zpuIdeGwVsGIPSbRgWKZ8WRk2C1K7GxlC61fe5qjRCL/5Qvp2wUvUVDS8v"
33+
crossorigin="anonymous">
34+
</script>
35+
36+
<!-- geotiff.js -->
37+
<script src="https://cdn.jsdelivr.net/npm/geotiff@3.0.5/dist-browser/geotiff.min.js"
38+
integrity="sha384-DO8rKWcIW99TbR6KkYDH8iWvF+phDTyWcniS1ZemcLljgA1LVIbw09us1D0jiAN9"
39+
crossorigin="anonymous">
40+
</script>
41+
42+
<!-- UTIFF -->
43+
<script src="https://cdn.jsdelivr.net/npm/utif@3.1.0/UTIF.js"
44+
integrity="sha384-RyBmXHdfZ/Uon+ud+/AqSyWpUWnKYt2tkRG/P4gWoRUGDU+qIAV3tGBPNlYTBZEF"
45+
crossorigin="anonymous">
46+
</script>
47+
2948
<!-- spinner-->
3049
<script src="https://cdnjs.cloudflare.com/ajax/libs/spin.js/2.3.2/spin.js"
31-
integrity="sha512-C7tgVIfPE0ivBKcs2WstAh5y7Njir2odGBjnuIa64SmVzZIoTb8kRrNursRzEv4bNcesPywtVAXqH1GmqRBmpg==" crossorigin="anonymous"></script>
50+
integrity="sha512-C7tgVIfPE0ivBKcs2WstAh5y7Njir2odGBjnuIa64SmVzZIoTb8kRrNursRzEv4bNcesPywtVAXqH1GmqRBmpg=="
51+
crossorigin="anonymous">
52+
</script>
3253
</head>
3354

3455
<body class="container">
@@ -41,7 +62,7 @@ <h1 class="page-title mapPreviewText">Map Preview</h1>
4162
<div class='preview-container'>
4263
<div class='preview-header'></div>
4364
<div class='preview'>
44-
<div id="map" style="width: 800px; height: 500px;"></div>
65+
<div id="map" style="width: 900px; height: 600px;"></div>
4566
</div>
4667
</div>
4768
</main>

‎previewers/betatest/MapShpPreview.html‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
<!DOCTYPE html>
21
<html>
32
<head>
43
<meta charset="utf-8">
@@ -12,20 +11,22 @@
1211
<!-- Latest compiled and minified CSS -->
1312
<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/3.4.1/css/bootstrap.min.css"
1413
integrity="sha384-HSMxcRTRxnN+Bdg0JdbxYKrThecOKuH5zCYotlSAcp1+c8xmyTe9GYg1l9a69psu" crossorigin="anonymous">
14+
1515
<!-- Optional theme -->
1616
<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/3.4.1/css/bootstrap-theme.min.css"
1717
integrity="sha384-6pzBo3FDv/PJ8r2KRkGHifhEocL+1X2rVCTTkUfGk7/0pbek5mMa1upzvWbrUbOZ" crossorigin="anonymous">
18+
1819
<!-- Leaflet -->
1920
<link rel="stylesheet" href="https://unpkg.com/leaflet@1.7.1/dist/leaflet.css"
2021
integrity="sha512-xodZBNTC5n17Xt2atTPuE1HxjVMSvLVW9ocqUKLsCC5CXdbqCmblAshOMAS6/keqq/sMZMZ19scR4PsZChSR7A==" crossorigin="anonymous"/>
2122
<script src="https://unpkg.com/leaflet@1.7.1/dist/leaflet.js"
2223
integrity="sha512-XQoYMqMTK8LvdxXYG3nZ448hOEQiglfqkJs1NOQV44cWnUrBc8PkAOcXy20w0vlaXaVUearIOBhiXZ5V3ynxwA==" crossorigin="anonymous"></script>
2324
<link type="text/css" rel="stylesheet" href="css/preview.css" />
25+
2426
<!-- ShapeJS-->
25-
<script src="https://cdn.rawgit.com/calvinmetcalf/shapefile-js/gh-pages/dist/shp.js"
26-
integrity="sha512-RT1pg7PTZ3R8amXsuOV3aJAVjBxenRKgmLg68ZMN6RMeRQWTrbxnPszM9+6/UVkmuRYaM0cg6R5lqwQJfJhVUw==" crossorigin="anonymous"></script>
27-
<script src="https://cdn.rawgit.com/calvinmetcalf/leaflet.shapefile/gh-pages/leaflet.shpfile.js"
28-
integrity="sha512-pZ4bO+wYEIa3xGxktY7N3CDNF4QfBlzmps/cqfOY2SZA7v1kH/y7rxQaqvk1W31NqjVTcyZkV0fPaOBGcCTOgg==" crossorigin="anonymous"></script>
27+
<script src="https://unpkg.com/shpjs@6.2.0/dist/shp.min.js"
28+
integrity="sha512-uew7ZF76ydiw8bbr370XgfbQ7+B9eHMWiFbh8MiZ0Y9772nN12Wx6I913bvWb3V8GZbhxt/pBEMBHnjrz1U7GQ==" crossorigin="anonymous"></script>
29+
2930
<!-- spinner-->
3031
<script src="https://cdnjs.cloudflare.com/ajax/libs/spin.js/2.3.2/spin.js"
3132
integrity="sha512-C7tgVIfPE0ivBKcs2WstAh5y7Njir2odGBjnuIa64SmVzZIoTb8kRrNursRzEv4bNcesPywtVAXqH1GmqRBmpg==" crossorigin="anonymous"></script>
@@ -42,7 +43,7 @@ <h1 class="page-title mapPreviewText">Map Preview</h1>
4243
<div class='preview-container'>
4344
<div class='preview-header'></div>
4445
<div class='preview'>
45-
<div id="map" style="width: 800px; height: 500px;"></div>
46+
<div id="map" style="width: 900px; height: 600px;"></div>
4647
</div>
4748
</div>
4849
</main>

0 commit comments

Comments
 (0)