cloud-map-spider

Feature

• Auto fetch and download cloud maps from NSMC and etc...
• Can be easily extended to other meteorological service sites
• Convert downloaded satellite images into a video (MP4) for playback or presentation

Usage

git clone https://github.com/Jaciezyt/cloud-map-spider.git
cd cloud-map-spider
uv venv
source ./venv/bin/activate # change on windows
uv pip install -r requirements.txt
python main.py

Request throttling

The downloader uses one shared HTTP session, browser-like headers, randomized request spacing, retries, and longer cooldowns for 418/429 responses. These defaults are intentionally conservative. You can tune them with environment variables:

CMS_REQUEST_DELAY_MIN_SECONDS=6 \
CMS_REQUEST_DELAY_MAX_SECONDS=15 \
CMS_418_COOLDOWN_SECONDS=180 \
CMS_REQUEST_RETRIES=5 \
python main.py

Useful options:

• CMS_USER_AGENT: override the default browser-like User-Agent
• CMS_REQUEST_DELAY_MIN_SECONDS / CMS_REQUEST_DELAY_MAX_SECONDS: random delay between queued requests
• CMS_REQUEST_RETRIES: retry count before surfacing an HTTP/network error
• CMS_418_COOLDOWN_SECONDS / CMS_429_COOLDOWN_SECONDS: host cooldown after anti-bot/rate-limit responses
• CMS_CONNECT_TIMEOUT_SECONDS / CMS_READ_TIMEOUT_SECONDS: request timeouts

Convert Downloaded Images to Video

python videomaker.py

Implement

1. poll: polling threads will be created for all targets
2. get_manifest: get manifest from target.url
3. parse_parset: parse manifest content to image url list
4. get_image: get image data; if failed, mark {url} with failed
5. save_temp: save image to temp directory, marking {url} with downloaded
6. do_exports: do exports as the user definition
   1. If exporter returns False, mark {url}.{export} with failed; Otherwise, mark {url}.{export} with success
   2. If ALL exporters are marked with success, mark {url} with success
7. clean: if {url} is marked with success, delete the image from temp directory

Legal Statement

This project is intended solely for meteorological research purposes.
It is not affiliated with NSMC or any other organization.
The contributors are not responsible for any misuse or abuse of this software.
Please use this tool legally and responsibly.