QtLocationPlugin: tile cache, fetch, and provider overhaul#14556
Open
HTRamsey wants to merge 1 commit into
Open
QtLocationPlugin: tile cache, fetch, and provider overhaul#14556HTRamsey wants to merge 1 commit into
HTRamsey wants to merge 1 commit into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Overhauls QGroundControl’s QtLocation tile stack: provider URL correctness (HTTPS/tokens/retina), fetch pipeline (dedup, adaptive concurrency, retries/circuit-breaker, stale-while-revalidate), and replaces the legacy multi-layer cache with a single SQLite-authoritative store (schema v4 + LRU), plus offline archive import support.
Changes:
- Reworked tile fetching: adaptive concurrency gating, request coalescing/dedup, Retry-After parsing, per-host circuit breaker, and viewport-priority queueing.
- Replaced/centralized caching into an SQLite-backed authoritative cache (schema/migrations split into
QGCTileDatabaseSchema, true LRU viaaccessed, validators + must-revalidate). - Added broad unit test coverage for providers, fetch/cache paths, worker/task semantics, and new helpers (metrics, circuit breaker, importer, fallback tiles).
Reviewed changes
Copilot reviewed 99 out of 99 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| test/Utilities/Network/QGCNetworkHelperTest.h | Adds test slots for transient error + Retry-After parsing. |
| test/Utilities/Network/QGCNetworkHelperTest.cc | Implements new Retry-After/transient-error unit tests. |
| test/Utilities/Database/QGCSqlHelperTest.h | Adds execPrepared test slots. |
| test/Utilities/Database/QGCSqlHelperTest.cc | Tests QGCSqlHelper::execPrepared success/fail paths. |
| test/UnitTestFramework/Fixtures/RAIIFixtures.h | Extends NetworkReplyFixture API for raw headers. |
| test/UnitTestFramework/Fixtures/RAIIFixtures.cc | Implements NetworkReplyFixture::setRawHeader. |
| test/QtLocationPlugin/UrlFactoryTest.h | Adds regression tests for retina hash + layer slug URL interpolation. |
| test/QtLocationPlugin/UrlFactoryTest.cc | Updates expectations/logs; adds retina-hash and layer-slug URL tests; enforces HTTPS scheme. |
| test/QtLocationPlugin/TileFetchMetricsTest.h | New unit test declarations for metrics counters/reset. |
| test/QtLocationPlugin/TileFetchMetricsTest.cc | New unit tests for TileFetchMetrics singleton behavior. |
| test/QtLocationPlugin/QtLocationTestBase.h | Adds shared helpers for QtLocation plugin tests. |
| test/QtLocationPlugin/QGeoTileFetcherQGCTest.h | New tests for request building, dedup, expiry parsing, concurrency gate. |
| test/QtLocationPlugin/QGeoTileFetcherQGCTest.cc | Implements fetcher unit tests (dedup/expiry/must-revalidate/concurrency). |
| test/QtLocationPlugin/QGeoTiledMapReplyQGCTest.h | New tests for reply cache/network/freshness behavior. |
| test/QtLocationPlugin/QGeoTiledMapReplyQGCTest.cc | Implements reply behavior tests (cache hit/miss, 304 handling, must-revalidate behavior, stale serving). |
| test/QtLocationPlugin/QGeoFileTileCacheQGCTest.h | New tests for cache no-op behavior and task creation. |
| test/QtLocationPlugin/QGeoFileTileCacheQGCTest.cc | Verifies cache does not shadow SQLite (get/insert no-op semantics). |
| test/QtLocationPlugin/QGCTileSetImporterTest.h | Declares importer tests (MBTiles/PMTiles + malformed cases). |
| test/QtLocationPlugin/QGCTileFallbackTest.h | Declares fallback-scaling/encoding tests. |
| test/QtLocationPlugin/QGCTileFallbackTest.cc | Tests fallback cropping/scaling and PNG encoding. |
| test/QtLocationPlugin/QGCMapTaskTest.h | Declares tests for map task types/signals/ownership. |
| test/QtLocationPlugin/QGCMapTaskTest.cc | Tests task semantics (signals, ownership transfer, progress). |
| test/QtLocationPlugin/QGCMapEngineManagerTest.h | Declares singleton/model behavior tests for map engine manager. |
| test/QtLocationPlugin/QGCMapEngineManagerTest.cc | Tests manager lists, totals update, selection helpers, rename behavior, task error labeling. |
| test/QtLocationPlugin/QGCHostCircuitBreakerTest.h | Declares circuit breaker unit tests. |
| test/QtLocationPlugin/QGCHostCircuitBreakerTest.cc | Tests breaker open/close thresholds and reset on success. |
| test/QtLocationPlugin/QGCCacheWorkerTest.h | Adds worker tests for post-stop enqueue rejection, totals, main-thread affinity. |
| test/QtLocationPlugin/QGCCacheWorkerTest.cc | Updates worker tests for new QSharedPointer tile fetch and new behaviors. |
| test/QtLocationPlugin/QGCCachedTileSetTest.h | Adds tests for paused state and status strings. |
| test/QtLocationPlugin/QGCCachedTileSetTest.cc | Implements paused/status/error string tests. |
| test/QtLocationPlugin/MockMapProvider.h | Adds mock provider for URL dispatch tests. |
| test/QtLocationPlugin/MapProviderTest.h | Adds tests for getTileURL dispatch + invalid URL behavior. |
| test/QtLocationPlugin/MapProviderTest.cc | Uses MockMapProvider to validate URL dispatch behavior. |
| test/QtLocationPlugin/CMakeLists.txt | Registers newly added QtLocation plugin unit tests/sources. |
| src/Viewer3D/Viewer3DTileReply.h | Updates cache-hit path to use QSharedPointer<QGCCacheTile>. |
| src/Viewer3D/Viewer3DTileReply.cc | Removes manual delete; adapts to shared ownership. |
| src/Utilities/Network/QGCNetworkHelper.h | Adds transient-error classification + Retry-After parsing API and constants. |
| src/Utilities/Network/QGCNetworkHelper.cc | Implements transient error classification and Retry-After parsing; updates UA string. |
| src/Utilities/Database/QGCSqlHelper.h | Adds execPrepared, richer SQLite pragma helpers, and new scoped-connection args. |
| src/Utilities/Database/QGCSqlHelper.cc | Implements extended pragmas + WAL checkpoint/optimize/vacuum helpers; adds busy timeout support. |
| src/Settings/AppSettings.h | Adds LINZ token setting. |
| src/Settings/AppSettings.cc | Registers LINZ token Fact. |
| src/Settings/App.SettingsGroup.json | Defines LINZ token setting and updates Custom URL example text. |
| src/QtLocationPlugin/TileFetchMetrics.h | Adds lightweight lock-free tile fetch metrics API. |
| src/QtLocationPlugin/TileFetchMetrics.cpp | Implements metrics counters, snapshot, and formatting. |
| src/QtLocationPlugin/QGeoTileFetcherQGC.h | Major fetcher changes: adaptive concurrency, coalescing, snapshot struct, viewport priority updates. |
| src/QtLocationPlugin/QGeoTileFetcherQGC.cpp | Implements concurrency gating, dedup/coalescing, expiry parsing, request building, and queue reprioritization. |
| src/QtLocationPlugin/QGeoTiledMapQGC.cpp | Adds SupportsSetBearing capability. |
| src/QtLocationPlugin/QGeoTiledMappingManagerEngineQGC.h | Tracks active maps and manages prefetch style updates. |
| src/QtLocationPlugin/QGeoTiledMappingManagerEngineQGC.cpp | Sets tile size based on retina, updates prefetch style on reachability changes, improves error handling. |
| src/QtLocationPlugin/QGeoServiceProviderPluginQGC.cpp | Replaces asserts with runtime thread/availability checks and better error handling. |
| src/QtLocationPlugin/QGeoMapReplyQGC.h | Extends reply to support coalesced results, retries, conditional headers, fallback tiles, and revalidation. |
| src/QtLocationPlugin/QGeoFileTileCacheQGC.h | Converts to QAbstractGeoTileCache subclass; SQLite-only authoritative cache; adds validator refresh API. |
| src/QtLocationPlugin/QGeoFileTileCacheQGC.cpp | Implements no-op get/insert and cache/validator task emission; improves old-cache wipe logic. |
| src/QtLocationPlugin/QGCTileSetImporter.h | Declares MBTiles/PMTiles import API. |
| src/QtLocationPlugin/QGCTileFallback.h | Declares fallback crop/scale/encode helpers. |
| src/QtLocationPlugin/QGCTileFallback.cpp | Implements fallback crop/scale and PNG encoding. |
| src/QtLocationPlugin/QGCTileDatabaseSchema.h | Adds schema create/migrate/drop/check API (standalone). |
| src/QtLocationPlugin/QGCTileDatabaseSchema.cpp | Implements schema v4 migrations, LRU indices, and legacy reset logic. |
| src/QtLocationPlugin/QGCTileCacheWorker.h | Refactors worker from QThread subclass to QObject + owned QThread; adds batching. |
| src/QtLocationPlugin/QGCTileCacheWorker.cpp | Implements new worker thread loop, batching, shutdown semantics, and init/stop behavior. |
| src/QtLocationPlugin/QGCTileCacheDatabase.h | Updates schema version, adds batch save, validator refresh, accessed bump, and filtered state updates. |
| src/QtLocationPlugin/QGCTile.h | Adds paused download state. |
| src/QtLocationPlugin/QGCMapUrlEngine.h | Adds initialization caching, retina helpers, and provider lookup maps. |
| src/QtLocationPlugin/QGCMapUrlEngine.cpp | Adds provider lookup caches, retina hash space offsetting, and warning de-dup. |
| src/QtLocationPlugin/QGCMapTasks.cpp | Moves execution logic into tasks; adds fallback/validator tasks and thread-affinity handling. |
| src/QtLocationPlugin/QGCMapTaskBase.h | Adds execute(QGCCacheWorker&) to tasks; expands task types. |
| src/QtLocationPlugin/QGCMapEngineManager.h | Cleans up Q_PROPERTY declarations, adds QML type registration hook. |
| src/QtLocationPlugin/QGCMapEngine.cpp | Registers QSharedPointer<QGCCacheTile> metatype; updates worker lifecycle management. |
| src/QtLocationPlugin/QGCHostCircuitBreaker.h | Adds process-wide per-host circuit breaker implementation. |
| src/QtLocationPlugin/QGCCacheTile.h | Adds HTTP validators (etag/last-modified/expires/mustRevalidate) and QSharedPointer metatype. |
| src/QtLocationPlugin/QGCCachedTileSet.h | Adds paused state and removes reply mutex with main-thread-only model. |
| src/QtLocationPlugin/QGCCachedTileSet.cpp | Implements pause/resume behavior, queued delivery for worker results, and adaptive concurrency caching. |
| src/QtLocationPlugin/Providers/TianDiTuProvider.h | Formatting + server-count constant. |
| src/QtLocationPlugin/Providers/TianDiTuProvider.cpp | Percent-encodes token and uses configured server count. |
| src/QtLocationPlugin/Providers/MapProvider.h | Adds validation helpers, OSM UA flag, settings helpers; converts zoom macro to constexpr. |
| src/QtLocationPlugin/Providers/MapProvider.cpp | Adds enum drift asserts, token/settings helpers, coordinate validation, and more format detection. |
| src/QtLocationPlugin/Providers/MapboxMapProvider.h | Formatting cleanup. |
| src/QtLocationPlugin/Providers/MapboxMapProvider.cpp | Adds retina-aware URLs and percent-encoded tokens. |
| src/QtLocationPlugin/Providers/GoogleMapProvider.h | Switches to HTTPS, server-count const, removes hard-coded scale. |
| src/QtLocationPlugin/Providers/GoogleMapProvider.cpp | Uses retina pixel scale and server count constant. |
| src/QtLocationPlugin/Providers/GenericMapProvider.cpp | Adds token encoding, fixes layer slug usage, LINZ token support, SSRF guard for custom URL, and VWorld math safety. |
| src/QtLocationPlugin/Providers/EsriMapProvider.h | Switches to HTTPS and formatting cleanup. |
| src/QtLocationPlugin/Providers/EsriMapProvider.cpp | Uses centralized settings access for token retrieval. |
| src/QtLocationPlugin/Providers/ElevationMapProvider.h | Formatting cleanup. |
| src/QtLocationPlugin/Providers/ElevationMapProvider.cpp | Adds inverted-bbox underflow guard. |
| src/QtLocationPlugin/Providers/BingMapProvider.h | Switches to HTTPS and adds server-count const. |
| src/QtLocationPlugin/Providers/BingMapProvider.cpp | Uses server-count constant. |
| src/QtLocationPlugin/CMakeLists.txt | Adds new plugin sources and links Qt6::Gui (retina/QGuiApplication usage). |
| src/QGCApplication.cc | Registers map-engine manager QML types during QML root initialization. |
Comment on lines
+702
to
+709
| const QDateTime when = QDateTime::fromString(QString::fromLatin1(raw), Qt::RFC2822Date); | ||
| if (when.isValid()) { | ||
| const qint64 deltaMs = QDateTime::currentDateTimeUtc().msecsTo(when.toUTC()); | ||
| if (deltaMs <= 0) { | ||
| return std::chrono::milliseconds::zero(); | ||
| } | ||
| return std::min(std::chrono::milliseconds(deltaMs), kMaxRateLimitDelay); | ||
| } |
Comment on lines
+411
to
+413
| const qint64 now = QDateTime::currentSecsSinceEpoch(); | ||
| const bool expired = (_cachedTile->expiresAt != 0) && (_cachedTile->expiresAt <= now); | ||
| const bool revalidatable = !_cachedTile->etag.isEmpty() || !_cachedTile->lastModified.isEmpty(); |
Comment on lines
+233
to
+235
| // A bare 304 carries no Expires/Cache-Control; keep the cached expiry rather than clobbering it to 0. | ||
| const qint64 newExpiresAt = (result.expiresAt == 0) ? _cachedTile->expiresAt : result.expiresAt; | ||
|
|
HTRamsey
force-pushed
the
feature/qtlocation-overhaul
branch
3 times, most recently
from
June 18, 2026 06:24
263bfa8 to
9522c8a
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #14556 +/- ##
==========================================
+ Coverage 25.47% 30.15% +4.68%
==========================================
Files 769 777 +8
Lines 65912 67173 +1261
Branches 30495 31159 +664
==========================================
+ Hits 16788 20258 +3470
+ Misses 37285 33238 -4047
- Partials 11839 13677 +1838
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 374 files with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
HTRamsey
force-pushed
the
feature/qtlocation-overhaul
branch
2 times, most recently
from
June 18, 2026 07:35
5e6b98f to
e8135f0
Compare
Providers: - HTTPS tile URLs; mapTypeId/server-count/format-detection correctness; LINZ Basemaps token via settings Tile fetch / network: - Enforce adaptive concurrency: gate concurrent leader GETs at the cellular/wifi budget (previously computed but never applied) - Single transfer-timeout authority (20s leader, chrono overload) - must-revalidate/no-cache: cache then lazily revalidate - Viewport-aware prioritization: drop pending off-screen tiles, keep in-flight, visible tiles ahead of prefetch backlog - 7-day default tile expiry; identifiable QGroundControl UA for OSM - Drop dead QNetworkRequest cache/HTTP2 attributes and unused include Tile cache (SQLite): - True LRU eviction via accessed-timestamp column (was FIFO by insert date) - mustRevalidate column; schema user_version 2->3 in-place migration - Fix WAL read-lock that blocked legacy-DB reset on DROP TABLE Import: - MBTiles (TMS->XYZ) and PMTiles v3 (Hilbert dir, gzip) import into the tile cache as offline sets Tests added across all areas.
HTRamsey
force-pushed
the
feature/qtlocation-overhaul
branch
from
June 18, 2026 09:28
e8135f0 to
8d4b8a8
Compare
Contributor
|
@HTRamsey Should this wait for 5.2? Concerned such a major change will get no bake time before 5.1 goes out? |
Member
Author
|
Yup that's fine |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Overhaul of the QtLocation plugin's tile cache, fetch pipeline, and providers.
Providers
mapTypeId/server-count/format-detection correctnessTile fetch / network
chronooverload)must-revalidate/no-cache: cache then lazily revalidateQNetworkRequestcache/HTTP2 attributes and unused includeTile cache (SQLite)
accessed-timestamp column (was FIFO by insert date)mustRevalidatecolumn; in-place schema migrations touser_version4DROP TABLEQGCTileDatabaseSchemanamespace, independently unit-testableQGeoFileTileCacheQGCnow subclassesQAbstractGeoTileCachedirectly so the SQLite store is the sole authoritative cache (no shadow flat-file/QCache3Q layers)QGCCachedTileSetdownload state is now main-thread-only (queued task-result delivery), removing the per-tile mutexImport
Tests
Notes