Troubleshoot¶
This page starts with common error messages and their usual causes. The sections after the table cover each problem.
Common error messages¶
| Error | Likely cause | See |
|---|---|---|
UnsupportedClassVersionError | Java older than 17 | Use Java 17 or later |
No suitable provider found for URI: ... | The backend module of the URI scheme is missing | Missing backend module |
ClassNotFoundException for a cloud SDK class | The backend module is missing | Missing backend module |
Unable to load credentials from any of the providers in the chain | No AWS credentials found | S3 credentials |
AccessDeniedException: access denied for key '...' | A missing permission, or a missing object | Access denied |
Authentication failed for URI: ..., status code: 403 | The http(s) URL of a private bucket, opened by the HTTP backend | Set storage.provider to s3, azure or gcs, or use the native scheme (s3://, az://, gs://) |
BlobStorageException: AuthenticationFailed | Wrong Azure credentials, or an expired SAS token | Azure authentication |
No Application Default Credentials found for Google Cloud Storage | No Google credentials found | GCS credentials |
NotFoundException | The object does not exist | Check the key and the URI of the container |
TransientStorageException: Connection timeout after ... | A network timeout | Connection timeouts |
Connection Manager failed to acquire a connection within the defined timeout | Too many S3 requests in flight | S3 requests waiting for a connection |
Response missing required ETag header | A caller-built crtBuilder S3 client against an S3-compatible service | S3-compatible services without an ETag header |
IOException: Too many open files | Many local readers in use at once | Too many open files |
Stale file handle | An NFS or SMB export changed under the reader | Stale NFS file handles |
OutOfMemoryError | Several CacheManager instances, or caching a streaming workload | Memory and the cache |
Dependencies¶
Dependency conflicts¶
Maven or Gradle reports conflicts with the AWS, Azure, or Google Cloud SDKs. Import io.tileverse:tileverse-bom: it imports the cloud SDK versions used to build and test the library. Declare it before any other cloud SDK BOM, because Maven takes a managed version from the first BOM declaring it.
<dependencyManagement>
<dependencies>
<dependency>
<groupId>io.tileverse</groupId>
<artifactId>tileverse-bom</artifactId>
<version>2.1.2</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
Missing backend module¶
Each cloud backend lives in its own module. Without it, a URI of its scheme fails to open with No suitable provider found for URI, and code that uses its SDK types fails with ClassNotFoundException. Add tileverse-storage-s3, tileverse-storage-azure, tileverse-storage-gcs, or tileverse-storage-all for all three. Getting Started lists the modules.
Credentials and access¶
S3 credentials¶
With no credential parameter, the S3 backend uses the AWS default credential chain, and reports Unable to load credentials ... when the chain finds none. Provide credentials in one of these ways:
- Environment variables:
- The credentials file,
~/.aws/credentials: - An IAM role, on EC2, ECS or EKS. Nothing to configure.
The chain never falls back to anonymous access. To read a public bucket, set storage.s3.anonymous=true. Authenticate lists all the options.
Access denied¶
AccessDeniedException means the identity lacks a permission on the object, such as s3:GetObject on S3. On S3, a missing object also returns 403 when the identity may not list the bucket.
On AWS, a request sent to the wrong region is sent again to the region of the bucket. Presigned URLs and S3-compatible services need the right region: set storage.s3.region.
Azure authentication¶
AuthenticationFailed means the credentials do not match the account. Check the connection string or the account key, and the expiry of a SAS token. The se field of the token holds its expiry time:
GCS credentials¶
With storage.gcs.anonymous unset, opening a GCS Storage fails with No Application Default Credentials found when no credentials are found. Set up Application Default Credentials:
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
# or, for your own account:
gcloud auth application-default login
To read a public bucket, set storage.gcs.anonymous=true. An AccessDeniedException caused by a GCS error 403 means the identity lacks a permission on the bucket.
Performance and memory¶
Slow reads¶
Each remote read pays a round trip. Cache the reads, with CachingRangeReader.of(reader) or with storage.caching.enabled=true on the configuration passed to StorageFactory.open. The file backend ignores that parameter. For parts of a file read again and again, such as a header or an index, declare block-aligned regions. Configure for performance describes both.
Memory and the cache¶
The byte-range cache shares the data pool budget, set by io.tileverse.cache.data.max-size, with the decoded vector tiles. An entry expires after 5 minutes without access. Caching Layers describes the pools and their settings. The cache stays empty until readers are wrapped in a CachingRangeReader or storage.caching.enabled is set.
- Each
CacheManager.newInstance()has its own two cache pools, with their own budgets. Readers built without an explicit manager share the default one: keep it that way. - A job reading a file front to back gains nothing from the cache, and fills it with bytes read once. Use the reader returned by
Storage.openRangeReaderdirectly. BlockAlignedRangeReader.alignWholeFile()turns each read inside the file into cached blocks. Declare only the regions read repeatedly, and let tile or row payloads pass through as exact ranges.
Low cache hit rate¶
The cache keys on the exact (offset, length) of a read. Two reads share an entry only when they ask for the same range, or when they fall in the same block of a block-aligned region. An entry unused for 5 minutes expires, which alone lowers a hit rate measured across idle periods.
import io.tileverse.cache.CacheStats;
import io.tileverse.storage.cache.CachingRangeReader;
if (reader instanceof CachingRangeReader cachingReader) {
CacheStats stats = cachingReader.getCacheStats();
System.out.printf("hit rate %.2f%%, %d hits, %d misses, %d evictions, %d entries%n",
stats.hitRate() * 100, stats.hitCount(), stats.missCount(), stats.evictionCount(), stats.entryCount());
}
The statistics cover the byte-range cache of the CacheManager, across the readers using it, not this reader alone.
Network¶
Connection timeouts¶
The HTTP backend reports TransientStorageException: Connection timeout after 5000 milliseconds. Raise storage.http.timeout-millis, 5000 by default:
Properties props = new Properties();
props.setProperty("storage.http.timeout-millis", "30000");
try (Storage storage = StorageFactory.open(parent, props);
RangeReader reader = storage.openRangeReader(leaf)) {
// ...
}
For S3, raise the io.tileverse.storage.s3-http-client.connection-timeout system property, 2 seconds by default; see System Properties. For a client setting without a parameter, build the SDK client yourself and pass it to the provider, as described in Configure for performance.
Proxy¶
The HTTP and S3 backends use the proxy settings of the JVM:
The S3 backend also reads the http_proxy, https_proxy and no_proxy environment variables, in lowercase only. For another backend, configure the proxy on its SDK client and pass the client to the provider, as described in Authenticate.
TLS certificates¶
For an HTTPS endpoint with a self-signed certificate or a private CA, the HTTP, Azure and GCS backends use the trust store of the JVM. Add the certificate to it:
The S3 backend uses the trust store of the operating system instead. For a MinIO or Ceph service with a private CA, install the CA in the operating system. The Java SDK does not read AWS_CA_BUNDLE.
During development, storage.http.trust-all-certificates=true disables the certificate checks of the HTTP backend; the other backends have no such setting. Never use it in production.
S3-compatible services without an ETag header¶
A gateway exporting an existing tree of files may serve objects without an ETag header: the objects placed directly in its backend have none, and the objects written through the S3 API have one. A Storage opened from a URI or a StorageConfig reads both. A client built with S3AsyncClient.crtBuilder() and passed to S3StorageProvider.open(URI, S3AsyncClient) fails with Response missing required ETag header. Pass a client built with S3AsyncClient.builder() instead.
S3 requests waiting for a connection¶
S3 requests fail with Connection Manager failed to acquire a connection within the defined timeout. The S3 storages opened from a URI or a StorageConfig send their requests through one connection pool per host, shared by the process. A request waits 30 seconds for a free connection, then fails. The SDK retries some of those failures, up to 3 times each; when many requests fail together, most of them get no retry.
Raise the pool size, io.tileverse.storage.s3-http-client.max-concurrency, or keep fewer requests in flight against the host: lower storage.batch.max-in-flight-fetches (default 8) or the number of concurrent readers. A multipart upload keeps an eighth of the pool in flight while it runs. System Properties gives the default pool size.
Local files¶
Too many open files¶
A server opening many local readers fails with IOException: Too many open files. A local reader holds file descriptors only while its channels are open: from the first read until storage.file.idle-timeout (default 60 seconds) elapses with no read in progress. A reader holds one descriptor for its channel. When batched reads into direct buffers run at once on one reader, they may open extra channels, and the reader keeps up to 4 of them open between reads. Descriptors pile up when many readers are read within the same minute, or when readers are opened and never closed.
- Close readers when a request is done with them. A closed reader never reopens its channel.
- Shorten the idle timeout, for example
storage.file.idle-timeout=PT10S. - Raise the descriptor limit (
ulimit -n) when the working set is that large.
Stale NFS file handles¶
Reads of a file on an NFS or SMB mount fail after the export was remounted or the file was replaced on the server. The operating system reports Stale file handle (Linux) or Stale NFS file handle (macOS, BSD).
Range reads recover by themselves: the reader drops the stale channel, opens the file again by its real path, and resumes after the bytes already read, up to three attempts per call. A read that still fails reports Read failed after 3 attempts: the mount stays stale.
An AccessDeniedException on a local file means the process cannot read it: check the permissions of the file.
Debug logging¶
Set the log level of io.tileverse.storage to DEBUG in your logging configuration. The SDKs log under software.amazon.awssdk (S3), com.azure (Azure) and com.google.cloud (GCS).
-Djdk.httpclient.HttpClient.log=requests,headers logs the requests and headers of the JDK HTTP client, used by the HTTP and Azure backends.
Getting help¶
Search the GitHub issues. To report a problem, include the library version, the Java version, the full error message and stack trace, and a minimal example.