feat(oauth2): implement IAM impersonation mTLS transport pinning and 401 recovery - #14212
macastelaz wants to merge 7 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces support for actor tokens in IdentityPoolCredentials (specifically for file-based JSON credential sources over mTLS endpoints) and adds certificate pinning with automatic certificate reloading on 401 Unauthorized errors. It also updates other credential classes to support passing a custom HttpTransportFactory during token refresh, masks actor tokens in logs, and adds corresponding unit tests. The review feedback suggests persisting the reloaded transport factory back to this.transportFactory after a successful retry to prevent subsequent refreshes from failing first, and preserving programmatically set actorTokenSupplier instances in the Builder copy constructor.
73a0301 to
4be79cb
Compare
4be79cb to
ce341c7
Compare
…401 recovery - Pin mTLS HttpTransportFactory across multi-step STS and IAM token exchanges so both requests use the exact same certificate snapshot within a single refresh cycle. - Add 401 Unauthorized recovery with automatic certificate reload from X509Provider and single-retry coordination in IdentityPoolCredentials and ImpersonatedCredentials. - Preserve custom non-default HttpTransportFactory instances when X509Provider is configured. - Add comprehensive unit tests across IdentityPoolCredentialsTest, ImpersonatedCredentialsTest, and OAuth2UtilsTest.
e09d76e to
dadeda6
Compare
…dentials and address review findings - Explicitly scope inner sourceCredentials to CLOUD_PLATFORM_SCOPE in ExternalAccountCredentials.buildImpersonatedCredentials and ImpersonatedCredentials.refreshAccessToken so STS issues tokens authorized to call IAM generateAccessToken even when downstream target scopes are configured via createScoped. - Ensure public no-arg ImpersonatedCredentials.refreshAccessToken delegates without overriding source credential transport settings. - Preserve custom actorTokenSupplier in IdentityPoolCredentials.Builder copy constructor when credentialSource is present. - Ensure HTTP response is closed in a finally block in ImpersonatedCredentials.refreshAccessToken. - Attach initial 401 exception as suppressed when the 401 retry attempt fails in IdentityPoolCredentials.refreshWithRetry. - Add unit tests in IdentityPoolCredentialsTest and ImpersonatedCredentialsTest covering scoped impersonation, custom actorTokenSupplier preservation, and retry exception chaining.
| } | ||
| if (this.impersonatedCredentials != null) { | ||
| return this.impersonatedCredentials.refreshAccessToken(); | ||
| return this.impersonatedCredentials.refreshAccessToken(cycleTransportFactory); |
There was a problem hiding this comment.
When serviceAccountImpersonationUrl is set, IdentityPoolCredentials.refreshWithRetry fetches the subject and actor tokens to build stsTokenExchangeRequest, calls exchangeExternalCredentialForAccessToken, and then immediately delegates to impersonatedCredentials.refreshAccessToken here while discarding stsTokenExchangeRequest. The inner credential then fetches the subject and actor tokens a second time in the same refresh cycle, or four times on a 401 retry. Can we short-circuit before fetching tokens when serviceAccountImpersonationUrl is present?
There was a problem hiding this comment.
Done! Added ImpersonatedCredentials impersonated = getImpersonatedCredentials();
if (impersonated != null) {
return impersonated.refreshAccessToken(cycleTransportFactory);
}
at the start of: IdentityPoolCredentials.refreshWithRetry, AwsCredentials.refreshAccessToken, and
PluggableAuthCredentials.refreshAccessToken before retrieveSubjectToken() / getActorToken() are called.
Subject and actor tokens (and AWS metadata requests) are now fetched only once per refresh attempt (or twice on a 401 retry). Kept the check inside exchangeExternalCredentialForAccessToken as a defensive fallback if called directly.
| * @return the refreshed access token | ||
| * @throws IOException if the token refresh fails | ||
| */ | ||
| public AccessToken refreshAccessToken(HttpTransportFactory transportFactory) throws IOException { |
There was a problem hiding this comment.
The default refreshAccessToken(HttpTransportFactory) delegates to refreshAccessToken() and silently drops the transport factory. All internal subclasses override it, but a third-party subclass would lose transport pinning without warning. Should we annotate this method or class with @InternalExtensionOnly per the repository versioning guidelines?
There was a problem hiding this comment.
Annotated public AccessToken refreshAccessToken(HttpTransportFactory cycleTransportFactory) with @InternalExtensionOnly across ExternalAccountCredentials, IdentityPoolCredentials, AwsCredentials, and PluggableAuthCredentials (and kept ImpersonatedCredentials.refreshAccessToken(HttpTransportFactory) package-private).
| } | ||
|
|
||
| /** | ||
| * Refreshes the access token using the specified transport factory. Default implementation |
There was a problem hiding this comment.
Nit: The Javadoc says the default implementation delegates to refreshAccessToken(), whereas all three subclasses do the reverse by delegating the no-arg method into this overload. Also, naming the parameter transportFactory in AwsCredentials.java:127 and ImpersonatedCredentials.java:601 shadows the class field, whereas IdentityPoolCredentials names it cycleTransportFactory.
There was a problem hiding this comment.
Updated the Javadoc on ExternalAccountCredentials.refreshAccessToken(HttpTransportFactory) to clarify that internal subclasses delegate refreshAccessToken() into this method while the default implementation delegates back to refreshAccessToken(), and renamed the parameter to cycleTransportFactory across all classes to avoid shadowing this.transportFactory.
| .refreshAccessToken(effectiveTransportFactory); | ||
| Credentials authCredentials = | ||
| intermediateAccessToken != null | ||
| ? OAuth2Credentials.create(intermediateAccessToken) |
There was a problem hiding this comment.
Wrapping intermediateAccessToken in OAuth2Credentials.create creates a bare credential with a null quotaProjectId, which drops the x-goog-user-project header from the IAM generateAccessToken request. Can we use OAuth2Credentials.newBuilder().setAccessToken(intermediateAccessToken).setQuotaProjectId(this.sourceCredentials.getQuotaProjectId()).build() so quota attribution is preserved?
There was a problem hiding this comment.
Replaced OAuth2Credentials.create(intermediateAccessToken) with a GoogleCredentials wrapper initialized via GoogleCredentials.newBuilder().setQuotaProjectId(this.sourceCredentials.getQuotaProjectId()).setUniverseDomain(this.sourceCredentials.getUniverseDomain()) so both quotaProjectId (x-goog-user-project) and universeDomain are preserved on the IAM request.
| } | ||
| AccessToken intermediateAccessToken = | ||
| (transportFactory == null) | ||
| ? ((ExternalAccountCredentials) this.sourceCredentials).refreshAccessToken() |
There was a problem hiding this comment.
In the null transport branch, calling refreshAccessToken directly instead of refreshIfExpired bypasses the token cache, lock, and change listeners on sourceCredentials for non-mTLS WIF impersonation. Should line 620 call refreshIfExpired and read getAccessToken, while keeping the uncached refreshAccessToken(effectiveTransportFactory) call on line 621 for pinned mTLS transports?
There was a problem hiding this comment.
Done! When cycleTransportFactory == null, we now call this.sourceCredentials.refreshIfExpired() and pass this.sourceCredentials directly to new HttpCredentialsAdapter(this.sourceCredentials) (preserving token caching, locking, and change listeners), while keeping the uncached refreshAccessToken(effectiveTransportFactory) call when cycleTransportFactory != null.
| "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/test@project.iam.gserviceaccount.com:generateAccessToken")) { | ||
| @Override | ||
| HttpTransportFactory createMtlsTransportFactory(KeyStore keyStore) { | ||
| usedKeyStores.add(keyStore); |
There was a problem hiding this comment.
In the five new retry and impersonation tests, createMtlsTransportFactory records usedKeyStores.add(keyStore) and returns () -> mockTransport, which is the exact same HttpTransport instance regardless of which keystore was passed. The tests would still pass even if refreshWithRetry passed the stale transport on retry. Can we return a distinct transport per keystore so the mock verifies that the retry request ran on the transport created from freshKeyStore, and add tests for a persistent 401 failure and a non-401 IAM error?
There was a problem hiding this comment.
Updated createMtlsTransportFactory(KeyStore) in the impersonation mTLS tests to return a distinct HttpTransport wrapper per KeyStore that records which KeyStore executed each request (requestKeyStores), and added "refreshAccessToken_impersonation_persistent401OnIam_throwsWithSuppressed" and "refreshAccessToken_impersonation_non401OnIam_doesNotRetry".
| } | ||
|
|
||
| @Test | ||
| void builder_actorToken_plainPublicTokenUrl_throws() throws Exception { |
There was a problem hiding this comment.
Nit: The five actor-token endpoint validation tests added at lines 1821 through 1928 duplicate the existing five tests at lines 2873 through 2981 in the same file. Can we remove the duplicate block?
There was a problem hiding this comment.
Removed the 5 duplicate actor-token endpoint validation tests.
| errorResponse.put("error", "invalid_token"); | ||
| errorResponse.put("error_description", "Invalid or expired client certificate."); | ||
| return new MockLowLevelHttpResponse() | ||
| .setStatusCode(statusCode) |
There was a problem hiding this comment.
The early return for non-200 status codes exits before reaching assertEquals(EXPECTED_GRANT_TYPE, grantType), so the failed first attempt never has its request form parameters verified. Can we move the grant_type assertion above the status code check?
There was a problem hiding this comment.
Moved the grant_type and subject_token assertions above if (statusCode != 200) so request parameters are verified on simulated error responses as well.
|
|
||
| @Test | ||
| void isUnauthorizedException_null_returnsFalse() { | ||
| org.junit.jupiter.api.Assertions.assertFalse(OAuth2Utils.isUnauthorizedException(null)); |
There was a problem hiding this comment.
Nit: This file already static-imports assertEquals and assertThrows at lines 35 and 36, but the new tests write out org.junit.jupiter.api.Assertions.assertFalse, org.junit.jupiter.api.Assertions.assertTrue, java.io.IOException, com.google.api.client.http.HttpResponseException, and com.google.api.client.http.HttpHeaders inline. Can we use static imports and top-level imports here?
There was a problem hiding this comment.
Replaced all inline fully qualified names with static and top-level imports in OAuth2UtilsTest.
| .getTransport() | ||
| .addStatusCodeAndMessage(HttpStatusCodes.STATUS_CODE_OK, ""); | ||
|
|
||
| java.util.concurrent.atomic.AtomicReference<HttpTransportFactory> capturedSourceTransport = |
There was a problem hiding this comment.
Nit: The line-wrapping changes to the existing private key string and universe-domain assertion message are unrelated formatting churn, and java.util.concurrent.atomic.AtomicReference is written out inline. Can we revert the unrelated rewrapping to keep the diff clean and add a top-level import for AtomicReference?
There was a problem hiding this comment.
Reverted the line-wrapping changes on SA_PRIVATE_KEY_PKCS8 and the universe-domain assertion message, and replaced inline fully qualified names with top-level imports.
…in IdentityPoolCredentials
…ss and FILE cert pinning
…InternalExtensionOnly
| try { | ||
| KeyStore mtlsKeyStore = this.x509Provider.getKeyStore(); | ||
| this.transportFactory = new MtlsHttpTransportFactory(mtlsKeyStore); | ||
| if (shouldUseMtlsTransportFactory()) { |
There was a problem hiding this comment.
After deserialization this only rebuilds the factory when shouldUseMtlsTransportFactory() is true, so an MtlsHttpTransportFactory subclass or a ServiceLoader-provided factory comes back keyless without a client certificate, whereas the old readObject always restored the keystore-backed factory. Should this also replace the factory when transportFactory instanceof MtlsHttpTransportFactory && !((MtlsHttpTransportFactory) transportFactory).hasKeyStore()?
| * @return the refreshed access token | ||
| * @throws IOException if the token refresh fails | ||
| */ | ||
| @InternalExtensionOnly |
There was a problem hiding this comment.
Does this overload need to be public? Commit f2d752e compiled cleanly with it package-private, and making it public @InternalExtensionOnly advertises a call target whose default implementation ignores transportFactory and whose IdentityPoolCredentials override skips 401 cert recovery.
| iamTransportFactory.getTransport().setExpireTime(getDefaultExpireTime()); | ||
| iamTransportFactory.getTransport().addStatusCodeAndMessage(HttpStatusCodes.STATUS_CODE_OK, ""); | ||
|
|
||
| IdentityPoolCredentials sourceCredentials = |
There was a problem hiding this comment.
The source credential here defaults to cloud-platform, so the new !currentScopes.contains(CLOUD_PLATFORM_SCOPE) branch at ImpersonatedCredentials.java:613 never runs. Can we pass .setScopes(Collections.singletonList("https://www.googleapis.com/auth/devstorage.read_only")) on the source builder so the test covers that branch?
| .setTokenUrl("https://sts.mtls.googleapis.com/v1/token") | ||
| .build(); | ||
| IdentityPoolCredentials deserialized = serializeAndDeserialize(regularCredentials); | ||
| assertTrue( |
There was a problem hiding this comment.
Nit: Can this assert ((MtlsHttpTransportFactory) deserialized.getTransportFactory()).hasKeyStore() too? Right now instanceof MtlsHttpTransportFactory passes even when readObject leaves the deserialized subclass with a null KeyStore.
| @Override | ||
| HttpTransportFactory createMtlsTransportFactory(KeyStore keyStore) { | ||
| usedKeyStores.add(keyStore); | ||
| return () -> transport; |
There was a problem hiding this comment.
Nit: createMtlsTransportFactory returns the same () -> transport instance for both ksA and ksB, so this test passes even if the 401 retry reuses ksA's transport. Could it return a per-keystore recording transport like the impersonation retry tests do?
| throw new IllegalArgumentException( | ||
| "Actor tokens are only supported for mTLS token exchanges. Please configure a certificate" | ||
| + " source or MtlsHttpTransportFactory."); | ||
| + " configuration in the credential source or provide an mTLS-enabled transport."); |
There was a problem hiding this comment.
Nit: The exception message says "or provide an mTLS-enabled transport", but isMtlsConfigured() at line 230 only accepts x509Provider != null or an MtlsHttpTransportFactory with hasKeyStore(). Should the message say "or provide an MtlsHttpTransportFactory constructed with a KeyStore"?
| return credential.refreshAccessToken(); | ||
| }); | ||
| Future<AccessToken> futureA = executor.submit(() -> credential.refreshAccessToken()); | ||
| Future<AccessToken> futureB = executor.submit(() -> credential.refreshAccessToken()); |
There was a problem hiding this comment.
Nit: With the new CyclicBarrier, getKeyStoreCount is deterministically 3, so line 2442 can assert assertEquals(3, ...) instead of >= 3 and verify that credential.getTransportFactory() is still mtlsTransport after the retry.
| } | ||
|
|
||
| @Test | ||
| void hasCertificateChanged_nullOrSameReference_returnsFalse() throws Exception { |
There was a problem hiding this comment.
Nit: hasCertificateChanged_nullOrSameReference_returnsFalse asserts assertTrue at lines 162-163 and only checks identical references. Could we also compare two distinct KeyStore instances loaded with the same cert so oldCerts.equals(newCerts) is tested directly?
| return new AccessToken("intermediate-sts-token-" + count, null); | ||
| } | ||
|
|
||
| @Override |
There was a problem hiding this comment.
Nit: The createScoped override here is never called because line 1554 already sets cloud-platform scope, and the 401 retry comes from HttpCredentialsAdapter.handleResponse on the null-transport path. Can we drop the unused createScoped override and rename the test to reflect that?
| sourceCredentials = | ||
| PluggableAuthCredentials.newBuilder((PluggableAuthCredentials) this) | ||
| .setServiceAccountImpersonationUrl(null) | ||
| .setScopes(Collections.singletonList(OAuth2Utils.CLOUD_PLATFORM_SCOPE)) |
There was a problem hiding this comment.
Because buildImpersonatedCredentials() clears serviceAccountImpersonationUrl on the inner sourceCredentials copy, getServiceAccountEmail() returns null when PluggableAuthCredentials.retrieveSubjectToken() runs on that copy, so GOOGLE_EXTERNAL_ACCOUNT_IMPERSONATED_EMAIL is omitted from envMap. Should buildImpersonatedCredentials() preserve the impersonated email on the PluggableAuthCredentials source copy?
🥞 Stacked PRs
Summary
Propagates the cycle-pinned mTLS transport factory into
ImpersonatedCredentialsduring Workload Identity Federation with Service Account Impersonation, ensuring the IAMgenerateAccessTokencall uses the exact same pinned client certificate as the STS token exchange.Also extends 401 recovery in
IdentityPoolCredentialsto catch unauthorized responses across both STS and IAM calls, reloading fresh certificates and re-executing the full refresh cycle.Test Coverage
oauth2_http(including transport pinning lifecycle, multi-cycle rotation, 401 retry on IAM, and suppressed error handling).com.spotify.fmt:fmt-maven-plugin:2.25).Manual Testing
Sample test "app": https://paste.googleplex.com/5495919190081536
Test results: https://paste.googleplex.com/5757145073713152
See b/542238030 for tracking.