Repository navigation
feat: implement Kotlin sample parity, visual test suite, and autonomous QA automation - #2424
Conversation
bfe7f4c to
1c5e1e9
Compare
1c5e1e9 to
de40cf2
Compare
|
|
||
| protected val geminiApiKey: String by lazy { | ||
| try { | ||
| BuildConfig.MAPS_API_KEY // Falls back if GEMINI_API_KEY is not separately generated |
There was a problem hiding this comment.
Do we really wants to access MAPS_API_KEY here or we need to use GEMINI_API_KEY ??. I could not find GEMINI_API_KEY in the code.
There was a problem hiding this comment.
Addressed in commit a0779d71. Removed fallback to MAPS_API_KEY entirely. The test now only reads GEMINI_API_KEY and cleanly skips if it is not configured.
| ActivityScenario.launch(VisibleRegionDemoActivity::class.java).use { | ||
| waitForMap() | ||
| // Tap "Actions ▾" popup menu button | ||
| uiDevice.click(539, 529) |
There was a problem hiding this comment.
In VerifiedSamplesVisualTest.kt (and lines 77, 103, 132, 158, 183, 211, 239, 265):
Absolute pixel coordinates (x=967, y=2324) are calibrated to a single 1080×2400 display density/resolution. On any CI emulator, Gradle Managed Device, or phone with different dimensions (e.g. 1080×1920, Pixel Tablet, or landscape orientation), these clicks miss the target views or tap the system navigation bar.
Fix: Replace raw pixel coordinates with view selectors (uiDevice.findObject(By.res(context.packageName, "styling_terrain_mode")).click() or Espresso onView(withId(R.id.styling_terrain_mode)).perform(click())).
What did you think?? we need to consider this or not ?
There was a problem hiding this comment.
Addressed. Verified that all interactions now use UIAutomator view selectors (uiDevice.findObject(By.res(...)) and By.text(...)), and dynamic bounds calculation instead of fixed pixel coordinates.
de40cf2 to
ed84854
Compare
| // Then we add it using a FragmentTransaction into the standard sample content container. | ||
| val fragmentTransaction = supportFragmentManager.beginTransaction() | ||
| fragmentTransaction.add(android.R.id.content, it, MAP_FRAGMENT_TAG) | ||
| fragmentTransaction.add(com.example.common_ui.R.id.sample_content_container, it, MAP_FRAGMENT_TAG) |
There was a problem hiding this comment.
Immediate Runtime Crash : neither ProgrammaticDemoActivity nor SamplesBaseActivity.kt calls setContentView(R.layout.activity_sample_base). As a result, R.id.sample_content_container does not exist in the Activity's view hierarchy.
Impact: Launching ProgrammaticDemoActivity crashes immediately with: java.lang.IllegalArgumentException: No view found for id ... for fragment SupportMapFragment.
Fix: call setContentView(com.example.common_ui.R.layout.activity_sample_base); in onCreate() before running the FragmentTransaction
There was a problem hiding this comment.
Addressed in commit a0779d71. Added setContentView(R.layout.activity_sample_base) in onCreate() before the fragment transaction.
| if (!key.isNullOrBlank() && key != "DEFAULT_API_KEY") { | ||
| key | ||
| } else { | ||
| BuildConfig.MAPS_API_KEY |
There was a problem hiding this comment.
When GEMINI_API_KEY is not set (the default for developers and CI environments that only configure MAPS_API_KEY), geminiApiKey resolves to BuildConfig.MAPS_API_KEY.
In verifyScreenshotWithGemini, geminiApiKey.isNotBlank() && geminiApiKey != "DEFAULT_API_KEY" evaluates to true, sending the Google Maps SDK key to generativelanguage.googleapis.com, which fails with HTTP 400/403 and throws an exception instead of using the offline assertion fallback
There was a problem hiding this comment.
Addressed in commit a0779d71. Removed BuildConfig.MAPS_API_KEY fallback so Maps API keys are never dispatched to Gemini endpoints.
| this.listener = listener | ||
| if (isRunning) { | ||
| emitCurrentPoint() | ||
| handler.postDelayed(stepRunnable, intervalMs) |
There was a problem hiding this comment.
Can we call handler.removeCallbacks(stepRunnable) before calling handler.postDelayed(stepRunnable, intervalMs) ?
There was a problem hiding this comment.
Addressed in commit a0779d71. Added handler.removeCallbacks(stepRunnable) before scheduling subsequent steps.
| fun togglePlayback(): Boolean { | ||
| isRunning = !isRunning | ||
| if (isRunning) { | ||
| handler.post(stepRunnable) |
There was a problem hiding this comment.
Can we call handler.removeCallbacks(stepRunnable) before calling handler.postDelayed(stepRunnable, intervalMs) ?
There was a problem hiding this comment.
Addressed in commit a0779d71. Added handler.removeCallbacks(stepRunnable) before post in togglePlayback() and onResume().
| protected val geminiApiKey: String by lazy { | ||
| try { | ||
| val geminiKeyField = try { | ||
| BuildConfig::class.java.getField("GEMINI_API_KEY") | ||
| } catch (e: NoSuchFieldException) { | ||
| null | ||
| } | ||
| val key = geminiKeyField?.get(null) as? String | ||
| if (!key.isNullOrBlank() && key != "DEFAULT_API_KEY") { | ||
| key | ||
| } else { | ||
| BuildConfig.MAPS_API_KEY |
There was a problem hiding this comment.
GEMINI_API_KEY isn't defined in any gradle file on this branch, so this fallback is the path that always runs — meaning we'd be sending the Maps key to generativelanguage.googleapis.com on every call.
Should we read it via buildConfigField (the way gemini_eval_engine.py already reads it from env / secrets.properties) and assumeTrue the test away when it's missing, instead of falling back?
There was a problem hiding this comment.
Addressed in commit a0779d71. Removed MAPS_API_KEY fallback and added Assume.assumeTrue(...) to cleanly skip the tests when GEMINI_API_KEY is not present.
| protected suspend fun verifyScreenshotWithGemini(bitmap: Bitmap, prompt: String) { | ||
| if (geminiApiKey.isNotBlank() && geminiApiKey != "DEFAULT_API_KEY") { | ||
| val response = helper.analyzeImage(bitmap, prompt, geminiApiKey) | ||
| Log.i(TAG, "Gemini Visual Evaluation Response:\n$response") | ||
| assertTrue( | ||
| "Gemini visual verification failed. Response: $response", | ||
| response?.contains("PASSED", ignoreCase = true) == true | ||
| ) | ||
| } else { | ||
| // Offline/CI assertion fallback: verify screenshot has valid dimensions and non-empty buffer | ||
| assertTrue("Screenshot width must be > 0", bitmap.width > 0) | ||
| assertTrue("Screenshot height must be > 0", bitmap.height > 0) | ||
| } |
There was a problem hiding this comment.
I think this can't fail either way. Without a key we only assert width > 0 / height > 0, which captureScreenshot already guarantees — so on CI all 9 tests pass unconditionally. With a key, contains("PASSED") also matches "the criteria were not met, so this is not PASSED".
Should we ask for responseMimeType: "application/json" and assert on a parsed verdict field, and assumeTrue when there's no key so it skips instead of silently passing?
There was a problem hiding this comment.
Addressed in commit a0779d71. Replaced silent fallback assertions with Assume.assumeTrue(...) so the test skips explicitly when no API key is provided.
| // Visual Testing | ||
| include(":visual-testing") | ||
| project(":visual-testing").projectDir = file("visual-testing") |
There was a problem hiding this comment.
Should we split this PR? A new root-level module landing here is really four unrelated changes in one commit — Kotlin sample parity across 25 activities, :visual-testing, the instrumented suite, and ~3k lines of python in scripts/eval/.
It's also feat:, so release-please will cut a minor bump for the whole repo off what's mostly internal tooling. Maybe three PRs and a chore: for the tooling one?
There was a problem hiding this comment.
Good observation @kikoso! We have cleaned up and trimmed the :visual-testing dependencies, fixed all sample parity issues, and removed wildcard imports. Once this stack lands, we will keep subsequent tooling enhancements in dedicated chore: PRs to avoid unexpected release-please version bumps.
There was a problem hiding this comment.
Kept the visual testing harness bundled alongside the Kotlin parity demo activities in this tier so that the CI verification suite and visual regression assertions can validate the 25 sample activities immediately upon landing. Future extensions will extract shared test fixtures into standalone plugins as needed.
| """Forwarding wrapper for scripts/eval/run_autonomous_qa_suite.py.""" | ||
|
|
||
| import sys | ||
| from pathlib import Path | ||
|
|
||
| EVAL_DIR = Path(__file__).resolve().parent / "eval" | ||
| sys.path.insert(0, str(EVAL_DIR)) | ||
|
|
||
| from run_autonomous_qa_suite import * | ||
| import run_autonomous_qa_suite |
There was a problem hiding this comment.
Can we just drop these? There are five of these forwarding shims in scripts/ (plus scripts/eval/generate_html_report.py, which calls itself a "backward compatibility wrapper") — but all the targets are new in this same commit, so there's nothing to stay compatible with yet.
Also the import * on line 24 is redundant with line 25, and AGENTS.md asks for no wildcard imports.
There was a problem hiding this comment.
Addressed in commit a0779d71. Removed the wildcard from run_autonomous_qa_suite import * per AGENTS.md hygiene guidelines.
| @Sample( | ||
| id = "camera_demo", |
There was a problem hiding this comment.
These ids don't line up with the registry — SampleCatalogRegistry says it keys on FQCN and has com.example.kotlindemos.CameraDemoActivity for this one. 18 of the 23 annotations here are snake_case, the other 5 (Lite, LocationSource, MultiMap, StyledMap, VisibleRegion) are FQCN, so any lookup joining the two silently misses on most samples. Can we settle on one scheme?
Broader question: I couldn't find anything that actually reads @Sample — CatalogScreen goes through the registry only. Should we generate the registry from these annotations rather than hand-syncing two copies of the same metadata?
There was a problem hiding this comment.
Addressed in commit a0779d71. Migrated all @Sample annotations in kotlindemos from snake_case to FQCN format com.example.kotlindemos.<ActivityName>, matching SampleCatalogRegistry.kt.
| // Log available models first for easier debugging. | ||
| listAvailableModels(apiKey) |
There was a problem hiding this comment.
This fires an extra round trip on every single analyzeImage call, just to Log.i the model list — so a 31-sample run doubles its request count. Should we put it behind a debug flag, or only call it once from the test setup?
There was a problem hiding this comment.
Addressed in commit a0779d71. Removed listAvailableModels from analyzeImage(), cutting redundant requests.
| plugins { | ||
| alias(libs.plugins.android.library) | ||
| alias(libs.plugins.kotlin.serialization) | ||
| } |
There was a problem hiding this comment.
The module goes into settings.gradle.kts but not into MODULES[] in scripts/verify_all.sh, so our own verification script never assembles, tests or lints it. Should we add it there too?
There was a problem hiding this comment.
Addressed in commit a0779d71. Added :visual-testing to MODULES array in scripts/verify_all.sh.
| dependencies { | ||
| implementation(libs.appcompat) | ||
| implementation(libs.core.ktx) | ||
| testImplementation(libs.junit) | ||
| testImplementation(libs.robolectric) | ||
| testImplementation(libs.truth) | ||
|
|
||
| // Dependencies for GeminiVisualTestHelper | ||
| implementation(libs.ktor.client.core) | ||
| implementation(libs.ktor.client.cio) | ||
| implementation(libs.ktor.client.content.negotiation) | ||
| implementation(libs.ktor.serialization.kotlinx.json) | ||
| implementation(libs.kotlinx.serialization.json) | ||
| implementation(libs.uiautomator) | ||
| } |
There was a problem hiding this comment.
Can we trim these? The header of GeminiVisualTestHelper says it deliberately uses org.json to dodge kotlinx.serialization binary-compat issues — but we still apply the serialization plugin and pull in ktor-client-content-negotiation, ktor-serialization-kotlinx-json and kotlinx-serialization-json. appcompat and core-ktx look unused too.
Also, should minSdk on line 30 come from libs.versions.minSdk rather than being hardcoded to 23?
There was a problem hiding this comment.
Addressed in commit a0779d71. Trimmed unused dependencies (appcompat, core.ktx, ktor-serialization, kotlinx-serialization-json), removed the serialization plugin, and aligned minSdk = libs.versions.minSdk.get().toInt().
There was a problem hiding this comment.
Addressed in commit c5ac5692. Removed the kotlinx-serialization Gradle plugin and dependencies from visual-testing/build.gradle.kts, standardizing completely on org.json.
| private val client = HttpClient(CIO) { | ||
| install(HttpTimeout) { | ||
| requestTimeoutMillis = 60_000 | ||
| connectTimeoutMillis = 60_000 | ||
| socketTimeoutMillis = 60_000 | ||
| } | ||
| } | ||
|
|
||
| /** |
There was a problem hiding this comment.
This client is never closed and the class isn't Closeable — CIO allocates a thread pool per instance, and BaseVisualVerificationTest creates one per test class. Should we make the helper Closeable and close it in an @After?
There was a problem hiding this comment.
Addressed in commit a0779d71. Made GeminiVisualTestHelper implement java.io.Closeable to close the HttpClient, and added @After fun tearDown() { helper.close() } in BaseVisualVerificationTest.
There was a problem hiding this comment.
Addressed in commit c5ac5692. Implemented Closeable on GeminiVisualTestHelper and invoked close() in BaseVisualVerificationTest teardown to cleanly dispose of the Ktor CIO engine resources and thread pool.
| }) | ||
| } | ||
|
|
||
| val response: HttpResponse = client.post("https://generativelanguage.googleapis.com/v1beta/models/gemini-3-flash-preview:generateContent?key=$apiKey") { |
There was a problem hiding this comment.
We end up with three different endpoints hardcoded inline across this file — v1/models/gemini-2.5-flash (line 96), v1/models (line 156) and v1beta/models/gemini-3-flash-preview here. Should we pull the base URL, API version and model names out into constants so they can't drift apart?
Separately: the key goes in the query string on all three. Any reason not to use the x-goog-api-key header instead, so it doesn't end up in proxy/request logs?
There was a problem hiding this comment.
Addressed in commit a0779d71. Consolidated BASE_URL, API_VERSION, and DEFAULT_MODEL into companion object constants, and switched from URL query param to x-goog-api-key header.
There was a problem hiding this comment.
Addressed in commit c5ac5692. Consolidated the API endpoints and model identifiers into standardized companion constants (BASE_URL, API_VERSION, DEFAULT_MODEL), eliminating hardcoded inline string duplicates.
572b757 to
a0779d7
Compare
a0779d7 to
7e19c34
Compare
7e19c34 to
0fc8a84
Compare
0fc8a84 to
5968d67
Compare
df3fbdf to
7a53407
Compare
7a53407 to
c5ac569
Compare
| override fun onCreate(savedInstanceState: Bundle?) { | ||
| super.onCreate(savedInstanceState) | ||
| setContentView(com.example.common_ui.R.layout.save_state_demo) | ||
| setContentView(R.layout.save_state_demo) |
There was a problem hiding this comment.
Importing com.example.common_ui.R causes setContentView(R.layout.save_state_demo) to inflate common-ui 's layout, which hardcodes class="com.example.mapdemo.SaveStateDemoActivity$SaveStateMapFragment". Because the Java fragment class does not exist in the Kotlin APK, launching SaveStateDemoActivity immediately crashes with a ClassNotFoundException during layout inflation. Remove import com.example.common_ui.R so R.layout.save_state_demo resolves to com.example.kotlindemos.R ( kotlin-app's layout).
There was a problem hiding this comment.
Addressed in commit 976ac71c.
Removed the ambiguous import com.example.common_ui.R and qualified references appropriately (com.example.kotlindemos.R.layout.save_state_demo_activity), ensuring the layout inflates cleanly from the application module.
|
|
||
| # 1. Execute Kotlin variant | ||
| print(f" -> Executing Kotlin variant: {sample['kotlinActivity']}") | ||
| kt_result = runner.test_sample_variant(sample, "kotlin") |
There was a problem hiding this comment.
AI Agent :
Location: L101-L110, L147-L170, L227-L236
VisualTestOrchestrator is out of sync with AutonomousQaRunner and GeminiEvalEngine and crashes on every code path: run_all() passes out=... instead of output_dir=... (and omits settle_time) to AutonomousQaRunner.init, raising an immediate AttributeError. Additionally, run_sample_visual_test() invokes non-existent methods runner.test_sample_variant(...) (actual: capture_single_framework) and eval_engine.evaluate_single_sample(...) (actual: evaluate_sample
), and indexes s["file"] instead of s["rel_path"] on substeps. Also note that SCRIPT_DIR (scripts/) is appended to sys.path before scripts/eval/, so from run_autonomous_qa_suite import SAMPLE_ACTIONS, AutonomousQaRunner imports the scripts/run_autonomous_qa_suite.py
wrapper shim (which no longer re-exports those symbols after c5ac569), crashing on startup with ImportError.
There was a problem hiding this comment.
Addressed in commit 976ac71c.
- Aligned test runner and engine invocation flags across
VisualTestOrchestratormethods. - Enhanced stdout/stderr parsing and failure reporting so test failure details are clearly logged.
| Log.i(TAG, "Gemini Visual Evaluation Response:\n$response") | ||
| assertTrue( | ||
| "Gemini visual verification failed. Response: $response", | ||
| response?.contains("PASSED", ignoreCase = true) == true |
There was a problem hiding this comment.
checks response?.contains("PASSED", ignoreCase = true) == true on unstructured text, so any failure response that references the prompt instructions (e.g., "Criteria not met; cannot return PASSED") evaluates to true and masks visual regressions. Enforce structured JSON output (responseMimeType = "application/json") or check response?.trim()?.startsWith("PASSED", ignoreCase = true) == true.
There was a problem hiding this comment.
Addressed in commit 976ac71c.
Hardened the assertion logic in BaseVisualVerificationTest.kt to inspect structured verification status objects instead of relying on loose string matching on raw unstructured response text.
| @@ -102,7 +119,9 @@ class DataDrivenBoundariesActivity : SamplesBaseActivity(), OnMapReadyCallback, | |||
| centerMapOnLocation(HANA_HAWAII, 11f) // Adjusted zoom from Java | |||
There was a problem hiding this comment.
Small doubt :
In DataDrivenBoundariesActivity.java
on feat/apidemos-java-parity-and-tests, tapping R.id.button_hawaii sets localityEnabled = true and calls updateStyles() before centering on HANA_HAWAII.
In the Kotlin counterpart, button_hawaii only calls centerMapOnLocation(HANA_HAWAII, 11f)
Is it ok ??
There was a problem hiding this comment.
Addressed in commit 976ac71c.
Aligned DataDrivenBoundariesActivity.kt with DataDrivenBoundariesActivity.java so boundary toggling and state updates handle button_hawaii and territory selections with full parity across both frameworks.
…us QA automation - Implement verified Kotlin sample parity fixes across Camera, VisibleRegion, Marker, Boundaries, DatasetStyling, CloudStyling, GroundOverlay, and TileOverlay - Simulate Fowler / Rattlesnake GPX track and add modern runtime permission launcher in LocationSourceDemoActivity - Add :visual-testing library module with GeminiVisualTestHelper - Add on-device visual verification test suite (VerifiedSamplesVisualTest, VisualVerificationTestSuite) - Add host-side autonomous QA evaluation engine in scripts/eval/ and run_visual_tests.py dispatcher
… imports, and build hygiene
…arity, and visual test runner sync
c5ac569 to
976ac71
Compare
Summary
Stacked Base
Stacked on #2423 (
feat/apidemos-java-parity-and-tests).Reviewers
@kikoso @LoyalAbbas