Can we "Test" the tests/examples, by running them on GitHub Actions? about testing-samples HOT 27 OPEN

Yes its on the roadmap to use GitHub Actions. FWIW all samples listed in projects.conf are tested regularly locally via the test_all.sh script.

from testing-samples.

Awesome to hear! Thanks for the insight. Looking forward to seeing these on Actions. Samples being run on Actions will also, in a way, help contribute to the stability of emulators on Github's infrastructure.

from testing-samples.

@djbrown I haven't taken a glance at your code, but the stability is likely related to the Github Infrastructure than your code setup. The macOS-12 agents can use the default hypervisor for VM acceleration, but the macOS VMs still don't have hardware acceleration, so emulators tend to have flakey startup and run pretty slow. Also, the ubuntu agents don't offer KVM or a default hypervisor, so those have been incredible slow.

But -- you can check this out because there is another official Android repo which is running instrumentation tests here https://github.com/android/architecture-samples/actions/runs/3461542977/workflow

from testing-samples.

@yuuki3655 It looks like something (GMD/AGP?) is producing a file with a double quote in it, which the uploader treats as invalid

from testing-samples.

That log file is produced by UTP core and the file name needs to be escaped. I filed a bug for tracking. https://issuetracker.google.com/263305468

from testing-samples.

FWIW - github/android/android_test has been using GMD+GitHub actions for a while now and it has been working well.
https://github.com/android/android-test/blob/9f05f8710fb0592b2575e46523010471e6cbf241/.github/workflows/ci.yml#L88

Although it is now using the paid large Linux runners - the same setup on the macos free tier machines had a ~10% flakiness rate when we last tried them.

from testing-samples.

either an OS issue or a case where machines are just underpowered.

I think the 'large runners' group is 4 core but I'm not sure.

from testing-samples.

This repo should have access to the 'large runners' pool. I believe I requested it at the same time as access for github/android/android_test.

from testing-samples.

@JoseAlcerreca for #450 (comment)

from testing-samples.

I tried to make GMD screenshot test with Github action work https://github.com/hannesa2/testing-samples/pulls but till now, I found no working action. Eg. hannesa2#1 with untouched code and local working gradle command

I run into

Execution failed for task ':app:nexusOneApi30Setup'.
> A failure occurred while executing com.android.build.gradle.internal.tasks.ManagedDeviceSetupTask$ManagedDeviceSetupRunnable
   > java.lang.IllegalStateException: Gradle was not able to complete device setup for: dev30_aosp_atd_x86_Nexus_One
     This could be due to having insufficient resources to provision the number of
     devices requested. Try running the test again and request fewer devices or
     fewer shards.

I've no clue why

from testing-samples.

@hannesa2 where ist the gradle task nexusOneApi30Setup or the device dev30_aosp_atd_x86_Nexus_One defined?
I couldn't find any related code in the repos (origin or your fork).

Over at GnuCash Pocket I use gradle-build-action and android-emulator-runner which works quite well... BUT I cannot get the instrumentation tests to behave deterministically..

from testing-samples.

If I have time this week, I'll submit a PR to run all of the samples in a build matrix.

from testing-samples.

@djbrown All my pull requests using this definition

from testing-samples.

run pretty slow

@mrk-han
No, all macOS in Github action have hardware acceleration installed. Ok, old slow x86, but it works. Ubuntu doesn't.
With Apple Silicon M1 I currently failed Malinskiy/action-android#70 to make it work.

In a lot of other repos I do Espresso tests, eg this or here and they work more or less stable. Only api30 devices tend to flakiness

from testing-samples.

@hannesa2 They are passing VM acceleration check because of the new default hypervisor framework that is now leveraged by the Emulator. If I am not mistake, this is not the same as GPU acceleration. (Hardware Acceleration has 2 facets, GPU and VM acceleration)

And yes, M1 is not supported yet unless there you configure a self-hosted M1 laptop, which I believe is still somewhat experimental.

from testing-samples.

Also, accel-check fails depending on where you run it from, the test done here: ReactiveCircus/android-emulator-runner#286 (comment)

New tooling ($ANDROID_HOME/emulator/emulator -accel-check) passes, but old tooling $ANDROID_HOME/platform-tools/emulator -accel-check) fails. Which, I assume, is because the newer emulator architecture leverages the default hypervisor provided by macOS.

from testing-samples.

Just put up a test PR on my branch here, likely will need more optimization. It's already failing because I accidentally removed the sed command 😆 But you can check out what I'm trying to do here: https://github.com/mrk-han/testing-samples/actions/runs/3466821782/jobs/5791077529

I'm wondering a few things:

@brettchabot What is the easiest way to add testOptions to our emulator after creating it with GMD? Is there a way to provide emulator options in the managedDevices{} block?

It's also somewhat common to change the config.ini of the emulator post-creation like-so:

          echo "y" | $ANDROID_HOME/tools/bin/sdkmanager --install 'system-images;android-31;default;x86_64'
          echo "y" | $ANDROID_HOME/tools/bin/sdkmanager --licenses
          
          # Create emulator
          $ANDROID_HOME/tools/bin/avdmanager create avd -n TestAvd123 -d pixel --package 'system-images;android-31;default;x86_64'
          $ANDROID_HOME/emulator/emulator -list-avds
          if false; then
          emulator_config=~/.android/avd/TestAvd123.avd/config.ini
          # The following madness is to support empty OR populated config.ini files,
          # the state of which is dependant on the version of the emulator used (which we don't control),
          # so let's be defensive to be safe.
          # Replace existing config (NOTE we're on MacOS so sed works differently!)
          sed -i .bak 's/hw.lcd.density=.*/hw.lcd.density=420/' "$emulator_config"
          sed -i .bak 's/hw.lcd.height=.*/hw.lcd.height=1920/' "$emulator_config"
          sed -i .bak 's/hw.lcd.width=.*/hw.lcd.width=1080/' "$emulator_config"
          # Or, add new config
          if ! grep -q "hw.lcd.density" "$emulator_config"; then
            echo "hw.lcd.density=420" >> "$emulator_config"
          fi
          if ! grep -q "hw.lcd.height" "$emulator_config"; then
            echo "hw.lcd.height=1920" >> "$emulator_config"
          fi
          if ! grep -q "hw.lcd.width" "$emulator_config"; then
            echo "hw.lcd.width=1080" >> "$emulator_config"
          fi
          echo "Emulator settings ($emulator_config)"
          cat "$emulator_config"
          fi

It would be nice to support that if possible for GMD with the ATP images.

Lastly, is there a way to specify SKIA rendering from the testOptions? adb shell 'su;setprop debug.hwui.renderer skiagl;stop;start'

Ideally on GitHub Actions we can run with macOS agents to use the default hypervisor, and then run with swiftshader_indirect GPU acceleration (because I don't think GPU Host is possible), and I'd like to experiment with SKIA rendering if that would help in the absence of GPU Acceleration.

from testing-samples.

@hannesa2 They are passing VM acceleration check because of the new default hypervisor framework that is now leveraged by the Emulator. If I am not mistake, this is not the same as GPU acceleration. (Hardware Acceleration has 2 facets, GPU and VM acceleration)

There is no need to put something on top of you stuff.
It was working and it's working. e.g https://github.com/hannesa2/LiveEdgeDetection/actions/runs/3424743076/jobs/5704811567

And yes, M1 is not supported yet unless there you configure a self-hosted M1 laptop, which I believe is still somewhat experimental.

I use it since months gitx/gitx#314 and it's fast.

Please can we skip this now and concentrate to make this GradleManagedDevice work with github actions

from testing-samples.

@hannesa2

There is no need to put something on top of you stuff.

I'm not sure what you mean by this, can you reiterate?

It was working and it's working. e.g

Yes -- Hypervisor works without HAXM because of the latest improvements by the emulator team to take advantage of the new 1st party hypervisor provided by apple https://developer.apple.com/documentation/hypervisor

e.g https://github.com/hannesa2/LiveEdgeDetection/actions/runs/3424743076/jobs/5704811567

Not sure this is a great example of the types of UI tests that will be run in this suite (You are just sleeping and taking a screenshot), but I am glad it has been stable for you!

Please can we skip this now and concentrate to make this GradleManagedDevice work with github actions

That's a lot of what this PR will help show

I use it since months gitx/gitx#314 and it's fast.

You're using M1 runners on MacStadium though, is that free with OSS?

from testing-samples.

Btw, I asked the maintainer of
malinskiy/action-android/emulator-run-cmd and
malinskiy/action-android/install-sdk

Malinskiy/action-android#76 (comment)
When he is right, it's a resource issue. The big question ist, how to measure and how to improve

from testing-samples.

@hannesa2

To make your setup work you need to provide the same config of emulator that is known to work as of now on GH to the GMD

This is literally what I just asked above.

Btw, I asked the maintainer of

Yes, his action has nothing to do with GMD and his opinion is subjective on where this problem should be solved.

When he is right, it's a resource issue. The big question ist, how to measure and how to improve

It is fairly well known that Github's default runners are not very powerful. Google has released the ATD images which will help reduce CPU load. This PR to get these core UI tests running on GitHub Actions is a step in the right direction to see if we can get these stable on Github's default runner infrastructure.

For any of your concerns with GMD stuff, you should create a separate issue. I asked @brettchabot here because the testOptions configuration with GMD will help out (see the example script I posted earlier for using sed to change the config.ini)

GMD is still very new and that is the biggest reason people are shying away from it for now. It requires alpha versions of tooling (8.0.0 alpha) to connect to firebase test lab. But it will be a huge improvement for the ecosystem once polished. Starting and running emulators on CI has been a pain for years.

from testing-samples.

Made a bit more progress, latest build is running here -> https://github.com/mrk-han/testing-samples/actions/runs/3483003666

mrk-han#1

from testing-samples.

@brettchabot Can you look into why these are failing? Is there an easy way around this?

Artifact path is not valid: /ui/espresso/AccessibilitySample/app/build/outputs/androidTest-results/managedDevice/nexusOneApi30/testlog/adb.additional_test_output" ]].23.ok.txt. Contains the following character:  Double quote "

from testing-samples.

@yuuki3655 @brettchabot Hope you are both doing well!

I updated my fork with main and checked again to see if this is still happening. I'm mainly wondering if this is going to cause problems with more projects using GMD/AGP on macOS agents with Github -- or if for some reason it'd be something specific to this repo which is causing the problem.

from testing-samples.

@brettchabot

Ahhh, that's interesting. Probably an OS issue then?

I'm so glad to see it using the Larger Runners. I've had to swap all of my projects to them as well, and the stability has been incredible.

Curious, is that group using the 4-Core Ubuntu runners? I think 2-Core will have KVM access soon which could decrease costs if they're also stable (assuming Google is paying for the Larger Runners)

from testing-samples.

Would this repo also be able to access those Larger Runners at this time, or would we need to have y'all's Github Org Owner add this repo to the list of repos with access to the runners first?

If we can get this on larger runners, I can put up a PR today or tomorrow and get this repo ported over to GHA.

from testing-samples.

I'll try to put up a PR asap then.

It could be beneficial to get Now In Android on the Larger Runners as well.

At the least, it would show a more proper way of getting reliable test results on GHA, and leave users less frustrated with emulators + CI who go to that repo for access to the "latest and greatest" ways of doing things.

from testing-samples.