8

How do I run flutter integration tests (flutter_driver) non locally within an infrastructure like AWS Device Farm, Saucelabs, Firebase Test Lab?

Running flutter_driver tests on my local machine works smoothly and easily however cloud based mobile testing services don't have the required dependencies installed to utilize Flutter CLI commands.

There are a few places in the Flutter repo that indicate this is possible, however I'm having a hard time finding exactly what needs to happen to make this possible (I have limited experience with Android & iOS integration testing).

A few references I've found:
A comment discussing an integration test in the flutter_gallery: https://github.com/flutter/flutter/issues/18879#issuecomment-400816050

A github issue to add a Firebase Test Lab integration test: https://github.com/flutter/flutter/issues/11718

What do I need to add to my flutter app to allow for remote integration testing inside AWS Device Farm, Saucelabs, Firebase Test Lab etc?

Thanks for your help.

Improve this question
3
  • 1
    Did you see this flutter.io/docs/testing#continuous-integration-and-testing? Commented Nov 14, 2018 at 21:24
  • It looks like someone else has attempted to do this here. I have not tested this repo yet, however, it looks somewhat promising: github.com/mmcc007/sylph Commented May 21, 2019 at 23:32
  • @AlexFallenstedt this lib is actually the exact implementation of the top answer just below. It is mentioned in the source code Commented Apr 9, 2020 at 22:28

2 Answers 2

Reset to default
5

Although we don't officially support the Flutter framework and Flutter tests on AWS Device Farm, we do have a way for you to run these tests using our "custom environment mode" and by pre-selecting a different, supported framework. My advice would be to do the following:

  • I have a very minimal dummy test for Appium Python, a supported test framework located at https://s3-us-west-2.amazonaws.com/aws-devicefarm-support/test_bundle_slim.zip. Download this from there, and, using the command line tool zip, add your local tests to this zip file, with a command such as "zip -r ~/Downloads/test_bundle_slim.zip flutter-tests/"
  • Go to AWS Device Farm, setup an automation test run, upload your app, then get to the screen where you select your test type. Choose Appium Python, upload your zip file, and AWS Device Farm will accept the test because of the original dummy Appium Python files we've left in the zip file.
  • Select "customize your test environment" below there, and a new window below with a YAML test spec file will appear. Put the following lines in the pre-test section of your test spec in place of the existing code used to start an Appium server:

{code}

  - >-
    if [ $DEVICEFARM_DEVICE_PLATFORM_NAME = "Android" ];
    then
        # Run EC2 setup code here
        curl https://storage.googleapis.com/flutter_infra/releases/beta/linux/flutter_linux_v0.11.3-beta.tar.xz -o flutter_linux_v0.11.3-beta.tar.xz
        tar xf flutter_linux_v0.11.3-beta.tar.xz
    fi

    if [ $DEVICEFARM_DEVICE_PLATFORM_NAME = "iOS" ];
    then
        # Run Mac setup code here
        curl https://storage.googleapis.com/flutter_infra/releases/beta/macos/flutter_macos_v0.11.3-beta.zip -o flutter_macos_v0.11.3-beta.zip
        unzip -qq flutter_macos_v0.11.3-beta.zip
    fi

  - mv flutter ~/flutter
  - echo "export PATH=$PATH:$HOME/flutter/bin">> ~/.bash_profile
  - export PATH=$PATH:$HOME/flutter/bin
  - flutter devices

{code}

  • Likewise, go to the test section of the test spec file and remove the python test command "bin/py.test" and replace it with your Flutter test command, such as "flutter drive --target=./flutter-tests/main.dart
  • Save and close this test spec file, and finish setting up the automation run by selecting your devices and setting up their state.

Good luck testing on AWS Device Farm! I have confirmed that these steps do work to setup Flutter properly on our testing platform.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

Anurag Goyal, Flutter integration tests require the standard adb shipped with the Android SDK. Device Farm over-writes the standard adb with a custom adb from Amazon which disables port forwarding, a feature required for flutter testing. AFAIK, the only way to get flutter integration tests to run is to use a private Device Farm which is much more expensive than the shared Device Farm. Is this correct?
Disregard last message. I was able to get a flutter integration test running on the shared Device Farm for android
Have you tried testing with iOS? I am not having any luck so far... Is there support available from AWS Device Farm?
2

Note: this addresses the assumptions that lead to your question. Not a direct answer to your question.

It is possible to run Flutter integration tests on the cloud using emulators only (no actual devices). The following article describes how to set it up. The article includes a working example on Travis-CI.

Flutter unit, widget and integration testing with IOS and Android emulators on Travis-CI

Source code on GitHub, with a link to the latest build, including integration test results running on both iOS and Android, can be found here. Feel free to clone and use as a starting point for your own project.

Running integration tests on actual devices is probably something you want to do if you are building a complex app with wide distribution (a lot of users), etc...

Improve this answer

1 Comment

mmcabe - How do you use ios and integration_test plugin with aws device farm? (eg not appium, not sylph, not old test driver framework)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.