The Tamarin acceptance tests can be used to verify your installation and or local changes that you have made to the source. To run acceptance tests on the android shell see the "Testing the Android Shell" heading below. Running the Tamarin test suite requires the following steps:
- Set the
AVMenvironment variable to the path of the AVMPlus executable (avmshell). Note: the name of theavmshellexecutable varies among the build systems; use the name of the AVM shell produced from the build process you used, above. - Set the
BUILTINABCenvironment variable to the path of thegenerated/builtin.abc. - You may build the .abc files build by downloading the latest asc.jar. The latest asc.jar can be downloaded from from ftp://ftp.mozilla.org/pub/js/tamarin...latest/asc.jar. Also the source to asc.jar may be downloaded and rebuilt from http://opensource.adobe.com/wiki/dis...exsdk/Flex+SDK.
- Set the
ASCenvironment variable, as described above. - Set the
SHELLABCenvironment variable to the path of thegenerated/shell_toplevel.abc.
$ cd tamarin-redux/test/acceptance $ export ASC=/Users/build/hg/tamarin-redux/utils/asc.jar $ export BUILTINABC=/Users/build/hg/tamarin-redux/generated/builtin.abc $ export SHELLABC=/Users/build/hg/tamarin-redux/generated/shell_toplevel.abc $ export AVM=/Users/build/hg/tamarin-redux/objdir-release/shell/avmshell $ python runtests.py Tamarin tests started: 2010-09-28 10:37:06.410676 current configuration: x64-mac-tvm-release avm version: 5260:6d1899261bac Executing 2532 tests against vm: /Users/build/hg/builds/5260-6d1899261bac/mac/avmshell_64 2532 running abcasm/abs_helper.as skipping... reason: 2531 running abcasm/adhoc.abs skipping... reason: inconsistencies in different debug output, need to implement regex matching of diffs 2527 running abcasm/branchToCommon.abs 2530 running abcasm/arithmetic.abs 2529 running abcasm/bkpt.abs 2528 running abcasm/bkptline.abs 2526 running abcasm/bug_476556.abs 2524 running abcasm/bug_491056.abs ... test run PASSED! Tests complete at 2010-09-28 10:39:26.751797 Start Date: 2010-09-28 10:38:07.221457 End Date : 2010-09-28 10:39:26.751797 Test Time : 0:01:19.530340 passes : 59291 failures : 0 expected failures : 75 tests skipped : 76
The -f flag tells runtests.py to force recompilation of all the scripts. For help on runtests.py options, see below or use python runtests.py -h.
Notes on using acceptance runtests.py
Threading
In order to speed up the execution time of the acceptance testsuite, the runtests script automatically detects the number of cores on the machine and sets an equivalent number of threads. This can be overrided with the threads flag:
python ./runtests.py --threads=3
Threading on cygwin / windows
Due to an issue with cygwin python, threading does not work, and threads will always be set to 1. In order to use threading on windows you must install windows python and invoke runtests using that python executable in cygwin. The locations of AVM, ASC, BUILTINABC and SHELLABC must also be defined using windows paths (replace backslashes [\] with forward slashes [/]):
export AVM=c:/location/of/avmshell.exe
export ASC=c:/location/of/asc.jar
...
c:/Python26/python.exe ./runtests.py --threads=3
Other flags of interest
(For a list of all flags, run runtests.py with -h/--help)
--nohtml : will suppress the creation of an html output file
--rebuildtests : will only rebuild test .abc's, will not attempt to run them. Can also be using in conjunction with --threads
-f --forcerebuild force rebuild all test files
-q --quiet : will print a period (.) for each test run regardless of results, then print the test run summary when finished
--ascargs= : additional arguments to pass to asc.jar
--vmargs= : additional arguments to pass to avmshell
--random : run tests in random order
--timeout : max time for testrun
--verify : run a verify pass instead of running abcs
--verifyonly : run a -Dverifyonly pass: only checks test exitcode (only works with debugger vms)
Testing the Android Shell
The instructions above apply to running tests on an android device as well, but with a few differences. You should set environment variables as described above except for AVM, which should be:
$ export AVM=$TAMARIN_BUILD_TOP/platform/android/android_shell.py
where $TAMARIN_BUILD_TOP is your main Tamarin repo folder. Note that AVM needs to point to android_shell.py which is a proxy for running the tests on the phone. The /android-public/android-vars.sh script has commented-out examples of the environment settings for running tests on the android avm shell. To recap the other environment settings:
- Set the BUILTINABC environment variable to the path of the generated/builtin.abc.
- You may build the .abc files build by downloading the latest asc.jar. The latest asc.jar can be downloaded from from ftp://ftp.mozilla.org/pub/js/tamarin...latest/asc.jar. Also the source to asc.jar may be downloaded and rebuilt from http://opensource.adobe.com/wiki/display/flexsdk/Flex+SDK.
- Set the ASC environment variable, as described above.
- Set the SHELLABC environment variable to the path of the generated/shell_toplevel.abc.
adb
'adb' stands for Android Debug Bridge. adb is used to access your android device and can run both direct and shell-type commands. If you've correctly edited and run the android-vars.sh script mentioned on the Tamarin Build Documentation page you should already be pathed to the adb executable in the public sdk/ndk, but if not it exists at /android-public/android-sdk-mac_86/platform-tools.
adb supports ssh-like commands to an android device. adb automatically connects to a phone if connected by usb. adb shell "ls /" is a good test to see if connected. The command:
$ adb devices
lists the device id of each phone. If you have more than 1 phone connected to usb, use "adb -s " for each phone. If this fails, do you have a dev OS installed? Under Settings->Applications->Development, is USB Debugging enabled?
Testsuite Setup
Example commands:
Copy avmshell to /data/local/tamarin. The shell can be named variously in the build but on the phone the executable must be 'avmshell'
$ adb shell 'mkdir /data/local/tamarin' $ adb push avmshell /data/local/tamarin/avmshell $ adb shell 'chmod 777 /data/local/tamarin/avmshell'
Copy android_runner.sh, if it doesn't already exist on the phone in /data/local/tamarin
$ adb push tamarin-redux/platform/android/android_runner.sh /data/local/tamarin/android_runner.sh $ adb shell 'chmod 777 /data/loca/android_runner.sh'
Test it out with a simple .abc or no args for usage (should return EXITCODE=0)
$ tamarin-redux/platform/android/android_shell.py hello.abc hello EXITCODE=0
Test it out by retrieving the version information of the shell on the android device
$ $AVM -Dversion shell 1.4 debug build 6299:455bca954565 features AVMSYSTEM_32BIT;AVMSYSTEM_UNALIGNED_INT_ACCESS;AVMSYSTEM_LITTLE_ENDIAN;AVMSYSTEM_ARM;AVMSYSTEM_UNIX; AVMFEATURE_JIT;AVMFEATURE_ABC_INTERP;AVMFEATURE_SELFTEST;AVMFEATURE_EVAL;AVMFEATURE_PROTECT_JITMEM; AVMFEATURE_SHARED_GCHEAP;AVMFEATURE_CACHE_GQCN;AVMFEATURE_SAFEPOINTS;AVMFEATURE_SWF12;AVMFEATURE_SWF13;AVMTWEAK_EXACT_TRACING;
Running Acceptance Tests
--androidthreads sets the number of threads to the number of phones found via usb. Running with 2 or more phones uses threads to run tests in parallel and reduces running times dramatically. You must, in any case, at least use --androidthreads or --threads=1 (or will fail due to multiple tests running).
$ cd tamarin-redux/test/acceptance $ ./runtests.py --androidthreads
Note on using an emulator: the emulator does seem to support hardware floating point but gives a BusError when running avmshell -Darm_vfp. It's unreliable, losing its adb connection frequently. An emulator is ok for testing a few abc files but for any consistency a real device is better.