diff options
Diffstat (limited to 'chromium/docs/website/site/developers/testing/isolated-testing/for-swes/index.md')
| -rw-r--r-- | chromium/docs/website/site/developers/testing/isolated-testing/for-swes/index.md | 224 |
1 files changed, 0 insertions, 224 deletions
diff --git a/chromium/docs/website/site/developers/testing/isolated-testing/for-swes/index.md b/chromium/docs/website/site/developers/testing/isolated-testing/for-swes/index.md deleted file mode 100644 index 06f9e9fe5eb..00000000000 --- a/chromium/docs/website/site/developers/testing/isolated-testing/for-swes/index.md +++ /dev/null @@ -1,224 +0,0 @@ ---- -breadcrumbs: -- - /developers - - For Developers -- - /developers/testing - - Testing and infrastructure -- - /developers/testing/isolated-testing - - Isolated Testing -page_name: for-swes -title: Isolated Testing for SWEs ---- - -[TOC] - -## Overview - -**Also See:** -[//docs/workflow/debugging-with-swarming.md](https://chromium.googlesource.com/chromium/src/+/HEAD/docs/workflow/debugging-with-swarming.md) - -This page explains how to convert a googletest executable into a fully isolated -test. This is done by describing *runtime dependencies* for a test. - -For information about the infrastructure itself and the roadmap, see [Isolated -Testing Infrastructure](/developers/testing/isolated-testing/infrastructure). - -Note: If you're a Googler who needs to perform a manual test on a platform for -which you cannot readily build (e.g. Windows Developer with no development -environment on your Mac book), follow [these instructions to get the build from -a try run](https://goto.google.com/runtrybuiltbits). - -### What's "Isolate"? - -#### **Goal: describe the exact list of files needed to run a executable.** - -The isolate project is all the file formats, tools and server code to make this -work **fast** and efficiently. Every test executable list the data files it -depends on, which allows the testers bots to only download the files they need -instead of the whole Chromium source tree. - -They are described in [BUILD.gn files](/developers/gn-build-configuration). - -### What's "Swarming"? - -**Goal: distribute tasks fast and efficiently in an heterogeneous fleet of -bots.** - -The swarming project is all the tools, server and bot code to run a step (like a -unit test) on a remote bot and get results back. It has native google-test -sharding capability. - -### What are the advantages? - -By reducing the amount of data that we need to transfer to the tester machines, -it becomes much easier to increase the number of tester bots that we have using -Swarming (part of [LUCI](https://github.com/luci/)). - -## Adding a new test - -* Make sure the test exist in a BUILD.gn file. -* Add missing items to the "data" section until the test runs locally - isolated via mb.py run. -* Add the test to the corresponding json file in //testing/buildbot/. -* All the required data files, executables started by the test - executable\* required by the test to succeed are listed as a - dependency. - * For example, `browser_tests.exe` needs `chrome.exe`, which needs - `resources.pak`, etc. - -Expectations of the tests: - -* Must **not** do directory walking or otherwise try to guess what - should be tested. -* Must **not** edit any input file. -* Must **not** write at all in the current directory. It must only use - the temporary directory for temporary files. - -## HowTos - -### Download binaries from a try job - -* Visit the try job build, e.g. from <https://ci.chromium.org/>. -* Search for \[trigger\] my_test and click Swarming task "shard #0" - link -* Scroll down a bit, follow the instructions in the **Reproducing this - Task Locally** section - -### Run a test isolated locally - -```none -echo gn > out/Release/mb_type # Must be done once to avoid mb.py from performing a clobber -tools/mb/mb.py run //out/Release base_unittests # Builds, generates .isolate (via gn desc runtime_deps), and runs (via "isolate.py run") -``` - -### Run a test built locally on Swarming - -#### 1. Eligibility and login - -Right now, only users Chromium team members can use the infrastructure directly. -Note that the whole [Swarming infrastructure is open -source](https://github.com/luci/luci-py) so if any other company would help to -recreate the same infrastructure internally, send us a note at -[infra-dev@chromium.org](https://groups.google.com/a/chromium.org/g/infra-dev) - -By login first, you have access tokens so that the following commands do not -have to constantly prompt for your identity. You only need to do this once: - -```none -python tools/swarming_client/auth.py login --service=https://isolateserver.appspot.com -python tools/swarming_client/auth.py login --service=https://chromium-swarm.appspot.com -``` - -If you are running through a text only session on a remote machine, append -argument `--auth-no-local-webserver` - -#### 2. Build & Generate .isolate file - -The isolate file describes what are the files that needs to be mapped on the -Swarming bot. It is generated via GN "data" and "data_deps" statements, and is -processed by mb.py: - -```none -ninja -C out/Release base_unittests -echo gn > out/Release/mb_type # Must be done once to avoid mb.py from performing a clobber -python tools/mb/mb.py isolate //out/Release base_unittests # Creates out/Release/base_unittests.isolate -``` - -#### 3. Compute .isolated file and upload it - -The isolated file contains the SHA-1 of each input files. It is archives along -all the inputs to the Isolate server. Since the isolate server is a -content-addressed cache, only the ones missing from the cache are uploaded. -Depending on your connection speed and the size of the executable, it may take -up to a minute: - -```none -python tools/swarming_client/isolate.py archive \ - -I https://isolateserver.appspot.com \ - -i out/Debug/base_unittests.isolate \ - -s out/Debug/base_unittests.isolated -``` - -#### 4. Trigger the task - -That's where a task is requested, specifying the isolated (tree of SHA-1 of -files to map in): - -```none -python tools/swarming_client/swarming.py trigger \ - -S chromium-swarm.appspot.com \ - -I isolateserver.appspot.com \ - -d os Android \ - -d pool Chrome \ - -d gpu none \ - -s <hash printed at step 3> -``` - -Wait for results. OS currently available: - -* Windows-7-SP1 (64 bits) -* Windows-8.1-SP0 -* Windows-2008ServerR2-SP1 (64 bits) -* Windows-10-15063 (64 bits) -* Ubuntu-14.04 and Ubuntu-16.04 (64 bits) -* Mac-10.9 to Mac-10.13 - -For other available --dimension values, look at Swarming bots e.g.: -<https://chromium-swarm.appspot.com/botlist>. - -That's it. Feel free to contact the team at -[infra-dev@chromium.org](https://groups.google.com/a/chromium.org/g/infra-dev) -for any chromium open source specific questions. - -**Notes:** - -* `-d pool Chrome` is needed! -* -d cpu x86-64 is useful to prevent your 64 bit build from being run - on 32-bit Windows trybots. - -### Additional Notes - -* Running an executable on a swarming bot is documented at in the - [user - guide](https://chromium.googlesource.com/infra/luci/luci-py.git/+/HEAD/appengine/swarming/doc/User-Guide.md). -* Monitor the running tasks by visiting - [chromium-swarm.appspot.com](https://chromium-swarm.appspot.com). - -## FAQ - -### I run a task on Swarming and it hangs there - -It is possible that all the bots are currently fully utilized. In this case, the -task is in PENDING state. - -### It seems tedious to list each test data file individually, can I just list src/ ? - -In theory yes, in practice please don't and keep the list to the strict minimum. -The goal is not to run the tests more slowly and having the bots download 20 gb -of data. Reasons includes: - -1. Isolate Server is optimized for < 50000 files scenario. There's a - 2ms/file cost per cache hit. So for example, layout tests are - currently out of the use case since there's > 80000 files. -2. It's always possible to go coarser but it's much harder to get back - stricter. - -### Where can I find the .isolated file? - -The .isolated files are generated when you build the isolation specific version -of a target, e.g. out/Debug or out/Release. The isolation target name is just -the normal target name with a _run added to the end. - -### I have an error, is it related to .isolate files? - -If you have a test that passes locally and fails on some trybots, then it could -be related. - -This error can be seen when a browser test js file is not found: - -TypeError: Cannot read property 'testCaseBodies' of undefined - -### Where should I file bugs? - -Swarming specific bugs can be filed on at <http://crbug.com> in [component -Infra>LUCI>TaskDistribution](https://bugs.chromium.org/p/chromium/issues/list?q=component%3AInfra%3ELUCI%3ETaskDistribution).
\ No newline at end of file |