summaryrefslogtreecommitdiffstats
path: root/docs/TestingGuide.rst
diff options
context:
space:
mode:
authorGreg Parker <gparker@apple.com>2017-01-25 02:26:03 +0000
committerGreg Parker <gparker@apple.com>2017-01-25 02:26:03 +0000
commit4a50c7f39c7ad7204fbff83e529c0b30365ee987 (patch)
tree3fb4fbd1e5cedc43e7106876042f9eb691da6a0a /docs/TestingGuide.rst
parent431645e04393da431451561fb889d71ea5a2bf4e (diff)
Reinstate "r292904 - [lit] Allow boolean expressions in REQUIRES and XFAIL
and UNSUPPORTED" This reverts the revert in r292942. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@293007 91177308-0d34-0410-b5e6-96231b3b80d8
Diffstat (limited to 'docs/TestingGuide.rst')
-rw-r--r--docs/TestingGuide.rst64
1 files changed, 36 insertions, 28 deletions
diff --git a/docs/TestingGuide.rst b/docs/TestingGuide.rst
index 26143febd073..99616770d8e9 100644
--- a/docs/TestingGuide.rst
+++ b/docs/TestingGuide.rst
@@ -387,23 +387,49 @@ depends on special features of sub-architectures, you must add the specific
triple, test with the specific FileCheck and put it into the specific
directory that will filter out all other architectures.
-REQUIRES and REQUIRES-ANY directive
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Some tests can be enabled only in specific situation - like having
-debug build. Use ``REQUIRES`` directive to specify those requirements.
+Constraining test execution
+---------------------------
+
+Some tests can be run only in specific configurations, such as
+with debug builds or on particular platforms. Use ``REQUIRES``
+and ``UNSUPPORTED`` to control when the test is enabled.
+
+Some tests are expected to fail. For example, there may be a known bug
+that the test detect. Use ``XFAIL`` to mark a test as an expected failure.
+An ``XFAIL`` test will be successful if its execution fails, and
+will be a failure if its execution succeeds.
.. code-block:: llvm
- ; This test will be only enabled in the build with asserts
+ ; This test will be only enabled in the build with asserts.
; REQUIRES: asserts
+ ; This test is disabled on Linux.
+ ; UNSUPPORTED: -linux-
+ ; This test is expected to fail on PowerPC.
+ ; XFAIL: powerpc
+
+``REQUIRES`` and ``UNSUPPORTED`` and ``XFAIL`` all accept a comma-separated
+list of boolean expressions. The values in each expression may be:
-You can separate requirements by a comma.
-``REQUIRES`` means all listed requirements must be satisfied.
-``REQUIRES-ANY`` means at least one must be satisfied.
+- Features added to ``config.available_features`` by
+ configuration files such as ``lit.cfg``.
+- Substrings of the target triple (``UNSUPPORTED`` and ``XFAIL`` only).
+
+| ``REQUIRES`` enables the test if all expressions are true.
+| ``UNSUPPORTED`` disables the test if any expression is true.
+| ``XFAIL`` expects the test to fail if any expression is true.
+
+As a special case, ``XFAIL: *`` is expected to fail everywhere.
+
+.. code-block:: llvm
+
+ ; This test is disabled on Windows,
+ ; and is disabled on Linux, except for Android Linux.
+ ; UNSUPPORTED: windows, linux && !android
+ ; This test is expected to fail on both PowerPC and ARM.
+ ; XFAIL: powerpc || arm
-List of features that can be used in ``REQUIRES`` and ``REQUIRES-ANY`` can be
-found in lit.cfg files.
Substitutions
-------------
@@ -520,24 +546,6 @@ their name. For example:
This program runs its arguments and then inverts the result code from it.
Zero result codes become 1. Non-zero result codes become 0.
-Sometimes it is necessary to mark a test case as "expected fail" or
-XFAIL. You can easily mark a test as XFAIL just by including ``XFAIL:``
-on a line near the top of the file. This signals that the test case
-should succeed if the test fails. Such test cases are counted separately
-by the testing tool. To specify an expected fail, use the XFAIL keyword
-in the comments of the test program followed by a colon and one or more
-failure patterns. Each failure pattern can be either ``*`` (to specify
-fail everywhere), or a part of a target triple (indicating the test
-should fail on that platform), or the name of a configurable feature
-(for example, ``loadable_module``). If there is a match, the test is
-expected to fail. If not, the test is expected to succeed. To XFAIL
-everywhere just specify ``XFAIL: *``. Here is an example of an ``XFAIL``
-line:
-
-.. code-block:: llvm
-
- ; XFAIL: darwin,sun
-
To make the output more useful, :program:`lit` will scan
the lines of the test case for ones that contain a pattern that matches
``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number