diff options
Diffstat (limited to 'test/README')
| -rw-r--r-- | test/README | 233 |
1 files changed, 233 insertions, 0 deletions
diff --git a/test/README b/test/README new file mode 100644 index 0000000..f387157 --- /dev/null +++ b/test/README @@ -0,0 +1,233 @@ +Writing APR tests + +All APR tests should be executable in 2 ways, as an individual program, or +as a part of the full test suite. The full test suite is controlled with +the testall program. At the beginning of the testall.c file, there is an +array of functions called tests. The testall program loops through this +array calling each function. Each function returns a CuSuite variable, which +is then added to the SuiteList. Once all Suites have been added, the SuiteList +is executed, and the output is printed to the screen. All functions in the +array should follow the same basic format: + +The Full Suite +-------------- + +/* The driver function. This must return a CuSuite variable, which will + * then be used to actually run the tests. Essentially, all Suites are a + * collection of tests. The driver will take each Suite, and put it in a + * SuiteList, which is a collection of Suites. + */ +CuSuite *testtime(void) +{ + /* The actual suite, this must be created for each test program. Please + * give it a useful name, that will inform the user of the feature being + * tested. + */ + CuSuite *suite = CuSuiteNew("Test Time"); + + /* Each function must be added to the suite. Each function represents + * a single test. It is possible to test multiple features in a single + * function, although no tests currently do that. + */ + SUITE_ADD_TEST(suite, test_now); + SUITE_ADD_TEST(suite, test_gmtstr); + SUITE_ADD_TEST(suite, test_localstr); + SUITE_ADD_TEST(suite, test_exp_get_gmt); + SUITE_ADD_TEST(suite, test_exp_get_lt); + SUITE_ADD_TEST(suite, test_imp_gmt); + SUITE_ADD_TEST(suite, test_rfcstr); + SUITE_ADD_TEST(suite, test_ctime); + SUITE_ADD_TEST(suite, test_strftime); + SUITE_ADD_TEST(suite, test_strftimesmall); + SUITE_ADD_TEST(suite, test_exp_tz); + SUITE_ADD_TEST(suite, test_strftimeoffset); + + /* You must return the suite so that the driver knows which suites to + * run. + */ + return suite; +} + +Building the full driver +------------------------ + +All you need to do to build the full driver is run: + + make testall + +To run it, run: + + ./testall + +Running individual tests +--------------------------------- + +It is not possible to build individual tests, however it is possible to +run individual tests. When running the test suite, specify the name of the +tests that you want to run on the command line. For example: + + ./testall teststr testrand + +Will run the Strings and Random generator tests. + +Reading the test suite output +----------------------------- + +Once you run the test suite, you will get output like: + +All APR Tests: + Test Strings: .... + Test Time: ............ + +16 tests run: 16 passed, 0 failed, 0 not implemented. + +Known test failures are documented in ../STATUS. + +There are a couple of things to look at with this. First, if you look at the +first function in this document, you should notice that the string passed to +the CuSuiteNew function is in the output. That is why the string should +explain the feature you are testing. + +Second, this test passed completely. This is obvious in two ways. First, and +most obvious, the summary line tells you that 16 tests were run and 16 tests +passed. However, the results can also be found in the lines above. Every +'.' in the output represents a passed test. + +If a test fails, the output will look like: + +All APR Tests: + Test Strings: .... + Test Time: ..F......... + +16 tests run: 15 passed, 1 failed, 0 not implemented. + +This is not very useful, because you don't know which test failed. However, +once you know that a test failed, you can run the suite again, with the +-v option. If you do this, you will get something like: + +All APR Tests: + Test Strings: .... + Test Time: ..F......... + +16 tests run: 15 passed, 1 failed, 0 not implemented. +Failed tests: +1) test_localstr: assert failed + +In this case, we know the test_localstr function failed, and there is an +Assert in this that failed (I modified the test to fail for this document). +Now, you can look at what that test does, and why it would have failed. + +There is one other possible output for the test suite (run with -v): + +All APR Tests: + Test Strings: .... + Test Time: ..N......... + +16 tests run: 15 passed, 0 failed, 1 not implemented. + +Not Implemented tests: + +Not Implemented tests: +1) test_localstr: apr_time_exp_lt not implemented on this platform + +The 'N' means that a function has returned APR_ENOTIMPL. This should be +treated as an error, and the function should be implemented as soon as +possible. + +Adding New test Suites to the full driver +------------------------------------------- + +To add a new Suite to the full driver, you must make a couple of modifications. + +1) Edit test_apr.h, and add the prototype for the function. +2) Edit testall.c, and add the function and name to the tests array. +3) Edit Makefile.in, and add the .lo file to the testall target. + +Once those four things are done, your tests will automatically be added +to the suite. + +Writing tests +------------- + +There are a couple of rules for writing good tests for the test suite. + +1) All tests can determine for themselves if it passed or not. This means +that there is no reason for the person running the test suite to interpret +the results of the tests. +2) Never use printf to add to the output of the test suite. The suite +library should be able to print all of the information required to debug +a problem. +3) Functions should be tested with both positive and negative tests. This +means that you should test things that should both succeed and fail. +4) Just checking the return code does _NOT_ make a useful test. You must +check to determine that the test actually did what you expected it to do. + +An example test +--------------- + +Finally, we will look at a quick test: + +/* All tests are passed a CuTest variable. This is how the suite determines + * if the test succeeded or failed. + */ +static void test_localstr(CuTest *tc) +{ + apr_status_t rv; + apr_time_exp_t xt; + time_t os_now; + + rv = apr_time_exp_lt(&xt, now); + os_now = now / APR_USEC_PER_SEC; + + /* If the function can return APR_ENOTIMPL, then you should check for it. + * This allows platform implementors to know if they have to implement + * the function. + */ + if (rv == APR_ENOTIMPL) { + CuNotImpl(tc, "apr_time_exp_lt"); + } + + /* It often helps to ensure that the return code was APR_SUCESS. If it + * wasn't, then we know the test failed. + */ + CuAssertTrue(tc, rv == APR_SUCCESS); + + /* Now that we know APR thinks it worked properly, we need to check the + * output to ensure that we got what we expected. + */ + CuAssertStrEquals(tc, "2002-08-14 12:05:36.186711 -25200 [257 Sat] DST", + print_time(p, &xt)); +} + +Notice, the same test can fail for any of a number of reasons. The first +test to fail ends the test. + +CuTest +------ + +CuTest is an open source test suite written by Asim Jalis. It has been +released under the zlib/libpng license. That license can be found in the +CuTest.c and CuTest.h files. + +The version of CuTest that is included in the APR test suite has been modified +from the original distribution in the following ways: + +1) The original distribution does not have a -v flag, the details are always +printed. +2) The NotImplemented result does not exist. +3) SuiteLists do not exist. In the original distribution, you can add suites +to suites, but it just adds the tests in the first suite to the list of tests +in the original suite. The output wasn't as detailed as I wanted, so I created +SuiteLists. + +The first two modifications have been sent to the original author of CuTest, +but they have not been integrated into the base distribution. The SuiteList +changes will be sent to the original author soon. + +The modified version of CuTest is not currently in any CVS or Subversion +server. In time, it will be hosted at rkbloom.net. + +There are currently no docs for how to write tests, but the teststr and +testtime programs should give an idea of how it is done. In time, a document +should be written to define how tests are written. + |