Security/Tizen 3.X Security Tests

From Tizen Wiki
Jump to: navigation, search

Project name: Security-tests

Introduction

Security-tests is a project and its repository designed for testing packages from domain Security. It contains tests for following packages:

  • cynara
  • key-manager
  • libsmack
  • libprivilege-control
  • security-server
  • security-manager

The tests use internal security-tests framework. Inner-tests for testing the security-tests framework mechanisms are also included.

Project Goals

This project's main task is to ensure that core security mechanisms operate properly and work together seamlessly. It is also a place for tests which exceed scope of internal unit tests, e.g. actions involving writing to storage or using external tools.

Contact information

Maintainers:

Zbigniew Jasiński

Developers:

Zofia Abramowska
Radosław Bartosiak
Piotr Bartosiewicz
José Bollo
Paweł Broda
Jacek Bukarewicz
Damian Chromejko
Jan Cybulski
Mariusz Domański
Michał Eljasiewicz
Sebastian Grabowski
Bartłomiej Grzelewski
Krzysztof Jackiewicz
Maciej Karpiuk
Łukasz Kostyra
Janusz Kozerski
Rafał Krypa
Marcin Lis
Adam Malinowski
Jung Minsun
Marcin Niesłuchowski
Jan Olszak
Paweł Polawski
Krzysztof Sasiak
Marek Smoliński
Paweł Wieczorek
Michał Witanowski
Łukasz Wojciechowski
Aleksander Zdyb

All discussion about security-tests should be held at the Dev on Tizen.org mailing lists.

Git repository for security-tests is available at: review.tizen.org/git.

To submit bugs please use the Tizen.org JIRA.

Features

API overview

Single package security-tests is provided, containing mulitple binaries (at least one per package) along with necessary files and automation scripts.

Running the project

Installation

When installing through Zypper, dependencies are resolved and installed automatically.

Manual installation with rpm needs some prerequisites, as the security-tests package depends on three packages: gdb, perf and libpcrecpp. These packages should be installed on device (with their dependencies) prior to running security-tests.

Installing following packages should make running tests possible:

  • gdb
  • perf
  • libpcrecpp
  • kernel-common-arm-default
  • libdw

Useful scripts

Three scripts for test automation are provided:

  1. test-performance-check.sh - runner for performance tests
  2. security-tests-all.sh - runner for all projects at once
  3. security-tests.sh - unified runner for each project binary

Tests

Tests may also be called explicitly. List below provides names of binaries for corresponding packages:

  • libsmack
    • libsmack-test
  • libprivilege-control
    • libprivilege-control-test
  • security-server
    • security-server-tests-client-smack
    • security-server-tests-stress
    • security-server-tests-server
    • security-server-tests-api-speed
    • security-server-tests-password
    • security-server-tests-privilege
    • security-server-tests-dbus
  • security-manager
    • security-manager-tests
  • cynara
    • cynara-test
  • security-tests
    • security-tests-inner-test

Options

Each test binary may be executed with following options:

Security-tests options
option parameter description
--output= summary/text/xml/html Print output in given format (example: test-binary --output=text --output=xml --file=output.xml)
--list N/A Show list of Test IDs
--listgroups N/A Show list of Test Group names
--listingroup= Group name Show list of Test IDs in given group
--regexp= Regular expression Run only tests with names matching regexp
--start= Test ID Start from specific Test ID
--group= Group name Run only tests from given group
--only-from-xml= XML file Run only tests specified in given XML file
--runignored N/A Run also ignored tests
--allowchildlogs N/A Allow to print logs from child process on screen
--help N/A Print help

Each test can end with one of three possible statuses:

  1. FAILED
  2. IGNORED
  3. OK

High Level Architecture

Branches

List of all branches for security-test is available at review.tizen.org.

Main branch is tizen. It is used for testing released packages (already present on images). There are also separate branches for following packages:

  • ckm,
  • cynara,
  • dbus,
  • security-manager.

Division was introduced in order not to stale project development. Previous workflow (single branch for all packages) slowed addition of new functionalities down, as they are not released without being tested first. These branches are periodically synchronized with main branch, usually around release of the new version of given package.

Code style

There are few recommendations to take into consideration while extending this project with new tests:

  1. While changing body of test cases, be sure to remove functions and global variables (if are not used by any other tests anymore)
  2. Use const variables instead of #define's
  3. Use mechanisms already provided in common library

Testing

Internal tests are invoked by executing security-tests-inner-test. All new functionalites introduced in framework or in common libraries should also be provided with test for them.

How to write new tests

Below all available macros are presented. In order to start writing new test quickly, jump to Usage example right away.

Macros

Security-tests privide different macros to simplify writing new tests:

  • Registering macros - Adding/removing those macro calls will add/remove test cases they provide. Tochange tests, change body of those macro calls. Registered tests are run within group alphabetically.
  • Assert macros - These are used within test registering macros. First failed assertion throws test failed exception. If another assertions fail, information about fail conditions and backtrace is cumulated and presented together with already thrown exception message.
  • Performence macros - Used for time measurement.
  • Message macros - Used to print error messages during test run time
  • Defer macros - Used to defer throwing TestException exceptions (TestFailed, TestIgnored) by catching them and rethrowing later. This mechanism can help in breaking test and passing test result from places where throwing exceptions is not allowed
Test group registering macro
Library Header Macro Description
dpl-test-framework test_runner.h RUNNER_TEST_GROUP_INIT Registers group of tests. Test are registered to this group until another group registering macro is called
Test registering macros
Library Header Macro Description
dpl-test-framework test_runner.h RUNNER_TEST Function registered by this macro will be run in the same process as framework. No forking allowed unless forked process does not throw any exception
dpl-test-framework test_runner_child.h RUNNER_CHILD_TEST Function registered by this macro will be run in child process. No forking allowed unless forked process does not throw any exception
dpl-test-framework test_runner_multiprocess.h RUNNER_MULTIPROCESS_TEST Function registered by this macro will be run in the same process as framework. Forking allowed. Exception of every process will be registered
tests-common tests_common.h RUNNER_TEST_SMACK Same as RUNNER_TEST but run only with smack enabled
tests-common tests_common.h RUNNER_TEST_NOSMACK Same as RUNNER_TEST but run only with smack disabled
tests-common tests_common.h RUNNER_CHILD_TEST_SMACK Same as RUNNER_TEST_CHILD but run only with smack enabled
tests-common tests_common.h RUNNER_CHILD_TEST_NOSMACK Same as RUNNER_TEST_CHILD but run only with smack disabled
tests-common tests_common.h RUNNER_MULTIPROCESS_TEST_SMACK Same as RUNNER_TEST_MULTIPROCESS but run only with smack enabled
tests-common tests_common.h RUNNER_MULTIPROCESS_TEST_NOSMACK Same as RUNNER_TEST_MULTIPROCESS but run only with smack disabled
Assert macros
Library Header Macro Description
dpl-test-framework test_runner.h RUNNER_ASSERT_MSG Assertion with message with backtrace information appended
dpl-test-framework test_runner.h RUNNER_ASSERT_ERRNO_MSG Assertion with message, error string and backtrace information appended
dpl-test-framework test_runner.h RUNNER_ASSERT_ERRNO Assertion with error string and backtrace information appended
dpl-test-framework test_runner.h RUNNER_FAIL_MSG Fail with message and backtrace information appended
dpl-test-framework test_runner.h RUNNER_ASSERT Assertion with backtrace information appended
dpl-test-framework test_runner.h RUNNER_IGNORED_MSG Assertion with message classified as ignored
Performence macros
Library Header Macro Description
dpl-test-framework test_runner.h RUNNER_PERF_TEST_BEGIN Begin time measurement
dpl-test-framework test_runner.h RUNNER_PERF_TEST_END End time measurement
Message macros
Library Header Macro Description
dpl-test-framework test_runner.h RUNNER_ERROR_MSG Print error message using red color
Defer macros
Library Header Macro Description
dpl-test-framework test_runner.h RUNNER_DEFER_TRYCATCH Catches thrown TestException exceptions and stores them in TestRunner structures for later use. This macro works only inside deferred scope defined by RUNNER_DEFER_SCOPE, otherwise it won't catch exceptions
dpl-test-framework test_runner.h RUNNER_DEFER_SCOPE Defines deferred scope. All RUNNER_DEFER_TRYCATCH macros used inside the scope catch and save TestException exceptions. After scope is left all saved exceptions take part in setting result of test. If there is no any uncaught exception then additionally first of saved exceptions is thrown.

Collectors

Collectors are classes which collect test results. Each class does it differently. Collectors can be registered by --output parameter (see Options section) but there is also another collector created to write summary.

Collectors
Library Header Class Description
dpl-test-framework test_results_collector_summary.h SummaryCollector Collector writing tests summary. Call SummaryCollector::Register() to register it

Usage example

   #include <test_runner.h> 
   #include <tests_common.h>
   #include <summary_collector.h>
   #include <sys/stat.h>
   #include <unistd.h>
   #include <fcntl.h>
   RUNNER_TEST_GROUP_INIT(foo_module)
   RUNNER_TEST_SMACK(bar_allways_fails)
   {
       RUNNER_ASSERT(false);
   }
   RUNNER_TEST(bar_allways_passses)
   {
       RUNNER_ASSERT(true);
   }
   RUNNER_TEST(bar_file1)
   {
       cosnt char *file = "bar_file1";
       int fd = TEMP_FAILURE_RETRY(open(file, O_RDONLY));
       RUNNER_ASSERT_ERRNO_MSG(fd != -1, "Cannot open " << file << " file");
       close(fd);
   }
   RUNNER_CHILD_TEST_NOSMACK(bar_file2_dropped_root)
   {
       RUNNER_ASSERT_ERRNO(setgid(5000) == 0);
       RUNNER_ASSERT_ERRNO(setuid(5000) == 0);
       cosnt char *file = "bar_file2";
       int fd = TEMP_FAILURE_RETRY(open(file, O_RDONLY));
       if(fd != -1) {
           close(fd);
           RUNNER_FAIL_MSG("file " << file << "should not be opened");
       }
       RUNNER_ASSERT_ERRNO_MSG(errno == EACCESS,
                           "Wrong errno on opening " << " file");
   }
   int main(int argc, char *argv[])
   {
       SummaryCollector::Register();
       return DPL::Test::TestRunnerSingleton::Instance().ExecTestRunner(argc, argv);
   }

Future release plan with dates (includes functionalities planned for a release)

Note about ongoing refactoring might be put here.