Ppinfra

From Tizen Wiki
Jump to: navigation, search

Wiki

This part was written 2015-05 describing actual state of pptest at that time.

Overview

PPtest setup was created and developed inside Intel lab network during 2012..2014, with aim to provide a parallel off-production environment for testing Tizen tools. The main idea has been: mimic production setup of tizen.org.

pptest consists of 3 main servers:

  1. OBS server for building: replica of build.tizen.org + backend.tizen.org
  2. Jenkins server for CI jobs: replica of robot.tizen.org
  3. Download server: replica of download.tizen.org


There are also worker hosts that serve as combined OBS builders and Jenkins workers corresponding to wNN worker hosts in tizen.org

There is no gerrit server in pptest, no source is hosted on pptest. PPtest uses gerrit server from tizen.org. PPtest OBS has main projects linked to master projects in OBS of tizen.org. Jenkins server CI jobs are triggered by events in pptest OBS. Build results are stored on the download server of pptest.

Testing of images

There are 3 testers connected to Jenkins server with connection method: linux-twjk (SSHLauncher) CATS-Tester-ppfarm (JNLPLauncher) Tester-00 (JNLPLauncher)

Differences from tizen.org

Because of some limitations, pptest is not exact replica (see diffs section below). Overall, network topology is quite different. pptest does not replicate the tizen.org separated/isolated networks model. In pptest model, all hosts are connected to lab network which is used as main interface (and interface towards the world). There is separate privately addressed segment where workers can connect. This isolated segment is controlled by OBS server which provides DNS and proxy services in that network, and acts as gateway to access workers. All 3 pptest main servers have 2nd interface connected to this isolated network.

====================================================================
Tizen.org                  |  pptest                
====================================================================
Workers:                   |
30 x baremetal hosts       |  10 x LXC-type VMs
                           |  in isolated network behind OBS server
                           |
Web front end:             |
Separate frontend host     |  OBS host acts as web frontend
                           |
OBS:                       |
separate front and         | both backend and frontend in same host
back ends                  |
====================================================================


PPtest special needs different from tizen.org

1. OBS projects need additional cleanup on OBS server: As pptest is less powerful in OBS build capacity than tizen.org, it can not complete building of all dynamically created (home:prerelease) OBS projects. Without explicit cleanup, it would face continuosly growing build queues. To avoid this, there is a special cron job belonging to pptest OBS server root user which periodically (once or twice per day) deletes home:prerelease:* projects that become older than N days, N being tunable with value 1..2

Server configuration

The versions of OS distributions and installed packages should match tizen.org respective values as close as possible.

1. OBS

VM with 8 vCPU, 20 GB of RAM, 2 TB of disk openSUSE 12.3, OBS 2.4.7

2. Jenkins

VM with 4 vCPU, 8 GB of RAM, 160 GB of disk openSUSE 13.1, Jenkins 1.580

3. Download

VM with 4 vCPU, 4 GB of RAM, 4 TB of disk openSUSE 13.1, apache 2.4.6


 -------------- end of new part of 2015-05 -------------------------------

Note: Below is "old" part of this document created during 2014 and discusses mainly desired state and features (reset feature, slots reservation etc), which were mostly not implemented.

About the so called "ppfarm" environment

Purposes

The main uses of the environment are:

  • Staging area
Before releasing any new SW to the production environment, it must be successfully tested in preproduction.
This comprises:
  • Jenkins scripts and Jobs
  • Backend tools involved in the generation of any binary deliverable (gbs, mic, etc.)
  • Jenkins framework and its plugins, OBS, OBS-related packages
The way it will be done is by resetting to the same configuration as production and then installing the content of the release.
  • Testing and Development
Users will be able to reset the environment to the same state as production. Then install over it their own development packages,to test the effect of the patches under development.


Utilization

  • Because there is a problem of potential clashes between packages developed concurrently by different people, the environment will be available in time-sharing, to one user only at any point in time. The user can be either an individual developer or one person deploying a synchronized set of changes, coming from several developers.
  • Pre-release staging has precedence over normal development.
  • In other cases, some fair queuing will be applied, to guarantee everyone a chance to test own work.
  • There should be 3-6 slots per day:
    • the 1st prioritized for PRC,
    • the last prioritized for Finland,
    • the middle one(s) free for all.
The prioritization is meant to be compatible with regular office hours for each site.


Implementation

The key part of the implementation consists of the reset feature, which must enable a random developer to re-align pre-production with production, no matter what was changed/deployed by the previous user of the environment.

  • The parts that must be realigned are:
    • Jenkins Master:
      • scripts
      • jobs
      • nodes configuration (?)
    • Jenkins Workers: none.
    • OBS:
      • service packages
      • pointers to projects in the production OBS
    • Download Server:
      • Must be wiped of all the previous custom builds.
      • Jenkins will then be re-triggered and reproduce locally the latest build.
  • The interface for resetting the environment will be implemented as Jenkins job on otc-tools, so that:
    • It will automatically acquire the user from the login credential
    • It will use the credential to implement exclusive access to the preproduction environment. Admins will be able to access the system 24/7 ,but solely for maintenance purpose.
    • It will be possible to keep under control the use of the resource, releasing it after the end of a session.


Customers

The customers for the resetting feature are:

  • developers in general
  • those who need to deploy one of the following projects:
    • jenkins-jobs
    • jenkins-scripts
    • tizen test robot
    • GBS
    • MIC