This part was written 2015-05 describing actual state of pptest at that time.
PPtest setup was created and developed inside Intel lab network during 2012..2014, with aim to provide a parallel off-production environment for testing Tizen tools. The main idea has been: mimic production setup of tizen.org.
pptest consists of 3 main servers:
- OBS server for building: replica of build.tizen.org + backend.tizen.org
- Jenkins server for CI jobs: replica of robot.tizen.org
- Download server: replica of download.tizen.org
There are also worker hosts that serve as combined OBS builders and Jenkins workers corresponding to wNN worker hosts in tizen.org
There is no gerrit server in pptest, no source is hosted on pptest.
PPtest uses gerrit server from tizen.org.
PPtest OBS has main projects linked to master projects in OBS of tizen.org.
Jenkins server CI jobs are triggered by events in pptest OBS.
Build results are stored on the download server of pptest.
There are 3 testers connected to Jenkins server with connection method:
Because of some limitations, pptest is not exact replica (see diffs section below). Overall, network topology is quite different. pptest does not replicate the tizen.org separated/isolated networks model. In pptest model, all hosts are connected to lab network which is used as main interface (and interface towards the world). There is separate privately addressed segment where workers can connect. This isolated segment is controlled by OBS server which provides DNS and proxy services in that network, and acts as gateway to access workers. All 3 pptest main servers have 2nd interface connected to this isolated network.
==================================================================== Tizen.org | pptest ==================================================================== Workers: | 30 x baremetal hosts | 10 x LXC-type VMs | in isolated network behind OBS server | Web front end: | Separate frontend host | OBS host acts as web frontend | OBS: | separate front and | both backend and frontend in same host back ends | ====================================================================
1. OBS projects need additional cleanup on OBS server:
As pptest is less powerful in OBS build capacity than tizen.org, it can not
complete building of all dynamically created (home:prerelease) OBS projects.
Without explicit cleanup, it would face continuosly growing build queues.
To avoid this, there is a special cron job belonging to pptest OBS server
root user which periodically (once or twice per day) deletes home:prerelease:* projects
that become older than N days, N being tunable with value 1..2
The versions of OS distributions and installed packages should match tizen.org respective values as close as possible.
VM with 8 vCPU, 20 GB of RAM, 2 TB of disk openSUSE 12.3, OBS 2.4.7
VM with 4 vCPU, 8 GB of RAM, 160 GB of disk openSUSE 13.1, Jenkins 1.580
VM with 4 vCPU, 4 GB of RAM, 4 TB of disk
openSUSE 13.1, apache 2.4.6
-------------- end of new part of 2015-05 -------------------------------
Note: Below is "old" part of this document created during 2014 and
discusses mainly desired state and features (reset feature, slots reservation etc),
which were mostly not implemented.
The main uses of the environment are:
- Staging area
- Before releasing any new SW to the production environment, it must be successfully tested in preproduction.
- This comprises:
- Jenkins scripts and Jobs
- Backend tools involved in the generation of any binary deliverable (gbs, mic, etc.)
- Jenkins framework and its plugins, OBS, OBS-related packages
- The way it will be done is by resetting to the same configuration as production and then installing the content of the release.
- Testing and Development
- Users will be able to reset the environment to the same state as production. Then install over it their own development packages,to test the effect of the patches under development.
- Because there is a problem of potential clashes between packages developed concurrently by different people, the environment will be available in time-sharing, to one user only at any point in time. The user can be either an individual developer or one person deploying a synchronized set of changes, coming from several developers.
- Pre-release staging has precedence over normal development.
- In other cases, some fair queuing will be applied, to guarantee everyone a chance to test own work.
- There should be 3-6 slots per day:
- the 1st prioritized for PRC,
- the last prioritized for Finland,
- the middle one(s) free for all.
- The prioritization is meant to be compatible with regular office hours for each site.
The key part of the implementation consists of the reset feature, which must enable a random developer to re-align pre-production with production, no matter what was changed/deployed by the previous user of the environment.
- The parts that must be realigned are:
- Jenkins Master:
- nodes configuration (?)
- Jenkins Workers: none.
- service packages
- pointers to projects in the production OBS
- Download Server:
- Must be wiped of all the previous custom builds.
- Jenkins will then be re-triggered and reproduce locally the latest build.
- Jenkins Master:
- The interface for resetting the environment will be implemented as Jenkins job on otc-tools, so that:
- It will automatically acquire the user from the login credential
- It will use the credential to implement exclusive access to the preproduction environment. Admins will be able to access the system 24/7 ,but solely for maintenance purpose.
- It will be possible to keep under control the use of the resource, releasing it after the end of a session.
The customers for the resetting feature are: