Tizen-Distro Workflow

From Tizen Wiki
Jump to: navigation, search

This page is work in progress and may be changed

Tizen-Distro is an effort to create and maintain a self-contained distribution repository similar to Poky of the Yocto project - based on a similar project structure and using BitBake as the build tool.
More information on Tizen-Distro here

This page has a developer view on the workflow proposal.

Backend workflow

The following is a proposal for combining the current Tizen workflow and repository layout with a new BitBake-based development model:

  • It does not try to change existing Tizen practices, but, add a BitBake workflow parallel to that.


Tizen development would still be done on per-package Git repositories in Tizen gerrit, changes would be "submitted" with Git tags etc.

Mapping to the current Tizen development model:

  • The combined tizen-distro repository reflects whatever has been currently integrated into Tizen image.


That is, the state of latest snapshot/daily/weekly/milestone release – iow it would reflect the accepted/* branches of packages.

Tizen-distro-backend-workflow.png

Tizen-distro-per-pkg-backend-workflow-sequence.png

Per-package Git repositories

The layout or maintenance of the per-package repositories would be virtually unchanged.
Regarding the repository content, the only change would be the addition of bitbake metadata (under packaging/ or a separate directory).
Otherwise, things would be more or less the same:

  • changes would be reviewed in Gerrit
  • changes would need to be submitted (submit tag) in order to get them integrated into Tizen
  • "joint packaging" (sources and packaging in the same branch) and "orphan packaging" (source code and packaging in separate branches) maintenance models would be supported

RPM vs. BitBake packaging meta data

One possible source of problems is that the Tizen per-package repositories will have two copies of the packaging meta data:

  • one for RPM (.spec)
  • one for BitBake (.bb)


For the time being, both of these meta data must be up-to-date and present in the repository because we need to support both GBS/OBS and BitBake based workflows.

In order to keep the metadata in sync, automatic meta data verification/update would be implemented.
The following automation would be in place whenever a change is submitted for review or a branch is updated:

Tizen-distro-spec-autoupdate.png

  1. Skip if no change in packaging metadata. This way, we do not force-introduce packaging changes by convertor when only source code is modified.
  2. Skip if rpm metadata has NOT been marked to be automatically maintained (e.g. by a special header section in the spec file)
  3. Otherwise, run bb-to-spec conversion and compare to existing rpm metadata. If delta is found:
    1. If a patcset was created/updated, upload a new version of the patcset with the auto-updated rpm packaging files.
    2. If a ref was updated (branch head updated via Gerrit merge or direct push) -> make a new commit with the auto-generated rpm metadata and push to Git automatically without review

The conversion would always be made in the context of the target branch.
I.e. the converter would be run in tizen_ivi branch of the tizen-distro if refs/heads/tizen_ivi of a per-pkg repo would be changed.

The general refs/heads/tizen branch would be mapped to Tizen Common.
This means that care must be taken so that profile-specific packages that are developed using the tizen branch are convertible in the Tizen Common context (e.g. by using features conditional).
In addition to the per-commit automation, there would be automatic periodic (daily/weekly?) check of the metadata delta.
A script would run through all the packages (those with automatically maintained rpm metadata) in tizen-distro repository, convert the packaging meta data and compare the result with the output of gbs export of the same package. Possible delta would be submitted for review. This would reveal delta caused by changes in the conversion process, e.g. update in distro configuration or in the coversion tool itself.

The intent would be to eventually convert all packages to use the "automatic mode", i.e. automatically updated rpm metadata.
The on/off setting for automatically updated rpm metadata allows for a transition period for maintainers to adapt to the .bb format. Allowing them to use .spec file as the primary packaging while .bb metadata is already present and bitbake build thus also possible. This transition period risks for differing build results between OBS/GBS and bitbake builds. The problem could be mitigated by periodic (weekly?) scanning of the packages that have manually maintained .spec files, see if packaging has changed and adjust .bb metadata accordingly - this would be (semi-)manual process done by REs/distro maintainers.

Maintenance of the combined (tizen-distro) repo

The Tizen-Distro combined repository would be maintained with the help of the combo-layer tool (similarly to Poky) and would be the responsibility of Tizen release engineers.
However, the maintenance would semi-automatic, with CI involved, as described in the "Overview" section above.

The CI automation would be two-fold:

  1. External repositories would involve manual gating. Full CI (build and test) would be run whenever a change in any external repository would be detected. RE(s) would be notified and if everything is green the RE would integrate the changes by running a "merge job" on the backend. Integration could be made fully automatic later, if seen appropriate and when sufficient test coverage is available.
  2. Changes in meta-tizen-* layers would be immediately relayed in the tizen-distro repository. This is required in order to keep the OBS and BitBake versions of Tizen in sync.

Gerrit review process is not useful for managing the layers in the tizen-distro repository level as changes cannot basically be rejected – it is only possible to create additional fixes (in the originating layer repos - i.e. the external repos directly, or, tizen-meta-x layers) and delay integration.
However, Gerrit could/should be used for changes that are specific to the combined repository, i.e. changes not coming from any "layer repository" but committed to the tizen-distro directly.

Meta-tizen layer(s)

The Tizen packages would/could be split into multiple (supposedly) independent meta-tizen-* layers. The package->layer mapping configuration would be maintained in Tizen Gerrit (scm/bb/git-layer-mapping) and automatically deployed to the backend server. One of the layers would contain Tizen distro configuration.

The meta-tizen-* layers are supposed to reflect whatever is currently integrated into Tizen, i.e. the accepted/* branches of packages.

Maintenance of the meta-tizen-* layers would be combination of fully automatic and manual maintenance.

  1. Package meta data
    All package meta-data would be maintained fully automatically. Whenever a submission would be accepted in Tizen (by a release engineer, with the repa tool) bitbake packaging metadata from the submission would be committed/pushed to the corresponding branch in the layer repository. The automation would be in place in order to make sure that meta-tizen layers are is always in sync with the Tizen OBS build.
  2. Other data
    Non-package-specific parts would be manually maintained. These could include configuration, additional scripts or bitbake classes etc. Review would be done normally via Gerrit.

All changes to package meta data must come from package-specific Git repositories. E.g. if zlib would need to be tweaked the changes would need to be integrated in the platform/upstream/zlib Git repository in tizen.org.

The recipe subdirectory layout in meta-tizen would be controlled by a configuration that would be maintained in gerrit and automatically deployed in the backend. The same Gerrit project as for package->layer mapping could be used.


Branching

The combined tizen-distro and individual meta-tizen-* repositories would have one branch per profile plus release/milestone branches, mapped similarly to the branches of per-package Git repositories in Tizen.
However, only accepted/* branches of the per-package repositories would be followed as they are the only branches more or less guaranteed to be in consistent state system wide.
The 'accepted/*' prefix would be removed from the branch names as it's redundant and possibly just confusing in the context of tizen-distro.

In addition to the accepted/* branches, the submit/* tags of the per-package repositories would be monitored.
However, the submit tags would not be mapped to tags but special refs "hidden" refs under refs/submit/*.
This is makes possible to run combine submissions from multiple packages under one ref and run test build on that.
The hidden ref would be used in order to not pollute the branch namespace.

The tag and branch mapping from per-package to meta-tizen-* repositories would be:

  • refs/heads/accepted/* -> refs/heads/*
  • refs/tags/submit/* -> refs/submit/*

Branches (tizen_common, tizen_ivi etc) would be independent and individually tested, similarly to the current state in Tizen.

Syncing of external repositories

One complication in the dual-workflow-dual-metadata (GBS BitBake) setup in Tizen are the external repositories.
That is, at least OE-Core at this point.
The BitBake build of Tizen (Tizen-Distro here) is supposed to be based on OE-Core.
Basically, it would be easy to keep OE-Core in sync with the combined tizen-distro repository if the per-package repositories and OBS builds were not present.
However, the OBS/GBS and BitBake builds of Tizen should be the same, and thus, the per-package Git repositories (of the OE-Core) must be updated in-sync with the combined distro-repository.
This is handled by the opaque "sync job" in the figure above.
There is an assumtion that very little Tizen-specific development of the core packages (from OE-Core) would be done.

In the first phase of the tizen-distro syncing would be handled manually by REs/core maintainers.

  1. Pull patches from oe-core -> tizen-distro on a test branch
  2. Find orphan bbappend files (i.e. for the cases oe-core has done version bump but Tizen's bb points to the old version)
  3. Rebase orphan bbappend files
  4. Run test build and fix problems
  5. When happy, merge the changes.

At this phase the OBS and BitBake builds might be slightly out-of-sync wrt to the OE-Core packages.

In a second phase some automation would be deployed.
At each sync time (e.g. weekly):

  1. pull patches from oe-core -> tizen-distro on a test branch
  2. test build on tizen-distro
    1. build
    2. run tests
  3. test rpm build
    1. auto-convert all packaging meta data (i.e. all packages that are in oe-core) from .bb to .rpm
    2. auto-push rpm packaging data changes for review in Tizen Gerrit under common Gerrit topic, e.g. 'oe-core-sync'
      1. common test build for all patches
      2. if needed, manually make Tizen-specific fixups (i.e. push fixup patches for review)
    3. when happy with results, batch-merge all 'oe-core-sync' changes in Gerrit
    4. group-submit all 'oe-core-sync'
  4. accept submission (per-pkg repos), merge patches in tizen-distro

The above automation assumes that the "orphan-packaging" maintenance model is used in the Tizen per-package Git repositories - i.e. only packaging data is present in the packaging branch (no upstream source code there).
This makes it possible to push rpm packaging changes for review and does not force into breaking Git history.