"Onboarding" new projects within CI environment
Index
Contents
Intro
This wiki page will try to list all the mandatory steps to "onboard" new projects allowed to run tests within the CI Environment. Target audience is CI admins (from the CentOS side), having "power" in Jenkins and other machines.
Ensure all users being onboarded are signed up to the ci-users list ( https://lists.centos.org/mailman/listinfo/ci-users )
Jenkins
Jenkins master
There is a jenkins master running on the jenkins.ci.centos.org (and known both internally and externally as https://ci.centos.org too). For a new project, we need to :
- create a new user (jenkins/manage/people) (use the project name, not individual names, can be added though if needed), assigning it a strong password, and also use a valid email address (see with the project)
- grant him the following rights (jenkins/manage/global security/ add the user to the matrix) :
- Slave (configure, disconnect, connect, build)
- Job (all)
- Run (all)
- View (all)
- Adding a slave (manage/manage nodes) :
- give it a name, like $project-ci-slave${number} and a description
- check the remote root directory (so basically where the scripts to be run will be located on the slave VM)
- assign also a label (to restrict some particular jobs to run only on/from that slave)
- select the "only build jobs with label ..."
- launch method: launch slave agents via java web start (see jenkins slave below)
Jenkins slave
It was decided to not let projects directly access the central jenkins master, but instead use some Virtual Machines (shared - like slave01.ci.centos.org - , or dedicated, depending on the justification/need) Each time that a jenkins admin create a slave, it gets a specific key to authenticate against the master and announce itself as "online". To know that key, in Jenkins, just use manage/manage nodes/click on the node (like $project-ci-slave01)/status and notice the command listed with -secret
At the agent side, one needs first to download the jenkins slave jar, and start it : (assuming that jenkins remote root dir is /home/$project/)
cd /home/$project
wget https://ci.centos.org/jnlpJars/slave.jar #only once
java -jar slave.jar -jnlpUrl https://ci.centos.org/computer/${project_slave_name}/slave-agent.jnlp -secret ${secret key from above}TODO : write :
- jenkins-slave init script (for CentOS 6)
- jenkins-slave systemd unit file (for CentOS 7)
Machine access
Each $project can get access to either the "shared" VM (actually slave01.ci.centos.org) or a dedicated VM for the jenkins slave (TBD on a project basis) All shell access are granted to
a common ssh Jumphost (see for example https://wiki.centos.org/TipsAndTricks/SshTips/JumpHost) : jump.ci.centos.org
- the shared or dedicated jenkins slave VM (where they need to configure/put their scripts for the jenkins jobs)
Everything can be granted via Ansible playbooks, actually available from the admin.ci.centos.org:/srv/upstream.git/CentOS_CI/ git repo The playbooks/config-users.yml can be used against specific VMs to add the users, once they are added in the local_users_list, to be modified at the hosts/host_vars/jump.ci.centos.org (and VM name) Attention, be sure that nobody has sudo right on the jump host !
Accounts for the Shared VM
- The users should each be given their user accounts on the jumphost corresponding to their username and ssh key provided in the bug
- Each user's SSH key should be added to the project user account on the shared VM
- When creating a new project, be sure to generate SSH keys for the project user and add those to the duffy user-keys file (see below)
Duffy credentials
Projects will have to use the Duffy middleware (see http://wiki.centos.org/QaWiki/CI/Duffy) to be able to request CentOS 5/6/7 i386/x86_64 nodes to run their jobs (nothing will run directly on the slave VM). The dedicated wiki page list how to use the Duffy middleware and request a node, or give it back to duffy (meaning that Duffy will then reinstall it and put it back in the allocation pool) Each project though need to be "known" by Duffy, and so needs :
- an api key
- ssh public keys that Duffy will need to inject in the node to let the jenkins run connect to that provisioned node.
Duffy uses mysql DB to store all those informations. So a CI admin just needs to add it in the duffy.users db/table (project being replaced with real name):
insert into users values(uuid(),'$project','$project',now(),10);
For the ssh keys, the /home/duffy/user-keys has to contain the $project file containing the ssh pub keys. Duffy will then inject dynamically those public keys into each machine given back through the api call.
Artifacts
Projects can publish artifacts from their builds (e.g. VM images) to a publicly accessible folder (available at http://artifacts.ci.centos.org/). This folder can be created using the storage role in ansible. Each project should be added to the rsync.secrets file (in ansible) using the unique part of their duffy key as a password, for example if the duffy key is 9c67d9c6-b5e2-11e4-b2af-525400ea212d their rsync password would be 9c67d9c6-b5e2. An rsync target should be created using ansible named according to the project.
Projects can sync whatever they wish to their new folder:
RSYNC_PASSWORD='9c67d9c6-b5e2' rsync -av files-in-workdir-to-sync/ <project>@artifacts.ci.centos.org::<project>/