TeamCity bootstrapping

February 21, 2017 &english @code #teamcity

This repository has born as a solution for the repeating and tedious Continuous Integration servers (TeamCity) installation across the company. It provides a way to quickly bootstrap and run TeamCity, automatically integrated with VCS of choice and setup reporting tool for many languages.

Using this bootstrap solution allows to:

  • Unify the approach to setup CI environment for new and existing projects.
  • Keep CI configuration in project’s source code repository.
  • Save time of technical leader or technical coordinator spent on initial an consequent CI configuration.
  • Skip basic CI configuration steps, as bootstrap provides pre-configured templates for each technology it supports.

Supported platforms and languages

Bootstrap supports both BitBucket and GitHub (even with 2FA enabled) as a VCS.

It can be used to provision CI environment for 5 main platforms:

  • Ruby
    • rspec as testing library
    • bundler and dependency manager
    • rubocop as style check library
    • metric_fu as code quality library
  • Java
    • maven or gradle as project manager
    • checkstyle as style check library
    • findbugs and pmd as code quality library
  • Python
    • pip3 as dependency manager
    • pytest as testing library
    • flake8 as style check library
  • Swift
    • swiftlint as style check library
    • taylor as code quality library
  • .NET
    • Xamarin
    • Core
  • JavaScript
  • PHP
  • Golang
  • Swift

System requirements

Since bootstrap highly depends on Docker and shell scripts, it works well on Linux and MacOS operating systems. Support of .NET uses Windows containers and PowerShell, so it works on Windows 10 and Windows Server 2016. Typical usage scenario suggests running bootstrap on virtual server inside company’s intranet. If you are planning to use your VM only as a CI server, there is no need to ask HD about full-featured Ubuntu LTS installation. HD has special lightweight image (based on Arch Linux) for running Docker-based systems (no additional installation steps are needed).

Installation

Bootstrap script heavily depends on core Linux tools: git, sed, grep, dialog and curl. Use the following shell commands to install dependencies:

  • Install docker and docker-compose according to official installation steps for your OS.
  • MacOS: brew install dialog curl git gnu-sed (make sure GNU version of sed appears earlier in your PATH)
  • Linux: apt-get update && apt-get install -y dialog curl git

Prerequisites:

In order for bootstrap to successfully integrate CI with VCS, you have to have:

  • an administrator access to the project’s source code repository (in case you don’t have such access, please create a HelpDesk request in Jira);
  • know project name (like LifeChurch), BitBucket project name, BitBucket repository name, names of team members you want to add as users of CI;
  • server to host your project’s CI (usually its internal VM, provided by MDIS, but can be an external server as well);
  • a cup of coffee.

Usage

  1. Run ssh-agent locally to cache default ssh key and forward them to the target VM machine. This is needed to give VM access to the bootstrap repository. Make sure you have public key added to your BitBucket account.
  2. Establish SSH connection to the project’s VM: ssh root@project-vm-name -o ForwardAgent=yes. If its typical Ubuntu or Debian machine, you’ll need to install some stuff so it’s better to be a root. In case of Arch, everything is already installed for you, so login as docker user: ssh docker@project-vm-name -o ForwardAgent=yes.
  3. Install prerequisites: apt-get update && apt-get install -y dialog curl git if not installed.
  4. Clone this repository: git clone ssh://git@git.itransition.com:7999/projectalude/teamcity-bootstrap.git.
  5. Enter teamcity-bootstrap cd teamcity-bootstrap folder and make any modifications to the Docker.$PLATFORM file to add/remove software inside TeamCity agent image.
  6. Run ./bootstrap.sh command to start provisioning process.
  7. Answer to the installation wizard’s questions and wait a couple of minutes.
  8. Commit all your changes to the project’s branch and push. You’ll find this helpful on CI server re-setup.

If you are not satisfied with the results, perform any modifications to the configuration files and/or scripts and re-execute ./bootstrap.sh command. Everything will be built from scratch again.

To stop provisioned environment, use following command:

$ docker-compose -f docker-compose.yml -f docker-compose.$PLATFORM.yml -f docker-compose.agents.$TOTAL_AGENTS.yml down

Update

Whenever new version of the TeamCity becomes available, new commits will appear in master branch of this repository. Updating procedure is easy: on your VM, you can merge master branch to your project’s branch and restart docker-compose services. This will build new TeamCity server image, all agent containers will be automatically updated upon start.

$ git checkout master
$ git pull origin master
$ git merge master
$ git checkout project_branch
$ git merge master
$ docker-compose -f docker-compose.yml -f docker-compose.$PLATFORM.yml -f docker-compose.agents.$TOTAL_AGENTS.yml build
$ docker-compose -f docker-compose.yml -f docker-compose.$PLATFORM.yml -f docker-compose.agents.$TOTAL_AGENTS.yml up -d

Passwords expiration

Since BitBucket provides only Basic Auth for using it’s API, TeamCity has to store password from team member internally. Fortunately, there is a way to securely store sensitive data inside TeamCity without exposing it on GUI or inside configuration Kotlin files. When you notice, that TeamCity has stopped reporting build statuses back to BitBucket, run update_commit_status_password.sh script. It will ask for the username (does not have to be the same as on initial bootstrap) and domain password. Script then will update secure password value inside TeamCity and make corresponding changes inside project repository.

Internals

Bootstrap consists of three main components:

  • Installation bash script.
  • Docker compose environment.
  • Template collection for each supported platform.

Installation script

Responsibilities:

  • Orchestration of docker-compose configuration to set run specific containers.
  • Communication with TeamCity via HTTP API to perform initial setup.
  • Communication with VCS (BitBucket or GitHub) to setup access to the code from CI by SSH keys.

To get the idea what exactly installation script is doing, lets see typical log messages:

Destroying previous versions of docker images.
Generating access keys.
Uploading public key to BitBucket.
Executing platform specific scripts.
Downloading PostgreSQL driver.
Waiting for TeamCity server to accept agreement...
Accepting license agreement.
Adding private key to the TeamCity.
Creating users.
Granting admin privileges to the main user.
Creating project.
Generating custom config.
TeamCity configuration dir does not exist, Generating from the template.
Creating VCS.
Enable configuration synchronization.
Authorizing agents.
Scheduling settings update.
Waiting for TeamCity server to accept configuration...
Assigning project to the default agent pool.
Triggering build.
TeamCity instance is ready!
Passwords for users:

docker-compose

docker-compose allows to specify many yml files to operate with. Bootstrap benefits from that feature by splitting configuration into three parts:

  • Main configuration - docker-compose.yml. Contains skeleton list of services: TeamCity server, its database, list of TeamCity agents, database for the application.
  • Agents technology configuration - docker-compose.TECHNOLOGY.yml. By selecting of the configuration files, bootstrap chooses appropriate image for the agent (each technology stack requires own set of tools).
  • Agents count configuration - docker-compose.agents.N.yml. By selecting of the configuration files, bootstrap limits number of TeamCity agents being started.

It wasn’t possible to define only one TeamCity agent service inside docker-compose.yml file and use scale parameter to control agents count. The problem hides in environment variables. Docker does not support specifying different set or even different values to the environment variables of the same service. But this is needed to differentiate agents between each other.

Templates

Template is a folder, which contains TeamCity configuration (build plans, steps to execute during builds, reports to collect, etc) in form of Kotlin files, combined into maven project. Installation script asks user for the technology and uses corresponding branch of the template collection repository. Then, script modifies the template by substituting placeholders for project’s name, repository address and etc. After substitution process, installation scripts commits build configuration to the project repository inside .teamcity folder.

TeamCity detects changes inside .teamcity folder and applies modification to the build process. So, there is no need to re-configure TeamCity using it’s administrative interface, all required changed can be made by editing .kt files.

Since TeamCity configuration is a simple maven project, it is possible to ensure, that modifications were safe (at least compilable) by executing mvn package command inside .teamcity folder before pushing to remote git repository. Loading this project in IntelliJ IDEA gives auto-completion of keys and properties. In case, when you don’t know what to change, it is possible to:

  1. Disable VCS settings synchronization for the project using TeamCity GUI.
  2. Make changes to the project configuration using TeamCity GUI.
  3. Re-enable VCS settings synchronization using TeamCity GUI. TeamCity will prompt to commit all changes back to project’s repository.

Contribution

Special thanks to:

  • Blinov, Alexey - for starting the project and investigation TeamCity configuration abilities.
  • Atroshko, Gennady - for early adoption, finding and fixing defects.
  • Buevich, Pavel - for early adoption, finding and fixing defects.
  • Katlinsky, Ilya - for introducing Gradle support and working on Java template.
  • Pashkovich, Anatoly - for Swift support and ability to work on MacOS VMs.
  • Pyatlin, Aleksey - for .NET support and ability to work on Windows VMs.
  • Sushko, Valery - for password generation feature, many code simplification ideas.
  • Sviridov, Andrei - for Python support, finding and fixing defects.
  • Vasilevsky, Evgeny - for early adoption, finding and fixing defects.

Ideas and Pull Requests are welcome.

Plans

  • Integrate sonar into bootstrap script.
  • Integrate with AD to manage users.
  • Introduce global dashboard.
  • Support branches, other than master for GitFlow lovers.
  • Cache invalidation
  • Why so strange name? It is an anagram for TeamCity Bootstrap.