Recently, I've discussed the benefits of putting microservices for a single application in a common repository. Let's now talk about how to deal with CI/CD. I'll focus on Gitlab, but similar techniques can be used for other tools.

Launching the pipelines

Our services are in the same repository now, and each has a pipeline. Each pipeline is specific to a service. Gitlab allows only one pipeline per project.

authentication/
├──.gitlab-ci.yml
catalog/
├──.gitlab-ci.yml
gateway/
├──.gitlab-ci.yml
order-backend/
├──.gitlab-ci.yml
order-ui/
├──.gitlab-ci.yml

There must be the main pipeline that includes the pipelines from each service. Gitlab has the include directive for that.

stages:
  - build
  - test
  - package
  - publish
  - deploy

include:
  - local: 'authentication/.gitlab-ci.yml'
  - local: 'catalog/.gitlab-ci.yml'
  - local: 'gateway/.gitlab-ci.yml'
  - local: 'order-backend/.gitlab-ci.yml'
  - local: 'order-ui/.gitlab-ci.yml'

Tweaking each service pipeline

Actually, it's not that simple and a few modifications have to be done to each pipeline in order to make things work.

If a service pipeline uses paths as in the artifacts and cache directives, these must be adapted to the new folder hierarchy and be prefixed by the service name (as it is the folder name)

  cache:
    key: "order-backend-$CI_BUILD_REF_NAME"
    policy: push
    paths:
      - order-backend/.gradle
      - order-backend/build

It is also a good idea to prefix things like keys too in order to avoid naming conflicts

scripts usually contain commands which are dependant on the working directory. One way to fix this would be to change the paths used in every command, but that is not always possible since some commands are path-relative. An easier and less intrusive way would be to cd into the service directory before executing the rest of the script.

Since this has to be done for every job, we can mutualize this by extending a common job

.order-backend:
  before_script:
    - cd order-backend
  only: &changes
    changes:
      - order-backend/**/*

order-backend-build-gradle:
  extends: .order-backend
  ...

Every job in the service gitlab-ci.yml file should extends the common hidden job.

A good technique to have a readable execution graph would be to prefix each job by the name of the service it is used for.

Making pipelines run only when needed

I've also declared in the .order-backend job that it should be executed only when changes are detected on order-backend/**/*. This is clearly an optimization for not running the pipeline if nothing has changed since the last push on this branch.

Caveats

New branches are insensitive to the changes directive. The whole pipeline is, therefore, run when creating a new branch, which can take a long time to initialize and have an impact on runner loads.

One way to avoid this (though I've not tested it), would be to create a new branch without any modifications and push it with -o ci.skip option, forcing the pipeline to be skipped. The following pushes would then check the changes from then.

One should give extra attention as to not merge into the master branch things that break the pipeline, as that would break the pipeline for all services. It would probably be a good idea to restrict pushes to master to the minimum or only to things which are not built (ie. documentation) and prefer to use branches. Gitlab allows creating a Merge Request and merging it directly if the pipeline succeeds.

If the branching and merging strategy are solid, then some jobs can be specific to certain branches. For example, tests can only be run only on branches since everything that is merged into master has already been tested. Branches can use Review Apps whereas tags can publish artifacts; master may be used to deploy to production if there is continuous deployment.

Ongoing work

Gitlab is working on better integration with monorepos. They even have an epic for it in their backlog. Keep up to date to see how monorepos are handled by then.