Recently, I've discussed the benefits of putting microservices for a single application in a common repository. Let's now talk about how to deal with CI/CD. I'll focus on Gitlab, but similar techniques can be used for other tools.
Launching the pipelines
Our services are in the same repository now, and each has a pipeline. Each pipeline is specific to a service. Gitlab allows only one pipeline per project.
authentication/ ├──.gitlab-ci.yml catalog/ ├──.gitlab-ci.yml gateway/ ├──.gitlab-ci.yml order-backend/ ├──.gitlab-ci.yml order-ui/ ├──.gitlab-ci.yml
There must be the main pipeline that includes the pipelines from each service. Gitlab has the include
directive for that.
stages: - build - test - package - publish - deploy include: - local: 'authentication/.gitlab-ci.yml' - local: 'catalog/.gitlab-ci.yml' - local: 'gateway/.gitlab-ci.yml' - local: 'order-backend/.gitlab-ci.yml' - local: 'order-ui/.gitlab-ci.yml'
Tweaking each service pipeline
Actually, it's not that simple and a few modifications have to be done to each pipeline in order to make things work.
If a service pipeline uses
paths as in the
cache directives, these must be adapted to the new folder hierarchy and be prefixed by the service name (as it is the folder name)
cache: key: "order-backend-$CI_BUILD_REF_NAME" policy: push paths: - order-backend/.gradle - order-backend/build
It is also a good idea to prefix things like
keys too in order to avoid naming conflicts
scripts usually contain commands which are dependant on the working directory. One way to fix this would be to change the paths used in every command, but that is not always possible since some commands are path-relative. An easier and less intrusive way would be to
cd into the service directory before executing the rest of the
Since this has to be done for every
job, we can mutualize this by extending a common
.order-backend: before_script: - cd order-backend only: &changes changes: - order-backend/**/* order-backend-build-gradle: extends: .order-backend ...
job in the service
gitlab-ci.yml file should
extends the common hidden
A good technique to have a readable execution graph would be to prefix each job by the name of the service it is used for.
Making pipelines run only when needed
I've also declared in the
job that it should be executed only when changes are detected on
order-backend/**/*. This is clearly an optimization for not running the pipeline if nothing has changed since the last push on this branch.
New branches are insensitive to the
changes directive. The whole pipeline is, therefore, run when creating a new branch, which can take a long time to initialize and have an impact on runner loads.
One way to avoid this (though I've not tested it), would be to create a new branch without any modifications and push it with
-o ci.skip option, forcing the pipeline to be skipped. The following pushes would then check the
changes from then.
One should give extra attention as to not merge into the
master branch things that break the pipeline, as that would break the pipeline for all services. It would probably be a good idea to restrict pushes to
master to the minimum or only to things which are not built (ie. documentation) and prefer to use branches. Gitlab allows creating a Merge Request and merging it directly if the pipeline succeeds.
If the branching and merging strategy are solid, then some jobs can be specific to certain branches. For example, tests can only be run only on branches since everything that is merged into
master has already been tested. Branches can use Review Apps whereas tags can publish artifacts;
master may be used to deploy to production if there is continuous deployment.
Gitlab is working on better integration with monorepos. They even have an epic for it in their backlog. Keep up to date to see how monorepos are handled by then.