Using gitlab CI_JOB_TOKEN for including remote projects

At my workplace, we recently migrated all our repositories to GitLab as part of the DevOps rationalizing program. The idea was to adopt uniform tools and practices across the organization, as previously, different teams were using various tools for SCM and build/deployment pipelines.

The migration had a couple of challenges due to missing features like group-level merge permissions and code quality gates (similar to Sonarway). Also, as we had some custom solutions running for many years, finding an alternative for those in the new world was additional overhead.

But even with these limitations, Gitlab as a platform is becoming a great choice with its built-in support for CI/CD. It is no longer just an SCM but a complete DevOps solution. It allows developers to create automated pipelines for building, testing, and deploying applications in addition to hosting the code, documentation, and project artifacts. GitLab CI/CD pipelines are defined in a .gitlab-ci.yml file, which specifies the jobs to be executed and the dependencies between them.

Problem Statement

Consider a scenario where you must include another repository in your build pipeline. One way is to use git submodules, but from my experience, I have found the process a bit cumbersome and difficult to manage in the long run.

Another option is to clone the remote repository as part of the current build process to include it at runtime. The remote repository can be any artifact, such as a jar stored in a package registry or another project. But if the remote repository is private, it cannot be directly cloned, and we need some authentication mechanism to establish our access.

What is CI_JOB_TOKEN?

GitLab provides a unique token for each job in a pipeline called CI_JOB_TOKEN. This token authenticates requests from within a job to other GitLab APIs such as container registry, artifact repository, or other project repositories.

CI_JOB_TOKEN is a feature of GitLab’s secure variables. These variables are encrypted and can only be accessed by the GitLab runner executing the job.

Benefits of using CI_JOB_TOKEN

Using CI_JOB_TOKEN provides several benefits, including:

1. Security: It provides secure authentication for requests made from within a job. This token is unique to each job and can only be used by that job. It ensures that sensitive data, such as access keys or passwords, are not exposed in plain text within the pipeline.

2. Simplified configuration: Using the job token simplifies the pipeline configuration by eliminating the need for hard-coded access keys or passwords. Instead, the token is automatically injected into the job environment, making it easy to use.

3. Fine-grained access control: It provides fine-grained access control by allowing access only to the specific resources required by the job adhering to the least privilege principle.

How to use CI_JOB_TOKEN to clone remote projects?

As mentioned in the official documentation, using this token to clone a remote private repository requires two steps:

1. Add the current project to the Allowlist of remote repositories to be accessed.
   1. Select Main menu > Projects and find your project.
   2. On the left sidebar, select Settings > CI/CD.
   3. Expand Token Access.
   4. Verify Allow access to this project with a CI_JOB_TOKEN is enabled.
   5. Under Allow CI job tokens from the following projects to access this project, add projects to the Allowlist.
2. Update the .gitlab-ci.yml to use the token in the clone command.

git clone https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com/example/others/some-name.git

GitLab will automatically inject the CI_JOB_TOKEN during pipeline execution. The injected token will have the same permission as the user running the job.

That is all for this post. If you want to share any feedback, please drop me an email, or contact me on any social platforms. I’ll try to respond at the earliest. Also, please consider subscribing feed for regular updates.