This document is a notes version of a talk given during a workshop about continous integration for scientific software at Institut Henri Poincaré in Paris, France. This workshop has been organized by the french group : calcul@listes.math.cnrs.fr and was about git, docker, jenkins, gitlab and travis.
We have seen major features of gitlab with an emphasis on continuous integration.
There is an other document about git that is complementary with this one (in French) but which is an adpatation of the slides I have shown in June 2015 at CentraleSupélec. Again it is more a notes version than a well elaborated document.
For the session, we worked on a template code based on django but you could used any other code or pseudocode you wanted.
conda create -y -n django python
source activate django
conda install -y django
django-admin startproject myproject
cd myproject
./manage.py startapp website
echo "class MyTests(TestCase):
def test_1(self):
self.assertEqual(1,1)" > website/tests.py
./manage.py test
function add(argv):
return argv[0] + argv[1]
We have seen the tasks below :
git lfs install
git lfs track "*.hdf5"
git add .gitattributes
git add data.hdf5
git commit -m "Add data file"
git push origin master
#...
git pull origin master
git lfs checkout
https://docs.gitlab.com/ce/ci/
https://docs.gitlab.com/ce/ci/quick_start/
https://docs.gitlab.com/runner/install/
Here is the output of the installation and registration of a runner. You need to find the coordinator URL and token registration on your gitlab instances in your project/settings or in the global administration panel.
sudo apt-get install gitlab-ci-multi-runner
sudo gitlab-ci-multi-runner register
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com )
https://gitlab.com
Please enter the gitlab-ci token for this runner
xxx
Please enter the gitlab-ci description for this runner
my-runner
INFO[0034] fcf5c619 Registering runner... succeeded
Please enter the executor: shell, docker, docker-ssh, ssh?
docker
Please enter the Docker image (eg. ruby:2.1):
ruby:2.1
INFO[0037] Runner registered successfully. Feel free to start it, but if it's
running already the config should be automatically reloaded!
https://docs.gitlab.com/ce/ci/yaml/
image | no | Use docker image, covered in Use Docker |
services | no | Use docker services, covered in Use Docker |
stages | no | Define build stages |
types | no | Alias for stages (deprecated) |
before_script | no | Define commands that run before each job’s script |
after_script | no | Define commands that run after each job’s script |
variables | no | Define build variables |
cache | no | Define list of files that should be cached between subsequent runs |
script | yes | Defines a shell script which is executed by Runner |
image | no | Use docker image, covered in Using Docker Images |
services | no | Use docker services, covered in Using Docker Images |
stage | no | Defines a job stage (default: test) |
type | no | Alias for stage |
variables | no | Define job variables on a job level |
only | no | Defines a list of git refs for which job is created |
except | no | Defines a list of git refs for which job is not created |
tags | no | Defines a list of tags which are used to select Runner |
allow_failure | no | Allow job to fail. Failed job doesn’t contribute to commit status |
when | no | Define when to run job. Can be on_success, on_failure, always or manual |
dependencies | no | Define other jobs that a job depends on so that you can pass artifacts between them |
artifacts | no | Define list of job artifacts |
cache | no | Define list of files that should be cached between subsequent runs |
before_script | no | Override a set of commands that are executed before job |
after_script | no | Override a set of commands that are executed after job |
environment | no | Defines a name of environment to which deployment is done by this job |
coverage | no | Define code coverage settings for a given job |
https://docs.gitlab.com/ce/ci/pipelines.html
https://docs.gitlab.com/ce/ci/variables
https://docs.gitlab.com/ce/ci/docker/using_docker_images.html
https://docs.gitlab.com/ce/ci/docker/using_docker_build.html
hello_world:
#script is the only mandatory keyword
script:
- echo "Hello World"
#tags will help to assign a job to a particular runner or a set of runners
tags:
- shell
In this script I install dependencies and execute tests.
run:
script:
- wget https://bootstrap.pypa.io/get-pip.py
- python get-pip.py --user
- /home/gitlab-runner/.local/bin/pip install --user django
- ./manage.py test
tags:
- shell
In this script I add stages for creating a pipeline of jobs with only test being used.
stages:
- test
- deploy
run_test:
stage: test
script:
- wget https://bootstrap.pypa.io/get-pip.py
- python get-pip.py --user
- /home/gitlab-runner/.local/bin/pip install --user django
- ./manage.py test
tags:
- shell
In this script I do use test and deploy stages. deploy stage is using the registry for pushing a new docker image. You can find information on the registry in the registry section of your project.
stages:
- test
- deploy
run_test:
stage: test
script:
- wget https://bootstrap.pypa.io/get-pip.py
- python get-pip.py --user
- /home/gitlab-runner/.local/bin/pip install --user django
- ./manage.py test
tags:
- shell
run_deploy:
stage: deploy
script:
#Here I use a login mechanism given by gitlab. The user is gitlab-ci-token and the password is given by the variable $CI_JOB_TOKEN
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.math.unistra.fr
- docker build -t registry.math.unistra.fr/bayol/tp-gitlab/image:latest .
- docker push registry.math.unistra.fr/bayol/tp-gitlab/image:latest
tags:
- shell
Here we add some variables by using some global variables that are set by the gitlab platform like $CI_COMMIT_REF_NAME
stages:
- build
- test
- release
variables:
CONTAINER_TEST_IMAGE: registry.math.unistra.fr/bayol/tp-gitlab/image:$CI_COMMIT_REF_NAME
CONTAINER_RELEASE_IMAGE: registry.math.unistra.fr/bayol/tp-gitlab/image:latest
before_script:
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.math.unistra.fr
build:
stage: build
script:
- docker build --pull -t $CONTAINER_TEST_IMAGE .
- docker push $CONTAINER_TEST_IMAGE
tags:
- shell
test1:
stage: test
script:
- docker pull $CONTAINER_TEST_IMAGE
- docker run $CONTAINER_TEST_IMAGE /manage.py test
tags:
- shell
release-image:
stage: release
script:
- docker pull $CONTAINER_TEST_IMAGE
- docker tag $CONTAINER_TEST_IMAGE $CONTAINER_RELEASE_IMAGE
- docker push $CONTAINER_RELEASE_IMAGE
only:
- master
tags:
- shell
Here we use the pages mechanism for publishing the index.html page that is available in the repository.
pages:
stage: deploy
script:
- mkdir .public
- cp index.html .public
- mv .public public
artifacts:
paths:
- public
only:
- master
tags:
- shell
stages:
- build
- test
- release
- deploy
variables:
CONTAINER_TEST_IMAGE: registry.math.unistra.fr/bayol/tp-gitlab/image:$CI_COMMIT_REF_NAME
CONTAINER_RELEASE_IMAGE: registry.math.unistra.fr/bayol/tp-gitlab/image:latest
before_script:
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.math.unistra.fr
build:
stage: build
script:
- docker build --pull -t $CONTAINER_TEST_IMAGE .
- docker push $CONTAINER_TEST_IMAGE
tags:
- shell
test1:
stage: test
script:
- docker pull $CONTAINER_TEST_IMAGE
- docker run $CONTAINER_TEST_IMAGE /manage.py test
tags:
- shell
release-image:
stage: release
script:
- docker pull $CONTAINER_TEST_IMAGE
- docker tag $CONTAINER_TEST_IMAGE $CONTAINER_RELEASE_IMAGE
- docker push $CONTAINER_RELEASE_IMAGE
only:
- master
tags:
- shell
# with also another deploy script for giving download to users
# deploy:
# stage: deploy
# script:
# - ./deploy.sh
# only:
# - master
pages:
stage: deploy
script:
- mkdir .public
- cp index.html .public
- mv .public public
artifacts:
paths:
- public
only:
- master
tags:
- shell
Here we use a “docker” runner for using an image of ananconda directly.
image: continuumio/anaconda:4.3.1
stages:
- test
run_test:
stage: test
script:
- conda create -y -n django
- source activate django
- conda install -y django
- ./manage.py test
tags:
- docker
Warning : docker in docker is “hype” but might not be suitable for you. Simpler is better. See https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/
image: docker:latest
services:
- docker:dind
stages:
- build
- test
- release
variables:
CONTAINER_TEST_IMAGE: registry.math.unistra.fr/bayol/tp-gitlab/image:$CI_COMMIT_REF_NAME
CONTAINER_RELEASE_IMAGE: registry.math.unistra.fr/bayol/tp-gitlab/image:latest
before_script:
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.math.unistra.fr
build:
stage: build
script:
- docker build --pull -t $CONTAINER_TEST_IMAGE .
- docker push $CONTAINER_TEST_IMAGE
test1:
stage: test
script:
- docker pull $CONTAINER_TEST_IMAGE
- docker run $CONTAINER_TEST_IMAGE /manage.py test
release-image:
stage: release
script:
- docker pull $CONTAINER_TEST_IMAGE
- docker tag $CONTAINER_TEST_IMAGE $CONTAINER_RELEASE_IMAGE
- docker push $CONTAINER_RELEASE_IMAGE
only:
- master