10

I have a list of CI jobs running in my GitLab and the Caching does not work as expected:

This is how my docu-generation job ends:

[09:19:33] Documentation generated in ./documentation/ in 4.397 seconds using gitbook theme
Creating cache angular...
00:02
WARNING: frontend/node_modules: no matching files  
frontend/documentation: found 136 matching files   
No URL provided, cache will be not uploaded to shared cache server. Cache will be stored only locally. 
Created cache
Job succeeded

I then start a deployment Job (to GitLab Pages) but it fails because it doesn't find the documentation-folder:

$ cp -r frontend/documentation .public/frontend
cp: cannot stat 'frontend/documentation': No such file or directory

this is the cache config of the generation:

generate_docu_frontend:
  image: node:12.19.0
  stage: build
  cache:
    key: angular
    paths:
      - frontend/node_modules
      - frontend/documentation
  needs: ["download_angular"]

and this is for deployment:

deploy_documentation:
  stage: deploy
  cache:
    - key: angular
      paths:
        - frontend/node_modules
        - frontend/documentation
      policy: pull
    - key: laravel
      paths:
        - backend/vendor
        - backend/public/docs
      policy: pull

does anyone know why my documentation folder is missing?

Improve this question
2
  • 2
    Looks like you have an issue in your shared cache config… No URL provided, cache will be not uploaded to shared cache server.. As a quick fix you could ensure that the same runner is used for both jobs. Commented Sep 23, 2021 at 11:32
  • Btw it sounds to me as if you would better use artifacts here. Commented Sep 23, 2021 at 11:33

1 Answer 1

Reset to default
13

The message in your job output No URL provided, cache will be not uploaded to shared cache server. Cache will be stored only locally. just means that your runners are not using Amazon S3 to store your cache, or something similar like Minio.

Without S3/Minio, the cache only lives on the runner that first ran the job and cached the resources. This means that the next time the job runs and it happens to be picked up by a different runner, it won't have the cache. In that case, you'd run into an error like this.

There's a couple ways around this:

  1. Configure your runners to use S3/Minio (Minio is an open source, free-to-use license if you're interested in hosting it yourself).
  2. Only use one runner (not a great solution since generally more runners means faster pipelines and this would slow things down considerably, though it would solve the cache problem).
  3. Use tags. Tags are used to ensure that a job runs on a specific runner(s). Let's say for example that 1 out of your 10 runners have access to your production servers, but all have access to your lower environment servers. Your lower-env jobs can run on any runner, but your Production Deployment job has to run on the one runner with prod access. You can do this by putting a Tag on the runner called let's say prod-access and putting the same tag on the prod deploy job. This will ensure that job will run on the runner with prod access. The same thing can be used here to ensure the cache is available.
  4. Use artifacts instead of cache. I'll explain this option below as it's really what you should be using for this use case.

Let's briefly explain the difference between Cache and Artifacts:

  • Cache is generally best used for dependency installation like npm or composer (for PHP projects). When you have a job that runs npm ci or composer install, you don't want it to run every since time your pipeline runs when you don't necessary change the dependencies as it wastes time. Use the cache keyword to cache the dependencies so that subsequent pipelines don't have to install the dependencies again.

  • Artifacts are best used when you need to share files or directories between jobs in the same pipeline. For example, after installing npm dependencies, you might need to use the node_modules directory in another job in the pipeline. Artifacts are also uploaded to the GitLab server by the runner at the end of the job, opposed to being stored locally on the runner that ran the job. All previous artifacts will be downloaded for all subsequent jobs, unless controlled with either dependencies or needs.

Artifacts are the better choice for your use case.

Let's update your .gitlab-ci.yml file to use artifacts instead of cache:

stages:
  - build
  - deploy

generate_docu_frontend:
  image: node:12.19.0
  stage: build
  script:
    - ./generate_docs.sh # this is just a representation of whatever steps you run to generate the docs
  artifacts:
    paths:
      - frontend/node_modules
      - frontend/documentation
    expire_in: 6 hours # your GitLab instance will have a default, you can override it like this
    when: on_success # don't attempt to upload the docs if generating them failed

deploy_documentation:
  stage: deploy
  script:
    - ls # just an example showing that frontend/node_modules and frontend/documentation are present
    - deploy.sh # whatever else you need to run this job
Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

Did you find any gitlab documentation for the bit about "just means that your runners are not using Amazon S3 to store your cache, or something similar like Minio."? Was having trouble debugging this using the gitlab docs
It doesn’t explicitly mention this warning, but the docs for shared/distributed are here. It does talk about distributes caching vs. caching on the runners directly.
Your distinction between artifacts and cache are almost identical. node_modules are the dependencies installed by npm ci, so I'm not sure what you would be cacheing if not that. From the url you linked Use cache for dependencies, like packages you download from the internet... and Use artifacts to pass intermediate build results between stages... ie, your built deployable/image/bin/executable etc... I'd argue it'd be faster to use cache than using artifacts since the runner can request from cache faster than it would from the gitlab host in most cases.
How I see it, based on my understanding of the term "Cache": Cache: Anything that is nice to have and makes your jobs faster. Your pipeline survives if the cache is missing. Artifacts: Anything that, if missing, would break your pipeline beyond recovery. If node_modules are absolutely required and can't be generated in your next pipeline step, use artifacts, because they are guaranteed to be there (as explained above)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.