Skip to content

AWS S3 Backup fails with SSL_CTX_load_verify_locations #3010

Closed
Closed
AWS S3 Backup fails with SSL_CTX_load_verify_locations#3010

Description

@fhaefemeier
fhaefemeier
opened on Sep 25, 2024

I configured automatic backups with storing the backup archive in AWS S3 (self hosted minio cluster). After upgrade the docker image from 17.1.1 to 17.3.2 the backup configuration fails with a SSL error SSL_CTX_load_verify_locations. The configuration is not changed. Server certificate of minio cluster is a Let's Encrypt certificate.

Configuration:

GITLAB_BACKUP_SCHEDULE: daily
GITLAB_BACKUP_TIME: "23:00"
GITLAB_BACKUP_SKIP: "uploads,artifacts,packages,builds,lfs,registry,ci_secure_files,terraform_state"
GITLAB_BACKUP_EXPIRY: 302400
AWS_BACKUPS: "true"
AWS_BACKUP_REGION: "de-local-1"
AWS_BACKUP_BUCKET: "gitlab-backup"
AWS_BACKUP_ACCESS_KEY_ID: <masked>
AWS_BACKUP_SECRET_ACCESS_KEY: <masked>
AWS_BACKUP_ENDPOINT: https://main.storage.<domain>

Log extract:

2024-09-23 23:00:33 +0200 -- Creating backup archive: 1727125223_2024_09_23_17.3.2_gitlab_backup.tar ... 
rake aborted!
Excon::Error::Socket: SSL_CTX_load_verify_locations: system lib (OpenSSL::SSL::SSLError)
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/ssl_socket.rb:139:in `initialize'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/ssl_socket.rb:139:in `new'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/ssl_socket.rb:139:in `initialize'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/connection.rb:474:in `new'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/connection.rb:474:in `socket'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/connection.rb:121:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/middlewares/mock.rb:57:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/middlewares/instrumentor.rb:34:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/middlewares/idempotent.rb:19:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/middlewares/base.rb:22:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/middlewares/base.rb:22:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/excon-0.99.0/lib/excon/connection.rb:286:in `request'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/fog-xml-0.1.3/lib/fog/xml/sax_parser_connection.rb:35:in `request'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/fog-xml-0.1.3/lib/fog/xml/connection.rb:7:in `request'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/fog-aws-3.18.0/lib/fog/aws/storage.rb:677:in `_request'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/fog-aws-3.18.0/lib/fog/aws/storage.rb:672:in `request'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/fog-aws-3.18.0/lib/fog/aws/requests/storage/initiate_multipart_upload.rb:29:in `initiate_multipart_upload'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/fog-aws-3.18.0/lib/fog/aws/models/storage/file.rb:324:in `multipart_save'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/fog-aws-3.18.0/lib/fog/aws/models/storage/file.rb:279:in `save'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/fog-core-2.1.0/lib/fog/core/collection.rb:50:in `create'
/home/git/gitlab/lib/backup/remote_storage.rb:26:in `upload'
/home/git/gitlab/lib/backup/manager.rb:279:in `upload'
/home/git/gitlab/lib/backup/manager.rb:151:in `run_all_create_tasks'
/home/git/gitlab/lib/backup/manager.rb:32:in `create'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:12:in `block in create_backup'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:75:in `lock_backup'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:10:in `create_backup'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:117:in `block (3 levels) in <top (required)>'
/home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'

Activity

  1. ymazzer commented on Sep 27, 2024

    @ymazzer
    ymazzer
    on Sep 27, 2024
    Contributor

    Same here, thought it was linked to openssl3 upcoming requirements.

    Any idea

  2. dimm0 commented on Sep 30, 2024

    @dimm0
    dimm0
    on Sep 30, 2024

    Same here with google s3

  3. ntnamazu commented on Sep 30, 2024

    @ntnamazu
    ntnamazu
    on Sep 30, 2024

    I have seen the same error message and a workaround like below to add symbolic link was posted at GitLab Forum. The post said that "Ruby is looking for /usr/lib/ssl/cert.pem and it wasn’t there". This method quite worked for me.

    ln -s /etc/ssl/certs/ca-certificates.crt /usr/lib/ssl/cert.pem

    Also, I implemented this workaround in PR #3016 as commit d8d7804, so if the PR is approved and merged, I think this issue would be resolved.

  4. ymazzer commented on Oct 1, 2024

    @ymazzer
    ymazzer
    on Oct 1, 2024
    Contributor

    I have seen the same error message and a workaround like below to add symbolic link was posted at GitLab Forum. The post said that "Ruby is looking for /usr/lib/ssl/cert.pem and it wasn’t there". This method quite worked for me.

    ln -s /etc/ssl/certs/ca-certificates.crt /usr/lib/ssl/cert.pem

    Also, I implemented this workaround in PR #3016 as commit d8d7804, so if the PR is approved and merged, I think this issue would be resolved.

    Just tested, works well. Thanks for the tip!

    Btw, would have been better to keep the fix in another PR.

  5. ymazzer
    added a commit that references this issue on Oct 24, 2024

    fix(sameersbn#3010) Add symbolic link to the correct location of ca-c…

    199070b
  6. sachilles
    added 3 commits that reference this issue on Oct 25, 2024

    Merge pull request #3027 from datailor-fr/f/3010-fix-ssl-issue

    8ce157e

    fix(#3010) Add symbolic link to the correct location of ca-cert file

    a8caafc

    fix(#3010) Add symbolic link to the correct location of ca-cert file

    9d3a90d

  7. ymazzer commented on Nov 27, 2024

    @ymazzer
    ymazzer
    on Nov 27, 2024
    Contributor

    @sachilles this issue is fixed, you can close it.

  8. sachilles commented on Nov 28, 2024

    @sachilles
    sachilles
    on Nov 28, 2024
    Collaborator

    @ymazzer Thanks for your reminder!

  9. sachilles
    closed this as completedon Nov 28, 2024
  10. kkimurak
    added a commit that references this issue on May 8, 2025

    Revert "fix(sameersbn#3010) Add symbolic link to the correct location…

    062075f
  11. sachilles
    added a commit that references this issue on May 15, 2025

    Remove symbolic link to the correct location of ca-cert file

    3d07804
  12. kkimurak
    added a commit that references this issue on May 20, 2025

    Remove symbolic link to the correct location of ca-cert file

    747e904
  13. kkimurak
    added a commit that references this issue on Jun 16, 2025

    Remove symbolic link to the correct location of ca-cert file

    5033aa2
  14. Jack12816
    added 2 commits that reference this issue on Dec 1, 2025

    fix(sameersbn#3010) Add symbolic link to the correct location of ca-c…

    d452060

    Remove symbolic link to the correct location of ca-cert file

    702d97b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions