Skip to content

getting 503 with 17.9.0 #3073

Description

@th-2021

Upgrading to 17.9.0 shows always 503.

My setup uses relative_url. I'm using nginx-ingress as reverse proxy.

The services are all running. There are no errors shown in the log.

Activity

  1. kkimurak commented on Feb 25, 2025

    @kkimurak
    Contributor

    Could not reproduced on my environment (also using relative_url but no reverse proxy and no https).

    Does the following command report any logs?

    $ docker exec -it <your-gitlab-container> supervisorctl tail gitlab_extensions:nginx
  2. th-2021 commented on Feb 25, 2025

    @th-2021
    ContributorAuthor
  3. kkimurak commented on Feb 25, 2025

    @kkimurak
    Contributor

    Hmm, I tried enabling self-signed SSL to force nginx to load another recently updated configuration (gitlab-ssl) and see if anything would happen, but the gitlab web UI appeared fine. It may be a proxy setting or nginx version issue, but I'm not sure.
    I'm sorry, but I don't think there's much I can do to help you with this.

  4. th-2021 commented on Feb 25, 2025

    @th-2021
    ContributorAuthor
  5. th-2021 commented on Feb 26, 2025

    @th-2021
    ContributorAuthor

    I found a difference between 17.8 and 17.9:

    17:8:

    root@prod-gitlab-5db74f4c59-n5lgc:/home/git/gitlab# curl  http://10.3.154.30/gitlab/users/sign_in
    <html>
    <head><title>301 Moved Permanently</title></head>
    <body>
    <center><h1>301 Moved Permanently</h1></center>
    <hr><center>nginx</center>
    </body>
    </html>
    

    17.9:

    root@staging-gitlab-6fd8896445-nhg4q:/home/git/gitlab# curl http://10.3.133.145/gitlab
    
    <!DOCTYPE html>
    <html>
    <head>
      <meta content="width=device-width, initial-scale=1, maximum-scale=1" name="viewport">
      <title>The page you're looking for could not be found (404)</title>
    ---
    

    I'm using port 80 for readiness probe.

  6. kkimurak commented on Feb 26, 2025

    @kkimurak
    Contributor

    If you get a 404 error, the settings may not have been loaded correctly.
    Since 17.9.0, the configuration file used at runtime has been renamed from /etc/nginx/sites-enabled/gitlab to /etc/nginx/conf.d/gitlab.conf (as part of work #3064).
    Are your nginx configurations loaded correctly? Could you please check if the output of nginx -T includes it?

    output on my env running sameersbn/gitlab:17.9.0
    nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
    nginx: configuration file /etc/nginx/nginx.conf test is successful
    # configuration file /etc/nginx/nginx.conf:
    
    user  nginx;
    worker_processes 1;
    
    error_log  /var/log/nginx/error.log notice;
    pid        /var/run/nginx.pid;
    
    
    events {
        worker_connections  1024;
    }
    
    
    http {
        include       /etc/nginx/mime.types;
        default_type  application/octet-stream;
    
        log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"';
    
        access_log  /var/log/nginx/access.log  main;
    
        sendfile        on;
        #tcp_nopush     on;
    
        keepalive_timeout  65;
    
        #gzip  on;
    
        include /etc/nginx/conf.d/*.conf;
    }
    
    # configuration file /etc/nginx/mime.types:
    
    types {
        text/html                                        html htm shtml;
        text/css                                         css;
        text/xml                                         xml;
        image/gif                                        gif;
        image/jpeg                                       jpeg jpg;
        application/javascript                           js;
        application/atom+xml                             atom;
        application/rss+xml                              rss;
    
        text/mathml                                      mml;
        text/plain                                       txt;
        text/vnd.sun.j2me.app-descriptor                 jad;
        text/vnd.wap.wml                                 wml;
        text/x-component                                 htc;
    
        image/avif                                       avif;
        image/png                                        png;
        image/svg+xml                                    svg svgz;
        image/tiff                                       tif tiff;
        image/vnd.wap.wbmp                               wbmp;
        image/webp                                       webp;
        image/x-icon                                     ico;
        image/x-jng                                      jng;
        image/x-ms-bmp                                   bmp;
    
        font/woff                                        woff;
        font/woff2                                       woff2;
    
        application/java-archive                         jar war ear;
        application/json                                 json;
        application/mac-binhex40                         hqx;
        application/msword                               doc;
        application/pdf                                  pdf;
        application/postscript                           ps eps ai;
        application/rtf                                  rtf;
        application/vnd.apple.mpegurl                    m3u8;
        application/vnd.google-earth.kml+xml             kml;
        application/vnd.google-earth.kmz                 kmz;
        application/vnd.ms-excel                         xls;
        application/vnd.ms-fontobject                    eot;
        application/vnd.ms-powerpoint                    ppt;
        application/vnd.oasis.opendocument.graphics      odg;
        application/vnd.oasis.opendocument.presentation  odp;
        application/vnd.oasis.opendocument.spreadsheet   ods;
        application/vnd.oasis.opendocument.text          odt;
        application/vnd.openxmlformats-officedocument.presentationml.presentation
                                                         pptx;
        application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
                                                         xlsx;
        application/vnd.openxmlformats-officedocument.wordprocessingml.document
                                                         docx;
        application/vnd.wap.wmlc                         wmlc;
        application/wasm                                 wasm;
        application/x-7z-compressed                      7z;
        application/x-cocoa                              cco;
        application/x-java-archive-diff                  jardiff;
        application/x-java-jnlp-file                     jnlp;
        application/x-makeself                           run;
        application/x-perl                               pl pm;
        application/x-pilot                              prc pdb;
        application/x-rar-compressed                     rar;
        application/x-redhat-package-manager             rpm;
        application/x-sea                                sea;
        application/x-shockwave-flash                    swf;
        application/x-stuffit                            sit;
        application/x-tcl                                tcl tk;
        application/x-x509-ca-cert                       der pem crt;
        application/x-xpinstall                          xpi;
        application/xhtml+xml                            xhtml;
        application/xspf+xml                             xspf;
        application/zip                                  zip;
    
        application/octet-stream                         bin exe dll;
        application/octet-stream                         deb;
        application/octet-stream                         dmg;
        application/octet-stream                         iso img;
        application/octet-stream                         msi msp msm;
    
        audio/midi                                       mid midi kar;
        audio/mpeg                                       mp3;
        audio/ogg                                        ogg;
        audio/x-m4a                                      m4a;
        audio/x-realaudio                                ra;
    
        video/3gpp                                       3gpp 3gp;
        video/mp2t                                       ts;
        video/mp4                                        mp4;
        video/mpeg                                       mpeg mpg;
        video/quicktime                                  mov;
        video/webm                                       webm;
        video/x-flv                                      flv;
        video/x-m4v                                      m4v;
        video/x-mng                                      mng;
        video/x-ms-asf                                   asx asf;
        video/x-ms-wmv                                   wmv;
        video/x-msvideo                                  avi;
    }
    
    # configuration file /etc/nginx/conf.d/gitlab.conf:
    ## GitLab
    ##
    ## Lines starting with two hashes (##) are comments with information.
    ## Lines starting with one hash (#) are configuration parameters that can be uncommented.
    ##
    ##################################
    ##        CONTRIBUTING          ##
    ##################################
    ##
    ## If you change this file in a Merge Request, please also create
    ## a Merge Request on https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests
    ##
    ###################################
    ##         configuration         ##
    ###################################
    ##
    ## See installation.md#using-https for additional HTTPS configuration details.
    
    upstream gitlab-workhorse {
      server localhost:8181 fail_timeout=0;
    }
    
    map $http_upgrade $connection_upgrade_gitlab {
        default upgrade;
        ''      close;
    }
    
    ## Obfuscate access_token and private_token in access log
    map $request_uri $obfuscated_request_uri {
        ~(.+\?)(.*&)?(private_token=|access_token=)[^&]*(&.*|$) $1$2$3****$4;
        default $request_uri;
    }
    log_format gitlab_access '$remote_addr - $remote_user [$time_local] '
                      '"$request_method $obfuscated_request_uri $server_protocol" $status $body_bytes_sent '
                      '"$http_referer" "$http_user_agent"';
    
    ## Normal HTTP host
    server {
      ## Either remove "default_server" from the listen line below,
      ## or delete the /etc/nginx/sites-enabled/default file. This will cause gitlab
      ## to be served if you visit any address that your server responds to, eg.
      ## the ip address of the server (http://x.x.x.x/)n 0.0.0.0:80 default_server;
      listen 0.0.0.0:80 default_server;
      listen [::]:80 default_server;
      server_name [manually masked]; ## Replace this with something like gitlab.example.com
      server_tokens off; ## Don't show the nginx version number, a security best practice
    
      ## See app/controllers/application_controller.rb for headers set
    
      ## Real IP Module Config
      ## http://nginx.org/en/docs/http/ngx_http_realip_module.html
      real_ip_header X-Real-IP; ## X-Real-IP or X-Forwarded-For or proxy_protocol
      real_ip_recursive off;    ## If you enable 'on'
      ## If you have a trusted IP address, uncomment it and set it
    
      add_header X-Accel-Buffering no;
    
      ## Individual nginx logs for this GitLab vhost
      access_log  /var/log/gitlab/nginx/gitlab_access.log gitlab_access;
      error_log   /var/log/gitlab/nginx/gitlab_error.log;
    
      location / {
        client_max_body_size 0;
        gzip off;
    
        ## https://github.com/gitlabhq/gitlabhq/issues/694
        ## Some requests take more than 30 seconds.
        proxy_read_timeout      300;
        proxy_connect_timeout   300;
        proxy_redirect          off;
        proxy_buffering         off;
    
        proxy_http_version 1.1;
    
        proxy_set_header    Host                $http_host;
        proxy_set_header    X-Real-IP           $remote_addr;
        proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
        proxy_set_header    X-Forwarded-Proto   $scheme;
        proxy_set_header    Upgrade             $http_upgrade;
        proxy_set_header    Connection          $connection_upgrade_gitlab;
    
        proxy_pass http://gitlab-workhorse;
      }
    
      error_page 404 /404.html;
      error_page 422 /422.html;
      error_page 500 /500.html;
      error_page 502 /502.html;
      error_page 503 /503.html;
      location ~ ^/(404|422|500|502|503)\.html$ {
        root /home/git/gitlab/public;
        internal;
      }
    
    }
    
  7. th-2021 commented on Feb 26, 2025

    @th-2021
    ContributorAuthor

    looks like the gitlab-pages config overrides the gitlab config.

  8. th-2021 commented on Feb 27, 2025

    @th-2021
    ContributorAuthor

    Thanks for helping me tracking down the issue. I can now workaround it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions