1

I have GitLab CE setup with my domain (Ubuntu) and the git clone with http not working but ssh works well.

git clone [email protected]:anshad/test.git works.

git clone http://gitlab.mydomain.com/anshad/test.git not working.

Cloning into 'test'...
Username for 'http://gitlab.mydomain.com': anshad
Password for 'http://[email protected]': 
fatal: unable to access 'http://gitlab.mydomain.com/anshad/test.git/': The requested URL returned error: 500

sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production says

System information
System:     Ubuntu 16.04
Current User:   git
Using RVM:  no
Ruby Version:   2.3.1p112
Gem Version:    2.5.1
Bundler Version:1.13.6
Rake Version:   10.5.0
Sidekiq Version:4.2.1

GitLab information
Version:    8.14.0-pre
Revision:   358e814
Directory:  /home/git/gitlab
DB Adapter: postgresql
URL:        http://gitlab.mydomain.com
HTTP Clone URL: http://gitlab.mydomain.com/some-group/some-project.git
SSH Clone URL:  [email protected]:some-group/some-project.git
Using LDAP: no
Using Omniauth: no

GitLab Shell
Version:    4.0.0
Repository storage paths:
- default:  /home/git/repositories/
Hooks:      /home/git/gitlab-shell/hooks/
Git:        /usr/bin/git

sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production says

Checking GitLab Shell ...

GitLab Shell version >= 4.0.0 ? ... OK (4.0.0)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ... 
Anshad Vattapoyil / test ... repository is empty
Running /home/git/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /home/git/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Reply by email ...

Reply by email is disabled in config/gitlab.yml

Checking Reply by email ... Finished

Checking LDAP ...

LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab ...

Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... yes
Init script up-to-date? ... yes
projects have namespace: ... 
Anshad Vattapoyil / test ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.1)
Your git bin path is "/usr/bin/git"
Git version >= 2.7.3 ? ... yes (2.7.4)
Active users: 1

Checking GitLab ... Finished

Apache virtual host configuration for the domain,

<VirtualHost *:80>
    ServerAdmin [email protected]
    ServerName gitlab.mydomain.com
    ServerAlias www.gitlab.mydomain.com

    ServerSignature Off

    ProxyPreserveHost On

    AllowEncodedSlashes NoDecode

    <Location />
       Require all granted
       ProxyPassReverse http://localhost:8080
       ProxyPassReverse http://gitlab.mydomain.com/
    </Location>

    RewriteEngine on

    RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f [OR]
    RewriteCond %{REQUEST_URI} ^/uploads/.*
    RewriteRule .* http://127.0.0.1:8080%{REQUEST_URI} [P,QSA,NE]

    DocumentRoot /home/git/gitlab/public

    LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common_forwarded
    ErrorLog /var/log/apache2/gitlab_error.log
    CustomLog /var/log/apache2/gitlab_forwarded.log common_forwarded
    CustomLog /var/log/apache2/gitlab_access.log combined env=!dontlog
    CustomLog /var/log/apache2/gitlab.log combined
 </VirtualHost>

Config /home/git/gitlab/config/gitlab.yml

production: &base

  gitlab:
    host: gitlab.mydomain.com
    port: 80 
    https: false 
    trusted_proxies:

    email_from: [email protected]
    email_display_name: GitLab
    email_reply_to: [email protected]
    email_subject_suffix: ''

    default_projects_features:
      issues: true
      merge_requests: true
      wiki: true
      snippets: true
      builds: true
      container_registry: true

  incoming_email:
    enabled: false
    address: "gitlab-incoming+%{key}@gmail.com"
    user: "[email protected]"
    password: "[REDACTED]"
    host: "imap.gmail.com"
    port: 993
    ssl: true
    start_tls: false
    mailbox: "inbox"

  artifacts:
    enabled: true
  lfs:
    enabled: true

  gravatar:

  cron_jobs:
    stuck_ci_builds_worker:
      cron: "0 0 * * *"
    expire_build_artifacts_worker:
      cron: "50 * * * *"
    repository_check_worker:
      cron: "20 * * * *"
    admin_email_worker:
      cron: "0 0 * * 0"
    repository_archive_cache_worker:
      cron: "0 * * * *"

  registry:


  gitlab_ci:

  ldap:
    enabled: false
    servers:
      main: 
        label: 'LDAP'
        host: '_your_ldap_server'
        port: 389
        uid: 'sAMAccountName'
        method: 'plain' # "tls" or "ssl" or "plain"
        bind_dn: '_the_full_dn_of_the_user_you_will_bind_with'
        password: '_the_password_of_the_bind_user'
        timeout: 10
        active_directory: true
        allow_username_or_email_login: false
        block_auto_created_users: false
        base: ''
        user_filter: ''
        attributes:
          username: ['uid', 'userid', 'sAMAccountName']
          email:    ['mail', 'email', 'userPrincipalName']
          name:       'cn'
          first_name: 'givenName'
          last_name:  'sn'

  omniauth:
    enabled: false
    allow_single_sign_on: ["saml"]
    block_auto_created_users: true
    auto_link_ldap_user: false
    auto_link_saml_user: false
    external_providers: []
    providers:

  shared:

  satellites:
    path: /home/git/gitlab-satellites/

  repositories:
    storages: # You must have at least a `default` storage path.
      default: /home/git/repositories/
  backup:
    path: "tmp/backups"   
  gitlab_shell:
    path: /home/git/gitlab-shell/
    hooks_path: /home/git/gitlab-shell/hooks/
    upload_pack: true
    receive_pack: true
  git:
    bin_path: /usr/bin/git
    max_size: 20971520 # 20.megabytes
    timeout: 10

  extra:

  rack_attack:
    git_basic_auth:

In /home/git/gitlab/unicorn.rb, I have

worker_processes 4
working_directory "/home/git/gitlab" # available in 0.94.0+
listen "/home/git/gitlab/tmp/sockets/gitlab.socket", :backlog => 1024
listen "127.0.0.1:8080", :tcp_nopush => true

The production.log says only this

Started POST "/ci/api/v1/builds/register.json" for 35.154.26.24 at 2016-11-19 08:16:45 +0000
Started GET "/anshad/test.git/info/refs?service=git-upload-pack" for 137.97.204.60 at 2016-11-19 08:16:47 +0000
Processing by Projects::GitHttpController#info_refs as */*
  Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"anshad", "project_id"=>"test.git"}
Filter chain halted as :authenticate_user rendered or redirected
Completed 401 Unauthorized in 23ms (Views: 0.3ms | ActiveRecord: 1.6ms)
Started POST "/ci/api/v1/builds/register.json" for 35.154.26.24 at 2016-11-19 08:16:48 +0000
Started POST "/ci/api/v1/builds/register.json" for 35.154.26.24 at 2016-11-19 08:16:51 +0000
Started POST "/ci/api/v1/builds/register.json" for 35.154.26.24 at 2016-11-19 08:16:54 +0000
Started GET "/anshad/test.git/info/refs?service=git-upload-pack" for 137.97.204.60 at 2016-11-19 08:16:56 +0000
Processing by Projects::GitHttpController#info_refs as */*
  Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"anshad", "project_id"=>"test.git"}
Filter chain halted as :authenticate_user rendered or redirected
Completed 401 Unauthorized in 25ms (Views: 0.3ms | ActiveRecord: 2.2ms)
Started GET "/anshad/test.git/info/refs?service=git-upload-pack" for 137.97.204.60 at 2016-11-19 08:16:57 +0000
Processing by Projects::GitHttpController#info_refs as */*
  Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"anshad", "project_id"=>"test.git"}
Completed 500 Internal Server Error in 98ms (ActiveRecord: 3.1ms)

JWT::DecodeError (Nil JSON web token):
  lib/gitlab/workhorse.rb:120:in `verify_api_request!'
  app/controllers/projects/git_http_client_controller.rb:154:in `verify_workhorse_api!'
  lib/gitlab/request_profiler/middleware.rb:15:in `call'
  lib/gitlab/middleware/go.rb:16:in `call'


Started POST "/ci/api/v1/builds/register.json" for 35.154.26.24 at 2016-11-19 08:16:57 +0000

Issue on GitLab

Improve this question
1
  • My GitLab was installed using source and nothing solved the issue. Now uninstalled everything and installed GitLab package from about.gitlab.com/downloads/#ubuntu1604 and works fine even with Apache. Commented Nov 19, 2016 at 16:35

2 Answers 2

Reset to default
2
+50

It's important to note that this system referenced in question was built from source code and supported nginx was replaced with Apache (not officially supported by gitlab).
Here is the deal - in the standard nginx config on my system I can see this

upstream gitlab-workhorse {
  server unix:/var/opt/gitlab/gitlab-workhorse/socket;
}

proxy_pass http://gitlab-workhorse;

Which means - it's using socket. Not a network port. If I try to see if the workhorse even listening on network - I will see that it's not.

ps -ef|grep -i workhorse
lsof -p pid

Would not show any network ports open by workhorse pid. So perhaps apache config is incorrect? It should be using socket instead of port?

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Removed everything and installed from package about.gitlab.com/downloads/#ubuntu1604, now it is working fine with Apache.
I have mistaken, I have tried all the possibilities.As you mentioned, the issue has no solution with apache and I should have nginx till gitlab comes with some config.
0

First, double-check your gitlab workhorse version and if it is compatible with your current GitLab installation.

Of all the GitLab issues you reference, the comments on 22484 seem the most promising:

In my case, workhorse's logs showed an error accessing ./.gitlab_workhorse_secret

After some digging, the fix was to add the following to the workhorse startup command line in /etc/systemd/system/gitlab-workhorse.service:

-secretPath /home/git/gitlab/.gitlab_workhorse_secret

For reference, the full ExecStart is now:

ExecStart=/home/git/gitlab/bin/daemon_with_pidfile /home/git/gitlab/tmp/pids/gitlab-workhorse.pid \
/home/git/gitlab-workhorse/gitlab-workhorse -listenUmask 0 -listenNetwork unix \
-listenAddr /home/git/gitlab/tmp/sockets/gitlab-workhorse.socket \
-authBackend http://127.0.0.1:8080 -authSocket /home/git/gitlab/tmp/sockets/gitlab.socket \
-documentRoot /home/git/gitlab/public -secretPath /home/git/gitlab/.gitlab_workhorse_secret \
>> /home/git/gitlab/log/gitlab-workhorse.log 2>&1

The other possibility is:

In my case 500 error was caused by bad nginx configuration in /etc/gitlab/gitlab.rb.

In case where I had something "before" the nginx, like in my case haproxy. I overlooked this fact. It is described in NGiNX settings.

In my case in haproxy sends backend to 8081 where is listening nginx now (originally I setted 8080 - default unicorn service) -
I was not able configure gitlab only with haproxy, without nginx layer.

So in my configuration was important

nginx['listen_port'] = 8081
nginx['listen_https'] = false

Note that both issues are for NGiNX (there is one when Apache2 is used)

There is also a mention about 403 (permission denied) errors:

We were able to resolve the 403 issue by enabling both HTTPS and SSH cloning; we only had SSH cloning enabled which seemed to be causing the problem. This can be changed by going to https:///admin/application_settings and double checking Enabled Git access protocols

Those conclusions are summarized in merge request 6843

But there is more:

Looking at the default files, it looks like there is some sort of confusion with upgrades and what the defaults should be.
With the default configuration file examples (init.d and nginx), gitlab-workhorse will listen on a Unix socket and not an IP:port.
The Nginx example config file does have some lines for Unix sockets, but the proxy pass goes to an address.

I thought setting gitlab up for the first time I needed point my nginx config to the Unicorn port bind as it was the only port I was seeing in netstat get setup when I started the gitlab services.
If you send the git clone request to Unicorn you will get the 500 error.
What I had to do is change gitlab-workhorse to listen to my lookback address and point Nginx there. That cleared up my HTTP 500 error with cloning.
See more with A Brief History of GitLab Workhorse

Improve this answer

1 Comment

I can't understand this solution, anyway fixed the issue by uninstalling and installing it using package.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.