Before describing the diagnosis and troubleshooting methods that I used to solve this problem, allow me to give you a little rundown on what this is all about.
GitHub: It is a wonderful tool that make managing and administering lots of Git repositories and their associated permissions on a centralized server quite easy. What if you want to do something like this on your own servers where your developers can store their projects and network? GitLab gives you complete control over your repositories and allows you to decide whether they are public, private or free. It’s a github clone to be more specific running on your own hardware.
To learn more about the minimum requirements to run your own gitlab service click here.
Hunting down the errors with GitLab:
One of our gitlab servers started throwing up “502 bad gateway” error one fine day. We have a Ubuntu server with nginx+gitlab+ISPconfig (ISPconfig is a free hosting control panel) to server as a gitlab server.
Troubleshooting:
The usual routine is to check the error logs pertaining to the service.
We found this from gitlab_erorr.log
2012/09/21 23:42:29 [error] 19689#0: *5 connect() to unix:/home/gitlab/gitlab/tmp/sockets/gitlab.socket failed (111: Connection refused) while connecting to upstream, client: ****, server: gitlab..***, request: “GET / HTTP/1.1”, upstream:
Bad Gateway means nginx wasn’t able to connect to the upstream socket.
Three possible reasons:-
- Misconfigured path to the socket
- Unicorn is not running.
- Unicorn crashes intermittently due to resource shortage
There was no resource shortage. Server was in ship-shape.
We tried to restart gitlab
root@server:/etc/nginx/sites-available# /etc/init.d/gitlab restart
Removing stale Unicorn web server pid. This is most likely caused by the web server crashing the last time it ran.
Removing stale Sidekiq web server pid. This is most likely caused by the Sidekiq crashing the last time it ran.
Starting the GitLab Unicorn web server…
master failed to start, check stderr log for details
Starting the GitLab Sidekiq event dispatcher…
The GitLab Unicorn webserver is not running.
The GitLab Sidekiq job dispatcher with pid 6296 is running.
The solution still eluded us. After a couple of minutes googling using “The GitLab Unicorn webserver is not running” as search keyword. We stumbled on the possibility of a port number mismatch/misconfiguration. We decided to check that.
Root Cause
We noticed that ISPconfig and Unicorn were running on same port 8080. Now that is the root cause of this problem. One of our associates had installed ISPconfig couple of hours before and inadvertently used the port number 8080 reserved for unicorn.
We changed ISP config port to 8081 from here:-
/etc/nginx/sites-available/ispconfig.vhost
/home/git/gitlab/config/unicorn.rb
Restarted ngnix and gitlab
Now both services are no longer sharing one port.
root@server:/home/git/gitlab/config# netstat -tulpn | grep :808
tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN 6692/unicorn.rb -E
tcp 0 0 0.0.0.0:8081 0.0.0.0:* LISTEN 6627/nginx
So, that was it! Double check before and after you install anything on the server. Make sure it does not adversely affect or tamper the functions of other services.
InstaCarma is an eminent provider of managed IT services and web hosting technical support where you can easily avail 24*7*365 uninterrupted services.
