Author	Nejat Hakan
eMail	nejat.hakan@outlook.de
PayPal Me	https://paypal.me/nejathakan

Git Server GitLab CE

Introduction

Welcome to this comprehensive guide on self-hosting GitLab Community Edition (CE). GitLab is far more than just a Git repository manager; it's a complete DevOps platform delivered as a single application. By self-hosting GitLab CE, you gain full control over your source code, development workflows, and CI/CD pipelines, ensuring data privacy and customization tailored to your specific needs. This is particularly valuable in academic settings or research groups where data sovereignty and experimentation are paramount.

GitLab CE is the open-source version of GitLab, offering a vast array of features suitable for individuals, small teams, and large organizations. It provides Git repository hosting, code reviews, issue tracking, activity feeds, wikis, continuous integration, and continuous deployment.

Self-hosting might seem daunting initially, but it offers significant advantages:

• Control: You own the data and the infrastructure. You decide on update schedules, security policies, and integrations.
• Privacy: Your source code and project data reside on your servers, not on a third-party platform.
• Customization: You can tweak configurations, integrate with internal systems, and manage resources according to your requirements.
• Cost: GitLab CE is free (as in freedom and beer), although you bear the infrastructure costs (server, storage, bandwidth).

This guide is structured progressively:

• Basic: Covers the fundamentals, installation, and initial configuration. Ideal for beginners wanting to get a functional instance up and running.
• Intermediate: Delves into user management, security hardening, and essential maintenance tasks like backups. Builds upon the basic setup for a more robust instance.
• Advanced: Explores powerful features like GitLab CI/CD, monitoring, troubleshooting, and scaling considerations. Aimed at users looking to leverage the full potential of GitLab.

Each section includes theoretical explanations followed by practical "Workshop" sessions designed as step-by-step tutorials. These workshops use real-world scenarios to help you solidify your understanding and gain hands-on experience. We assume you have a basic understanding of Linux command-line operations, networking concepts, and version control with Git. Let's begin the journey of mastering your own GitLab CE instance!

1. Understanding GitLab CE Fundamentals

Before diving into installation, it's crucial to understand the core concepts behind GitLab and its architecture. This foundational knowledge will help you manage and troubleshoot your instance effectively.

GitLab revolves around Git, the distributed version control system. At its heart, GitLab provides a central place to store Git repositories. A repository contains all your project files and the entire revision history.

Key GitLab concepts include:

• Projects: A project in GitLab encapsulates a repository, along with issues, merge requests, CI/CD pipelines, wikis, and other features related to a specific software project or piece of work.
• Groups: Groups are collections of projects and users. They allow you to organize related projects together and manage permissions efficiently at a higher level. Groups can also have subgroups, creating a hierarchical structure.
• Users: These are individual accounts that can log in to GitLab, contribute to projects, create issues, etc. Each user has a profile and specific permissions.
• Permissions: GitLab employs a role-based access control system. Users are assigned roles (e.g., Guest, Reporter, Developer, Maintainer, Owner) within projects or groups, determining what actions they can perform.

GitLab Components: GitLab isn't a single program but a suite of components working together. The Omnibus package, which we'll use for installation, bundles these components and manages them for you. Understanding these helps in diagnostics:

• Nginx: A high-performance web server that acts as a reverse proxy, handling incoming HTTP/HTTPS requests and directing them to the appropriate GitLab service. It also serves static assets.
• PostgreSQL: The relational database management system used to store metadata, including user information, permissions, issue data, merge request details, and CI/CD pipeline information. The actual Git data is not stored in the database.
• Redis: An in-memory data structure store used for caching user sessions, background job queues, and other transient data requiring fast access.
• Sidekiq: A background job processing framework that handles asynchronous tasks like sending emails, running CI/CD jobs (via runners), updating merge requests, and repository housekeeping. It uses Redis to manage its job queue.
• Gitaly: A GitLab component responsible for handling all Git repository access. It acts as a Git RPC service, allowing other GitLab components to interact with Git repositories without needing direct filesystem access. This design enhances performance and scalability, especially in distributed setups.
• Puma: An application server (formerly Unicorn) that runs the core GitLab Rails application code, handling dynamic requests and business logic.

System Requirements: Self-hosting requires adequate hardware resources. GitLab's requirements depend heavily on the number of users and the workload (especially CI/CD usage). The official documentation provides minimums, but for a smooth experience, especially for learning and moderate use (e.g., a university course or small research group), consider:

• CPU: 4 cores is the recommended minimum (supports ~100 users). More cores are beneficial, especially with heavy CI/CD usage.
• RAM: 8 GB RAM is the recommended minimum. More RAM significantly improves performance, especially caching. For CI/CD, memory usage can spike. 16 GB offers a much better experience.
• Disk: Sufficient storage is crucial. The OS and GitLab itself require around 15-20 GB. However, your Git repositories will consume the most space. Start with at least 50-100 GB of fast storage (SSD recommended) and plan for growth. Ensure you have space for backups as well.
• OS: A supported Linux distribution (e.g., Ubuntu, Debian, CentOS). We'll focus on Ubuntu LTS in the workshops.

Workshop Preparing the Ground

This workshop guides you through setting up a basic, secure Linux environment suitable for hosting GitLab CE. We'll use Ubuntu Server 22.04 LTS as the example OS and VirtualBox for virtualization.

Goal: Create a virtual machine, perform initial updates, and apply basic security hardening.

Steps:

1. Install VirtualBox: If you don't have it, download and install VirtualBox and the VirtualBox Extension Pack from virtualbox.org.
2. Download Ubuntu Server ISO: Get the latest Ubuntu Server LTS ISO image from ubuntu.com. We'll use 22.04 LTS.
3. Create a New Virtual Machine (VM):
   • Open VirtualBox and click "New".
   • Name: GitLab-Server (or similar).
   • Type: Linux.
   • Version: Ubuntu (64-bit).
   • Memory size: Allocate at least 8192 MB (8 GB) RAM. 16384 MB (16 GB) is better if your host machine allows.
   • Hard disk: Select "Create a virtual hard disk now". Choose "VDI (VirtualBox Disk Image)". Select "Dynamically allocated". Set the size to at least 50 GB (100 GB recommended). Click "Create".
4. Configure VM Settings:
   • Select the newly created GitLab-Server VM and click "Settings".
   • System -> Processor: Allocate at least 4 CPU cores.
   • Network -> Adapter 1: Ensure "Enable Network Adapter" is checked. Attach it to "Bridged Adapter" (allows the VM to get its own IP address on your local network, making it easily accessible) or "NAT" (simpler if network configuration is complex, but requires port forwarding for external access). For simplicity in this workshop, "Bridged Adapter" is often easier if your network allows it.
   • Storage: Select the empty CD drive under "Storage Devices". Click the CD icon on the right and choose "Choose a virtual optical disk file...". Select the downloaded Ubuntu Server ISO.
   • Click "OK".
5. Install Ubuntu Server:
   • Start the GitLab-Server VM.
   • Follow the on-screen prompts for the Ubuntu Server installation.
     • Choose your language.
     • Select keyboard layout.
     • Choose "Ubuntu Server" (not minimized).
     • Network configuration: If using Bridged Adapter, it should get an IP via DHCP. Note this IP address down! If using NAT, it will likely get an IP like 10.0.2.15.
     • Configure proxy (leave blank unless needed).
     • Configure mirror (use the default).
     • Storage configuration: Use the entire disk, set up as an LVM group (default is fine).
     • Confirm destructive action.
     • Profile setup: Choose a username (e.g., gitlabadmin), server name (e.g., gitlab-server), and a strong password. Remember these credentials!
     • Install OpenSSH server: Check the box to install the OpenSSH server. This is crucial for remote management.
     • Featured Server Snaps: Do not install any snaps for now (like Docker).
     • Wait for the installation to complete.
     • Reboot when prompted (VirtualBox should automatically eject the ISO).
6. Initial Server Update:
   • Log in to the server using the credentials you created (either directly in the VirtualBox console or via SSH from your host machine: ssh gitlabadmin@<VM_IP_ADDRESS>).
   • Update the package lists and upgrade installed packages:

     sudo apt update
     sudo apt upgrade -y
     sudo apt autoremove -y # Remove unused packages
     

   • Reboot if necessary (e.g., if a kernel update occurred): sudo reboot
7. Basic Firewall Setup (UFW):
   • Uncomplicated Firewall (ufw) is a user-friendly frontend for iptables.
   • Allow essential ports: SSH (22), HTTP (80), and HTTPS (443). GitLab will need these.

     sudo ufw allow ssh     # Or sudo ufw allow 22/tcp
     sudo ufw allow http    # Or sudo ufw allow 80/tcp
     sudo ufw allow https   # Or sudo ufw allow 443/tcp
     

   • Enable the firewall:

     sudo ufw enable
     

     • Confirm with 'y'.
   • Check the status:

     sudo ufw status verbose
     

     • You should see the allowed ports and the default policy (deny incoming, allow outgoing).
8. (Optional but Recommended) SSH Key Authentication:
   • Password-based SSH is less secure than key-based authentication.
   • On your host machine (not the VM), generate an SSH key pair if you don't have one:

     ssh-keygen -t rsa -b 4096 -C "your_email@example.com"
     

     • Follow prompts (accept default file location, optionally set a passphrase).
   • Copy the public key to your GitLab server VM:

     # Replace gitlabadmin@<VM_IP_ADDRESS> with your VM's user and IP
     ssh-copy-id gitlabadmin@<VM_IP_ADDRESS>
     

     • Enter your VM user's password when prompted.
   • Try logging in via SSH again; it should now use the key (and prompt for the passphrase if you set one) instead of the password.
   • (Optional Hardening): Once key login works, you can disable password authentication on the VM. Edit the SSH daemon config: sudo nano /etc/ssh/sshd_config. Change PasswordAuthentication yes to PasswordAuthentication no. Save the file (Ctrl+X, Y, Enter). Restart the SSH service: sudo systemctl restart sshd. Test login again before logging out to ensure you don't lock yourself out!

You now have a basic, updated, and slightly hardened Ubuntu Server VM ready for the GitLab CE installation. Keep the VM's IP address handy.

2. Installing GitLab CE Omnibus Package

The recommended and most straightforward way to install GitLab CE is using the official Omnibus package. This package bundles all the necessary components (Nginx, PostgreSQL, Redis, etc.) and dependencies, managing them through a simple command-line tool (gitlab-ctl). This approach significantly simplifies installation and upgrades.

Advantages of Omnibus:

• Simplicity: Single package installation manages complex dependencies.
• Consistency: Ensures components are compatible and configured correctly to work together.
• Management: Provides gitlab-ctl for starting, stopping, reconfiguring, upgrading, and backing up GitLab.
• Upgrades: Simplifies the upgrade process significantly compared to manual installations.

Installation Steps (Ubuntu/Debian):

These steps are performed on the Ubuntu Server VM prepared in the previous workshop.

1. Install Dependencies: GitLab requires some essential packages like curl, ca-certificates, and postfix (for sending emails, though we'll configure external SMTP later if needed).

   sudo apt update
   sudo apt install -y curl openssh-server ca-certificates postfix
   

   • During the postfix installation, you'll be asked for a mail configuration type. Select 'Internet Site'.
   • For the 'System mail name', enter the domain name you intend to use for GitLab (e.g., gitlab.yourdomain.com). If you don't have a domain yet or are just testing locally, you can use the server's hostname (e.g., gitlab-server) or even localhost, but email notifications might not work correctly without proper configuration later.

2. Add the GitLab Package Repository: GitLab maintains its own APT repository. We need to add it to our system.

   curl -sS https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash
   

   • This command downloads a script from GitLab and executes it. The script detects your OS, adds the appropriate repository source, and imports the GPG key to verify package authenticity. Always be cautious when piping curl to bash, but packages.gitlab.com is the official source.

3. Install GitLab CE: Now, install the GitLab CE package. This step might take several minutes as it downloads and unpacks the large Omnibus package.

   • Crucial: Before running the install command, determine the URL you will use to access GitLab. This is configured via an environment variable.
     • If using a real domain name (e.g., gitlab.example.com) that resolves to your VM's IP address:

       # Make sure DNS is set up correctly for gitlab.example.com to point to <VM_IP_ADDRESS>
       sudo EXTERNAL_URL="https://gitlab.example.com" apt install gitlab-ce
       

       (We use https:// anticipating enabling Let's Encrypt later. If you stick to HTTP initially, use http://)
     • If using the VM's IP address directly (for local testing):

       # Replace <VM_IP_ADDRESS> with the actual IP
       sudo EXTERNAL_URL="http://<VM_IP_ADDRESS>" apt install gitlab-ce
       

       (Using HTTP here is simpler for initial IP-based access.)
   • The EXTERNAL_URL tells GitLab how users will access it. It's used to configure Nginx and generate correct URLs within the application. Setting it before installation ensures the initial configuration is correct.
   • The apt install gitlab-ce command downloads and installs the package.

4. Initial Configuration (gitlab-ctl reconfigure): Although the installation command triggers an initial configuration, sometimes it's good practice to run it manually, especially if you need to adjust EXTERNAL_URL later. The first run can take a significant amount of time (5-15 minutes or more) as it sets up all the bundled services, databases, secrets, and configuration files based on /etc/gitlab/gitlab.rb.

   sudo gitlab-ctl reconfigure
   

   • You will see a lot of output as Chef (the configuration management tool used internally by Omnibus) applies the settings. Look for "GitLab Reconfigured!" at the end.

5. Check GitLab Status: Verify that all GitLab components are running.

   sudo gitlab-ctl status
   

   • You should see output showing run: for services like nginx, postgresql, redis, sidekiq, gitaly, puma, etc.

Workshop Your First GitLab CE Installation

This workshop applies the steps above to install GitLab CE on the VM you prepared.

Goal: Install GitLab CE using the Omnibus package and perform the initial login.

Prerequisites: The Ubuntu Server VM from the previous workshop, running and accessible via SSH. Know the VM's IP address.

Steps:

1. SSH into your VM:

   ssh gitlabadmin@<VM_IP_ADDRESS>
   

2. Install Dependencies:

   sudo apt update
   sudo apt install -y curl openssh-server ca-certificates postfix
   

   • Select Internet Site for Postfix configuration.
   • Use the VM's IP address or a planned hostname for the system mail name (e.g., <VM_IP_ADDRESS> or gitlab-local).
3. Add GitLab Repository:

   curl -sS https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash
   

4. Install GitLab CE (using IP Address):
   • Let's assume your VM's IP is 192.168.1.150. Replace this with your actual VM IP address.

     sudo EXTERNAL_URL="http://192.168.1.150" apt install gitlab-ce
     

   • Wait patiently for the installation to complete. This involves downloading several hundred megabytes and initial configuration.
5. Monitor Reconfiguration (Optional but informative): If the installation finishes quickly without extensive configuration output, or if you want to ensure everything is set, run:

   sudo gitlab-ctl reconfigure
   

   • Watch the output. This can take several minutes.
6. Verify Services:

   sudo gitlab-ctl status
   

   • Ensure all key services show run:. If any service shows down or fails to start, you'll need to check logs (covered later).
7. Initial Login:
   • Open a web browser on your host machine (the one running VirtualBox, not the VM itself).
   • Navigate to the EXTERNAL_URL you configured during installation (e.g., http://192.168.1.150).
   • You should see the GitLab login screen.
   • The default administrator username is root.
   • GitLab automatically generates a temporary password for the root user during the first reconfigure. You need to retrieve this password from the server. It's stored in a file for 24 hours. Run this command on the VM via SSH:

     sudo cat /etc/gitlab/initial_root_password
     

   • Copy the displayed password (be careful with special characters).
   • Go back to the browser, enter root as the username and paste the retrieved password. Click "Sign in".
   • GitLab will immediately prompt you to change the password. Choose a new, strong password for the root administrator account and save it securely.
8. Explore the Admin Area:
   • After changing the password, you'll be logged into GitLab.
   • Click the wrench icon (Admin Area) in the top navigation bar (or Menu -> Admin).
   • Briefly explore the sections available: Overview (Dashboard), Monitoring, Settings, etc. This is your central control panel for managing the GitLab instance.

Congratulations! You have successfully installed GitLab CE and logged in as the administrator. Your self-hosted Git server is now operational.

3. Basic GitLab Configuration and Usage

With GitLab installed, the next step is understanding basic configuration and how to perform fundamental tasks like creating users, groups, and projects, and interacting with them using Git.

The Main Configuration File: gitlab.rb

Almost all configuration for an Omnibus GitLab installation is managed through a single file: /etc/gitlab/gitlab.rb. This file contains numerous settings, most of which are commented out by default, indicating GitLab is using its standard defaults.

• Editing: Use a text editor with root privileges (e.g., sudo nano /etc/gitlab/gitlab.rb).
• Applying Changes: After saving changes to gitlab.rb, you must run sudo gitlab-ctl reconfigure for the changes to take effect. This command reads the file, translates the settings into configuration files for individual components (Nginx, Puma, PostgreSQL, etc.), and restarts services as needed.
• Syntax: The file uses Ruby syntax, but primarily involves setting variable values (e.g., external_url 'http://gitlab.example.com'). Be careful with syntax – incorrect Ruby can cause reconfigure to fail.
• Key Settings (Examples):
  • external_url: We set this during installation. It's fundamental.
  • Email settings (gitlab_rails['smtp_enable'], etc.): For configuring GitLab to send notification emails via an external SMTP server.
  • HTTPS settings (letsencrypt['enable'], etc.): For enabling automatic SSL via Let's Encrypt.
  • Backup settings (gitlab_rails['backup_path'], gitlab_rails['backup_keep_time']): For configuring backup location and retention.
  • Resource tuning (e.g., Puma worker counts, database connection pools): For performance optimization (more advanced).

Managing Users, Groups, and Projects (Web UI)

While GitLab offers APIs and command-line tools, the web interface is the most common way to manage basic entities. Log in as the root user (or another admin user).

• Creating Users:
  • Navigate to Menu -> Admin -> Overview -> Users.
  • Click "New user".
  • Fill in the user's details (Name, Username, Email).
  • Choose an initial password or let GitLab send a password reset link (requires email to be configured).
  • Adjust access level (Regular or Admin). Grant Admin privileges sparingly.
  • Set project limits if desired.
  • Click "Create user".
• Creating Groups:
  • Navigate to Menu -> Groups -> Your groups.
  • Click "New group".
  • Give the group a name (e.g., Research Team Alpha), an optional description, and set its visibility level (Private, Internal, Public).
  • Choose a URL path for the group.
  • Click "Create group".
  • You can add members to the group via the Group information -> Members section, assigning roles.
• Creating Projects:
  • Projects usually belong to a user or a group.
  • Navigate to the group where you want to create the project (or your personal namespace via Menu -> Projects -> Your projects).
  • Click "New project".
  • Choose "Create blank project".
  • Enter a project name (e.g., Data Analysis Scripts), an optional description.
  • The project slug (URL path) is usually derived from the name.
  • Select the group/user namespace for the project.
  • Choose the Visibility Level:
    • Private: Only explicit members can see and access the project.
    • Internal: Any logged-in user on your GitLab instance can see and clone the project.
    • Public: Anyone (even without an account) can see and clone the project. Use with caution on self-hosted instances unless intended.
  • Optionally initialize the repository with a README file.
  • Click "Create project".

Basic Git Operations with GitLab

Once a project exists, users interact with it using standard Git commands. The project's main page provides the URLs needed for cloning.

1. Clone: Get a local copy of the repository. Use HTTPS or SSH.
   • HTTPS: git clone http://<your_gitlab_url>/<group_or_username>/<project_name>.git
     • You'll be prompted for your GitLab username and password (or a Personal Access Token, which is more secure and recommended).
   • SSH: git clone git@<your_gitlab_url>:<group_or_username>/<project_name>.git
     • Requires you to have added an SSH public key to your GitLab user profile (User Settings -> SSH Keys).
2. Add & Commit: Make changes locally.

   cd <project_name>
   echo "My first change" > newfile.txt
   git add newfile.txt
   git commit -m "Add initial data file"
   

3. Push: Send your local commits to the GitLab server.

   git push origin <branch_name> # e.g., git push origin main or master
   

   • You might need to authenticate again (HTTPS password/token or SSH key passphrase).
4. Pull: Get the latest changes from the GitLab server.

   git pull origin <branch_name>
   

Workshop Creating Your First Project

This workshop walks through creating a user, group, and project, then performing basic Git operations to push code to it.

Goal: Set up a simple project structure, add a user, and push a file using Git via HTTPS.

Prerequisites: GitLab CE installed and accessible. Logged in as the root user. Git installed on your local machine (the one you'll use to interact with GitLab, not necessarily the VM).

Steps:

1. Create a New User:
   • In GitLab, go to Menu -> Admin -> Overview -> Users.
   • Click "New user".
   • Name: Test Developer
   • Username: tdev
   • Email: Use a real email address you can access if you want password reset functionality (requires email setup in GitLab), or use a dummy one like tdev@test.local for now.
   • Password: Set a password directly (e.g., StrongP@ssw0rd!). Uncheck "Send password reset link".
   • Access level: Keep as Regular.
   • Click "Create user".
2. Log out and Log in as the New User:
   • Click the user avatar (top right) -> Sign out.
   • Log in as tdev with the password you set. You might be asked to change it on first login if policy requires.
3. Create a New Group:
   • As the tdev user, go to Menu -> Groups -> Your groups.
   • Click "New group" -> "Create group".
   • Group name: My Awesome Research
   • Visibility level: Private
   • Click "Create group".
4. Create a New Project within the Group:
   • You should be redirected to the new group's page. Click "New project".
   • Select "Create blank project".
   • Project name: Lab Notebook
   • Project slug: Should default to lab-notebook.
   • Visibility Level: Keep Private (inherits from group default).
   • Initialize repository with a README: Check this box. It makes cloning easier.
   • Click "Create project".
5. Clone the Project via HTTPS:
   • On the project's main page, click the blue "Clone" button.
   • Copy the "Clone with HTTPS" URL (e.g., http://192.168.1.150/my-awesome-research/lab-notebook.git).
   • Open a terminal or Git Bash on your local machine (not the VM).
   • Navigate to a directory where you want to store the project.
   • Run the git clone command:

     # Replace with the actual URL from GitLab
     git clone http://192.168.1.150/my-awesome-research/lab-notebook.git
     

   • You will be prompted for your username (tdev) and password (StrongP@ssw0rd!).
   • A new directory lab-notebook should be created, containing the README.md file.
6. Make a Change, Commit, and Push:
   • Change into the project directory:

     cd lab-notebook
     

   • Create a new file:

     echo "Experiment results from today." > results_log.txt
     

   • Check the status:

     git status
     

     • It should show results_log.txt as an untracked file.
   • Add the file to staging:

     git add results_log.txt
     

   • Commit the change:

     git commit -m "Add initial results log"
     

     • If this is your first commit on this machine, Git might ask you to configure your user name and email. Follow the instructions it provides (using git config --global user.name "Your Name" and git config --global user.email "you@example.com"). Then, run the git commit command again.
   • Push the changes back to GitLab:

     # Assuming your default branch is 'main' (check with 'git branch')
     git push origin main
     

   • Enter your username (tdev) and password again when prompted.
7. Verify in GitLab:
   • Refresh the project page in your web browser.
   • You should see results_log.txt listed in the files.
   • Check the commit history (Repository -> Commits) to see your new commit.

You have successfully created GitLab entities, cloned a repository, made local changes, and pushed them back to your self-hosted server. This covers the fundamental workflow of using GitLab as a Git repository manager.

4. Managing Users, Groups, and Permissions

Effective management of users, their access levels, and how projects are organized into groups is crucial for maintaining security and order within your GitLab instance, especially as it grows.

User Roles Deep Dive:

GitLab employs a granular role-based access control (RBAC) system. Roles define what actions a user can perform within the scope of a project or group. The main roles, from least to most privileged, are:

• Guest: Can view the project (if not private), create issues, and leave comments. Cannot see or access the code repository. Useful for external stakeholders or bug reporters.
• Reporter: Includes Guest permissions, plus the ability to view the code repository, view CI/CD pipelines, pull code, view snippets, and manage issue labels. Cannot push code. Good for non-developer team members like QA or project managers who need visibility.
• Developer: Includes Reporter permissions, plus the ability to push code to non-protected branches, create and merge merge requests, manage CI/CD jobs, create snippets, and manage branches. This is the standard role for most software developers.
• Maintainer: Includes Developer permissions, plus the ability to push to protected branches, manage project settings (members, integrations, deploy keys, runners), protect branches, edit project details, and delete issues. Often assigned to team leads or key project maintainers.
• Owner: Includes Maintainer permissions, plus the ability to delete the project, transfer the project, manage group memberships (if the project is in a group and the user is also a group Owner), and manage all aspects of the project. Typically assigned to the project creator or administrators. Group Owners have similar overarching powers within their group.

Permissions Inheritance:

Permissions flow downwards in a logical way:

• Group Membership: When you add a user to a group, you assign them a role for that group.
• Project Membership: Users automatically inherit the maximum role they have from any parent groups the project belongs to. For example, if a user is a Developer in Group A and Group A has Subgroup B, and Project X is inside Subgroup B, the user is implicitly a Developer in Project X.
• Direct Project Membership: You can also add users directly to a project with a specific role. If a user is added directly to a project with a higher role than they inherit from the group (e.g., Maintainer in the project but only Developer in the group), the higher project-specific role takes precedence for that project only. If the project role is lower, they still retain their higher inherited group role.

This system allows flexible structuring. You can grant baseline access via group membership and elevate permissions for specific individuals on critical projects.

LDAP/AD Integration (Conceptual Overview):

For organizations already using directory services like Microsoft Active Directory (AD) or other LDAP servers for user management, GitLab CE can integrate with them.

• How it works: You configure GitLab (/etc/gitlab/gitlab.rb) with the connection details for your LDAP/AD server. When a user tries to log in, GitLab attempts to authenticate them against the directory server. If successful, a GitLab account is automatically provisioned (or linked to an existing one).
• Benefits: Centralized user management (create/disable users in one place), consistent credentials, group membership synchronization (map LDAP groups to GitLab roles).
• Configuration: Involves setting parameters like gitlab_rails['ldap_enabled'] = true, server host, port, bind DN (user for querying LDAP), password, base DN (where to search for users), and attribute mappings (e.g., mapping sAMAccountName to GitLab username). This is an advanced topic typically covered under specific integration guides.

SSH Key Management:

While HTTPS authentication with passwords or Personal Access Tokens works, SSH is often preferred by developers for command-line Git operations.

• How it works: A user generates an SSH key pair (public and private key) on their local machine. They upload the public key to their GitLab user profile (User Settings -> SSH Keys). The private key remains securely on their local machine.
• Authentication: When performing a Git operation over SSH (e.g., git clone git@gitlab.example.com:...), the Git client uses the private key to sign a challenge sent by the GitLab server. GitLab verifies this signature using the stored public key, granting access without requiring a password.
• Security: Generally considered more secure than password-based authentication, especially when keys are protected with passphrases. Eliminates the need to transmit passwords over the network for Git operations.

Workshop Advanced User and Group Management

This workshop explores creating users with different roles, structuring groups, managing memberships, and setting up SSH key authentication.

Goal: Create a simulated team structure with varying access levels and enable SSH access for a developer.

Prerequisites: GitLab CE running, logged in as root or an admin user. Git installed locally. Access to the tdev user account created previously.

Steps:

1. Create More Users:
   • As admin, navigate to Menu -> Admin -> Overview -> Users.
   • Create two new users:
     • User 1:
       • Name: Project Manager
       • Username: pm_user
       • Email: pm@test.local
       • Password: Set a strong password (e.g., PM_P@ssw0rd!)
       • Access level: Regular
       • Click "Create user".
     • User 2:
       • Name: Senior Developer
       • Username: sr_dev
       • Email: srdev@test.local
       • Password: Set a strong password (e.g., SrDev_P@ssw0rd!)
       • Access level: Regular
       • Click "Create user".
2. Manage Group Membership and Roles:
   • Navigate to the My Awesome Research group created earlier (Menu -> Groups -> Your groups -> My Awesome Research).
   • Go to Group information -> Members.
   • You'll see tdev is likely listed as Owner (since they created it).
   • Invite the new users:
     • In the "Invite members" section, search for Project Manager (pm_user).
     • Select the Reporter role.
     • Click "Invite".
     • Search for Senior Developer (sr_dev).
     • Select the Maintainer role.
     • Click "Invite".
     • Optionally, invite the root user (or your admin user) as an Owner for administrative convenience.
   • Observe: You now have members with different roles within the group. These roles will be inherited by the Lab Notebook project inside this group.
3. Verify Project Roles (Inheritance):
   • Navigate to the Lab Notebook project (Menu -> Projects -> Your projects -> My Awesome Research / Lab Notebook).
   • Go to Project information -> Members.
   • You should see tdev (Owner), pm_user (Reporter), and sr_dev (Maintainer) listed under "Members inherited from group My Awesome Research". Their roles match those assigned at the group level.
4. Add SSH Key for tdev:
   • Log out of GitLab as admin.
   • Log in as the tdev user.
   • On your local machine (where you run git commands), open a terminal or Git Bash.
   • Generate an SSH key pair if you don't already have one specifically for GitLab (it's good practice to use different keys for different services):

     # Use a descriptive filename like id_rsa_gitlab
     ssh-keygen -t rsa -b 4096 -C "tdev@test.local" -f ~/.ssh/id_rsa_gitlab
     

     • Press Enter to accept the default file path (or modify if desired).
     • Enter a strong passphrase when prompted. This encrypts your private key on disk, adding an extra layer of security. Remember this passphrase!
   • Display the public key:

     cat ~/.ssh/id__rsa_gitlab.pub
     

   • Copy the entire output (starting with ssh-rsa and ending with your comment tdev@test.local).
   • In the GitLab web UI (logged in as tdev), click your avatar (top right) -> Edit profile -> SSH Keys.
   • Paste the copied public key into the "Key" field.
   • Give it a Title (e.g., "My Work Laptop").
   • Leave "Usage type" as "Authentication & Signing".
   • Click "Add key".
5. Test SSH Access:
   • On the Lab Notebook project page in GitLab, click the blue "Clone" button.
   • This time, copy the "Clone with SSH" URL (e.g., git@192.168.1.150:my-awesome-research/lab-notebook.git). Note: Replace 192.168.1.150 with your GitLab server's actual hostname or IP as configured in external_url for SSH.
   • On your local machine, navigate to a different directory (outside the one cloned via HTTPS).
   • Run the git clone command using the SSH URL:

     # Replace with the actual SSH URL from GitLab
     git clone git@192.168.1.150:my-awesome-research/lab-notebook.git lab-notebook-ssh
     

   • The first time you connect via SSH, you'll likely see a host authenticity warning. Type yes to continue.
   • If you set a passphrase for your SSH key, you will be prompted to enter it now.
   • The repository should clone successfully without asking for your GitLab username or password.
6. Test Permissions (Conceptual):
   • Log in to GitLab as pm_user (Project Manager, Role: Reporter). Try navigating to the Lab Notebook project's Repository -> Files. They should be able to view files but will not see options to edit or upload directly, nor will they be able to push changes via Git. They can create Issues.
   • Log in as sr_dev (Senior Developer, Role: Maintainer). They should have full developer access plus access to Settings -> Repository (e.g., to manage protected branches).

This workshop demonstrated how to leverage GitLab's role system through group memberships and how to set up the more secure and convenient SSH key authentication for developers.

5. Securing Your GitLab Instance

Running a publicly accessible server, especially one holding valuable source code, requires careful attention to security. This section covers essential steps to harden your GitLab CE installation.

HTTPS with Let's Encrypt:

The most crucial security enhancement is enabling HTTPS (HTTP over SSL/TLS). This encrypts traffic between users' browsers and your GitLab server, protecting login credentials, code, and other sensitive data from eavesdropping. Omnibus GitLab has excellent built-in support for obtaining free SSL certificates from Let's Encrypt.

• How it works: Let's Encrypt is a free, automated, and open Certificate Authority (CA). GitLab's integration automates the process of requesting, obtaining, installing, and renewing SSL certificates.
• Prerequisites:
  1. Your GitLab server must be accessible from the public internet on ports 80 (for the validation challenge) and 443 (for HTTPS traffic).
  2. You must use a real, registered domain name (e.g., gitlab.yourdomain.com) in your external_url. Let's Encrypt cannot issue certificates for IP addresses or internal-only hostnames.
  3. DNS records for your chosen domain must correctly point to your GitLab server's public IP address.
• Configuration (/etc/gitlab/gitlab.rb):
  1. Set the external_url to use https and your public domain name:

     external_url 'https://gitlab.yourdomain.com'
     

  2. Enable Let's Encrypt integration:

     letsencrypt['enable'] = true
     

  3. (Optional but Recommended) Provide an email address for Let's Encrypt notifications (e.g., expiration warnings):

     letsencrypt['contact_emails'] = ['your_email@yourdomain.com']
     

  4. (Optional) If you prefer Let's Encrypt certificates to renew automatically slightly before they expire (recommended), you can enable auto-renewal (it's often enabled by default with letsencrypt['enable'] = true):

     # Default is true when letsencrypt['enable'] = true
     # letsencrypt['auto_renew'] = true
     # Configure renewal check time (hour, minute, day_of_month)
     # letsencrypt['auto_renew_hour'] = 0
     # letsencrypt['auto_renew_minute'] = 30
     # letsencrypt['auto_renew_day_of_month'] = "*/7" # Check every 7 days
     

• Applying: Run sudo gitlab-ctl reconfigure. GitLab will contact Let's Encrypt, perform the validation challenge (usually via HTTP-01 on port 80), obtain the certificate, configure Nginx to use it, and set up automatic renewal.

Firewall Configuration:

Ensure your server's firewall only allows necessary traffic. Based on the workshop setup using ufw:

• Required Ports:
  • 22/tcp (SSH): For server administration. Consider restricting access to specific IP addresses if possible.
  • 80/tcp (HTTP): Required for Let's Encrypt validation (HTTP-01 challenge) and potentially for redirecting HTTP users to HTTPS.
  • 443/tcp (HTTPS): For the main GitLab web traffic and Git over HTTPS.
  • (Optional) 2224/tcp or another port if you configure GitLab Shell to use a different port for Git over SSH (default uses port 22).
• Commands (Review):

  sudo ufw status # Check current rules
  sudo ufw allow ssh
  sudo ufw allow http
  sudo ufw allow https
  # Deny other ports implicitly by enabling ufw
  sudo ufw enable
  

Two-Factor Authentication (2FA):

2FA adds a significant layer of security to user accounts by requiring a second form of verification (usually a time-based one-time password, TOTP, from an authenticator app) in addition to the password.

• Enabling: Can be enabled and enforced instance-wide by an administrator.
  • Navigate to Menu -> Admin -> Settings -> General.
  • Expand the "Sign-in restrictions" section.
  • Check the box "Require all users to set up Two-Factor Authentication".
  • (Optional) Set a grace period (Grace period for users to set up two-factor authentication (hours)) during which users can still log in without 2FA configured after the setting is enabled. Set to 0 to enforce immediately.
  • Save changes.
• User Setup: When enforced, users logging in without 2FA configured will be prompted to set it up using an authenticator app (like Google Authenticator, Authy, etc.) on their smartphone. They scan a QR code or enter a secret key, and the app generates codes that change every 30-60 seconds. They must enter a valid code to complete login. Users should also save the provided recovery codes in a safe place.

GitLab Security Best Practices:

• Regular Updates: GitLab releases updates frequently, including security patches. Keep your instance up-to-date. Follow the official upgrade documentation carefully. Omnibus package upgrades are usually straightforward (sudo apt update, sudo apt install gitlab-ce, sudo gitlab-ctl reconfigure).
• Principle of Least Privilege: Assign users the minimum role necessary for their tasks. Avoid granting Admin privileges unnecessarily.
• Strong Passwords: Enforce strong password policies (Admin -> Settings -> General -> Sign-in restrictions).
• Monitor Logs: Regularly review logs in /var/log/gitlab/ for suspicious activity.
• Limit Instance Visibility: If your GitLab instance is only for internal use, restrict access at the network level (firewall rules) if possible. Use Internal or Private visibility for projects unless Public is explicitly needed.
• Disable Public Sign-up: Unless intended, disable public user registration (Admin -> Settings -> General -> Sign-up restrictions -> uncheck "Sign-up enabled").

Rate Limiting and Abuse Reporting:

GitLab includes features to mitigate brute-force attacks and spam:

• Rate Limiting: (Admin -> Settings -> Network -> User and IP rate limits) Configure limits on actions like login attempts, API requests, and Git operations per IP address or user.
• Abuse Reports: Users can report abusive behavior; administrators can review these reports (Admin -> Monitoring -> Abuse Reports).

Workshop Securing GitLab with HTTPS and 2FA

This workshop guides you through enabling HTTPS using Let's Encrypt and enforcing Two-Factor Authentication.

Goal: Secure communication with your GitLab instance using SSL/TLS and enhance account security with 2FA.

Prerequisites:

• GitLab CE installed and running.
• A registered domain name (e.g., mygitlab.example.org). You cannot use an IP address for Let's Encrypt.
• DNS configured so your domain name points to the public IP address of your GitLab VM.
• Your GitLab VM must be reachable from the public internet on ports 80 and 443. This might involve configuring port forwarding on your router if your VM is behind NAT (like in a typical home setup). This network configuration is beyond the scope of the GitLab setup itself but is crucial for Let's Encrypt.

If you cannot meet the public accessibility/domain name prerequisites: You can skip the Let's Encrypt part, but understand that your instance will remain on insecure HTTP. You can still practice the 2FA part.

Steps:

1. (If using Let's Encrypt) Configure DNS and Firewall:
   • Log in to your domain registrar or DNS provider's control panel.
   • Create an A record pointing your chosen subdomain (e.g., gitlab.yourdomain.com) to the public IP address of your GitLab server.
   • Ensure your firewall (on the server ufw and any network firewall/router) allows incoming traffic on ports 80 and 443 from the internet.
   • Wait for DNS propagation (can take minutes to hours). You can check using dig gitlab.yourdomain.com or nslookup gitlab.yourdomain.com from an external network.
2. Edit gitlab.rb for HTTPS:
   • SSH into your GitLab VM.
   • Edit the configuration file:

     sudo nano /etc/gitlab/gitlab.rb
     

   • Find the external_url line. Change it to use https and your domain name:

     # Replace with your actual domain
     external_url 'https://gitlab.yourdomain.com'
     

   • Find the letsencrypt['enable'] line (it might be commented out). Uncomment it and set it to true:

     letsencrypt['enable'] = true
     

   • (Recommended) Find letsencrypt['contact_emails']. Uncomment it and add your email address:

     # Replace with your actual email
     letsencrypt['contact_emails'] = ['admin@yourdomain.com']
     

   • Save the file (Ctrl+X, Y, Enter).
3. Apply Configuration and Obtain Certificate:
   • Run the reconfigure command:

     sudo gitlab-ctl reconfigure
     

   • Watch the output carefully. You should see logs related to Let's Encrypt certificate acquisition. It might mention creating challenges, waiting for verification, and installing the certificate. This can take a minute or two.
   • If it succeeds, GitLab (Nginx) will automatically restart and begin serving traffic over HTTPS.
4. Test HTTPS Access:
   • Open your web browser and navigate to https://gitlab.yourdomain.com (using your domain).
   • Your browser should show a padlock icon, indicating a secure connection. You might be automatically redirected from HTTP if you try that.
   • Log in as root (or another admin user).
5. Enforce Two-Factor Authentication (2FA):
   • As admin, navigate to Menu -> Admin -> Settings -> General.
   • Expand the "Sign-in restrictions" section.
   • Check the box "Require all users to set up Two-Factor Authentication".
   • For this workshop, set the grace period to 0 for immediate enforcement. Warning: Ensure you have an authenticator app ready before doing this for the admin account!
   • Click "Save changes".
6. Set Up 2FA for Admin Account:
   • You might be immediately prompted to set up 2FA upon saving, or upon your next login attempt. If not immediately prompted, log out and log back in as root.
   • You will see a "Two-Factor Authentication" setup screen.
   • Open an authenticator app on your phone (e.g., Google Authenticator, Authy, Duo Mobile).
   • Scan the QR code displayed by GitLab using the app.
   • The app will add the account and start generating 6-digit codes.
   • Enter the current code from the app into the "PIN code" field on the GitLab page.
   • Click "Register with two-factor app".
   • Crucially: GitLab will now show you a list of recovery codes. Copy these codes and store them in a very safe place (e.g., a password manager, printed document in a secure location). These are one-time use codes that allow you to log in if you lose access to your authenticator app.
   • Check the box acknowledging you've saved the codes and proceed.
7. Test 2FA Login:
   • Log out of GitLab.
   • Log in again as root. After entering your password, you will be prompted for the 2FA code from your authenticator app. Enter the current code to complete the login.
8. (Optional) Test 2FA Enforcement for Other Users:
   • Log out.
   • Try logging in as the tdev user (or pm_user, sr_dev). They will also be forced through the 2FA setup process before they can access GitLab.

You have now significantly improved the security of your GitLab instance by enabling encrypted connections with HTTPS and enforcing strong authentication with 2FA. Remember to keep your system and GitLab updated for ongoing security.

6. Backups and Restoration

One of the most critical aspects of managing any server, especially one containing vital project data like GitLab, is implementing a robust backup and recovery strategy. Data loss due to hardware failure, accidental deletion, or security incidents can be catastrophic. GitLab provides tools to facilitate backups, but understanding the process and limitations is key.

Importance of Backups:

• Data Recovery: Restore your GitLab instance after hardware failure, software bugs, or human error.
• Disaster Recovery: Recover data if your primary server location becomes unavailable (e.g., fire, flood).
• Migration: Backups are often used as part of the process when migrating GitLab to new hardware or infrastructure.
• Testing: Restore backups to a separate test environment to verify backup integrity or test upgrades safely.

GitLab Backup Command (gitlab-backup create)

The Omnibus package includes a command-line tool specifically for creating backups.

• Usage:

  sudo gitlab-backup create
  

• What it Backs Up: The command creates a compressed tar archive containing:
  • Database: A full SQL dump of the PostgreSQL database (users, permissions, issues, MRs, CI/CD metadata, etc.).
  • Repositories: All Git repository data (bare repositories stored on the server, typically under /var/opt/gitlab/git-data/repositories/).
  • Attachments/Uploads: Files attached to issues, merge requests, comments, user avatars, etc. (typically under /var/opt/gitlab/gitlab-rails/uploads/).
  • CI/CD Job Artifacts: Files generated and saved by CI/CD jobs (typically under /var/opt/gitlab/gitlab-rails/shared/artifacts/).
  • Container Registry Data: If the registry is enabled and used (typically under /var/opt/gitlab/gitlab-rails/shared/registry/).
  • Pages Content: If GitLab Pages is used (typically under /var/opt/gitlab/gitlab-rails/shared/pages/).
  • Other data depending on features used (LFS objects, Terraform states, etc.)
• What it Does NOT Back Up:
  • Configuration Files: Crucially, it does not back up /etc/gitlab/gitlab.rb or any manually modified configuration files.
  • Secrets: It does not back up the GitLab secrets file (/etc/gitlab/gitlab-secrets.json). This file contains database encryption keys and other sensitive information. Without this file, a restored backup is useless.
  • SSL Certificates: Custom SSL certificates or Let's Encrypt certificates are not included.
  • System Files: OS-level configurations, SSH host keys, etc.
• Backup Location: By default, backups are created in /var/opt/gitlab/backups/. The filename includes a timestamp and the GitLab version (e.g., 1678886400_2023_03_15_15.8.1-ce_gitlab_backup.tar).
• Permissions: The resulting backup file is owned by the git user and typically has restricted permissions (0600).

Configuring Backup Settings (gitlab.rb)

You can customize backup behavior in /etc/gitlab/gitlab.rb:

• Backup Path: Change the directory where backups are stored. Ensure the git user has write permissions to this location.

  gitlab_rails['backup_path'] = "/mnt/backups/gitlab" # Example: mount an external disk here
  

• Retention Policy: Automatically delete old backups. Time is specified in seconds (e.g., 604800 seconds = 7 days).

  gitlab_rails['backup_keep_time'] = 604800
  

  • When gitlab-backup create runs, it will delete any backups in the backup_path older than this time before creating the new one.
• Backup Upload (Advanced): Configure automatic uploading of backups to cloud storage (AWS S3, Google Cloud Storage). Requires additional configuration parameters.

  # Example for S3 (requires setting up connection details)
  # gitlab_rails['backup_upload_connection'] = {
  #   'provider' => 'AWS',
  #   'region' => 'eu-west-1',
  #   'aws_access_key_id' => 'YOUR_ACCESS_KEY_ID',
  #   'aws_secret_access_key' => 'YOUR_SECRET_ACCESS_KEY'
  # }
  # gitlab_rails['backup_upload_remote_directory'] = 'my-gitlab-backups' # S3 bucket name
  

• Cron Job for Automation: GitLab doesn't automatically schedule backups. You need to set up a cron job manually.
  • Edit the root user's crontab: sudo crontab -e
  • Add a line to run the backup command at a specific time (e.g., daily at 2:30 AM):

    # Example: Run GitLab backup daily at 2:30 AM
    30 2 * * * /opt/gitlab/bin/gitlab-backup create CRON=1
    

    • CRON=1 suppresses command output unless there's an error, suitable for cron jobs.

Restoration Process (gitlab-backup restore)

Restoring a GitLab backup requires careful steps:

1. Prerequisites:
   • A running GitLab instance of the exact same version and type (CE/EE) from which the backup was taken. You cannot restore a backup from GitLab 15.8 onto a GitLab 15.9 instance directly.
   • The backup tar file (<timestamp>_<gitlab_version>_gitlab_backup.tar) placed in the configured backup directory (/var/opt/gitlab/backups/ by default). Permissions must allow the git user to read it.
   • The original /etc/gitlab/gitlab-secrets.json file from the instance where the backup was taken.
   • (Recommended) The original /etc/gitlab/gitlab.rb file for reference, although secrets are the critical part.
2. Prepare for Restore:
   • Stop the services that interact with the database:

     sudo gitlab-ctl stop puma
     sudo gitlab-ctl stop sidekiq
     # Verify they stopped
     sudo gitlab-ctl status
     

   • Crucially: Restore the secrets file. Copy the original /etc/gitlab/gitlab-secrets.json from your secure backup location to /etc/gitlab/ on the target server, replacing the existing one. Ensure permissions are correct (sudo chown root:root /etc/gitlab/gitlab-secrets.json; sudo chmod 0600 /etc/gitlab/gitlab-secrets.json).
   • (Optional but Recommended) Restore /etc/gitlab/gitlab.rb if needed.
3. Run the Restore Command:
   • Specify the backup timestamp you want to restore (from the filename, excluding the rest):

     # Replace <timestamp> with the correct one, e.g., 1678886400_2023_03_15
     sudo gitlab-backup restore BACKUP=<timestamp>
     

   • The command will ask for confirmation before wiping database tables, etc. Type yes to proceed.
   • This process can take time depending on the backup size. It restores the database, repositories, uploads, etc.
4. Reconfigure and Restart:
   • Run reconfigure to ensure all configurations are applied correctly based on /etc/gitlab/gitlab.rb and the restored state:

     sudo gitlab-ctl reconfigure
     

   • Restart the GitLab instance:

     sudo gitlab-ctl restart
     

5. Verification:
   • Run a health check:

     sudo gitlab-rake gitlab:check SANITIZE=true
     

   • Log in via the web UI and verify that projects, users, issues, and repository contents are restored correctly.

Disaster Recovery Strategy:

• Securely Back Up Secrets: Store copies of /etc/gitlab/gitlab.rb and especially /etc/gitlab/gitlab-secrets.json in a secure, separate location (e.g., password manager, encrypted USB drive, secure cloud storage). Update these whenever you run gitlab-ctl reconfigure as secrets can change.
• Off-Site Backups: Ensure your main backup .tar files are copied regularly to a different physical location or reliable cloud storage. This protects against local disasters (fire, theft, hardware failure). Use tools like rsync, rclone, or GitLab's built-in upload features.
• Test Restores: Periodically practice restoring a backup to a temporary, non-production server to ensure your backup files and procedures are valid.

Workshop Implementing a Backup Strategy

This workshop focuses on creating manual backups, configuring automated backups via cron, and understanding the critical role of the secrets file. Simulating a full restore can be complex and risky on a live instance, so we will focus on the backup creation and discuss the restore steps.

Goal: Manually create a backup, configure daily backups, and securely handle configuration/secrets files.

Prerequisites: GitLab CE running. Access via SSH.

Steps:

1. Securely Copy Configuration and Secrets:

   • These files are essential for restoration. Copy them now before making any changes.
   • On the GitLab server VM (via SSH):

     # Create a secure temporary directory in the admin user's home
     mkdir ~/gitlab_secrets_backup
     chmod 700 ~/gitlab_secrets_backup
     
     # Copy the files
     sudo cp /etc/gitlab/gitlab.rb ~/gitlab_secrets_backup/
     sudo cp /etc/gitlab/gitlab-secrets.json ~/gitlab_secrets_backup/
     
     # Set ownership to the admin user
     sudo chown -R $(whoami):$(whoami) ~/gitlab_secrets_backup/
     

   • Crucially: Use scp or another secure method to transfer the contents of ~/gitlab_secrets_backup/ from the VM to a secure location on your host machine or a password manager.

     # Run this on your HOST machine, replace VM_IP and path if needed
     scp -r gitlabadmin@<VM_IP_ADDRESS>:~/gitlab_secrets_backup/ .
     

   • Verify the files are copied securely, then remove the temporary directory on the VM:

     rm -rf ~/gitlab_secrets_backup/
     

   • Remember: You need to repeat this secure copying process whenever you run gitlab-ctl reconfigure as the secrets might change.

2. Manually Create a Backup:

   • Run the backup command:

     sudo gitlab-backup create
     

   • Observe the output. It will show progress for dumping the database, backing up repositories, uploads, etc.
   • Note the location of the backup file (default: /var/opt/gitlab/backups/).
   • List the contents of the backup directory:

     sudo ls -lh /var/opt/gitlab/backups/
     

   • You should see a .tar file with a timestamp and GitLab version in its name. Note its size.

3. Configure Backup Path and Retention (Optional):

   • Let's pretend we want to store backups in /srv/gitlab-backups and keep them for 14 days.
   • Create the directory and set permissions for the git user:

     sudo mkdir -p /srv/gitlab-backups
     sudo chown git:git /srv/gitlab-backups
     sudo chmod 700 /srv/gitlab-backups # Restrict access
     

   • Edit gitlab.rb:

     sudo nano /etc/gitlab/gitlab.rb
     

   • Find or add these lines, uncommenting and modifying as needed:

     gitlab_rails['backup_path'] = "/srv/gitlab-backups"
     gitlab_rails['backup_keep_time'] = 1209600 # 14 days in seconds (14 * 24 * 60 * 60)
     

   • Save the file.
   • Apply the configuration:

     sudo gitlab-ctl reconfigure
     

   • (Important): Since reconfigure was run, securely copy /etc/gitlab/gitlab-secrets.json and /etc/gitlab/gitlab.rb again!

4. Set Up Automated Backups with Cron:

   • Edit the root user's crontab:

     sudo crontab -e
     

     • If prompted, choose an editor (like nano).
   • Add the following line at the bottom to schedule a backup every night at 3:00 AM:

     # GitLab Backup Job
     0 3 * * * /opt/gitlab/bin/gitlab-backup create CRON=1
     

     • Syntax: minute hour day_of_month month day_of_week command
     • CRON=1 suppresses output unless there's an error.
   • Save and close the crontab file. Cron will automatically pick up the new job.
   • You can verify the cron job exists: sudo crontab -l

5. Simulate/Discuss Restore:

   • Do NOT perform these restore steps on your primary instance unless you intend to overwrite it. This is for understanding the process.
   • Scenario: Imagine your server failed, and you built a new VM with the exact same GitLab version.
   • Steps you would take:
     1. Install GitLab CE (same version) on the new VM. Run sudo gitlab-ctl reconfigure once.
     2. Stop Puma and Sidekiq: sudo gitlab-ctl stop puma sidekiq.
     3. Securely transfer the original /etc/gitlab/gitlab-secrets.json (that you backed up in Step 1) to /etc/gitlab/ on the new VM, replacing the one generated during installation. Set correct permissions (sudo chmod 0600 /etc/gitlab/gitlab-secrets.json).
     4. Securely transfer the backup .tar file (e.g., 1678886400_2023_03_15_15.8.1-ce_gitlab_backup.tar) to the configured backup directory on the new VM (e.g., /var/opt/gitlab/backups/ or /srv/gitlab-backups/ if you changed it). Ensure the git user can read it (sudo chown git:git <backup_file>).
     5. Run the restore command, specifying the backup timestamp: sudo gitlab-backup restore BACKUP=1678886400_2023_03_15. Confirm with yes.
     6. Run reconfigure: sudo gitlab-ctl reconfigure.
     7. Restart GitLab: sudo gitlab-ctl restart.
     8. Perform checks: sudo gitlab-rake gitlab:check SANITIZE=true and verify via the web UI.

You now have a manual backup, automated nightly backups configured, and crucially, you've secured the essential configuration and secrets files required for a successful restore. Remember to regularly test your restore process in a non-production environment if possible.

7. GitLab CI/CD Basics

One of GitLab's most powerful features is its integrated Continuous Integration (CI) and Continuous Deployment/Delivery (CD) capability. GitLab CI/CD allows you to automate the building, testing, and deployment of your projects directly from your repository.

Core Concepts:

• Continuous Integration (CI): The practice of frequently merging code changes from multiple developers into a central repository, after which automated builds and tests are run. This helps identify integration issues, bugs, and regressions early in the development cycle.
• Continuous Deployment/Delivery (CD):
  • Continuous Delivery: Extends CI by automating the release process, so that tested code can be deployed to a staging or production environment with a manual trigger.
  • Continuous Deployment: Goes one step further by automating the deployment to production for every change that passes the CI pipeline.
• Pipeline: A pipeline defines the entire CI/CD process for your project. It's composed of stages and jobs. Pipelines are typically triggered automatically on commits or merge requests but can also be run manually.
• Stages: Define the logical phases of a pipeline (e.g., build, test, deploy). Jobs within the same stage can run in parallel (if enough runners are available), while stages run sequentially (the next stage only starts if all jobs in the previous stage succeed).
• Jobs: The smallest unit of work in a pipeline. Each job executes a specific script (e.g., compile code, run unit tests, build a Docker image, deploy to a server). Jobs run within the context of a GitLab Runner.
• .gitlab-ci.yml: A YAML file located in the root directory of your repository. This file defines the structure and order of your pipeline (stages, jobs, scripts, rules, variables, etc.). It's the heart of your CI/CD configuration.
• GitLab Runner: An agent (separate application) that executes the jobs defined in your .gitlab-ci.yml. Runners can be installed on different machines (including the GitLab server itself, separate VMs, or Kubernetes clusters) and connect back to the GitLab instance. They poll the GitLab server for pending jobs, execute them, and send the results (logs, artifacts) back.
• Artifacts: Files generated by a job (e.g., compiled binaries, test reports, Docker images) that can be saved and passed to jobs in later stages or downloaded via the GitLab UI.

GitLab Runner Architecture:

• Runner Types:
  • Shared Runners: Available to all projects in a GitLab instance (if enabled by the admin). Useful for general tasks, managed centrally.
  • Group Runners: Available to all projects within a specific group. Useful for sharing runners among related projects.
  • Specific Runners: Tied to individual projects. Useful for jobs with specific requirements (e.g., special hardware, licenses, deployment credentials).
• Executors: Determine how the Runner executes the job scripts. Common executors include:
  • Shell: Executes scripts directly on the machine where the Runner is installed. Simple but requires careful management of dependencies on the Runner machine.
  • Docker: Uses Docker containers to run jobs. Each job runs in a clean, isolated environment defined by a specified Docker image. Manages dependencies effectively.
  • Docker Machine: Automatically provisions VMs (e.g., on cloud providers) to run jobs using Docker.
  • Kubernetes: Runs jobs as pods within a Kubernetes cluster. Highly scalable and fault-tolerant.
  • VirtualBox/Parallels: Runs jobs inside virtual machines.

Basic .gitlab-ci.yml Structure:

# Define the stages the pipeline will run through
stages:
  - build
  - test
  - deploy # Example stage, might not be used initially

# Job 1: Compile the code (runs in the 'build' stage)
build_job:
  stage: build # Assign job to the 'build' stage
  script:
    - echo "Compiling the code..."
    # Add actual compilation commands here, e.g., make, mvn compile, npm run build
    - echo "Compile complete."
  # (Optional) Define artifacts to save from this job
  # artifacts:
  #   paths:
  #     - build/output/ # Path to the compiled files/binaries

# Job 2: Run unit tests (runs in the 'test' stage)
test_job:
  stage: test # Assign job to the 'test' stage
  script:
    - echo "Running unit tests..."
    # Add actual test commands here, e.g., make test, mvn test, npm test
    - echo "Tests finished."
  # (Optional) Specify dependencies if this job needs artifacts from 'build_job'
  # needs:
  #   - build_job

# Job 3: Example deployment job (runs in the 'deploy' stage)
deploy_job:
  stage: deploy
  script:
    - echo "Deploying application..."
    # Add actual deployment commands here (e.g., scp, ansible, kubectl apply)
    - echo "Deployment finished."
  # Often restricted to run only on specific branches (e.g., main/master)
  # rules:
  #   - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH # Only run on the default branch

When you commit a .gitlab-ci.yml file to your repository, GitLab automatically detects it and tries to start a pipeline using available Runners that match the job's tags (if specified).

Workshop Your First CI/CD Pipeline

This workshop guides you through installing a GitLab Runner, registering it with your GitLab instance, creating a simple project, and defining a basic CI/CD pipeline using .gitlab-ci.yml.

Goal: Set up a GitLab Runner and create a pipeline that runs simple "build" and "test" jobs for a project.

Prerequisites:

• GitLab CE running and accessible.
• Logged in as an admin user (to view Runner settings) and the tdev user (to work on the project).
• The Lab Notebook project created earlier (or create a new empty project).
• Ability to run commands with sudo on the machine where the Runner will be installed (this can be the GitLab server VM itself for simplicity, or a separate machine).

Steps:

1. Install GitLab Runner (on the GitLab Server VM):

   • SSH into your GitLab server VM.
   • Follow the official instructions for Linux distributions (we'll use the steps for Debian/Ubuntu):
     • Download the binary for your system architecture (usually amd64):

       # Download the binary for your system
       sudo curl -L --output /usr/local/bin/gitlab-runner "https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64"
       

     • Give it execute permissions:

       sudo chmod +x /usr/local/bin/gitlab-runner
       

     • Create a Linux user for the Runner service:

       sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash
       

     • Install as a service and start it:

       sudo /usr/local/bin/gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
       sudo /usr/local/bin/gitlab-runner start
       

     • Verify the service is running: sudo /usr/local/bin/gitlab-runner status (or sudo systemctl status gitlab-runner if systemd is used by the install script).

2. Register the Runner: Runners need to be registered with a GitLab instance to pick up jobs. Registration can create a Shared, Group, or Specific Runner. Let's register this one as a Specific Runner for the Lab Notebook project.

   • Get Registration Info from GitLab:
     • Log in to GitLab as the tdev user (or any user with Maintainer/Owner access to the project).
     • Navigate to the Lab Notebook project: Menu -> Projects -> Your projects -> My Awesome Research / Lab Notebook.
     • Go to Settings -> CI/CD.
     • Expand the "Runners" section.
     • In the "Specific runners" area, you'll find the registration URL (e.g., https://gitlab.yourdomain.com/ or http://<IP_ADDRESS>/) and the registration token. Copy these two values.
   • Run the Registration Command on the VM:
     • Execute the following command in the VM's terminal:

       sudo gitlab-runner register
       

     • The command will prompt you for information:
       • Enter the GitLab instance URL: Paste the URL you copied from the GitLab UI.
       • Enter the registration token: Paste the token you copied.
       • Enter a description for the runner: e.g., Project Lab Notebook Runner
       • Enter tags for the runner (comma-separated): Tags are used to select specific runners for jobs. Enter something like shell, linux, development. Press Enter.
       • Enter optional maintenance note: Leave blank, press Enter.
       • Enter an executor: This is crucial. For simplicity now, enter shell. Press Enter.
     • If successful, it will say "Runner registered successfully."
   • Verify in GitLab: Refresh the Settings -> CI/CD -> Runners page in GitLab. The newly registered runner should appear under "Specific runners" with a green circle indicating it's connected.

3. Create the .gitlab-ci.yml File:

   • On your local machine, navigate to your lab-notebook project directory (the one you cloned earlier, either via HTTPS or SSH).
   • Create a new file named .gitlab-ci.yml in the root of the project directory.
   • Add the following content:

     stages:          # List of stages for jobs, executed in the specified order
       - build
       - test
     
     build_job:       # This job runs in the build stage
       stage: build
       script:
         - echo "Checking environment..."
         - pwd # Print working directory
         - ls -la # List files
         - echo "Simulating build process..."
         - mkdir build_output # Create a dummy output directory
         - echo "Build successful!" > build_output/status.txt
     
     test_job:        # This job runs in the test stage
       stage: test
       script:
         - echo "Checking for build output..."
         - cat build_output/status.txt # Try to read the output from build stage (will fail initially)
         - echo "Simulating test process..."
         - sleep 5 # Wait 5 seconds
         - echo "Tests passed!"
     

     • Note: This initial version won't share the build_output directory between stages. We'll fix that with artifacts next.

4. Commit and Push the .gitlab-ci.yml File:

   • In your local terminal, inside the lab-notebook directory:

     git add .gitlab-ci.yml
     git commit -m "Add initial CI pipeline configuration"
     git push origin main # Or your default branch name
     

5. Observe the Pipeline:

   • Go back to your Lab Notebook project in the GitLab web UI.
   • Navigate to CI/CD -> Pipelines.
   • You should see a new pipeline running (or recently finished) for your latest commit. It will likely show the build_job as passed and the test_job as failed.
   • Click on the pipeline status icon (e.g., "failed").
   • Click on the test_job. You'll see the job log. It probably failed on the cat build_output/status.txt command because that directory doesn't exist in the test_job's execution environment. Each job runs independently unless artifacts are used.

6. Fix the Pipeline with Artifacts:

   • Edit the .gitlab-ci.yml file locally again.
   • Modify the build_job to declare build_output as an artifact:

     stages:
       - build
       - test
     
     build_job:
       stage: build
       script:
         - echo "Checking environment..."
         - pwd
         - ls -la
         - echo "Simulating build process..."
         - mkdir build_output
         - echo "Build successful!" > build_output/status.txt
       artifacts:
         paths:
           - build_output/ # Declare the directory as an artifact
     
     test_job:
       stage: test
       script:
         - echo "Checking for build output..."
         - ls -la # See if the artifact is present
         - cat build_output/status.txt # This should now work
         - echo "Simulating test process..."
         - sleep 5
         - echo "Tests passed!"
       # Optional: Declare dependency explicitly (often inferred)
       # needs:
       #   - build_job
     

7. Commit and Push Again:

   git add .gitlab-ci.yml
   git commit -m "Fix pipeline: Use artifacts to pass build output"
   git push origin main
   

8. Observe the Successful Pipeline:
   • Go back to CI/CD -> Pipelines in GitLab.
   • A new pipeline should start. This time, both the build_job and test_job should pass.
   • Click into the test_job log; you should see the ls -la command listing the build_output directory, and the cat command successfully printing "Build successful!".
   • On the main pipeline view, you might also see options to download artifacts.

You have successfully set up a GitLab Runner and created a working CI/CD pipeline that automatically executes predefined scripts on code changes, demonstrating the fundamental workflow of GitLab CI/CD.

8. Monitoring and Troubleshooting GitLab

Keeping your GitLab instance running smoothly requires understanding how to monitor its health and how to diagnose problems when they arise. GitLab provides built-in tools and extensive logging for this purpose.

Built-in Monitoring (Prometheus & Grafana):

Omnibus GitLab includes optional components for advanced monitoring:

• Prometheus: A time-series database and monitoring system. GitLab services are instrumented to expose metrics (CPU usage, memory consumption, request latency, job queue lengths, etc.) in a format Prometheus can scrape.

• Grafana: A visualization platform. GitLab bundles pre-configured Grafana dashboards that connect to the Prometheus data source, providing graphical insights into GitLab's performance and health.

• Enabling (/etc/gitlab/gitlab.rb):

  # Enable Prometheus
  prometheus['enable'] = true
  
  # Enable Grafana (optional, but provides dashboards)
  grafana['enable'] = true
  
  # Optional: Set scrape interval, retention time, etc.
  # prometheus['scrape_interval'] = 15
  # prometheus['storage_retention'] = '15d'
  
  # Optional: Set Grafana external URL if different from main GitLab URL
  # grafana['external_url'] = 'https://grafana.gitlab.yourdomain.com/' # Requires separate DNS/proxy setup
  # If not set, Grafana is usually accessible via /-/grafana on the main GitLab URL
  
  # Optional: Set Grafana admin password (default is 'admin', change recommended)
  # grafana['initial_root_password'] = 'YOUR_STRONG_GRAFANA_PASSWORD'
  

• Applying: sudo gitlab-ctl reconfigure
• Accessing: If grafana['external_url'] is not set, Grafana is typically available at https://gitlab.yourdomain.com/-/grafana. Log in (default user admin, default pass admin unless changed in gitlab.rb). You'll find pre-built dashboards for various GitLab components (Overview, Rails, Sidekiq, Gitaly, etc.).
• Resource Usage: Note that enabling Prometheus and Grafana consumes additional server resources (CPU, RAM, Disk for metrics storage).

Essential GitLab Logs:

Logs are invaluable for diagnosing errors. Most GitLab component logs reside under /var/log/gitlab/. Key logs include:

• nginx/gitlab_access.log & nginx/gitlab_error.log: Nginx logs showing incoming web requests, status codes, and errors related to web serving or proxying. Useful for 5xx errors.
• gitlab-rails/production.log: The main log for the GitLab Ruby on Rails application. Records web requests processed by Puma, API calls, database queries, and application-level errors. Often the first place to look for application issues.
• gitlab-rails/application.log & gitlab-rails/application_json.log: Contain structured (JSON) logs providing more detailed context for application events.
• sidekiq/current: Logs from the Sidekiq background job processor. Useful for diagnosing issues with email sending, repository imports, CI/CD job processing (scheduling aspects), and other background tasks.
• gitaly/current: Logs from the Gitaly service, which handles Git repository access. Check here for errors related to cloning, pushing, fetching, or other Git operations failing.
• postgresql/current: Logs from the bundled PostgreSQL database.
• redis/current: Logs from the bundled Redis server.
• puma/puma_stdout.log & puma/puma_stderr.log: Standard output and error streams from the Puma application server running the Rails code.
• gitlab-shell/gitlab-shell.log: Logs related to Git operations performed over SSH.
• gitlab-runner/current (if Runner installed as service): Logs specific to the GitLab Runner service itself (not the jobs it runs). Job logs are viewed via the GitLab UI.
• reconfigure/<timestamp>.log: Logs generated during gitlab-ctl reconfigure runs. Useful for debugging configuration failures.

Key gitlab-ctl Commands for Troubleshooting:

The gitlab-ctl command is your primary tool for managing and interacting with the Omnibus GitLab services.

• sudo gitlab-ctl status: Shows the running status (run: or down:) and uptime of all managed services. Essential first check.
• sudo gitlab-ctl start | stop | restart [service_name]: Manages individual services (e.g., sudo gitlab-ctl restart nginx, sudo gitlab-ctl stop sidekiq). sudo gitlab-ctl restart (no service name) restarts everything.
• sudo gitlab-ctl tail [service_name]: Tails the logs for specified services (e.g., sudo gitlab-ctl tail gitlab-rails, sudo gitlab-ctl tail nginx gitaly). Press Ctrl+C to stop. Very useful for watching logs in real-time while reproducing an issue.
• sudo gitlab-ctl reconfigure: Re-applies configuration from /etc/gitlab/gitlab.rb. Use after any change to that file. Also useful if configuration seems inconsistent.
• sudo gitlab-ctl show-config: Displays the final configuration that GitLab will generate based on /etc/gitlab/gitlab.rb and defaults. Useful for verifying settings before a reconfigure.
• sudo gitlab-ctl check-config: Performs a quick syntax check on /etc/gitlab/gitlab.rb without applying changes.
• sudo gitlab-rake gitlab:check SANITIZE=true: Runs a comprehensive GitLab application health check, verifying database connectivity, repository access, permissions, and more. SANITIZE=true prevents sensitive information from being displayed.
• sudo gitlab-rails console: Starts a Rails console connected to your GitLab instance. For advanced debugging and data manipulation (use with extreme caution!).

Common Issues and First Steps:

• 500 Internal Server Error: Usually an application-level error. Check gitlab-rails/production.log and potentially puma/puma_stderr.log for specific error messages or stack traces around the time the error occurred.
• 502 Bad Gateway: Often means Nginx couldn't connect to the upstream Puma/Workhorse service. Possible causes: Puma is down, overloaded, or crashing. Check gitlab-ctl status, Puma logs (puma/*), and potentially nginx/gitlab_error.log. Resource exhaustion (CPU/RAM) can also cause this.
• High Resource Usage (CPU/RAM): Identify the culprit process using tools like top or htop. Check Prometheus/Grafana dashboards if enabled. Could be Puma (heavy web traffic), Sidekiq (many background jobs), Gitaly (heavy Git activity), or PostgreSQL. Check corresponding logs for errors or unusual activity. May require tuning settings in gitlab.rb (e.g., Puma workers, Sidekiq concurrency) or upgrading hardware.
• Failed Upgrades: Check the reconfigure log for the specific timestamp of the failed upgrade attempt. Address the errors reported there. Consult the official GitLab upgrade documentation for version-specific notes.
• CI/CD Jobs Stuck/Pending: Ensure GitLab Runner service is running (sudo gitlab-runner status or sudo systemctl status gitlab-runner). Check if the Runner is registered and active in the GitLab UI (Admin -> CI/CD -> Runners or Project/Group -> Settings -> CI/CD -> Runners). Ensure job tags match Runner tags. Check sidekiq/current log for job scheduling issues. Check Runner logs for execution errors.
• Git Operations Failing (SSH/HTTPS):
  • SSH: Check gitlab-shell/gitlab-shell.log on the server. Verify user's SSH key is correctly added to GitLab. Check SSH configuration (/etc/ssh/sshd_config) and firewall rules.
  • HTTPS: Check nginx/* logs and gitlab-rails/production.log. Ensure HTTPS/Let's Encrypt is configured correctly if used. Check for password/token issues.

Workshop Monitoring and Debugging

This workshop involves enabling built-in monitoring (if resources permit), using gitlab-ctl commands to inspect the system, and simulating a minor configuration issue to practice troubleshooting.

Goal: Explore monitoring tools, use diagnostic commands, and practice identifying and fixing a configuration error.

Prerequisites: GitLab CE running. Access via SSH. Sufficient resources (especially RAM > 8GB) if enabling Prometheus/Grafana.

Steps:

1. (Optional but Recommended) Enable Prometheus & Grafana:

   • Check Resources: Ensure your VM has adequate RAM (ideally 10GB+ allocated if enabling monitoring).
   • Edit gitlab.rb:

     sudo nano /etc/gitlab/gitlab.rb
     

   • Uncomment or add and set to true:

     prometheus['enable'] = true
     grafana['enable'] = true
     # Optional: Change Grafana default password
     # grafana['initial_root_password'] = 'MyGrafanaPass123!'
     

   • Save the file.
   • Reconfigure:

     sudo gitlab-ctl reconfigure
     

     • This may take longer as it sets up the monitoring services.
   • (Important): Securely copy /etc/gitlab/gitlab-secrets.json and /etc/gitlab/gitlab.rb again after reconfiguring.
   • Access Grafana: Open https://gitlab.yourdomain.com/-/grafana (or your specific Grafana URL) in your browser. Log in (user admin, password admin or the one you set). Explore the available dashboards (e.g., "GitLab Omnibus Overview"). Note: It might take a few minutes for data to populate.

2. Use gitlab-ctl for Inspection:

   • Check Status:

     sudo gitlab-ctl status
     

     • Verify all expected services are run:. Note the PIDs and uptime.
   • Tail Logs: Simulate some activity (e.g., browse GitLab UI, push a small change) and watch logs in real-time. Open two SSH terminals to the VM if possible.
     • Terminal 1: sudo gitlab-ctl tail nginx gitlab-rails
     • Terminal 2: sudo gitlab-ctl tail sidekiq gitaly
     • Perform actions in the GitLab web UI and watch the logs scroll. Press Ctrl+C in each terminal to stop tailing.
   • Check Configuration:

     sudo gitlab-ctl check-config
     # View the applied config (can be long output)
     # sudo gitlab-ctl show-config | less
     

3. Simulate a Configuration Error:

   • Let's introduce a typo into gitlab.rb that will cause reconfigure to fail.
   • Edit the file:

     sudo nano /etc/gitlab/gitlab.rb
     

   • Find a simple, commented-out setting, like # gitlab_rails['time_zone'] = 'UTC'.
   • Uncomment it but introduce a syntax error (e.g., remove the closing quote):

     gitlab_rails['time_zone'] = 'UTC # Missing closing quote
     

   • Save the file.

4. Attempt Reconfigure and Observe Failure:

   • Run the reconfigure command:

     sudo gitlab-ctl reconfigure
     

   • This time, the command should fail fairly quickly. Look closely at the output. It will likely show:
     • A Ruby syntax error message.
     • The specific line number in /etc/gitlab/gitlab.rb causing the problem.
     • Information about a Chef run failure.
   • Note the error message and the line number indicated.

5. Debug and Fix the Error:

   • Based on the error message ("syntax error", line number X), re-edit the configuration file:

     sudo nano +<line_number> /etc/gitlab/gitlab.rb
     

     • (Replace <line_number> with the actual number from the error output. nano +<number> opens the file directly at that line).
   • Examine the line. You should spot the missing quote.
   • Correct the syntax:

     gitlab_rails['time_zone'] = 'UTC' # Fixed quote
     

   • Save the file.

6. Verify Fix and Reconfigure Successfully:

   • (Optional) Run a quick syntax check first:

     sudo gitlab-ctl check-config
     

     • This should now pass without errors.
   • Run reconfigure again:

     sudo gitlab-ctl reconfigure
     

   • It should now complete successfully ("GitLab Reconfigured!").

7. Perform a Health Check:

   • Run the GitLab check task:

     sudo gitlab-rake gitlab:check SANITIZE=true
     

   • Review the output. Most checks should pass (show green "yes" or "OK"). Some minor warnings might appear depending on your setup, but major components like DB connection, Gitaly access, etc., should be fine.

This workshop demonstrated how to enable monitoring, use gitlab-ctl for diagnostics, and follow a common troubleshooting pattern: observe failure -> check logs/output -> identify root cause (config error) -> fix -> verify.

9. Advanced GitLab Configuration and Scaling

Beyond the basics, GitLab offers a wealth of features and configuration options. As your usage grows, you might also need to consider scaling strategies and high availability.

Customizing GitLab Settings:

Many features can be enabled and configured via /etc/gitlab/gitlab.rb. Remember to run sudo gitlab-ctl reconfigure after each change.

• Email Settings (SMTP): Crucial for notifications, password resets, and CI/CD emails. GitLab defaults to using sendmail (via Postfix installed earlier), but using an external SMTP provider (like SendGrid, Mailgun, AWS SES, or your organization's mail server) is more reliable.

  # Disable bundled Postfix if using external SMTP
  # postfix['enable'] = false
  
  gitlab_rails['smtp_enable'] = true
  gitlab_rails['smtp_address'] = "smtp.example.com"
  gitlab_rails['smtp_port'] = 587
  gitlab_rails['smtp_user_name'] = "your_smtp_username"
  gitlab_rails['smtp_password'] = "your_smtp_password"
  gitlab_rails['smtp_domain'] = "example.com" # The domain sending emails
  gitlab_rails['smtp_authentication'] = "login" # Or plain, cram_md5
  gitlab_rails['smtp_enable_starttls_auto'] = true
  gitlab_rails['smtp_tls'] = false # Set to true if using port 465 (SMTPS)
  gitlab_rails['smtp_openssl_verify_mode'] = 'peer' # Or none, client_once
  
  # Set the email address GitLab uses to send emails
  gitlab_rails['gitlab_email_from'] = 'gitlab@yourdomain.com'
  gitlab_rails['gitlab_email_display_name'] = 'GitLab Instance Name'
  

• Integrated Mattermost: Include a Mattermost instance (open-source Slack alternative) bundled with GitLab for team chat integrated with GitLab projects.

  mattermost_external_url 'https://mattermost.yourdomain.com' # Needs separate DNS
  # mattermost['enable'] = true # Usually enabled by setting the URL
  # Additional Mattermost config (email, auth, etc.)
  

• Container Registry: Enable an integrated Docker container registry, allowing you to host private Docker images directly within your GitLab projects. Requires HTTPS to be configured for Docker client authentication.

  registry_external_url 'https://registry.yourdomain.com' # Often uses a port or subdomain
  # Example using the main domain + port 5050
  # registry_external_url 'https://gitlab.yourdomain.com:5050'
  
  # The following are usually handled automatically when registry_external_url is set
  # gitlab_rails['registry_enabled'] = true
  # registry['enable'] = true
  # registry['registry_http_addr'] = "localhost:5000" # Internal listening address
  
  # If using the same domain as GitLab but different port, configure Nginx proxy:
  # registry_nginx['enable'] = true
  # registry_nginx['listen_port'] = 5050
  # registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/gitlab.yourdomain.com.crt" # Use GitLab's main cert
  # registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/gitlab.yourdomain.com.key"
  

• Other Settings: Explore /etc/gitlab/gitlab.rb for options related to Git LFS, GitLab Pages, OAuth providers, repository storage paths, security headers, and much more.

Scaling and High Availability (HA) Concepts:

A single-node Omnibus installation is suitable for many use cases, but for larger organizations or mission-critical deployments, scaling and HA become important. GitLab supports various architectures:

• Vertical Scaling: Increase the resources (CPU, RAM, Disk I/O) of the single server running GitLab. Simplest approach, but has limits and provides no redundancy.
• Horizontal Scaling (Component Separation): Run different GitLab components (e.g., PostgreSQL, Redis, Gitaly, Sidekiq/Puma nodes) on separate servers. This distributes the load and allows independent scaling of components. The Omnibus package supports configuring connections to external services.
  • Example: External PostgreSQL (gitlab.rb on GitLab node)

    # Disable bundled PostgreSQL
    postgresql['enable'] = false
    # Configure connection to external DB
    gitlab_rails['db_adapter'] = 'postgresql'
    gitlab_rails['db_encoding'] = 'unicode'
    gitlab_rails['db_database'] = 'gitlabhq_production'
    gitlab_rails['db_host'] = 'your_postgres_server_ip_or_hostname'
    gitlab_rails['db_port'] = 5432
    gitlab_rails['db_username'] = 'gitlab'
    gitlab_rails['db_password'] = 'gitlab_db_password'
    

• High Availability (HA): Aims to eliminate single points of failure, ensuring GitLab remains available even if one component or server fails. This is significantly more complex to set up and manage. Key components for HA include:
  • Application Nodes: Multiple servers running Puma/Sidekiq, load balanced (e.g., using HAProxy or a cloud load balancer).
  • PostgreSQL HA: Requires a clustered PostgreSQL setup (e.g., using Patroni, Pgpool-II, or cloud provider managed services like AWS RDS with Multi-AZ).
  • Redis HA: Requires Redis Sentinel for leader election and failover.
  • Gitaly Cluster (using Praefect): Gitaly, which manages Git data, needs its own HA solution. Praefect acts as a router and replication manager, distributing Git operations across multiple Gitaly nodes and ensuring data replication.
• GitLab Geo (Premium Feature): Not strictly HA, but provides read-only secondary sites in different geographical locations, improving Git performance for distributed teams and offering disaster recovery capabilities.

Setting up full HA is an advanced undertaking requiring deep knowledge of each component, networking, and load balancing. Consult the official GitLab Reference Architectures documentation for detailed guidance.

GitLab Environment Toolkit (GET): For complex, multi-node deployments (including HA and Geo), GitLab provides the GET, an Ansible-based toolkit to help provision and configure the necessary infrastructure according to reference architectures.

Workshop Enabling the Container Registry

This workshop guides you through enabling and configuring the integrated Docker Container Registry.

Goal: Set up the Container Registry, log in using Docker, and push/pull an image to a project's registry.

Prerequisites:

• GitLab CE running.
• HTTPS must be enabled and working correctly for your GitLab instance (external_url starts with https://). Docker requires a secure connection to registries unless explicitly configured otherwise (which is not recommended).
• Docker installed and running on your local machine (the one you use to interact with GitLab).
• Logged in to GitLab as tdev or another user with Developer+ permissions on the Lab Notebook project.

Steps:

1. Choose Registry URL Strategy: Decide how users will access the registry. Common options:

   • Subdomain: registry.gitlab.yourdomain.com (Requires separate DNS A record pointing to your GitLab server IP). Cleanest approach.
   • Main Domain + Port: gitlab.yourdomain.com:5050 (Uses the existing DNS record but a non-standard port). Simpler DNS-wise.
   • We will use the Port option (5050) for this workshop as it avoids extra DNS setup.

2. Configure gitlab.rb for Registry:

   • SSH into your GitLab VM.
   • Edit the configuration file:

     sudo nano /etc/gitlab/gitlab.rb
     

   • Find the registry_external_url setting. Uncomment and set it using your main GitLab domain and port 5050:

     # Replace gitlab.yourdomain.com with your actual domain
     registry_external_url 'https://gitlab.yourdomain.com:5050'
     

   • Ensure the Nginx proxy for the registry is enabled (usually automatic when using a port on the main domain, but good to verify):
     • Find registry_nginx['enable']. If commented out or set to false, uncomment/set it to true.
     • Ensure registry_nginx['listen_port'] is set (or defaults) to the port used in registry_external_url (5050).
     • Verify that registry_nginx['ssl_certificate'] and registry_nginx['ssl_certificate_key'] are uncommented and pointing to your main GitLab SSL certificate files (usually handled automatically by Omnibus when using Let's Encrypt or standard paths). Example defaults:

       # registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/gitlab.yourdomain.com.crt"
       # registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/gitlab.yourdomain.com.key"
       

       You typically don't need to change these if Let's Encrypt is managing your main cert.
   • Ensure the main registry components are enabled (usually automatic when registry_external_url is set):

     # gitlab_rails['registry_enabled'] = true
     # registry['enable'] = true
     

   • Save the file (Ctrl+X, Y, Enter).

3. Reconfigure GitLab:

   • Apply the changes:

     sudo gitlab-ctl reconfigure
     

   • Watch the output for steps related to configuring the registry and its Nginx proxy.
   • (Important): Securely copy /etc/gitlab/gitlab-secrets.json and /etc/gitlab/gitlab.rb again.

4. Verify Firewall:

   • Ensure the port you configured (5050 in this case) is allowed through the firewall on the GitLab server:

     sudo ufw status # Check if 5050/tcp is listed as ALLOW IN
     # If not allowed:
     sudo ufw allow 5050/tcp
     sudo ufw reload
     

5. Log in to the Registry using Docker:

   • On your local machine (where Docker is installed), open a terminal or command prompt.
   • Use the docker login command, specifying your registry URL (domain + port):

     # Replace gitlab.yourdomain.com with your actual domain
     docker login gitlab.yourdomain.com:5050
     

   • You will be prompted for:
     • Username: Enter your GitLab username (e.g., tdev).
     • Password: Do NOT use your regular GitLab password. You need to use either:
       • A Personal Access Token (PAT): Recommended. Go to GitLab UI -> User Settings (Avatar) -> Access Tokens. Create a token with at least read_registry and write_registry scopes. Copy the generated token immediately (it won't be shown again) and use it as the password.
       • A Deploy Token: Project-specific or Group-specific token designed for CI/CD or script access. Go to Project/Group -> Settings -> Repository -> Deploy Tokens. Create one with read_registry and write_registry scopes. Use the generated token username and password.
       • For simplicity in this workshop, let's quickly create a PAT for tdev.
   • If login is successful, you'll see "Login Succeeded".

6. Prepare a Docker Image:

   • Create a simple Dockerfile in a new directory on your local machine:

     mkdir my-docker-app
     cd my-docker-app
     nano Dockerfile
     

   • Add the following content to Dockerfile:

     FROM alpine:latest
     CMD ["echo", "Hello from GitLab Registry!"]
     

   • Save the file.

7. Build and Tag the Docker Image:

   • Build the image. Critically, you need to tag it with the full path to your GitLab project's registry: <registry_url>/<group_path>/<project_path>/<image_name>:<tag>
   • Run the build command:

     # Replace gitlab.yourdomain.com, my-awesome-research, lab-notebook with your actual values
     docker build -t gitlab.yourdomain.com:5050/my-awesome-research/lab-notebook/my-app:v1.0 .
     

     • . indicates the build context is the current directory (where Dockerfile is).

8. Push the Image to the Registry:

   • Push the tagged image:

     docker push gitlab.yourdomain.com:5050/my-awesome-research/lab-notebook/my-app:v1.0
     

   • Docker will upload the image layers to your GitLab server's registry.

9. Verify in GitLab UI:

   • Go to your Lab Notebook project in the GitLab web UI.
   • Navigate to Packages & Registries -> Container Registry.
   • You should see your my-app image listed with the v1.0 tag. Click on it to see details, including the push timestamp and commands to pull it.

10. Pull the Image (Test):

    • (Optional) Remove the local copy of the image first to ensure you're pulling from the registry:

      docker rmi gitlab.yourdomain.com:5050/my-awesome-research/lab-notebook/my-app:v1.0
      

    • Pull the image back from your GitLab registry:

      docker pull gitlab.yourdomain.com:5050/my-awesome-research/lab-notebook/my-app:v1.0
      

    • Run a container from the pulled image:

      docker run --rm gitlab.yourdomain.com:5050/my-awesome-research/lab-notebook/my-app:v1.0
      

      • It should output: Hello from GitLab Registry!

You have successfully enabled the GitLab Container Registry, authenticated with it using Docker, and pushed/pulled a container image associated with your project. This provides a private, integrated registry for your development workflow.

Conclusion

Throughout this guide, we've journeyed from the fundamental concepts of GitLab CE to installing, configuring, securing, managing, and utilizing its advanced features like CI/CD and the Container Registry on your own self-hosted instance.

You began by preparing a suitable Linux environment and installing GitLab using the convenient Omnibus package. We covered essential initial setup, including setting the external URL and understanding the core components managed by gitlab-ctl.

Progressing to intermediate topics, you learned to manage users, groups, and permissions effectively using GitLab's role-based access control. We emphasized security by implementing HTTPS with Let's Encrypt and enforcing Two-Factor Authentication. Furthermore, you established a crucial backup strategy, understanding what gitlab-backup includes and the critical importance of safeguarding the gitlab-secrets.json file alongside regular backups.

In the advanced sections, you were introduced to the transformative power of GitLab CI/CD, setting up a GitLab Runner and creating your first automated pipeline defined by .gitlab-ci.yml. We explored essential monitoring techniques using built-in tools like Prometheus/Grafana and practiced troubleshooting common issues by analyzing logs and utilizing gitlab-ctl diagnostic commands. Finally, we touched upon advanced configuration possibilities, such as enabling the Container Registry, and discussed conceptual approaches to scaling and achieving high availability for larger deployments.

By completing the workshops, you've gained practical, hands-on experience with real-world tasks involved in administering a GitLab instance. You now possess a solid foundation for managing your own DevOps platform.

Next Steps and Further Learning:

• Explore GitLab Features: Dive deeper into features we only touched upon, such as Merge Requests, Code Review workflows, Issue Boards, Wikis, GitLab Pages, Security Scanning (DAST, SAST - some features require higher tiers or manual integration in CE), and more complex CI/CD scenarios (variables, environments, deployment strategies).
• Official GitLab Documentation: The official documentation (docs.gitlab.com) is the definitive resource. It's comprehensive, well-maintained, and covers every aspect of GitLab in detail. Pay special attention to the Administration section and the CI/CD documentation.
• GitLab Community Forum: Engage with other GitLab users and administrators at forum.gitlab.com for help and discussions.
• Experiment: Set up test projects, experiment with different .gitlab-ci.yml configurations, try out integrations, and don't be afraid to break things (preferably in a test environment or VM snapshot) to learn how to fix them.
• Reference Architectures: If considering larger or HA deployments, study the official GitLab Reference Architectures.

Self-hosting GitLab CE is a rewarding endeavor that grants you complete control over your software development lifecycle. Keep learning, stay updated with new releases, and tailor your instance to perfectly fit your needs.