Linux Services: Overview and Management

Linux services are processes that run in the background and provide specific functionality to the system or to users. They typically start automatically at boot time and continue running until the system is shut down, handling tasks like networking, logging, printing, and more.

Service Management Systems

Linux distributions primarily use one of three service management systems:

System	Used In	Key Commands
systemd	Ubuntu (16.04+), Debian (8+), RHEL/CentOS (7+), Fedora	systemctl
SysVinit	Older distributions, some specialized systems	service, chkconfig
Upstart	Ubuntu (9.10 to 15.04)	initctl, start, stop

This documentation focuses primarily on systemd, the most widely used service manager in modern Linux distributions.

systemd Architecture

systemd is not a single daemon but a collection of programs, daemons, libraries, technologies, and kernel components. Unlike traditional init systems, systemd aims to standardize service management and extends much deeper into system operations.

Key characteristics of systemd:

• Provides parallel startup of services
• Manages dependencies between services
• Implements socket and D-Bus activation
• Offers on-demand starting of services
• Tracks processes using Linux control groups
• Maintains system state in dynamically managed directories

Understanding systemd Units

In systemd, all managed entities are known as "units." Units are categorized by type, with .service files defining services.

Common Unit Types

Unit Type	File Extension	Purpose
Service	.service	System services
Socket	.socket	IPC socket for service activation
Target	.target	Group of units (similar to runlevels)
Timer	.timer	Scheduled activation of units
Mount	.mount	Filesystem mount points
Device	.device	Kernel device units

Unit File Locations

systemd unit files are stored in several directories:

• /lib/systemd/system/: Units provided by installed packages
• /etc/systemd/system/: Units created by the system administrator
• /run/systemd/system/: Runtime units

Basic Service Management Commands

Viewing Services

# List all active services
systemctl list-units --type=service

# List all services, including inactive
systemctl list-units --type=service --all

# View detailed information about a specific service
systemctl status nginx.service

Controlling Services

# Start a service
sudo systemctl start nginx.service

# Stop a service
sudo systemctl stop nginx.service

# Restart a service
sudo systemctl restart nginx.service

# Reload a service's configuration without restarting
sudo systemctl reload nginx.service

# Check if a service is active
systemctl is-active nginx.service

Managing Service Startup

# Enable a service to start at boot
sudo systemctl enable nginx.service

# Disable a service from starting at boot
sudo systemctl disable nginx.service

# Check if a service is enabled
systemctl is-enabled nginx.service

# Enable and immediately start a service
sudo systemctl enable --now nginx.service

Creating a Custom Service

You can create your own systemd service by defining a unit file:

Example: Simple Web Server Service

1. Create a service file:

sudo nano /etc/systemd/system/mywebapp.service

2. Add the following content:

[Unit]
Description=My Custom Web Application
After=network.target

[Service]
Type=simple
User=webuser
WorkingDirectory=/var/www/myapp
ExecStart=/usr/bin/python3 app.py
Restart=on-failure

[Install]
WantedBy=multi-user.target

3. Reload systemd to recognize the new service:

sudo systemctl daemon-reload

4. Enable and start the service:

sudo systemctl enable --now mywebapp.service

Understanding Service Files

A service file consists of three main sections:

[Unit] Section

Contains metadata and dependencies:

[Unit]
Description=Nginx Web Server
Documentation=https://nginx.org/en/docs/
After=network.target network-online.target
Wants=network-online.target

Common options:

• Description: Human-readable description
• Documentation: URLs to documentation
• After: Units that must start before this one
• Requires: Hard dependencies
• Wants: Soft dependencies

[Service] Section

Defines how the service runs:

[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t -q -g 'daemon on;'
ExecStart=/usr/sbin/nginx -g 'daemon on;'
ExecReload=/usr/sbin/nginx -g 'daemon on;' -s reload
ExecStop=/bin/kill -s QUIT $MAINPID
TimeoutStopSec=5
KillMode=mixed

Common options:

• Type: How systemd treats the process (simple, forking, oneshot, etc.)
• ExecStart: Command to start the service
• ExecStop: Command to stop the service
• Restart: When to restart (always, on-failure, on-abort, etc.)
• User/Group: User/group to run the service as
• Environment: Environment variables

[Install] Section

Defines how and when the service is enabled:

[Install]
WantedBy=multi-user.target

Common options:

• WantedBy: Target that wants this service when enabled
• RequiredBy: Target that requires this service when enabled
• Alias: Alternative names for the unit

Understanding Unit Statuses

When managing services with systemd, you'll encounter different unit statuses that indicate the current state of a service. Understanding these statuses is essential for proper troubleshooting.

Status	Meaning
loaded	Unit file has been processed
active	Running successfully
inactive	Not running but configured properly
enabled	Set to start at boot
disabled	Not set to start at boot
failed	The unit has failed in some way
masked	Administratively blocked from being started
static	Cannot be enabled, but may be started by another unit

These statuses can be checked using the systemctl status command:

systemctl status nginx.service

Example output:

● nginx.service - NGINX Web Server
   Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2023-01-01 12:00:00 UTC; 3h ago
     Docs: https://nginx.org/en/docs/
  Process: 1234 ExecStart=/usr/sbin/nginx (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/nginx.service
           ├─1234 nginx: master process /usr/sbin/nginx
           └─1235 nginx: worker process

The enabled and disabled states apply only to unit files with an [Install] section. When enabled, the unit will start automatically at boot according to its target dependencies.

Service Types

The Type= setting in the [Service] section affects how systemd manages the service:

Type	Description	Example Use Case
simple	Main process is executed directly	Single-process applications
forking	Process forks, parent exits	Traditional daemons
oneshot	Process runs once and exits	Quick configuration tasks
notify	Process signals when ready	Services with readiness notification
idle	Process starts after other jobs finish	Low-priority background services
exec	Like simple but waits for exec()	Applications with complex startup

Managing Service Logs

systemd includes a unified logging system called journald:

# View logs for a specific service
journalctl -u nginx.service

# View logs for a service since boot
journalctl -u nginx.service -b

# Follow logs in real-time (like tail -f)
journalctl -u nginx.service -f

# View logs within a time range
journalctl -u nginx.service --since "2023-01-01" --until "2023-01-02"

Analyzing Service Dependencies

# List dependencies of a service
systemctl list-dependencies nginx.service

# List services that depend on a service
systemctl list-dependencies --reverse nginx.service

Managing Services with Socket Activation

Socket activation allows systemd to create the listening socket but only start the service when a connection arrives:

# Create a socket unit file
sudo nano /etc/systemd/system/myapp.socket

Example socket file:

[Unit]
Description=My Application Socket

[Socket]
ListenStream=8080
Accept=no

[Install]
WantedBy=sockets.target

Then create the corresponding service file:

[Unit]
Description=My Application
Requires=myapp.socket

[Service]
Type=simple
ExecStart=/usr/local/bin/myapp
StandardInput=socket
StandardError=journal

[Install]
WantedBy=multi-user.target

Managing Service Resource Limits

systemd allows setting resource limits for services:

[Service]
# Limit CPU and memory
CPUQuota=30%
MemoryLimit=512M

# Limit disk I/O
IOWeight=500
IODeviceWeight=/dev/sda 1000

# Other limits
LimitNOFILE=16384
LimitNPROC=4096

SysVinit Commands (for older systems)

If you're working on an older Linux system that uses SysVinit:

# Start a service
sudo service nginx start

# Stop a service
sudo service nginx stop

# Restart a service
sudo service nginx restart

# Check service status
sudo service nginx status

# Enable a service to start at boot
sudo chkconfig nginx on

# Disable a service from starting at boot
sudo chkconfig nginx off

Upstart Commands (for Ubuntu 9.10-15.04)

If you're working on an Ubuntu system from 9.10 to 15.04:

# Start a service
sudo start nginx

# Stop a service
sudo stop nginx

# Restart a service
sudo restart nginx

# Check service status
sudo status nginx

Converting Between Service Systems

SysVinit to systemd

If you need to convert an old SysVinit script to a systemd service:

# Use the systemd-sysv-generator tool
systemd-sysv-generator

# Manual conversion example
# Original: /etc/init.d/myservice
# Converted: /etc/systemd/system/myservice.service

[Unit]
Description=My Legacy Service
After=network.target

[Service]
Type=forking
ExecStart=/etc/init.d/myservice start
ExecStop=/etc/init.d/myservice stop
PIDFile=/var/run/myservice.pid

[Install]
WantedBy=multi-user.target

Troubleshooting Services

Common Issues and Solutions

1. Service fails to start:

   Check logs for error messages:

   journalctl -u myservice.service -b

2. Service starts but exits immediately:

   Check for configuration errors or missing dependencies:

   systemctl status myservice.service

3. Service won't enable at boot:

   Verify the [Install] section has the correct target:

   systemctl cat myservice.service | grep WantedBy

4. Service dependencies not loading:

   Check and fix the dependency chain:

   systemctl list-dependencies myservice.service

Debug Techniques

# Run a service in debug mode (if supported)
sudo systemctl start myservice.service --log-level=debug

# Run service executable manually to check for errors
sudo -u serviceuser /path/to/executable --options

# Check for permission issues
ls -la /path/to/service/directory

Best Practices

1. Security Hardening:

   [Service]
   # Run as non-root user
   User=appuser
   Group=appgroup
   
   # Restrict capabilities
   CapabilityBoundingSet=CAP_NET_BIND_SERVICE
   
   # Restrict file system
   ProtectSystem=strict
   ProtectHome=true
   PrivateTmp=true
   ReadWritePaths=/var/lib/myapp

2. Restart Policies:

   [Service]
   # Automatically restart on failure
   Restart=on-failure
   # Wait 10 seconds before restart
   RestartSec=10
   # Maximum 5 restarts in 30 seconds
   StartLimitInterval=30
   StartLimitBurst=5

3. Dependency Management: Use Wants= instead of Requires= for soft dependencies to prevent cascading failures.

Hands-on Lab Exercises

These practical exercises will help you gain experience with Linux service management. Each exercise builds upon knowledge from the previous ones.

Exercise 1: Exploring Existing Services

Objective: Become familiar with listing and examining service information

1. List all currently active services:

   systemctl list-units --type=service

2. Find specific information about the SSH service:

   systemctl status sshd.service

3. Check if the web server (Apache or Nginx) is installed and running:

   systemctl status apache2.service
   # or
   systemctl status nginx.service

4. List all services that start at boot:

   systemctl list-unit-files --type=service --state=enabled

5. Examine the logs for a specific service (e.g., SSH):

   journalctl -u sshd.service -n 20

Exercise 2: Controlling System Services

Objective: Practice starting, stopping, and restarting services

1. If Apache or Nginx is installed, stop the service:

   sudo systemctl stop apache2.service  # or nginx.service

2. Verify the service has stopped:

   systemctl is-active apache2.service

3. Start the service again:

   sudo systemctl start apache2.service

4. Reload the service configuration without full restart:

   sudo systemctl reload apache2.service

5. Restart the service completely:

   sudo systemctl restart apache2.service

Exercise 3: Creating a Custom Service

Objective: Create, configure and manage your own systemd service

1. Create a simple Python web server script:

   mkdir -p ~/myserver
   cd ~/myserver
   
   # Create a simple Python HTTP server script
   cat > server.py << 'EOF'
   #!/usr/bin/env python3
   from http.server import HTTPServer, SimpleHTTPRequestHandler
   import os
   
   # Change to the script directory
   os.chdir(os.path.dirname(os.path.abspath(__file__)))
   
   # Create sample index.html
   with open('index.html', 'w') as f:
       f.write('<html><body><h1>My Custom Service is Running!</h1></body></html>')
   
   # Run server
   server = HTTPServer(('localhost', 8000), SimpleHTTPRequestHandler)
   print('Starting server at http://localhost:8000')
   server.serve_forever()
   EOF
   
   # Make the script executable
   chmod +x server.py
   
   # Test run it (press Ctrl+C to stop after confirming it works)
   ./server.py

2. Create a systemd service file:

   sudo nano /etc/systemd/system/mywebserver.service

3. Create a setup script that will run before the service starts:

   cat > ~/myserver/setup.sh << 'EOF'
   #!/bin/bash
   echo "[$(date)] Pre-start setup running..." > /tmp/mywebserver_pre.log
   echo "Checking for required directory structure..." >> /tmp/mywebserver_pre.log
   mkdir -p ~/myserver/logs
   echo "Pre-start setup complete." >> /tmp/mywebserver_pre.log
   EOF
   
   chmod +x ~/myserver/setup.sh

4. Create a post-start notification script:

   cat > ~/myserver/post-start.sh << 'EOF'
   #!/bin/bash
   echo "[$(date)] Post-start actions running..." > /tmp/mywebserver_post.log
   echo "Service should now be available at http://localhost:8000" >> /tmp/mywebserver_post.log
   # Could send notification, update status file, etc.
   echo "Post-start actions complete." >> /tmp/mywebserver_post.log
   EOF
   
   chmod +x ~/myserver/post-start.sh

5. Add the following content to the service file with ExecStartPre and ExecStartPost:

   [Unit]
   Description=My Python Web Server
   After=network.target
   
   [Service]
   Type=simple
   User=$USER
   WorkingDirectory=/home/$USER/myserver
   
   # Run setup script before starting the service
   ExecStartPre=/home/$USER/myserver/setup.sh
   
   # Main service
   ExecStart=/home/$USER/myserver/server.py
   
   # Run notification script after service starts
   ExecStartPost=/home/$USER/myserver/post-start.sh
   
   Restart=on-failure
   
   [Install]
   WantedBy=multi-user.target

6. Reload systemd to recognize the new service:

   sudo systemctl daemon-reload

7. Start your custom service:

   sudo systemctl start mywebserver.service

8. Check the status of your service:

   systemctl status mywebserver.service

9. Test your web server using curl or a web browser:

   curl http://localhost:8000

10. Check the pre-start and post-start logs:

    cat /tmp/mywebserver_pre.log
    cat /tmp/mywebserver_post.log

11. View the service logs:

    journalctl -u mywebserver.service

12. Enable your service to start at boot:

    sudo systemctl enable mywebserver.service

Exercise 4: Modifying and Troubleshooting Services

Objective: Practice modifying service behavior and troubleshooting

1. Modify your web server service to run on a different port:

   # Edit the Python file
   nano ~/myserver/server.py

   Change the port from 8000 to 8080 in the HTTPServer line.

2. Restart your service to apply changes:

   sudo systemctl restart mywebserver.service

3. Intentionally introduce an error by modifying the service file:

   sudo nano /etc/systemd/system/mywebserver.service

   Change ExecStart=/home/$USER/myserver/server.py to ExecStart=/home/$USER/myserver/server_typo.py

4. Reload systemd and try to start the service:

   sudo systemctl daemon-reload
   sudo systemctl restart mywebserver.service

5. Check the status and logs to identify the problem:

   systemctl status mywebserver.service
   journalctl -u mywebserver.service -n 20

6. Fix the error and restart the service:

   sudo nano /etc/systemd/system/mywebserver.service
   # Correct the typo in ExecStart
   sudo systemctl daemon-reload
   sudo systemctl restart mywebserver.service

Exercise 5: Advanced Service Configuration

Objective: Experiment with advanced service features

1. Add memory and CPU limits to your service:

   sudo nano /etc/systemd/system/mywebserver.service

   Add the following lines to the [Service] section:

   # Resource limits
   MemoryLimit=100M
   CPUQuota=20%

2. Configure environment variables for your service: Add to the [Service] section:

   Environment="PORT=8000" "DEBUG=true"

3. Create an override configuration:

   sudo systemctl edit mywebserver.service

   This will open a new file. Add:

   [Service]
   # Override restart settings
   Restart=always
   RestartSec=5

4. Reload and restart to apply all changes:

   sudo systemctl daemon-reload
   sudo systemctl restart mywebserver.service

5. Clean up after completing exercises:

   sudo systemctl stop mywebserver.service
   sudo systemctl disable mywebserver.service
   sudo rm /etc/systemd/system/mywebserver.service
   sudo systemctl daemon-reload

Summary

Managing Linux services is primarily done through systemd in modern distributions. The systemctl command provides a unified interface for starting, stopping, enabling, and monitoring services. Custom services can be created by defining unit files in the /etc/systemd/system/ directory.

Understanding service management is essential for Linux system administration, allowing for proper control of background processes, application deployment, and system monitoring.

Understanding systemd Targets

In systemd, targets are units that group other units together and provide synchronization points during boot. They replace traditional SysVinit runlevels with a more flexible mechanism.

Common Targets

Target	Description	SysVinit Equivalent
poweroff.target	System shutdown	Runlevel 0
rescue.target	Single-user mode	Runlevel 1
multi-user.target	Multi-user, non-graphical	Runlevel 3
graphical.target	Multi-user, graphical	Runlevel 5
reboot.target	System reboot	Runlevel 6
emergency.target	Emergency shell	-

Managing Targets

# View current target
systemctl get-default

# Change default target
sudo systemctl set-default multi-user.target

# Switch to a target immediately
sudo systemctl isolate rescue.target

# List available targets
systemctl list-units --type=target

The isolate command is particularly important - it activates the specified target and its dependencies while deactivating all other units.

Service Customization and Overrides

One of systemd's powerful features is the ability to customize existing services without modifying the original unit files.

Creating Override Files

Instead of editing vendor-provided unit files (which may be overwritten during updates), you can create overrides:

# Create override directory and file
sudo systemctl edit nginx.service

This command creates a directory at /etc/systemd/system/nginx.service.d/ and opens an editor for the file override.conf. Any settings you add will override the original unit file settings.

Example override to change the timeout and add memory limits:

[Service]
TimeoutStartSec=30s
MemoryLimit=512M

After creating or modifying an override file, reload systemd:

sudo systemctl daemon-reload
sudo systemctl restart nginx.service

Viewing Combined Configuration

To see the effective configuration after all overrides are applied:

systemctl cat nginx.service

Masking Services

To completely prevent a service from being started (even manually):

sudo systemctl mask bluetooth.service

This creates a symlink from the service file to /dev/null, making it impossible to start. To unmask:

sudo systemctl unmask bluetooth.service

Further Reading

• systemd Documentation
• Red Hat systemd Guide
• Arch Linux systemd Wiki
• Digital Ocean systemd Guide
• UNIX and Linux System Administration Handbook, 5th Edition - Chapter 2.7 on systemd