Building a Production-Grade Automated Deployment Pipeline: From Zero to Deployed
Introduction
As a DevOps intern, I was tasked with creating an automated deployment pipeline that mirrors real world workflows. The challenge? Build a robust bash script that automates the complete deployment of a Dockerized application to a remote server, including server setup, container orchestration, and reverse proxy configuration.
This article walks through my entire journey from setting up AWS infrastructure via CLI to deploying a containerized application accessible to the world. If you're learning DevOps or preparing for DevOps roles, this hands on guide will give you practical skills that employers actually look for.
Tech Stack: AWS EC2, Docker, Nginx, Bash scripting, GitHub Actions concepts
Time to Complete: ~2-3 hours
Difficulty: Intermediate
Table of Contents
1. Project Overview & Requirements {#overview}
What We're Building
An automated deployment script (deploy.sh) that:
✅ Clones/updates a GitHub repository with authentication
✅ Establishes SSH connection to remote server
✅ Installs Docker, Docker Compose, and Nginx
✅ Transfers application files via rsync
✅ Builds Docker images and runs containers
✅ Configures Nginx as a reverse proxy
✅ Validates deployment with health checks
✅ Provides comprehensive logging
✅ Supports cleanup operations
Why This Matters
In real world DevOps, automation isn't optional it's essential. This project demonstrates:
Infrastructure as Code principles
Idempotent operations (safe to run multiple times)
Error handling and rollback strategies
Security best practices (SSH keys, token handling)
Documentation and maintainability
2. Setting Up AWS Infrastructure via CLI {#aws-setup}
Installing AWS CLI
Instead of using the AWS Console, l did everything via CLI the DevOps way.
First l created a folder for the project called devops-deployment
Configuring AWS Credentials
# Configure AWS CLI with IAM credentials
aws configure
# Inputs:
# AWS Access Key ID: AKIA...
# AWS Secret Access Key: wJalr...
# Default region: us-east-1
# Default output: json
Creating SSH Key Pair
# Generate SSH key pair for EC2 access
aws ec2 create-key-pair \
--key-name devops-key \
--query 'KeyMaterial' \
--output text > ~/.ssh/devops-key.pem
# Set correct permissions
chmod 400 ~/.ssh/devops-key.pem
Security Note: Never commit SSH keys to version control!
Setting Up Security Group
bash
# Create security group
aws ec2 create-security-group \
--group-name devops-sg \
--description "Security group for DevOps deployment"
# Allow SSH (port 22)
aws ec2 authorize-security-group-ingress \
--group-name devops-sg \
--protocol tcp \
--port 22 \
--cidr 0.0.0.0/0
# Allow HTTP (port 80)
aws ec2 authorize-security-group-ingress \
--group-name devops-sg \
--protocol tcp \
--port 80 \
--cidr 0.0.0.0/0
# Allow application port (3000)
aws ec2 authorize-security-group-ingress \
--group-name devops-sg \
--protocol tcp \
--port 3000 \
--cidr 0.0.0.0/0
I ran the command below to get AMI ID
aws ec2 describe-images \
--owners 099720109477 \
--filters "Name=name,Values=ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*" \
--query 'sort_by(Images, &CreationDate)[-1].ImageId' \
--output text \
--region us-east-1
Launching EC2 Instance
# Launch Ubuntu 22.04 t2.micro instance
aws ec2 run-instances \
--image-id ami-0c7................ \
--count 1 \
--instance-type t2.micro \
--key-name devops-key \
--security-groups devops-sg \
--tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=devops-server}]'
# Get public IP address
aws ec2 describe-instances \
--filters "Name=tag:Name,Values=devops-server" \
--query "Reservations[*].Instances[*].PublicIpAddress" \
--output text
Testing SSH Connection
# Connect to instance
ssh -i ~/.ssh/devops-key.pem ubuntu@54.227.183.150
# Output:
# ubuntu@ip-172-31-17-224:~$
Key Takeaway: Infrastructure setup via CLI is faster, repeatable, and scriptable compared to clicking through AWS Console.
Next l Prepared my GitHub Repository
Since l will be Creating a simple test application with a Dockerfile that l will deploy, l prepare a GitHub Repository for it .
In my terminal:
# Go back to your project folder
cd ~/devops-deployment
# Create a new directory for your app
mkdir test-app
cd test-app
# Initialize git
git init
3. Creating the Application {application}
Simple Node.js Express Application
// server.js
const express = require('express');
const app = express();
const PORT = process.env.PORT || 3000;
app.get('/', (req, res) => {
res.json({
message: 'DevOps Deployment Successful!',
timestamp: new Date().toISOString(),
environment: 'production'
});
});
app.get('/health', (req, res) => {
res.json({ status: 'healthy' });
});
app.listen(PORT, '0.0.0.0', () => {
console.log(`Server running on port ${PORT}`);
});
Package Configuration
json
{
"name": "devops-test-app",
"version": "1.0.0",
"description": "DevOps deployment automation demo",
"main": "server.js",
"scripts": {
"start": "node server.js"
},
"dependencies": {
"express": "^4.18.2"
}
}
Dockerfile
FROM node:18-alpine
WORKDIR /app
COPY package*.json ./
RUN npm install --production
COPY . .
EXPOSE 3000
CMD ["npm", "start"]
Why Node.js? Lightweight, quick to build, easy to demonstrate. The same principles apply to any Dockerized application (Python, Go, Java, etc.).
Below are the command code l made to create the application files just copy and paste and you are good to go.
# Create package.json
cat > package.json << 'EOF'
{
"name": "devops-test-app",
"version": "1.0.0",
"description": "Simple app for DevOps deployment",
"main": "server.js",
"scripts": {
"start": "node server.js"
},
"dependencies": {
"express": "^4.18.2"
}
}
EOF# Create server.js
cat > server.js << 'EOF'
const express = require('express');
const app = express();
const PORT = process.env.PORT || 3000;
app.get('/', (req, res) => {
res.json({
message: 'DevOps Deployment Successful!',
timestamp: new Date().toISOString(),
environment: 'production'
});
});
app.get('/health', (req, res) => {
res.json({ status: 'healthy' });
});
app.listen(PORT, '0.0.0.0', () => {
console.log(`Server running on port ${PORT}`);
});
EOF
# Create Dockerfile
cat > Dockerfile << 'EOF'
FROM node:18-alpine
WORKDIR /app
COPY package*.json ./
RUN npm install --production
COPY . .
EXPOSE 3000
CMD ["npm", "start"]
EOF
# Create .gitignore
cat > .gitignore << 'EOF'
node_modules/
npm-debug.log
.env
*.log
EOF
# Create README
cat > README.md << 'EOF'
# DevOps Test Application
Simple Node.js application for deployment automation testing.
## Endpoints
- `/` - Main endpoint
- `/health` - Health check
## Port
- Default: 3000
EOF
To Verify that all Files were created run the command
ls -la
Next l Created GitHub Repository (via browser )
l visited: https://github.com/new and did the following
Repository name:
devops-deployment-taskDescription:
Automated deployment script for HNG DevOps Stage 1l Choose: Public
l didn’t check any initialization options and l went ahead and
Clicked "Create repository"
GitHub showed me commands but l ignored them for now.
l went ahead to get GitHub Personal Access Token
Still in the browser:
l went to: https://github.com/settings/tokens
Click "Generate new token" → "Generated a new token (classic)"
Note:
DevOps Deployment TaskExpiration:
30 daysSelect scopes:
- ✅
repo(all checkboxes under it)
- ✅
Scrolled down, and clicked "Generate token" l copied the token and saved it temporarily and l also saved it somewhere save.
# In your terminal (don't worry, it won't commit this)
echo "export GITHUB_TOKEN='ghp_YOUR_TOKEN_HERE'" > ~/.github_token
source ~/.github_token
After generating the token l went ahead to push everything to github. In your terminal (still in test-app folder):
# Add all files
git add .
# Commit
git commit -m "Initial commit: Node.js app with Dockerfile"
# Add remote (replace YOUR_USERNAME with your GitHub username)
git remote add origin https://github.com/YOUR_USERNAME/devops-deployment-task.git
# Push using token
git push https://${GITHUB_TOKEN}@github.com/YOUR_USERNAME/devops-
i saved everything with the command
cat >> ~/server-details.txt << EOF
GitHub Repo: https://github.com/YOUR_USERNAME/devops-deployment-task
GitHub Token: ${GITHUB_TOKEN}
Branch: main
App Port: 3000
EOF
cat ~/server-details.txt
4. Building the Automation Script {automation}
Script Structure
#!/bin/bash
# Core Components:
# 1. Configuration & Variables
# 2. Helper Functions (logging, error handling)
# 3. Input Collection & Validation
# 4. Repository Operations
# 5. Remote Server Setup
# 6. Docker Deployment
# 7. Nginx Configuration
# 8. Validation & Health Checks
# 9. Cleanup Functions
I created deploy.sh with the above basic structures, first i added the logging system input Validation SSH & Remote Server Configuration and other component to complete a working Script.
Logging System
LOG_FILE="deploy_$(date +%Y%m%d_%H%M%S).log"
log() {
local message="$1"
local timestamp=$(date +'%Y-%m-%d %H:%M:%S')
echo -e "${BLUE}[${timestamp}]${NC} ${message}" | tee -a "$LOG_FILE"
}
success() {
local message="$1"
local timestamp=$(date +'%Y-%m-%d %H:%M:%S')
echo -e "${GREEN}[${timestamp}] ✓ ${message}${NC}" | tee -a "$LOG_FILE"
}
error_exit() {
local message="$1"
local exit_code="${2:-1}"
local timestamp=$(date +'%Y-%m-%d %H:%M:%S')
echo -e "${RED}[${timestamp}] ✗ ERROR: ${message}${NC}" | tee -a "$LOG_FILE"
exit "$exit_code"
}
Benefits:
Timestamped logs for debugging
Color-coded output for readability
Persistent log files for audit trail
Input Validation
# Example: IP address validation
while true; do
echo -ne "${BLUE}Enter Server IP Address: ${NC}"
read SERVER_IP
if [[ "$SERVER_IP" =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]; then
success "Server IP: $SERVER_IP"
break
else
warn "Invalid IP format"
fi
done
Why Validate? Catch errors early before wasting time on failed deployments.
5. SSH & Remote Server Configuration {ssh-config}
Testing SSH Connectivity
log "Testing SSH connection..."
if ssh -i "$SSH_KEY" -o ConnectTimeout=10 -o StrictHostKeyChecking=no \
"$SSH_USER@$SERVER_IP" "echo 'SSH OK'" > /dev/null 2>&1; then
success "SSH connection established"
else
error_exit "SSH connection failed" 5
fi
Remote Server Setup
ssh -i "$SSH_KEY" "$SSH_USER@$SERVER_IP" 'bash -s' <<'REMOTESCRIPT'
set -e
echo "[REMOTE] Updating packages..."
sudo apt-get update -y
echo "[REMOTE] Installing Docker..."
if ! command -v docker &> /dev/null; then
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker $USER
fi
echo "[REMOTE] Installing Docker Compose..."
if ! command -v docker-compose &> /dev/null; then
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" \
-o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
fi
echo "[REMOTE] Installing Nginx..."
if ! command -v nginx &> /dev/null; then
sudo apt-get install -y nginx
fi
sudo systemctl enable docker nginx
sudo systemctl start docker nginx
REMOTESCRIPT
Key Concept: Idempotent checks (if ! command -v) ensure the script doesn't fail when run multiple times.
6. Docker Deployment Automation {docker-deploy}
File Transfer via rsync
log "Transferring files..."
rsync -avz --progress \
-e "ssh -i $SSH_KEY -o StrictHostKeyChecking=no" \
--exclude='.git' \
--exclude='node_modules' \
./ "$SSH_USER@$SERVER_IP:~/app/"
Why rsync? Only transfers changed files, much faster than scp for updates.
Building and Running Container
ssh -i "$SSH_KEY" "$SSH_USER@$SERVER_IP" bash <<DEPLOYEOF
set -e
cd ~/app
echo "[REMOTE] Stopping old containers..."
sudo docker stop app-container 2>/dev/null || true
sudo docker rm app-container 2>/dev/null || true
echo "[REMOTE] Building Docker image..."
sudo docker build -t devops-app:latest .
echo "[REMOTE] Running container..."
sudo docker run -d \
--name app-container \
-p $APP_PORT:3000 \
--restart unless-stopped \
devops-app:latest
sleep 5
sudo docker ps | grep app-container
DEPLOYEOF
Best Practice:
|| trueprevents script failure if container doesn't exist--restart unless-stoppedensures container survives rebootsSleep gives container time to initialize
7. Nginx Reverse Proxy Setup {nginx-setup}
Why Reverse Proxy?
Serves traffic on standard port 80
Hides internal port (3000)
Enables SSL termination (future)
Load balancing capability
Professional production setup
Nginx Configuration
ssh -i "$SSH_KEY" "$SSH_USER@$SERVER_IP" bash <<NGINXEOF
sudo tee /etc/nginx/sites-available/devops-app > /dev/null <<'CONF'
server {
listen 80;
server_name _;
location / {
proxy_pass http://localhost:$APP_PORT;
proxy_set_header Host \$host;
proxy_set_header X-Real-IP \$remote_addr;
proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
}
location /health {
proxy_pass http://localhost:$APP_PORT/health;
proxy_set_header Host \$host;
}
}
CONF
# Replace variable with actual port
sudo sed -i "s/\\\$APP_PORT/$APP_PORT/g" /etc/nginx/sites-available/devops-app
# Enable site
sudo ln -sf /etc/nginx/sites-available/devops-app /etc/nginx/sites-enabled/
sudo rm -f /etc/nginx/sites-enabled/default
# Test and reload
sudo nginx -t
sudo systemctl reload nginx
NGINXEOF
Result: Application accessible at http://54.227.183.150 (no port needed!)
8. Implementing Idempotency & Cleanup {#idempotency}
Idempotency Principles
# Check before action
if [ -d "$REPO_NAME" ]; then
log "Repository exists, updating..."
git pull
else
log "Cloning repository..."
git clone "$REPO_URL"
fi
Why Important? Script can be run multiple times without breaking things.
Make Executable
chmod +x deploy.sh
Now Run It
./deploy.sh
```
**Enter your details:**
- Repo: `https://github.com/Nweke-cloud/devops-deployment-task`
- Token: [your token]
- Branch: `master`
- Username: `ubuntu`
- IP: `54.227.183.150`
- Key: [Enter]
- Port: `3000`
- Proceed: `yes`
Expected Output
### **Test in Browser**
l Opened my browser and tested itand it worked fine:
1. **Main endpoint (no port needed!):**
```
http://54.227.183.150
```
2. **Health endpoint:**
```
http://54.227.183.150/health
l added the Cleanup Function
cleanup_deployment() {
echo "Cleaning up deployment on $SSH_USER@$SERVER_IP..."
ssh -i "$SSH_KEY" "$SSH_USER@$SERVER_IP" bash <<'CLEANUPEOF'
sudo docker stop app-container 2>/dev/null || true
sudo docker rm app-container 2>/dev/null || true
sudo docker rmi devops-app:latest 2>/dev/null || true
rm -rf ~/app
sudo rm -f /etc/nginx/sites-enabled/devops-app
sudo systemctl reload nginx
CLEANUPEOF
echo "✓ Cleanup completed"
}
if [ "$1" = "--cleanup" ]; then
cleanup_deployment
fi
Test Cleanup Function
# make it Executable
chmod +x deploy.sh
# Test cleanup
./deploy.sh --cleanup
```
**Enter your SSH details when prompted.**
You should see:
```
Starting cleanup process...
Cleaning up deployment on ubuntu@54.227.183.150...
Stopping and removing containers...
Removing Docker image...
Removing application files...
Removing Nginx configuration...
Cleanup completed!
✓ All resources removed successfully
9. Testing & Validation {testing}
Automated Health Checks
log "Testing application endpoint..."
RESPONSE=$(ssh -i "$SSH_KEY" "$SSH_USER@$SERVER_IP" \
"curl -s -o /dev/null -w '%{http_code}' http://localhost")
if [ "$RESPONSE" = "200" ]; then
success "Application is responding (HTTP $RESPONSE)"
else
error_exit "Application health check failed (HTTP $RESPONSE)" 9
fi
Make Executable and Run
chmod +x deploy.sh
./deploy.sh # Fresh deployment
Manual Testing
# From local machine
curl http://54.227.183.150
# Expected output:
{
"message": "DevOps Deployment Successful!",
"timestamp": "2025-10-22T20:15:30.123Z",
"environment": "production"
}
# Health check
curl http://54.227.183.150/health
# Expected output:
{"status":"healthy"}
Verifying Container Status
ssh -i ~/.ssh/devops-key.pem ubuntu@54.227.183.150
# Check running containers
sudo docker ps
# Check logs
sudo docker logs app-container
# Check Nginx
sudo systemctl status nginx
10. Lessons Learned & Best Practices {lessons}
What Worked Well
CLI First Approach: Using AWS CLI made infrastructure reproducible
Comprehensive Logging: Timestamped logs saved hours of debugging
Input Validation: Caught errors before expensive operations
Idempotent Design: Script could run multiple times safely
Modular Structure: Easy to add features or modify sections
Challenges Faced
Challenge 1: Here-Document Syntax Errors
Problem: Nested heredocs in bash caused parsing errors
Solution: Used unique delimiters for each heredoc level.
cat > script.sh << 'OUTER'
ssh server << 'INNER'
commands
INNER
OUTER
Challenge 2: Variable Substitution in Remote Scripts
Problem: Variables weren't expanding correctly in SSH heredocs
Solution: Used proper quoting and sed for replacements
ssh server bash <<EOF # Unquoted = variables expand
echo "$LOCAL_VAR"
EOF
ssh server bash <<'EOF' # Quoted = no expansion
echo "\$REMOTE_VAR"
EOF
Security Best Practices Implemented
✅ SSH Keys Instead of Passwords
chmod 400 ~/.ssh/devops-key.pem # Strict permissions
✅ GitHub PAT for Authentication
# Never hardcode tokens
read -s GIT_PAT # Hidden input
REPO_URL="https://${GIT_PAT}@github.com/..."
✅ Principle of Least Privilege
Security groups only allow necessary ports
IAM user with minimal required permissions
✅ No Credentials in Version Control
# .gitignore
*.pem
*.log
.env
Performance Optimizations
rsync for File Transfer: Only transfers changed files
Docker Layer Caching: Speeds up subsequent builds
Conditional Installations: Skip already-installed software
Parallel Operations: Where possible (future improvement)
Final Architecture
┌─────────────────┐
│ Developer │
│ Machine │
└────────┬────────┘
│
│ ./deploy.sh
│
▼
┌─────────────────────────────────┐
│ GitHub Repository │
│ - deploy.sh │
│ - Dockerfile │
│ - Application Code │
└────────┬────────────────────────┘
│
│ git clone + rsync
│
▼
┌─────────────────────────────────┐
│ AWS EC2 (Ubuntu 22.04) │
│ 54.227.183.150 │
│ ┌──────────────────────────┐ │
│ │ Nginx (Port 80) │ │
│ │ Reverse Proxy │ │
│ └────────┬─────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────┐ │
│ │ Docker Container │ │
│ │ Node.js App (Port 3000) │ │
│ └──────────────────────────┘ │
└─────────────────────────────────┘
Results & Metrics
Deployment Speed:
Manual deployment: ~15-20 minutes
Automated deployment: ~3-5 minutes
Improvement: 75% faster
Error Reduction:
Manual process: 3-5 errors per deployment
Automated process: 0 errors (after script is debugged)
Improvement: 100% error reduction
Reproducibility:
Manual: Requires documentation, human memory
Automated: One command, consistent results
Improvement: Infinite reproducibility
Key Takeaways for Aspiring DevOps Engineers
Technical Skills Demonstrated
Bash Scripting: Advanced scripting with error handling, logging, and modularity
Cloud Infrastructure: AWS CLI, EC2, Security Groups, IAM
Containerization: Docker image building, container orchestration
Web Servers: Nginx configuration, reverse proxy setup
Version Control: Git operations, authentication with PAT
Networking: Understanding ports, proxies, security groups
Linux Administration: Package management, systemd services
DevOps Principles Applied
Automation First: Eliminate manual, error-prone processes
Infrastructure as Code: Everything scriptable and version-controlled
Idempotency: Operations produce same result regardless of how many times executed
Observability: Comprehensive logging and monitoring
Security: Never sacrifice security for convenience
Documentation: README as important as the code itself
Repository & Live Demo
GitHub: github.com/Nweke-cloud/devops-deployment-task
Live Application: http://54.227.183.150
What's Next?
Potential Improvements
Conclusion
Building this automated deployment pipeline taught me that DevOps isn't just about knowing tools it's about understanding systems, anticipating failures, and creating reliable, repeatable processes.
Every manual step is an opportunity for automation. Every deployment is an opportunity to improve the process. Every error is an opportunity to add better validation.
If you're learning DevOps, I encourage you to build projects like this. Don't just follow tutorials—build something end-to-end, encounter real problems, and solve them. That's where real learning happens.
The best way to learn DevOps is to DO DevOps.
Connect With Me
Questions? Comments? Improvements? Drop them below! I'm always learning and would love to hear your perspectiThis project was completed as part of the HNG DevOps Internship Stage 1. Special thanks to the HNG team for providing such practical, hands-on challenges that simulate real-world DevOps work.
