Part 1 — The Roles: Who Does What

People often blur these three together. They are separate concerns:

Piece	What it actually is	Responsibility
CI/CD	A practice / methodology	The rules: integrate often, build once, test automatically, deploy repeatably
Jenkins	An automation server	The orchestrator — watches Git, runs the steps, gates approvals, reports results
Docker	A packaging & runtime technology	The artifact — produces one immutable image that runs identically everywhere

A one-line summary:

CI/CD is the strategy. Jenkins executes the strategy. Docker is what gets built, moved, and run.

1.1 CI vs CD vs CD

Continuous Integration (CI)
    Developers merge to main frequently.
    Every commit triggers: build + automated tests.
    Goal: catch integration breakage in minutes, not weeks.

Continuous Delivery (CD)
    Every passing build produces a deployable artifact,
    automatically pushed to a registry and deployed to Dev/QA.
    Production deploy requires a human click.
    Goal: always be *able* to release.

Continuous Deployment (CD)
    Same as above, but Production deploy is automatic too — no button.
    Goal: release many times a day.

Most enterprise teams run Continuous Delivery: everything automated up to Production, with a manual approval gate before Production.

1.2 Why Docker Makes CI/CD Actually Work

Without Docker, CI/CD has a persistent weakness — the build agent's environment drifts from the production server's environment. You tested on agent config X and deployed onto server config Y.

Docker fixes this by making the artifact be the environment:

The image contains the runtime, dependencies, and app together.
The exact bytes tested in QA are the exact bytes running in Production.
Rollback becomes "redeploy the previous image digest" — seconds, not a rebuild.
Jenkins agents themselves can be containers, so build environments are reproducible too.

Part 2 — The Full Flow

2.1 End-to-End Pipeline

   Developer
      │  git push / open PR
      ▼
   Git (GitHub / GitLab / Bitbucket)
      │  webhook
      ▼
┌─────────────────────────────────────────────┐
│                 JENKINS                     │
│                                             │
│  1. Checkout source                         │
│  2. Restore + Build                         │
│  3. Unit tests            ──── fail ──▶ ✗   │
│  4. Static analysis / SonarQube             │
│  5. docker build  ──▶  myapp:git-a1b2c3d    │
│  6. Image scan (Trivy)    ──── HIGH ──▶ ✗   │
│  7. docker push  ──▶  Registry              │
└─────────────────────────────────────────────┘
      │
      ▼
   Container Registry  (ACR / ECR / Docker Hub / Harbor)
      │
      │  the SAME image digest flows down
      ├──────────────▶ Deploy to DEV      (automatic)
      │                    │ smoke tests
      ├──────────────▶ Deploy to QA       (automatic)
      │                    │ integration + API tests
      │                    │ ⏸ manual approval
      └──────────────▶ Deploy to PRODUCTION
                           │ health checks
                           │ rollback on failure

2.2 The Golden Rule: Build Once, Promote Everywhere

This is the single most important idea in the whole document.

Wrong — rebuilding per environment:

Build for Dev   →  image A   (NuGet resolved Tuesday)
Build for QA    →  image B   (base image patched Wednesday)
Build for Prod  →  image C   (a transitive dependency bumped)

Three different artifacts. QA signed off on image B; Production runs image C. You tested something you never shipped.

Right — one artifact, promoted:

Build once  →  myapp@sha256:9f2c...
                 │
                 ├─▶ Dev          (ASPNETCORE_ENVIRONMENT=Development)
                 ├─▶ QA           (ASPNETCORE_ENVIRONMENT=QA)
                 └─▶ Production   (ASPNETCORE_ENVIRONMENT=Production)

Only environment variables and secrets differ between environments — never the image.

2.3 Where Configuration Enters

IMAGE (immutable, built once)
    application code
    .NET runtime
    dependencies
              +
ENVIRONMENT (injected at deploy time, per environment)
    ASPNETCORE_ENVIRONMENT
    ConnectionStrings__Default
    JWT__Secret
    Feature flags
              =
RUNNING CONTAINER

If you ever find yourself putting an environment name inside a Dockerfile, something has gone wrong.

Part 3 — Jenkins Fundamentals

3.1 Jenkins Architecture

        Jenkins Controller (master)
        - Stores jobs, config, credentials
        - Schedules work, renders UI
        - Should NOT run builds itself
                 │
      ┌──────────┼──────────┐
      ▼          ▼          ▼
   Agent 1    Agent 2    Agent 3
   (Linux)   (Docker)   (Windows)
   Executes the actual build steps

Best practice: the controller never executes builds. Set its executor count to 0. Builds run on agents — ideally ephemeral Docker containers or Kubernetes pods that are destroyed after each run, so no state leaks between builds.

3.2 Freestyle Jobs vs Pipeline

	Freestyle Job	Pipeline (`Jenkinsfile`)
Definition	Clicked together in the UI	Code, committed to the repo
Versioned	No	Yes — reviewed like any code
Complex flow	Painful	Natural (parallel, conditional, retry)
Recommendation	Legacy only	Use this

Always use a Jenkinsfile checked into the repository. The pipeline changes with the code that it builds.

3.3 Declarative Pipeline Skeleton

pipeline {
    agent any

    environment {
        REGISTRY = 'myregistry.azurecr.io'
        IMAGE    = 'albasher/marketplace-api'
    }

    options {
        timeout(time: 30, unit: 'MINUTES')
        buildDiscarder(logRotator(numToKeepStr: '30'))
        disableConcurrentBuilds()
    }

    stages {
        stage('Build')  { steps { echo 'build'  } }
        stage('Test')   { steps { echo 'test'   } }
        stage('Deploy') { steps { echo 'deploy' } }
    }

    post {
        success { echo 'OK' }
        failure { echo 'Broken' }
        always  { cleanWs() }
    }
}

Key blocks:

agent — where it runs (any node, a labelled node, or a Docker container)
environment — variables available to all stages
options — timeouts, log retention, concurrency control
stages / stage / steps — the actual work
post — always/success/failure/unstable hooks (notifications, cleanup)

3.4 Triggers

triggers {
    // Preferred: Git webhook pushes to Jenkins (instant, no wasted polling)
    githubPush()

    // Fallback when webhooks are not possible
    pollSCM('H/5 * * * *')

    // Nightly rebuild — catches base-image CVEs even with no code change
    cron('H 2 * * *')
}

Prefer webhooks over polling. Polling wastes agent capacity and delays feedback.

Part 4 — Docker + Jenkins in Practice

4.1 Three Ways Docker Shows Up in Jenkins

It is important to keep these separate in your head:

Docker as the build environment — the pipeline runs inside a container, so every build gets a clean, identical toolchain.
Docker as the artifact — the pipeline produces an image and pushes it to a registry.
Docker as the deployment target — the pipeline runs that image on a server or orchestrator.

Docker as the build environment

pipeline {
    agent {
        docker {
            image 'mcr.microsoft.com/dotnet/sdk:8.0'
            args  '-v $HOME/.nuget:/root/.nuget'
        }
    }
    stages {
        stage('Build') {
            steps { sh 'dotnet build -c Release' }
        }
    }
}

No .NET SDK needs to be installed on the agent. Upgrading .NET becomes a one-line change in the Jenkinsfile.

Different tools per stage

stages {
    stage('Backend') {
        agent { docker { image 'mcr.microsoft.com/dotnet/sdk:8.0' } }
        steps { sh 'dotnet test' }
    }
    stage('Frontend') {
        agent { docker { image 'node:20-alpine' } }
        steps { sh 'npm ci && npm run build' }
    }
}

4.2 Building Images Inside Jenkins — the Docker Socket Problem

To run docker build from inside a containerised agent, that agent needs access to a Docker daemon. Three options:

Approach	How	Verdict
Docker-outside-of-Docker (DooD)	Mount `/var/run/docker.sock` into the agent	Fast, common — but socket access ≈ root on the host. Only on trusted agents.
Docker-in-Docker (DinD)	Run a privileged `docker:dind` sidecar	Isolated, but `--privileged` is its own risk; slower (no shared layer cache)
Daemonless builders (Kaniko, Buildah, BuildKit rootless)	Build images without a Docker daemon	Preferred for shared/multi-tenant Jenkins

If you use DooD, never expose that agent to untrusted pull requests — a malicious PR can build a Jenkinsfile that owns the host.

4.3 Credentials — Never Hardcode

stage('Push') {
    steps {
        withCredentials([usernamePassword(
            credentialsId: 'acr-credentials',
            usernameVariable: 'REG_USER',
            passwordVariable: 'REG_PASS'
        )]) {
            sh '''
                echo "$REG_PASS" | docker login $REGISTRY -u "$REG_USER" --password-stdin
                docker push $REGISTRY/$IMAGE:$TAG
                docker logout $REGISTRY
            '''
        }
    }
}

Rules:

Store every secret in the Jenkins Credentials store (or better, Vault / Key Vault via a plugin).
Use --password-stdin, never -p $PASSWORD — the latter lands in process lists and logs.
Use single-quoted sh '''...''' so Groovy does not interpolate the secret into the script text (which would print it in the console log).
Always docker logout afterwards.

4.4 Tagging Strategy

environment {
    SHORT_SHA = "${GIT_COMMIT.take(7)}"
    TAG       = "${env.BUILD_NUMBER}-${SHORT_SHA}"
}

myapp:142-a1b2c3d     ← immutable, traceable to an exact commit  ✔ deploy this
myapp:1.4.2           ← semantic release tag                      ✔
myapp:latest          ← convenience pointer only                  ✘ never deploy

Deploy by digest (myapp@sha256:...) where the platform supports it. A digest is the only truly unambiguous reference.

Part 5 — A Complete Jenkinsfile

A realistic pipeline for a .NET API, covering build → test → image → scan → push → promote.

pipeline {
    agent { label 'linux && docker' }

    environment {
        REGISTRY   = 'myregistry.azurecr.io'
        IMAGE_NAME = 'albasher/marketplace-api'
        SHORT_SHA  = "${env.GIT_COMMIT.take(7)}"
        TAG        = "${env.BUILD_NUMBER}-${env.GIT_COMMIT.take(7)}"
        IMAGE      = "${REGISTRY}/${IMAGE_NAME}:${TAG}"
    }

    options {
        timeout(time: 45, unit: 'MINUTES')
        buildDiscarder(logRotator(numToKeepStr: '30'))
        disableConcurrentBuilds()
        timestamps()
    }

    triggers { githubPush() }

    stages {

        stage('Checkout') {
            steps {
                checkout scm
                sh 'git log -1 --oneline'
            }
        }

        stage('Build & Test') {
            agent {
                docker {
                    image 'mcr.microsoft.com/dotnet/sdk:8.0'
                    reuseNode true
                }
            }
            steps {
                sh 'dotnet restore'
                sh 'dotnet build -c Release --no-restore'
                sh '''
                    dotnet test -c Release --no-build \
                      --logger "trx;LogFileName=results.trx" \
                      --collect:"XPlat Code Coverage"
                '''
            }
            post {
                always {
                    mstest testResultsFile: '**/*.trx', failOnError: false
                }
            }
        }

        stage('Static Analysis') {
            steps {
                withSonarQubeEnv('sonar-server') {
                    sh 'dotnet sonarscanner begin /k:"marketplace-api"'
                    sh 'dotnet build -c Release'
                    sh 'dotnet sonarscanner end'
                }
            }
        }

        stage('Quality Gate') {
            steps {
                timeout(time: 10, unit: 'MINUTES') {
                    waitForQualityGate abortPipeline: true
                }
            }
        }

        stage('Docker Build') {
            steps {
                sh """
                    DOCKER_BUILDKIT=1 docker build \
                      --build-arg BUILD_VERSION=${TAG} \
                      --cache-from ${REGISTRY}/${IMAGE_NAME}:cache \
                      -t ${IMAGE} \
                      -f src/MyApp.Api/Dockerfile .
                """
            }
        }

        stage('Image Scan') {
            steps {
                sh """
                    trivy image --exit-code 1 \
                      --severity HIGH,CRITICAL \
                      --ignore-unfixed ${IMAGE}
                """
            }
        }

        stage('Push') {
            steps {
                withCredentials([usernamePassword(
                    credentialsId: 'acr-credentials',
                    usernameVariable: 'REG_USER',
                    passwordVariable: 'REG_PASS')]) {
                    sh '''
                        echo "$REG_PASS" | docker login $REGISTRY -u "$REG_USER" --password-stdin
                        docker push $IMAGE
                        docker logout $REGISTRY
                    '''
                }
            }
        }

        stage('Deploy to Dev') {
            when { branch 'develop' }
            steps {
                sh "./deploy.sh dev ${IMAGE}"
                sh "./smoke-tests.sh https://dev-api.example.com"
            }
        }

        stage('Deploy to QA') {
            when { branch 'main' }
            steps {
                sh "./deploy.sh qa ${IMAGE}"
                sh "./integration-tests.sh https://qa-api.example.com"
            }
        }

        stage('Approve Production') {
            when { branch 'main' }
            steps {
                timeout(time: 24, unit: 'HOURS') {
                    input message: "Deploy ${TAG} to Production?",
                          submitter: 'release-managers'
                }
            }
        }

        stage('Deploy to Production') {
            when { branch 'main' }
            steps {
                // Same image that passed QA — never rebuilt
                sh "./deploy.sh prod ${IMAGE}"
                sh "./health-check.sh https://api.example.com"
            }
            post {
                failure {
                    echo 'Health check failed — rolling back'
                    sh './rollback.sh prod'
                }
            }
        }
    }

    post {
        success {
            slackSend color: 'good',
                      message: "✅ ${env.JOB_NAME} #${env.BUILD_NUMBER} — ${TAG} deployed"
        }
        failure {
            slackSend color: 'danger',
                      message: "❌ ${env.JOB_NAME} #${env.BUILD_NUMBER} failed at ${env.STAGE_NAME}"
        }
        always {
            sh "docker rmi ${IMAGE} || true"
            cleanWs()
        }
    }
}

5.1 What Each Stage Buys You

Stage	Catches
Build & Test	Compilation errors, logic regressions
Static Analysis	Code smells, security hotspots, coverage drops
Quality Gate	Prevents merging code below the agreed bar
Docker Build	Broken Dockerfile, missing files, bad `.dockerignore`
Image Scan	Vulnerable OS packages and dependencies
Push	Makes the artifact available and traceable
Smoke / Integration tests	Wiring problems that unit tests cannot see
Approval gate	Business control over release timing
Health check + rollback	Limits blast radius of a bad deploy

Part 6 — Branching and Environment Mapping

A common, workable model:

feature/*  ──▶ Build + Test only (no image push)
                └── PR checks must pass to merge

develop    ──▶ Build + Test + Image + Push + Deploy to DEV
                └── smoke tests

main       ──▶ Build + Test + Image + Push + Deploy to QA
                └── integration tests
                └── ⏸ manual approval
                └── Deploy to PRODUCTION (same image)

hotfix/*   ──▶ Fast path: Build + Test + Image + approval + PROD

In the Jenkinsfile this is expressed with when:

stage('Push') {
    when { anyOf { branch 'main'; branch 'develop' } }
    steps { /* ... */ }
}

Feature branches deliberately do not push images — otherwise the registry fills with thousands of throwaway tags.

Part 7 — Deployment Strategies

How the new image actually replaces the old one.

7.1 Recreate

Stop old  →  Start new

Simple; causes downtime. Acceptable for internal tools.

7.2 Rolling Update

v1 v1 v1 v1  →  v2 v1 v1 v1  →  v2 v2 v1 v1  →  v2 v2 v2 v2

No downtime. Both versions run simultaneously — so the database schema must be backward compatible.

7.3 Blue-Green

Blue (v1)  ◀── 100% traffic
Green (v2) ◀── 0%, warmed up and tested

           switch load balancer

Blue (v1)  ◀── 0%   (kept for instant rollback)
Green (v2) ◀── 100%

Instant rollback; costs double infrastructure during the switch.

7.4 Canary

v2 ◀── 5% traffic   → watch error rate & latency
v2 ◀── 25%          → still healthy?
v2 ◀── 100%

Lowest risk; needs good metrics and traffic-splitting support.

7.5 Database Migrations

The hardest part of zero-downtime deploys. Use expand/contract:

Release 1: ADD the new nullable column        (old code still works)
Release 2: Deploy code writing to both        (dual-write)
Release 3: Backfill data
Release 4: Deploy code reading only the new column
Release 5: DROP the old column

Never combine a destructive schema change with the code change that depends on it in a single release — that makes rollback impossible.

Part 8 — Best Practices

8.1 Pipeline Design

Fail fast. Order stages cheapest-and-most-likely-to-fail first: lint → unit tests → build → integration → image scan.
Keep pipelines under ~10 minutes for the CI portion. Longer than that and developers stop watching them.
Pipeline as code. Jenkinsfile lives in the repo, reviewed in PRs.

Parallelise independent work:

stage('Verify') {
    parallel {
        stage('Unit')  { steps { sh 'dotnet test --filter Category=Unit' } }
        stage('Lint')  { steps { sh 'dotnet format --verify-no-changes' } }
        stage('Scan')  { steps { sh 'trivy fs .' } }
    }
}

Extract shared logic into Shared Libraries (vars/buildDotnetImage.groovy) instead of copy-pasting 300-line Jenkinsfiles across 20 repos.
Make every stage idempotent — re-running must be safe.
Always cleanWs() in post { always }; leftover workspace state causes builds that pass or fail for no visible reason.

8.2 Docker in CI

Build once per commit. Never rebuild for QA or Production.
Tag with the commit SHA; deploy by digest.
Use BuildKit and --cache-from a registry cache layer to keep builds fast.
Scan before push, not after — do not publish known-vulnerable images.
Prune agents on a schedule (docker system prune -af --filter "until=72h"); Docker will silently eat the agent's disk otherwise.
Do not use latest anywhere in the pipeline.

8.3 Jenkins Operations

Controller runs no builds (0 executors). Agents do the work.
Ephemeral agents — a fresh container/pod per build, destroyed afterwards. No state leaks.
Secrets in the Credentials store or Vault — never in the Jenkinsfile, never in job config, never echoed.
Back up JENKINS_HOME (jobs, credentials, plugin state). Or better, manage it with JCasC (Jenkins Configuration as Code) so the whole controller is reproducible from a YAML file.
Pin plugin versions and update deliberately — plugin drift is a leading cause of "the pipeline broke and nobody changed anything".
Set timeouts on every job. A hung build holds an executor forever.
Restrict PR builds from forks. Untrusted code running on an agent with registry credentials and a Docker socket is a full compromise.
Rotate credentials and scope them narrowly — a push token should push to one repository, not the whole registry.

8.4 Testing Strategy in the Pipeline

        ▲  Fewer, slower, more realistic
        │      Smoke tests (post-deploy, against the real environment)
        │      E2E / API tests (QA)
        │      Integration tests (real DB in a container)
        │      Unit tests (fast, isolated, thousands)
        ▼  More, faster, cheaper

Run integration tests against a real database in a container rather than mocks:

stage('Integration Tests') {
    steps {
        sh 'docker compose -f docker-compose.test.yml up -d'
        sh 'dotnet test --filter Category=Integration'
    }
    post {
        always { sh 'docker compose -f docker-compose.test.yml down -v' }
    }
}

8.5 Observability of the Pipeline Itself

Track and act on:

Build duration trend — slow pipelines get bypassed
Failure rate and flaky test rate — a flaky suite trains people to hit "Retry" and ignore real failures
Mean time to recovery — how fast a bad deploy gets rolled back
Deployment frequency — the core DORA metric

8.6 Common Mistakes

Mistake	Consequence
Rebuilding per environment	QA validated an artifact that never reached Production
Deploying `:latest`	Nobody can tell what is actually running; rollback is guesswork
Secrets in `Jenkinsfile` or job config	Leaked in console logs and in Git history
Builds on the controller	One bad build can take down all of Jenkins
Long-lived, hand-configured agents	Snowflake agents; "works on agent-3 only"
No timeouts	Hung jobs exhaust the executor pool
Skipping tests "just this once" to ship	The gate becomes optional, then decorative
Manual production steps outside the pipeline	Undocumented drift; nobody can reproduce Production
No rollback path	A bad deploy becomes an outage instead of an inconvenience
Ignoring flaky tests	The whole suite loses credibility

Part 9 — Putting It All Together

The mental model to walk away with:

Docker answers:   "What exactly are we shipping?"       → an immutable image
Jenkins answers:  "How does it get built and moved?"    → an automated pipeline
CI/CD answers:    "Why do we work this way?"            → fast, safe, repeatable releases

And the four rules that matter most:

Build once, promote the same artifact through Dev → QA → Production.
Configuration comes from the environment, never from the image.
Every gate is automated — tests, scans, quality gates — with a human only at the Production approval.
Always have a rollback path, and rehearse it.