Securing your Container Supply Chain with Harbor and Trivy

By: Aayush Pokharel

CNCF Kathmandu - 22nd June Meetup

About Me

  • Aayush Pokharel
  • KU BSc.CS (2019)
  • DevOps Engr.
  • STARTsmall Pvt. Ltd.

Today's Topic of Discussion

  • Why Secure the Container Supply Chain?
  • Harbor: Your Cloud-Native Registry
  • Harbor Infrastructure and Replication
  • Trivy: Vulnerability Scanning Done Right
  • SBOMs and Compliance
  • Integrating Harbor and Trivy in CI/CD
  • Best Practices for DevOps
  • Conclusion

Why Secure the Container Supply Chain?

Modern Supply Chains

  • Containers enable rapid development and deployment.
  • Multi-cloud, hybrid environments increase complexity.
  • Dependencies span OSS, vendor images, and internal builds.

Risks

  • Vulnerabilities: 80% of images have known CVEs (Snyk 2024).
  • Malicious Images: Cryptominers on Docker Hub (e.g., 2020 incidents).
  • Compliance: SOC2, HIPAA require audit trails.

Real-World Attacks

  • SolarWinds (2020): Compromised build pipeline affected 18,000+ orgs.
  • Codecov (2021): Bash uploader script tampered, exposing secrets.

Harbor: Your Cloud-Native Registry

What is Harbor?

  • CNCF-graduated, OCI-compliant open-source registry.
  • Enterprise features missing in Docker Hub or basic registries.
  • Focus: Security, governance, and scalability.

Why DevOps Love It

  • Integrates with Kubernetes, Helm, and CI/CD pipelines.
  • Supports multi-tenancy for large teams.
  • Extensible via plugins (e.g., scanners, authentication).

Core Features

  • RBAC with LDAP/SSO integration.
  • Image signing (Notary v2, Cosign in Harbor 2.8+).
  • Built-in scanning (Trivy, Clair).
  • Replication for geo-distributed teams.
  • Immutable tags and retention policies.
  • REST API for automation.

Harbor Infrastructure

Components

  • NGINX: Reverse proxy for load balancing and SSL.
  • Harbor Core: Manages auth, metadata, and policies.
  • Registry: Docker/OCI distribution for image storage.
  • Database: Postgres for project and user data.
  • Storage Backend: S3, Azure Blob, filesystem, or Swift.
  • Jobservice: Handles async tasks (e.g., replication, scanning).

Harbor’s Replication Capabilities

Replication Modes

  • Push: Harbor → external registries (e.g., prod to DR site).
  • Pull: External → Harbor (e.g., sync trusted base images).

Supported Registries

  • Docker Hub
  • Quay.io
  • GHCR
  • AWS ECR
  • other Harbor instances

Use Cases

  • Geo-redundancy: Sync US and EU registries.
  • Trusted Sources: Pull Alpine images from Docker Hub.
  • Promotion: Dev → test → prod with policy checks.

Metrics

  • Track replication status via Prometheus endpoint.

Trivy Scanner

Vulnerability Scanning Done Right

What is Trivy?

  • Open-source scanner by Aqua Security.
  • Scans images, filesystems, Git repos, and SBOMs.
  • Lightweight: ~50MB binary, <1s startup.

Why DevOps Choose Trivy

  • Fast: Scans a 500MB image in ~10s (vs. Clair’s ~30s).
  • Broad coverage: OS (Alpine, Ubuntu) and app deps (npm, pip).
  • No daemon required, ideal for CI pipelines.

Integration with Harbor

  • Runs as a scanner adapter via HTTP API.
  • Auto-scans on push or scheduled.
  • Reports: CVE ID, severity, package, fix version.

How Trivy Works

Scanning Process

  • Pulls vuln DBs: NVD, RedHat, Debian, GitHub advisories.
  • Extracts image layers using skopeo.
  • Analyzes OS packages and app dependencies.
  • Matches against CVEs with severity scoring.
  • Outputs JSON, table, or SARIF reports.

Supported Targets

  • OS: apk, apt, yum, dnf.
  • Languages: Python (pip), Node.js (npm), Go, Java (Maven).
  • IaC: Dockerfiles, Terraform, Kubernetes manifests.

SBOMs and Compliance

What is an SBOM?

  • SBOM: Software Bill of Materials.
  • Lists all image components.
  • Formats:
    • CycloneDX (JSON/XML)
    • SPDX (JSON)
  • Trivy generates SBOMs in <5s for typical images.

Why It Matters

  • Compliance: Required for FedRAMP, CISA guidelines.
  • Zero-Day Response: Identify affected deps (e.g., Log4Shell).
  • Auditability: Track transitive dependencies.

Harbor Integration

  • Store SBOMs as artifacts in Harbor.
  • Query via API:
GET /api/v2.0/projects/{name}/artifacts/{digest}

Container Image Vulnerabilities

Common Issues

  • Outdated Libraries: e.g., OpenSSL < 3.0.7.
  • Misconfigured Images: Root users, exposed ports.
  • CVEs: ~20% of prod images have Critical vulns (Aqua 2024).

Harbor’s Reporting

  • Dashboard shows severity (Low to Critical).
  • Drill-down: Affected package, fix version, layer.
  • Export reports for audits.

Example Scenario

  • Image myapp:1.0 has CVE-2024-5678 (Critical).
  • Harbor blocks deployment until fixed version is pushed.

Best Practices for DevOps

Image Hygiene

  • Use minimal base images (e.g., alpine).
  • Scan images in CI and before deployment.
  • Enable Harbor’s auto-scan on push.

Policy Enforcement

  • Block High/Critical CVEs via Harbor policies.
  • Enforce immutable tags:
    PUT /api/v2.0/projects/{name}/immutable
  • Use signed images (Cosign/Notary v2)

SBOM and Audit

  • Generate SBOMs for all prod images.
  • Enable Harbor audit logs: Track pushes, pulls, scans.

Trusted Sources

  • Replicate only from verified registries.
  • Use Harbor’s proxy cache for Docker Hub.

Monitoring

  • Expose Harbor/Trivy metrics to Prometheus.
  • Alert on scan failures or replication delays.

Conclusion:

Let's Shift Left on Security

2024 Open Source Security Report: Slowing Progress and New Challenges for DevSecOps

SolarWinds (2020) Supply Chain Attack — Summary What happened: A nation-state-level supply chain attack where hackers compromised SolarWinds' Orion software, inserting a backdoor (SUNBURST malware) into official software updates. Method: Attackers (linked to Russian APT29/Cozy Bear) inserted malicious code into Orion updates (v2019.4.5220 to v2020.2.1 HF1), which were downloaded by ~18,000 customers. The malware stayed dormant, then contacted command-and-control servers and enabled lateral movement across networks. Impact: Breached U.S. government agencies (DHS, Treasury, Commerce), tech companies (Microsoft, FireEye), and many private sector organizations. It exposed critical systems and led to widespread espionage. Key Lessons: Trust in software updates can be exploited. Highlights the need for supply chain security, zero trust architecture, cloud monitoring, and use of Software Bill of Materials (SBOM). Detection requires behavioral monitoring beyond signature-based tools.

Codecov (2021) Supply Chain Attack — Summary What happened: Attackers compromised Codecov’s Bash Uploader script, a tool used by developers to upload code coverage reports, and modified it to exfiltrate sensitive environment variables (like API keys, credentials, tokens). Method: Attackers gained access to Codecov’s Docker image building process using leaked credentials. They modified the Bash Uploader script to send environment variables to a remote server. The malicious version was live for over 2 months (Jan 31 – Apr 1, 2021). Impact: Thousands of organizations using the script unknowingly leaked secrets and credentials. Affected companies included HashiCorp, Twilio, and Rapid7. Attackers may have accessed private source code and internal systems through stolen credentials.