The Ideal CI CD Pipeline Concepts Overview

Ideal Pipeline

  • Source
    • Git version control
    • Require X reviewers
    • Independent Dev DB and any dependencies are replicated for Dev copies
  • Build
    • Should be Toggle-able for when you dont want something to run through the gamut
    • Compile Source and Dependencies
    • Unit tests
    • Coverage > 90%
  • Test
    • Independent Test DB
    • Should be Toggle-able for when you dont want something to run through the gamut
    • Integration tests
      • If you have an API then test a POST request followed by a GET request
      • If you have external service dependencies, test them too
      • In essence, test the core functionality of your app so that changed behavior will easily show up
  • Prod (1box)
    • Independent Prod DB
    • 1 Box (have incremental roll out to a single machine or a small fraction of them so impacts are smaller and more readily identified and rolled back)
    • Alarms on Errors, Latency, and Key business Metrics can indicate a needed rollback
      • Robust system can monitor these and run an automated rollback
    • Bake Period (24hrs)
      • Testing that the change is still good live in prod
      • Anomaly Detection or Error counts + Latency Breaches over the time period
    • Canary (in a coal mine)
      • Have a cron job test the API at a regular interval (POST/GET/UPDATE/DELETE etc.)
      • Should the cron job fail then it can be an alarm.
  • Prod
    • The other 90% of the traffic
    • Also has all the Alarms, canary, etc. from the 1box too
    • Traffic split through DNS?

Backlinks