Providing Feedback

Provide feedback at the survey

Experimental

Content in this chapter is experimental and will evolve based on user feedback.

Leave feedback on the conventions by creating an issue in the kubectl GitHub repository.

Also provide feedback on new kubectl docs at the survey

TL;DR

Decouple changes to Config to be deployed to separate Environments.

Branch Structure Based Layout

Motivation

Why use branches? Decouple changes that are rolled out with releases (e.g. new flags) from changes that are rolled out in response to production events (e.g. resource tuning).

Branch Structure

The convention shown here should be changed and adapted as needed.

Branch Type Name Deployed to a Cluster Purpose Example Config Change Example Branch Name
Base No. Merged into other Branches only. Changes that should be rolled out as part of a release. Add pubsub topic flag master, release-1.14, i1026
Deploy Yes. - Manually or Continuously. Base + Changes required to respond to "production" events (or dev, staging, etc). Increase memory resources - e.g. for crashing Containers deploy-test, deploy-staging, deploy-prod

Use with techniques described in Directories and Branches

Workflow Example

Diagram

Scenario

  1. Live Prod App version is v1
  2. v2 changes committed to Base Branch Config
  3. v2 rolled out to Staging
    • Deployed by continuous deployment
  4. Live Prod App requires change to v1 (unrelated to v2)
    • Change memory resources in Prod
  5. Prod Branch Config Updated at v1
    • Deployed immediately by continuous deployment
  6. v2 changes rolled out separately
    • Tag on Base Branch merged into Prod Branch
    • Prod Branch continuously deployed

Created with Raphaël 2.1.4

Description

Note: Starting version of Application is v1

  1. Developer Bob introduces new app flag for release with v2
    • e.g. PubSub topic name
  2. Bob updates the Base Config with the new flag
    • Add staging topic for Staging (e.g. staging-topic)
    • Add prod topic for Prod (e.g. prod-topic)
    • Flag should be rolled out with v2 release
  3. v2 is cut
    • Base tagged with v2 tag
  4. v2 rolled out to Staging
    • Merge v2 Tag -> Staging Branch
    • Deploy Staging Branch to Staging Clusters
  5. SRE Alice identifies issue in Prod (at v1)
    • Fix is to increase memory of containers
  6. Alice updates the Prod branch Config by increasing memory resources
    • Changes go directly into Prod Branch without going into Base
  7. v1 changes rolled out to Prod (v1++)
    • Include Alice's changes, but not Bob's
  8. v2 rolled out to Prod
    • Merge v2 Tag -> Prod Branch
    • Deploy Prod Branch to Prod Clusters

Techniques:

  • Add new required flags and environment variables to the Resource Config in the Base branch at the time they are added to the code.
    • Will be rolled out when the code is rolled out.
  • Adjust flags and configuration to the Resource Config in the Deploy branch in the deploy directory.
    • Will be rolled out immediately independent of versions.
  • Merge code from the Base branch to the Deploy branches to perform a Rollout.

Directory and Branch Layout

Structure:

  • Base branch (e.g. master, app-version, etc) for Config changes tied to releases.
  • Separate Deploy branches for separate Environments (e.g. deploy-<env>).
    • A new Directory in each branch with will contain overlay customizations - e.g. deploy-<env>.

Base Branch: master

tree
.
├── bases
│   ├── ...
├── prod
│   ├── bases
│   │   ├── ...
│   ├── us-central
│   │   ├── kustomization.yaml
│   │   └── backend
│   │       └── deployment-patch.yaml
│   ├── us-east
│   │   └── kustomization.yaml
│   └── us-west
│       └── kustomization.yaml
├── staging
│   ├── bases
│   │   ├── ...
│   └── us-west
│       └── kustomization.yaml
└── test
    ├── bases
    │   ├── ...
    └── us-west
        └── kustomization.yaml

Deploy Branches:

Prod Branch: deploy-prod

tree
.
├── bases # From Base Branch
│   └── ...
└── deploy-prod # Prod deploy folder
│   ├── us-central
│   │   ├── kustomization.yaml # Uses bases: ["../../prod/us-central"]
│   ├── us-east
│   │   └── kustomization.yaml # Uses bases: ["../../prod/us-east"]
│   └── us-west
│       └── kustomization.yaml # Uses bases: ["../../prod/us-west"]
├── prod # From Base Branch
│   └── ...
├── staging # From Base Branch
│   └── ...
└── test # From Base Branch
    └── ...

Staging Branch: deploy-staging

tree
.
├── bases # From Base Branch
│   ├── ...
├── deploy-staging # Staging deploy folder
│   └── us-west
│       └── kustomization.yaml # Uses bases: ["../../staging/us-west"]
├── prod # From Base Branch
│   └── ...
├── staging # From Base Branch
│   └── ...
└── test # From Base Branch
    └── ...

Test Branch: deploy-test

tree
.
├── bases # From Base Branch
│   ├── ...
├──deploy-test # Test deploy folder
│   └── us-west
│       └── kustomization.yaml # Uses bases: ["../../test/us-west"]
├── prod # From Base Branch
│   └── ...
├── staging # From Base Branch
│   └── ...
└── test # From Base Branch
    └── ...

Rollback Workflow Example

Summary of rollback workflow with Branches:

  1. Live Prod App version is v1
  2. Changes are introduced to Base Branch Config
    • To be released with version v2
  3. Release v2 is cut to be rolled out
    • Tag Base v2 and build artifacts (e.g. images)
  4. Changes are introduced into the Base Branch Confiug
    • To be released with version v3
  5. v2 is pushed to Prod (eventually)
    • v2 Tag merged into Prod Branch
  6. v2 has issues in Prod and must be rolled back
    • v2 changes are rolled back in new commit to Prod Branch
  7. Base Branch is unaffected
    • Fix introduced in v3

Note: New changes committed to the Base for "v3" did not make the rollback from "v2" -> "v1" more challenging, as they had not been merged into the Prod Branch.

Diagram

Created with Raphaël 2.1.4