CI (Continuous Integration) and CD (Continuous Delivery/Continuous Deployment) are practices in modern software development aimed at improving the process of integrating, testing, and deploying code changes.
 |
| image source: A Crash Course in CI/CD - ByteByteGo Newsletter |
Continuous Integration (CI)
Continuous Integration is a software development practice where developers regularly merge (integrate) their code changes into a shared repository multiple times a day. Each merge triggers an automated build and testing process to detect integration issues early.
Key Aspects:
- Frequent Commits: Developers commit code changes frequently, at least daily.
- Automated Builds: Each commit triggers an automated build to compile the code.
- Automated Testing: Automated tests (unit, integration, functional) are run to verify that the new code does not break existing functionality.
- Unit tests verify that code changes didn't introduce any regression at the function level. They should be fast and should be run on the dev machine before the code gets pushed to the remote (e.g. as part of git commit hooks) and also on CI server (which ensures they are 100% executed and also that they are executed in a non-local environment so there is not chance of having "it works on my machine" conflict).
- Integration tests verify that all components/modules of the product work together. Example:
- API endpoint /createOrder indeed creates an order with all attributes and this can be verified by verifying /getOrder response content
- Immediate Feedback: Developers receive immediate feedback on the build and test status, allowing them to address issues promptly.
- Shared Repository: All code changes are merged into a central repository (e.g., Git).
Stages:
- Locally (on dev machine):
- code is added to the repository locally, to the feature branch
- as part of the commit, unit tests are run locally
- code is pushed to the remote
- CI server:
- detects new commit
- runs unit tests
- builds the output binary, package or Docker image
- runs integrations tests
Benefits:
- Improve the feedback loop
- Faster feedback on business decisions is another powerful side effect of CI. Product teams can test ideas and iterate product designs faster with an optimized CI platform. Changes can be rapidly pushed and measured for success. Bugs or other issues can be quickly addressed and repaired. [What is Continuous Integration | Atlassian]
- Early detection of bugs and integration issues. - Before new code is merged it must pass the CI test assertion suite which will prevent any new regressions.
- Reduced integration problems.
- Improved code quality.
- Faster development cycles.
- Enables Scaling
- CI enables organizations to scale in engineering team size, codebase size, and infrastructure. By minimizing code integration bureaucracy and communication overhead, CI helps build DevOps and agile workflows. It allows each team member to own a new code change through to release. CI enables scaling by removing any organizational dependencies between development of individual features. Developers can now work on features in an isolated silo and have assurances that their code will seamlessly integrate with the rest of the codebase, which is a core DevOps process. [What is Continuous Integration | Atlassian]
Challenges:
- Adoption and installation
- Technology learning curve
Best Practices:
- Test Driven Development (TDD) - the practice of writing out the test code and test cases before doing any actual feature coding.
- Pull requests and code reviews
- Pull requests:
- critical practice to effective CI
- created when a developer is ready to merge new code into the main codebase
- notifies other developers of the new set of changes that are ready for integration
- an opportune time to kick off the CI pipeline and run the set of automated approval steps. An additional, manual approval step is commonly added at pull request time, during which a non-stakeholder engineer performs a code review of the feature
- foster passive communication and knowledge share among an engineering team. This helps guard against technical debt.
- Optimizing pipeline speed
- Given that the CI pipeline is going to be a central and frequently used process, it is important to optimize its execution speed.
- It is a best practice to measure the CI pipeline speed and optimize as necessary.
Continuous Delivery (CD)
Key Aspects:
- Automated Deployment Pipeline: Code changes go through an automated pipeline, including build, test, and packaging stages.
- Deployable State: The codebase is always ready for deployment to production.
- Manual Release: Deployment to production is triggered manually, ensuring final checks and balances.
- Staging Environment: Changes are deployed to a staging environment for final validation before production.
Benefits:
- Reduced deployment risk.
- Faster and more reliable releases.
- High confidence in code quality and stability.
- Easier and more frequent releases.
Continuous Deployment (CD)
Continuous Deployment takes Continuous Delivery a step further by automatically deploying every code change that passes the automated tests to production without manual intervention.
Key Aspects:
- Automated Deployment: Every code change that passes all stages of the pipeline (build, test, package) is automatically deployed to production.
- Monitoring and Alerting: Robust monitoring and alerting systems are essential to detect and respond to issues quickly.
- latency
- performance
- resource utilization
- KPIs/Key business parameters e.g. number of new installs, number of install re-tries, number of engagements, monetization, usage of various features
- errors/failures/warnings in logs
- Rollbacks and Roll-forwards: Mechanisms to roll back or roll forward changes in case of failures.
- ideally, rollbacks would be automated
Benefits:
- Accelerated release cycle.
- Immediate delivery of new features and bug fixes.
- Continuous feedback from the production environment.
- Higher customer satisfaction due to faster updates.
 |
| image source: Azure CI CD Pipeline Creation with DevOps Starter |
CI/CD Pipeline
A CI/CD pipeline is a series of automated processes that help deliver new software versions more efficiently.
The typical stages include:
- Source Code Management
- Developers commit code to a shared repository.
- Build
- The code is compiled and built into a deployable format (e.g., binary, Docker image).
- Automated Testing
- Automated tests are run to ensure the code functions correctly (unit tests, integration tests, functional tests).
- Packaging
- The build artifacts are packaged for deployment.
- Deployment
- Continuous Delivery: Artifacts are deployed to a staging environment, and deployment to production is manual.
- Continuous Deployment: Artifacts are automatically deployed to production.
- Monitoring
- The deployed application is monitored for performance and errors
CI/CD Pipeline Development: Build and maintain a continuous integration/continuous deployment (CI/CD) pipeline to automate the testing and deployment of code changes.
 |
| image source: EP71: CI/CD Pipeline Explained in Simple Terms |
Tools for CI/CD
- CI Tools:
- Jenkins
- GitHub Actions - The most popular free CI/CD platform
- Linting & Testing
- GitLab CI
- CircleCI
- Travis CI
- CD Tools:
- Spinnaker
- ArgoCD
- Tekton
- AWS CodePipeline
- Testing Tools:
- JUnit
- Selenium
- Cypress
- pytest
- Build Tools:
- Maven
- Gradle
- npm
- Docker
- Monitoring Tools:
- Prometheus
- Grafana
- ELK Stack
By implementing CI/CD practices, development teams can achieve faster delivery cycles, higher code quality, and a more reliable deployment process.
How to enhance automation and scalability in CI/CD?
Enhancing automation and scalability in CI/CD practices involves implementing strategies and tools that streamline processes, reduce manual intervention, and ensure that the system can handle increasing workloads effectively. Here are some key practices to achieve these goals:
CI/CD Practices for Enhanced Automation and Scalability:
- Automated Testing:
- Unit Testing: Automated tests for individual units of code ensure that changes don’t break functionality.
- Integration Testing: Tests that verify the interaction between different parts of the application.
- End-to-End (E2E) Testing: Simulate real user scenarios to ensure the application works as expected.
- Continuous Testing: Running tests automatically on every code change.
- Pipeline as Code:
- Define CI/CD pipelines using code (e.g., YAML files) stored in version control.
- This makes it easier to track changes, review pipeline modifications, and replicate environments.
- Example: In TeamCity it is possible to enable storing build configurations as a Kotlin code, in a dedicated repository
- Infrastructure as Code (IaC):
- Use tools like Terraform, Ansible, or CloudFormation to manage infrastructure.
- IaC allows for automated provisioning, scaling, and management of environments.
- Containerization:
- Use Docker or similar containerization technologies to create consistent environments.
- Containers ensure that applications run the same way regardless of where they are deployed, simplifying scaling and deployment.
- Orchestration:
- Use Kubernetes or other orchestration tools to manage containerized applications.
- Orchestration tools help in scaling applications automatically based on demand.
- Parallel Execution:
- Run tests and build processes in parallel to reduce overall pipeline execution time.
- This is especially useful for large test suites and complex builds.
- Caching:
- Implement caching for dependencies, build artifacts, and other frequently used resources to speed up the CI/CD pipeline.
- Cache mechanisms reduce the time required for repetitive tasks.
- Artifact Management:
- Use artifact repositories like JFrog Artifactory or Nexus to store build artifacts.
- Proper artifact management ensures reliable and consistent deployments.
- Environment Consistency:
- Ensure development, testing, staging, and production environments are as similar as possible.
- Consistent environments reduce the likelihood of environment-specific bugs.
- Monitoring and Logging:
- Implement monitoring and logging throughout the CI/CD pipeline.
- Use tools like Prometheus, Grafana, ELK Stack, or Splunk to gain insights and quickly identify issues.
- Feature Toggles:
- Use feature toggles to control the release of new features without deploying new code.
- This allows for safer and more controlled feature releases and rollbacks.
- Scalable Architecture:
- Design applications to be stateless and horizontally scalable.
- Use microservices architecture to break down applications into smaller, manageable services that can be scaled independently.
- Automated Rollbacks:
- Implement automated rollback mechanisms in case of deployment failures.
- This ensures quick recovery from failed deployments without manual intervention.
- Security Automation:
- Integrate security checks into the CI/CD pipeline using tools like Snyk, OWASP ZAP, or Aqua Security.
- Automated security scanning helps in identifying vulnerabilities early.
By adopting these practices, organizations can achieve a highly automated, reliable, and scalable CI/CD pipeline that supports rapid and safe software delivery.
Example Workflow for Automation and Scalability
Here’s a high-level example of a CI/CD workflow that incorporates some of these practices:
github-actions-example.yaml:
name: CI/CD Pipeline
on: [push, pull_request]
jobs:
build:
runs-on: ubuntu-latest
strategy:
matrix:
node-version: [12, 14, 16]
steps:
- uses: actions/checkout@v2
- name: Cache dependencies
uses: actions/cache@v2
with:
path: ~/.npm
key: ${{ runner.os }}-node-${{ matrix.node-version }}-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-node-${{ matrix.node-version }}-
- name: Setup Node.js
uses: actions/setup-node@v2
with:
node-version: ${{ matrix.node-version }}
- run: npm install
- run: npm test
deploy:
needs: build
runs-on: ubuntu-latest
steps:
- name: Deploy to Staging
run: |
# Deployment commands
echo "Deploying to staging environment..."
- name: Automated Tests on Staging
run: npm run test:e2e
promote:
needs: deploy
runs-on: ubuntu-latest
if: success()
steps:
- name: Deploy to Production
run: |
# Production deployment commands
echo "Deploying to production environment..."
