CI/CD tools and developer Experience

Our CI/CD tools are fast and reliable.

What are CI/CD tools?

Continuous Integration and Continuous Delivery/Deployment (CI/CD) tools are automation systems that enable development teams to frequently integrate code changes into a shared repository (CI) and reliably deliver applications to production environments (CD). These tools form the backbone of modern software delivery pipelines, automating building, testing, and deployment processes.

CI/CD tools include platforms like Jenkins, GitLab CI/CD, GitHub Actions, CircleCI, and Azure DevOps, among others. Each offers varying capabilities for automating workflows, running tests, managing infrastructure, and coordinating releases.

How can fast and reliable CI/CD tools improve developer experience and productivity?

Reduced wait times and context switching

When CI/CD tools work efficiently, developers spend less time waiting for builds and test results. This minimization of wait times is crucial for maintaining flow and productivity.

"When we started analyzing our processes, we noticed significant issues with our CI system. When builds run quickly, people review pull requests promptly. But when CI is slow, developers forget about the PR, switch to other tasks, and the code only gets merged the next day or even later."

Head of Engineering at Software Development Company

This quote highlights how slow CI pipelines force developers to context-switch, disrupting their workflow and delaying code integration.

Increased deployment confidence

Reliable CI/CD tools give developers confidence to deploy more frequently without fear of breaking production, encouraging innovation and experimentation.

Faster feedback loops

Quick, dependable CI/CD pipelines provide rapid feedback on code quality, enabling developers to catch and fix issues early in the development cycle.

Enhanced team collaboration

Effective automation reduces manual coordination overhead, allowing teams to work asynchronously while maintaining code quality standards.

Why are fast and reliable CI/CD tools important?

Fast and reliable CI/CD tools are crucial for successful dev teams because they directly impact developer productivity, software quality, and release velocity. When these tools falter, the consequences ripple throughout the organization.

"We experienced a massive backlog of builds. That situation made me truly understand the critical importance of maintaining a stable and reliable CI/CD pipeline. Developers and tech leaders were extremely frustrated, sending angry emails about the situation. It was a powerful lesson about the importance of reliable infrastructure."

Deputy Head of Development at Financial Trading Software Provider

The emotional reaction described here demonstrates how deeply CI/CD performance affects team morale and productivity. When build queues grow lengthy, the entire delivery process stalls, creating frustration across all levels of the organization.

Reliable CI/CD pipelines also:

• Reduce deployment risk: By automating testing and verification steps
• Increase software quality: Through consistent application of quality gates
• Support agile practices: Enabling small, frequent releases
• Free developer cognitive capacity: Automating repetitive tasks
• Create audit trails: Documenting what changed, when, and by whom

How to measure CI/CD performance?

DevEx surveys

The Network Perspective DevEx Survey question "Our CI/CD tools are fast and reliable" provides valuable insights into developers' actual experiences with your CI/CD pipeline. This subjective assessment complements objective metrics by capturing pain points that might not be evident in the numbers alone.

Regular surveys help:

1. Identify perception gaps: Situations where metrics look good but developers still experience pain
2. Prioritize improvements: Focus on the issues most impacting developer experience
3. Track sentiment changes: Monitor how perception evolves after improvements
4. Compare across teams: Identify teams struggling with CI/CD issues

"We've observed a significant trend regarding decision-making processes. It's been tremendously beneficial having our director join the team because major infrastructure or DevOps decisions are frequently made from the top down with substantial budgets allocated. However, without developer surveys or other methods to directly gather feedback from the engineers about their actual experiences, these expensive initiatives often completely miss the mark on addressing real problems."

Engineering Manager at Retail Company

Combining survey feedback with objective metrics provides a comprehensive view of your CI/CD health, helping you make informed decisions that truly improve developer experience.

Key CI/CD metrics to track

Queue time: The wait before a build starts executing

"We've developed what I call proxy or synthetic metrics. Since we struggled to identify the ideal metrics for evaluating build time in our CI/CD system, we decided to use queue time instead. We track the number of builds waiting in the queue as a proxy metric for velocity and latency."

Deputy Head of Development at Financial Trading Software Provider

Build time: Duration from build start to completion Success rate: Percentage of successful builds vs. failures Deployment frequency: How often code is deployed to production Recovery time: How quickly failures are fixed

These metrics should be regularly monitored and discussed with engineering teams. However, remember that different products may require different metrics:

"The DORA software delivery metrics are primarily optimized for web services, microservices, and frontend development. However, when you're shipping mobile or desktop software, the nuances are significantly different and require different measurement approaches."

Senior Principal Scientist at Software Development Company

What matters more: speed or predictability in CI/CD?

While speed is important, predictability often matters more to developers:

"Let me give you an example. We had a discussion about CI/CD optimization with the goal of reducing build times to just a few seconds. That's one perspective. But the reality is that you either need a massive budget to achieve that, or you end up with highly unpredictable build times. The real issue isn't about having builds that sometimes take 5 minutes and other times take 50 minutes. That's not the point. The key insight is this: if you ask your developers, they'll tell you they're fine with a 30-minute build time, as long as it's consistently 30 minutes every time. That consistency allows them to plan their work before and after the build, rather than sitting around waiting and checking if the build will complete or fail."

Engineering Manager at Retail Company

Predictable build and deployment times allow developers to plan their work efficiently. A CI/CD process that consistently takes 20 minutes is better than one that sometimes takes 5 minutes and sometimes 45 minutes.

Common CI/CD challenges and how to address them

Long build queues

When too many builds are submitted simultaneously, developers face long wait times.

Solution: Monitor queue sizes and scale infrastructure accordingly.

"This system was developed about 2-3 years ago when we encountered severe overload issues with our CI/CD infrastructure. We had launched many new projects, but our server capacity and builder resources remained unchanged. This resulted in enormous build queues, forcing engineers to wait extended periods for their builds to complete, which created significant frustration. We resolved the immediate issue by purchasing additional servers and implementing various upgrades. However, after addressing the crisis, we recognized we needed to prevent similar situations in the future. That's when we established our metrics system, with queue size being the primary indicator we monitor."

Professional

Flaky tests causing false failures

Unreliable tests that sometimes pass and sometimes fail erode trust in the CI/CD system.

Solution: Identify and fix flaky tests; consider quarantining them until fixed.

Configuration drift

Differences between local, test, and production environments lead to "works on my machine" problems.

Solution: Use containerization and infrastructure-as-code to ensure consistency.

Tool customization overhead

Maintaining heavily customized CI/CD solutions can become a burden.

"We experienced this with our CI/CD pipelines that we built on top of Jenkins. It was initially excellent and functioned well. The story is partly a success but also partly a failure. We extensively customized Jenkins, creating a schema-based rendering system similar to GitLab's approach. For the Jenkins pipeline, this worked wonderfully for 5-6 years. However, eventually, we found ourselves spending excessive time on customization. There were constant requests: 'I want to add this feature,' 'I want to change that,' and so on. This left us with two unsatisfactory options: either dedicate someone full-time to maintenance—sometimes even that wasn't enough—or stick with a packaged solution that didn't evolve. Neither option made anyone happy."

Director of Engineering at Software Development Company

Solution: Consider managed solutions that reduce maintenance overhead; balance customization with sustainability.

How to improve CI/CD performance and reliability?

Optimize build and test processes

• Break monolithic pipelines into smaller, parallelizable steps
• Implement test splitting and parallelization
• Use incremental builds when possible
• Leverage caching for dependencies, build artifacts, docker, layers, etc.

Establish a dedicated platform team

Many organizations benefit from having specialists focused on CI/CD reliability:

"We maintain a dedicated platform engineering team responsible for our entire CI/CD build process and toolchain. This approach has allowed us to fully automate the process for the majority of our developers. For them, it's remarkably simple: they create a pull request, the system automatically builds it, and they receive the results without any manual intervention."

Chief API Officer at Digital Banking Software Provider

The ROI of investing in CI/CD improvements

Investing in CI/CD improvements yields substantial returns through:

• Developer time savings: Reducing wait times multiplied across all developers
• Faster time-to-market: Enabling more frequent releases
• Higher quality: Catching bugs earlier when they're cheaper to fix
• Better morale: Eliminating a common source of developer frustration

One organization achieved remarkable efficiency with minimal staffing:

"Throughout a yeas, our system processed over 1 million builds, which represents an extremely high volume by any standard. Remarkably, just two to two-and-a-half DevOps engineers manage this entire infrastructure. When you consider the scale and efficiency, I believe this demonstrates an exceptional return on investment."

Head of Engineering, Financial Solutions Platforms

How to measure the impact of CI/CD improvements?

To assess the effectiveness of your CI/CD optimization efforts:

1. Establish a baseline: Measure current performance before making changes
2. Set clear improvement targets: Define what success looks like
3. Implement changes: Make targeted improvements
4. Measure the impact: Compare against the baseline

"For instance, when implementing platform engineering or CI/CD tooling, establish a clear agreement with your internal customers. Start by acknowledging the existing problem, then measure it precisely. Analyze the feedback and comments to determine the best approach for assistance. Once you've implemented the solution, measure the outcomes to quantify the improvement."

Strategic Sales Executive at Software Development Company