A Brief History of Time

In order to understand DevOps, we have to travel back in time to the old age when there was nothing but programmers.

As the Tao teaches us:

The programmers of old were mysterious and profound. We cannot fathom their thoughts, so all we do is describe their appearance:

  • Aware, like a fox crossing the water
  • Alert, like a general on the battlefield
  • Kind, like a hostess greeting her guests
  • Simple, like uncarved blocks of wood
  • Opaque, like black pools in darkened caves

The Programmer gave birth to machine language. Machine language gave birth to the assembler. The assembler gave birth to the compiler. Now there are thousands of languages. Each language has its purpose, however humble. Each language expresses the Yin and Yang of software. Each language has its place within the software.

At the time, software development offices were mostly called labs, and programmers were scientists. In order to create a good application, programmers had to fully understand the problem the application was solving. They had to know where the application will be used and what kind of system will be running it. In essence, programmers knew everything about their application from specification, through development, to deployment and support.

And then human nature kicked in, and we started to ask for more. More speed, more features, more users, more everything.

Being modest, humble, and peaceful creatures, programmers had very little chance to survive such burst of needy users always asking for more. The best chance to prevail was to start evolving in different directions, and create castes. Soon, programmers were extinct as forefathers of new breeds called developers, software engineers, network administrators, database developers, web developers, system architects, QA engineers, and many more. Evolving fast and adapting to challenges from the outside world became part of their DNA. New caste could evolve in a matter of weeks. Web developers became back-end developers, front-end developers, PHP developers, Ruby developers, Angular developers … it was all going to hell.

Soon they all forgot that they came from the same father, a Programmer. A simple and peaceful scientist that just wanted to make the world a better place. They started fighting each other, claiming that each of them is true descendant of “The Programmer” and that their blood is more pure than others.

As time passed by, they reduced their interaction to a minimum and spoke to each other only when they really had to. They stopped celebrating success of their distant family members, and eventually even stopped sending a postcard every once in awhile.

If they only searched their feelings, they would discover that the spark of the Programmer was in their hearts, waiting to shine and bring peace to the galaxy.

The Programmer

In their selfish and egocentric race to conquer the world, programmer’s descendants forgot the very purpose of their job - solving problems for their clients. Clients started to feel the pain of such behavior as projects got delayed, too expensive, or even failed.

Every once in awhile, a bright star would shine and someone would get an inspiration to try and make peace among The Descendants. They came up with Waterfall. This was a brilliant idea, as it utilized the fact that different groups of developers communicated only when necessary. When one group finished their part of the job, it communicated with the next group to send work down through the process and never looked back at it.



This worked for a while, but as usual, humans (Clients) wanted more again. They wanted to take more active part in the whole process of creating software, give their opinion more often, and ask for changes even when it was too late.

Software projects became so prone to failure that it was accepted as an industry standard. Statistics showed that over 50% of projects were failing, and it seems like there was nothing to be done about it (ZDNet, CNet)

Every generation had a few open-minded individuals who knew that all these groups of developers have to find a way to work together, communicate, and be flexible to assure their clients will receive the best possible solution. There are traces of these attempts even as early as 1957, by colleagues of the great John Von Neumann. However we had to wait until early 2001 to start the revolution, when a dirty dozen created an Agile Manifesto.


The Agile Manifesto is based on twelve principles:

  1. Our highest priority is to satisfy the customer
    through early and continuous delivery
    of valuable software.
  2. Welcome changing requirements, even late in
    development. Agile processes harness change for
    the customer's competitive advantage.
  3. Deliver working software frequently, from a
    couple of weeks to a couple of months, with a
    preference to the shorter timescale.
  4. Business people and developers must work
    together daily throughout the project.
  5. Build projects around motivated individuals.
    Give them the environment and support they need,
    and trust them to get the job done.
  6. The most efficient and effective method of
    conveying information to and within a development
    team is face-to-face conversation.
  7. Working software is the primary measure of progress.
  8. Agile processes promote sustainable development.
    The sponsors, developers, and users should be able
    to maintain a constant pace indefinitely.
  9. Continuous attention to technical excellence
    and good design enhances agility.
  10. Simplicity--the art of maximizing the amount
    of work not done--is essential.
  11. The best architectures, requirements, and designs
    emerge from self-organizing teams.
  12. At regular intervals, the team reflects on how
    to become more effective, then tunes and adjusts
    its behavior accordingly.
  1. 我们最重要的目标,是通过持续不断地及早交付有价值的软件使客户满意。

  2. 欣然面对需求变化,即使在开发后期也一样。为了客户的竞争优势,敏捷过程掌控变化。

  3. 经常地交付可工作的软件,尽量按周而不是按月,倾向于采取较短的周期。

  4. 业务人员和开发人员必须相互合作,项目中的每一天都不例外。

  5. 激发个体的斗志,以他们为核心搭建项目。提供所需的环境和支援,辅以信任,从而达成目标。

  6. 不论团队内外,传递信息效果最好效率也最高的方式是面对面的交谈。

  7. 可工作的软件是进度的首要度量标准。

  8. 敏捷过程倡导可持续开发。责任人、开发人员和用户要能够共同维持其步调稳定延续。

  9. 坚持不懈地追求技术卓越和良好设计,敏捷能力由此增强。

  10. 以简洁为本,它是极力减少不必要工作量的艺术。

  11. 最好的架构、需求和设计出自于自管理团队。

  12. 团队定期地反思如何能提高成效,并依此调整自身的举止表现。

Agile manifesto was the first big step in bringing peace to the Galaxy and restoring balance to the Force. For the first time in a very long time, connecting all stakeholders in software development process was based on the cultural and “human” way, as much as on procedural and mechanized manner. People started to talk to each other, meet on a regular basis, and exchange ideas and comments all the time. They realized that they have much more in common that they thought, and clients became part of the team, not just some external factor that was expected to send the money and ask no questions.


There were still a few hurdles to overcome, but the future seemed brighter than ever. Being agile means being open and willing to adapt to constant changes. However, with all too many changes, it is difficult to stay focused on the ultimate goal and delivery. And that is how Lean Software Development came to life.

Hooked on LSD (pun intended) and risking to be exiled and outcast, some of the descendants looked for solutions outside of their caste and the software industry. They found salvation in the works of a major car manufacturer. Toyota Production System was amazing in its simplicity and it was so obvious that its lean manufacturing can be easily applied to software development.

There are 7 principles of lean:

  1. Eliminate Waste
  2. Build Quality In
  3. Create Knowledge (Amplify learning)
  4. Defer Commitment (Decide as late as possible)
  5. Deliver as fast as possible
  6. Respect People (Empower the team)
  7. Optimize the Whole








Added on top of Agile, Lean principles supported the mentality and the focus on doing the right things, while being flexible during the whole process.

Once Agile and Lean were adopted by software development teams, it took just one more step to apply the whole set of principles to IT as a Whole - which finally brought us to DevOps!

Enter DevOps - Three lane highway

The old-school view of software development teams included business analysts, system architects, front-end developers, back-end developers, testers, and so on. Optimizing software development process, including agile and lean principles, was mostly focused on these. Once the application was “production ready”, it was sent over to “Operations” including systems engineers, release engineers, DBAs, network engineers, security professionals, etc. Removing the great wall between Dev and Ops is the main driving force of DevOps.

DevOps is the result of implementing lean principles to the whole IT value stream. IT value stream extends development into production, combining all ‘distant relatives’ that descended from the original Programmer.

The best explanation of DevOps is given by Gene Kim, and If you haven’t read The Phoenix Project I suggest you take some time and do it.

You should not hire a DevOps engineer and DevOps should not be a new department in your IT. DevOps is a culture, mindset, and it is part of IT as the Whole. There are no tools that will make your IT a DevOps organization, and no level of automation will empower your teams to deliver maximum value to your clients.


DevOps is usually referred as method of three ways, but I see them as three lanes of the same highway. You start in lane one, then you speed up and switch to lane two, and eventually you are speeding by in the third lane.

  • Lane one - Performance of the system as whole is main focal point and it is emphasized over performance of any individual element of the system
  • 系统的大局观最重要,而且在性能考量方面大过任何子元素。
  • Lane two - Make sure that there is a constant feedback loop sent upstream, and not ignored
  • 确保存在向上游的反馈通道并能行成闭环。确保任何反馈不被忽略。
  • Lane three - Nurture experiments, constant improvement, and fail fast
  • 不断试验、持续改进、快速试错

Lane one - Getting up to speed

Understanding the importance of the whole system, and prioritizing work items properly is the first thing an IT organization has to learn when adopting DevOps principles. No one in the IT value stream is allowed to create a bottleneck and reduce the flow of work.

DevOps - Understanding the whole system

Assuring uninterrupted workflow is the ultimate goal for everyone included in the process. Regardless of the role a person or a team has, it is imperative that they seek to achieve profound understanding of the system. Adopting this way of thinking has a direct impact on quality, as defects are never sent down the stream because they would cause bottlenecks.

Making sure that work is not stalling is not enough. A productive organization should always seek to increase the flow. There are numerous methodologies to increase the flow. You can find a solution in Theory Of Constraints, Six Sigma, Lean, or Toyota Production System. Feel free to pick any of these, come up with your own, or combine them.

DevOps principles do not care what team you belong to, if you are a System Architect, DBA, QA, or network administrator. The same rules apply to everyone, and everyone is expected to follow two simple principles:

  • Keep the system flow uninterrupted
  • Increase and optimize workflow at all times

Lane two - Gear up

Uninterrupted system flow is one-directional, and it is expected to go from development to operations. In an ideal world this means that software is created quickly with high quality, deployed to production, and delivering value to clients.

However, DevOps is not Utopia, and if one-directional delivery was possible the waterfall method would be sufficient. Evaluating deliverables and communicating up the flow is very important to assure quality. First “upstream” communication channel that has to be implemented is Ops to Dev.


Giving a high-five to yourself is easy, but asking for feedback and giving feedback is the way to see the real picture. It is imperative that each small step downstream is followed by an instant upstream confirmation.

It does not matter how you establish the feedback loop. You can invite developers to join support team meetings, or bring in the network admin to weekly sprint planning. As long as your feedback is in place, and comments are picked-up and handled, your organization is speeding down the DevOps highway.

Lane three - Warp 10

DevOps fast lane is not for weak minded. To get on the DevOps fast lane your organization must be mature enough. It is all about taking risks and learning from failure, continually experimenting, and accepting that repetition and practice is the prerequisite to mastery. Quite often you will hear the term Kata, and this is for a reason. Continuous training and repetition leads to mastery because it makes complex moves become a routine.

But before you start combining moves, it is imperative that you master each individual step first.

“What is appropriate for the Master is not appropriate for the novice. You must understand Tao before transcending structure.”


DevOps Third Way/Fast Lane include allocating time for continuous experimenting on a daily basis, constantly rewarding the team for taking risks, and introducing faults into the system to increase resilience.

In order to assure that your organization will not implode when implementing these kinds of measures, you must create frequent feedback loops between all teams, while making sure that all bottlenecks are clear and system flow is uninterrupted.

Implementing these measures makes your organization alert at all times and capable of responding to challenges quickly and effectively.

Summary - DevOps organization checklist

Here is a checklist that you can use to validate how DevOps enabled your IT organization is. Please, feel free to comment on the article and add your own points.

  • There is no wall between development team and operations team. Both are part of the same process
  • Work that is sent over from one team to another is always verified and top quality
  • There is no “piling” of work, and all bottlenecks are handled
  • Development team is not using Operations time for its activities, because deployment and maintenance are part of the same time box
  • Development team does not deliver the code for deployment at 5PM on Fridays, leaving operations to clean up their work over the weekend
  • Development environments are standardized, and operations can easily reproduce and scale them into production
  • Development team can deliver new versions as they find appropriate and operations can easily deploy them into production
  • There are clear communication lines between all teams
  • All team members have time to experiment and work on improvement of the system
  • Defects are introduced (or simulated) and handled in the system on a regular basis. Lessons learned from each experiment are documented and shared with all relevant people. Incident handling is a routine and mostly handled in a known way
  • 开发团队和运营团队之间没有隔阂,两者同属一体。
  • 每个环节的工作在交给下游团队之前都经过质量检查并且保证是最佳质量。
  • 工作没有“堆积”,所有的瓶颈都得到了处理。
  • 开发团队不会占用运维团队时间,因为部署和维护是可以同时并行的。
  • 开发团队不会在周五下午5点提供部署代码,而是让运维在周末擦屁股
  • 开发环境是标准化的,运维可以轻松地重现并将其扩展到生产环境中
  • 开发团队可以根据需要提供新版本,并且可以轻松地将操作部署到生产环境中
  • 所有团队之间都有明确的沟通渠道,所有团队成员都有时间进行实验并努力改进系统在系统中定期引入(或模拟)和处理缺陷。
  • 每个实验的经验教训都记录在案,并与所有相关人员分享。事故处理成为日常工作的一部分,而且都有对应处理方案。


Using modern DevOps Tools like Chef, Docker, Ansible, Packer, Troposphere, Consul, Jenkins, SonarQube, AWS, etc. does not mean that you are applying DevOps principles. DevOps is a way of thinking. We are all part of the same process, we share the same time and deliver value together. Every person that takes part in the software delivery process can speed up or slow down the entire system. A bug created during development can bring down the system, as well as the wrong firewall configuration.

All of the people are part of DevOps, and once your organization understands that, tools and technology stack that will help you manage it will magically show up.