The Yin and the Yang of managing complex technical projects

Nadia Clifford
Sr Director, Engineering Program Management
Automatic Summary

Managing Complexity Mindfully: A Lesson from Yin and Yang Philosophy

Hello everyone, I'm Nadia Clifford, the Singing Director of Engineering Program Management at Outfront Media. Today, I will share my insights on managing complexity in tech careers using the Yin and Yang philosophy.

Stress in Tech Careers

Working with complex systems and cutting-edge technology can be stressful. Years ago, I experienced a severe burnout, which led me on a journey to mindfulness. In this process, I discovered that applying the ancient Yin and Yang philosophies to my work made it more manageable, efficient and less stressful. Over time, I realized this practice was not only beneficial to me, but also to everyone involved in the project.

The Ever-Increasing Complexity of Technology

Over the last two decades, the complexity of technology has increased exponentially. Comparing my career's starting point in 1997, where I computerized a manual process, to my current job managing a network of thousands of cloud-enabled smart IOT devices, the scale and complexity have differed immensely.

Management Methods Evolution

Just as technology has evolved, the methods of managing it have also steadily transformed, from waterfall in the seventies to Agile Manifesto in 2001. With increased complexity after 20 years since Agile was introduced, it may be time to expand our mindset and rethink our management methods.

The Challenges of Managing Complexity

Complexity theory provides a deep understanding of why modern tech can be challenging to manage. According to the theory, a system can be considered complex if it consists of entities that are diverse, connected, interdependent and adaptable. Interestingly, our complex technology initiative perfectly fits into this definition of a complex system since the individual entities involved include multiple software components, infrastructure components, and above all, people.

The Intricacies of Complex Systems

  • The properties of a system cannot be inferred from its components. Meaning, even if we know the individual components, we cannot fathom how the system will behave as a whole.
  • The responses of systems to changes can be unpredictable.
  • The system entities can modify the system itself- it takes a life of its own.

Therefore, rigorously controlling a complex system is nearly impossible.

Turning to Mindfulness

As the complexity of our lives mirrors that of technology, stress levels rise. Consequently, society has been turning to the millennia-old practice of mindfulness to cope. Interestingly, mindfulness has become mainstream at the same time that our lives' complexity has increased. One prominent concept from ancient philosophies is the concept of Yin and Yang, which has direct applications to our modern lives.

Yin and Yang

Yin and Yang is a 5000-year-old principle originating from Taoism. It describes opposites that are complementary to each other and form a dynamic whole. Balancing Yin and Yang energies is the ultimate goal, with an imbalance leading to stress and hardship.

Applying Yin and Yang to Complex Technology

When we apply the concept of balance to complex technology projects, it helps in managing complexity and, consequently, stress. For example, applying structure (an ordered state) may limit the flow. Therefore, there may be times when we need to let go of structure to let ideas flow and emerge. The idea is to strike a balance between structure and flow, low-level detail and visioning, scope and time, action and reflection, and change and acceptance.

Dynamic Balance

However, dynamic balance doesn't mean a 50-50 divide all the time. It signifies a fluctuating balance, moving from 80-20 to 70-30 to 50-50, and so on. The natural, healthy state is for the balance to move.

In Conclusion...

Being aware of the opposing yet complementary energies allows us to purposefully shift the balance in our work environments for better results. As Nadia Clifford said in her presentation, "When we become much wiser at purposefully shifting this balance in our work environments." If you wish to learn more or chat further about this enlightening topic, please reach out to me on LinkedIn.

Thank you for your time, and I hope this lesson from Yin and Yang philosophy helps you manage your work's complexity more mindfully.


Video Transcription

All right. Hello everyone. I am Nadia Clifford. I'm a singing director of Engineering Program Management at Outfront Media. Thank you so much for joining my session today.I am delighted to be here and honored to have this opportunity to present my thoughts on managing complexity, mindfully using the Yin and Yang philosophy. I have been very interested in complex systems. All my career have been fortunate to work with some cutting edge tech and as we all know, tech careers can get very stressful. Uh a few years ago, I was seriously burned out and I had to find a way to manage stress in my life and that eventually put me on a journey to mindfulness. And at some point, I realized that if I applied these ancient philosophies of Yin and Yang to that I was learning about to my work and not just my personal life and to the way I approach technology initiatives, things become much more manageable, efficient, less stressful and not just for me personally, but for everyone working on these projects.

So I did not want to keep this discovery a secret and I really would love others to benefit from what I realized. So here we are. So we can all appreciate how the complexity of technology has increased exponentially even over the last two decades or so. And that can be easily shown in the example of our own individual careers. So if we look at my first job in 1997 I had to build a Microsoft access application to computerize a manual process. It was paper fax swivel chair operation. And I hope I'm not the only person here who remembers what a fax machine looked like. But here's a little reminder for you. And just to compare to my current job, um I'm working on a network of thousands of cloud enabled smart IOT devices and they run on New York City subway trains, which means underground with spotty Wi Fi with fluctuating power supply. And nonetheless, they're connected to the cloud, they're synchronized and we manage them remotely. So if you just compare the two, the scale and the level of complexity are vastly vastly different. And so as complexity of tax attack has been increasing, the management methods have also been evolving.

We started with waterfall in the seventies, then we had agile Manifesto that was introduced in 2001, but it has been 20 years. And if we think how much complexity has increased since 2001, it makes one pause and think is it time to expand that mindset again? So let's talk about it. What makes complexity so difficult to manage. I'm sure we've all been through situations when we changed something small in a system over here and it went boom in a big way over there. Right. So why does it happen? So first we're going to get a little bit nerdy and I'm going to use science specifically complexity theory to explain why modern tech can be so difficult to manage. And then I'm going to introduce a very different concept, Yin and Yang. And this is an idea that is 5000 years old and it stems from Taoism and we will explore what it can teach us about managing complexity. And then I'm going to offer some practical ways to apply this new old thinking in your work. So now let's look at how science defines what a complex system is. So according to the complexity theory, a system can be considered complex if it consists of entities that are diverse, connected interdependent and adaptable. So now that we defined a complex system, let's see how that definition compares to a large technology initiative. But first, let's see what kinds of entities we may have in a complex technology initiative.

So we have multiple software components, some are written in house, some are third party libraries. We have back and front end, we have infrastructure components such as servers, balance loaders network, we may have hardware devices, firmware. And above all, we have people and people can also be organized into entities such as teams, departments, vendors, customers and users, you get the idea. So are these entities diverse? Yes, of course. Are they connected? Yes, they're all part of our ecosystem. Are they interdependent? Absolutely.

Teams can depend on each other. Front end depends on back end and application behavior depends on user behavior, et cetera. And are they adaptable? Yes. As simple as an if then statement in your code, if a happens then do this else, do something else. This is basic adaptability.

And if we talk about people, then adaptability just seems almost infinite. So we can safely conclude that a tech initiative meets the definition of a complex system according to science. So now that we have an overview of the system, we're ready for a little bit more detail.

So what do scientists say about complex systems, system properties cannot be inferred from its components. What it means? We cannot tell how the system is going to behave as a whole. Even if we know its individual components, system may not respond according to defined rules, even if we just focus on people, right? And not tech, we have social norms, we have processes, but we can never predict with 100% certainty how someone is going to act response to change can be unpredictable whether they're local within the system or global outside of the system. System in entities can modify the system itself and its self organizing because of adaptability. So basically, it takes a life of its own. So rigorously controlling a complex system is impossible. So this is a typical graph for a complex system, we're essentially plotting predictability against number of variables and predictability starts at 100% when number of variables is one and then it plummets to zero when number of variables is just seven. So if you think back on that work cloud that I shown of entities in the tech initiative, we have way more than seven and each of those things has variables within it.

So let's summarize predictability and control fall dramatically as complexity rises, which essentially means it's impossible to know with any certainty. The effects of a single action in a complex system, complexity increases with the number of variables, which means that larger chunks of work are even less predictable.

So now that we've established the problem with complexity, let's talk about mindfulness. Leave each moment, practice being mindful, be present. Hold on. I have to take this, I love this picture. So it's not just tech, right? Our lives have also been getting more and more complex in the recent decades. And as complexity rises so does rise the human stress. And it's no wonder that as a society, we have started turning to mindfulness. And isn't it interesting how mindfulness starts to come into the mainstream?

At the same time as the complexity of our collective lives starts increasing. And these concepts are thousands of years old and they have withstood the test of time. Very well. They're really good at um managing the most complex system of all, which is life itself. They're simple, powerful yet not very easy to implement. And doesn't that sound familiar, ever tried implementing agile? Um So I don't claim to be an expert in ancient philosophies, but I have been extremely interested in one concept and how it applies to our modern lives, which is the concept of Yin and Yang. And this is the image that illustrates it. It comes from Taoism and is almost 5000 years old. And it says that opposites um that are complementary to each other, that form a dynamic whole and they give rise to each other and they are part of each other and basically balancing the two is the ultimate goal. And when these energies are out of balance, that creates stress and hardship. So let's look at some of the examples of Yin and Young energies. You will see that they're not really opposites, they complement each other. We cannot have light without dark, we cannot give something without someone else receiving. There's no fullness without an emptiness to fill and there's no exhale without an inhale. And this image really doesn't do this concept justice. So let's take a look at a video.

So notice how these two energies are not static at all. They're constantly moving and replacing each other. They're in a dynamic balance. And when these energies are not in a state of a natural balance that creates stress, for example, imagine maximizing light and dark at the same time, it's hard to do but possible, right? So maybe we blink a bright light on and off in a dark room. And how long would you be able to stay calm in a room like that? That'd be very stressful, right? So now compare that with experiencing a sunrise when the amount of light is increasing, while the amount of dark is decreasing, it's a peaceful experience that is a natural dynamic balance. It may be a crude example, but it illustrates the concept. So now I'm asking a radical question, can this ancient philosophy be applied to a complex system? And I found a quote from a renowned expert in complexity theory and the author of at home in the universe, Stuart Kaufman, the fate of all complex adapting systems in the biosphere from single cells to economies is to evolve to a natural state between order and chaos, a grand compromise between structure and surprise.

And doesn't that sound a lot like the concept of Yin and Yang and the idea of an ultimate balance between the two. So let's apply the concept of balance between Yin and Yang to complex technology projects. I picked out several complementary energies and I hope that will help you discover how you could get better at managing complexity and therefore managing stress. And it's been proven many times. Over that humans and systems perform better longer when not under stress. So applying structure moves the system into a more ordered state away from a complex state. It could be a good rigorous process, it could be disciplined agile ceremonies. But then when we have a lot of structure, that means the flow diminishes. So an unconstrained flow of ideas and creativity may be something we need to innovate. If we apply the idea of balancing young and young, then perhaps there are times when we need to let go of the structure and let the ideas flow and emerge. I love starting new development with a mind mapping session. That's a collaborative brainstorm to build out a visual tree of ideas. A great way to uncover hidden barriers and open up a flow of ideas. And we have literally shortened projects by several months just because we allowed ideas to flow. And only after a mind mapping exercise, would we move into a structured planning process, balancing structure and flow? Let's look at balancing low level detail with visioning.

I often run into situations when the leadership team wants to have an annual road map sketched out. But development teams are too worried about figuring out all the details before they say how long something would take. In this case, I always have the teams provide high level relative sizing estimates. Let go of the details. Let's zoom up. So is it small medium, extra, extra large you set the expectations that these t-shirt estimates are, you know, gut feel rather than precision. But then when you're ready to execute, that's when you focus on the low level detail. That's when you drill into uh the specifics and narrow down your time horizon scope and time. In my experience, people often do not understand that these two are opposing energies in a typical development team. You have two forces that are naturally pushing against each other product management, pushing for more features and development, pushing for more time to build quality when that picture becomes out of balance. That's when we get into trouble. I have mentioned t-shirt sizing for an annual road map that is usually based on high level understanding of scope and essentially it creates a high level time boundary.

But the more we dig into the specific initiative, the better sense we get on the true scope of it and potential timing of delivery. And this is when it's critical to have those crucial conversations as an organization. What are we in a position to adjust time or scope? If your timeline is hard, your scope is it should be negotiable. If your scope is non negotiable, your timeline should be moving. It's a natural balance of things. So this is one of my favorites and our modern hustle culture. We don't usually allow enough time for reflection.

We're usually executing on exhale, forgetting to inhale uh pause and reflect. And that's when we lose our balance and become very stressed. This applies not only to individuals but also to entire teams and organizations. We cannot always be hustling. We have to balance that with periods of reflection and strategizing. And it's not only scrum teams that need to have those moments of reflection in their retros, executive leadership teams can benefit from those even more as that has the potential to affect the entire organization.

And one more important thing to note that dynamic balance doesn't mean things are always 5050. It means that they flow from 8020 to 7030 to 5050 et cetera. The natural healthy state is for the balance to move. There's no time for thinking. We have to make a management decision, haven't we all been there? So most of my career, I've been a change agent. I know how hard change can be. I also know how hard it can be to accept a situation you desperately want to change. That's another opportunity to fine tune that balance, to know when it's appropriate to push for change. And maybe when it's time to step back, assess and respect what is in place and works already. Whenever I start a new job, I see so many things that I would love to jump in and improve. I literally have to restrain myself to wait and see. And sure enough after I spend maybe a few weeks trying to understand the present realities of this organization I'm much better positioned to understand why things are done in a certain way. And I'm much wiser at how to move the needle in the direction I want to move. So when we become aware of these opposing and at the same time, complementary energies, we become much wiser at purposefully shifting this balance in our work environments. And thank you very much for attending today. I hope you have some takeaways.

Maybe you've thought of some imbalances in your own systems. Please do reach out to me, let me know what you think. Um And if you want to chat about it further, find me on linkedin. You can either scan this QR code or you can just search for Nadia Clifford and thank you very much for your time today and I hope you enjoy the rest of the conference.