DevOps and sharing

 

This is a written version of a talk I gave at DevOps Days Kiel.
You can find the slides here.
My computer failed me during the presentation and hung up
on Libre Office, so I had to go the first 5 minutes
without slides.
Pro tip: keep your speaker notes on a different device
than your slides. Can save your ass.

Introduction

I am not a tech lead. I am not a product manager. I’m a bit of a software engineer, a sysadmin, and I guess a coordinator. The things I will talk about – the problems and my proposed solutions – come from my experience as someone who is not in a leadership position. As I am not a pure sysadmin but rather someone working across teams, I am a bit of a binding agent. I wholeheartedly hope the contents of this talk will be relatable to you; I can tell you that preparing this presentation was a learning experience for me. As during my whole professional career (about 13 years) I’ve done a fair amount of jobs in different fields including NGOs, election campaigns, translations, media and project management, I believe I can have a few valuable interdisciplinary insights.

Right now I hold the hilarious title of “DevOps Heroine” at Acrolinx. For non-native English speakers: a heroine is a female hero. I started working at Acrolinx in the beginning of this year. Earlier I was a junior software engineer at a small Polish company called Three of Coins. Three of Coins is a consulting company, focused on security and infrastructure. Acrolinx sells software for professional, advanced text editing. There’s a lot of natural language processing and computational linguistics involved. It’s pretty fascinating. Technologies we use include Java, Python, Clojure, Node.js, Ember, Ant, Maven, Gradle, Jenkins, Docker, lambdacd… and many more. We test and build or software on Linux, RedHat, Windows and MacOS. We have a team of around 20 developers (but I can never remember how much exactly) – most of them on site – and some of them are computational linguists. I won’t go into more detail, since this is not a company pitch.

What I mean by “people skills”

I chose “people skills” that I want to talk about based on the Wikipedia article on soft skills and an article about people skills in Portland Business Journal. It’s important to list them first, since “people skills” became this umbrella term that people have different ideas of. Thus, people skills in the scope of this talk would be:

  • communication abilities – clear and effective communication. “Effective” means we understand and we’re understood
  • personal habits
  • cognitive or emotional empathy – the ability to feel what the other person is feeling
  • leadership traits – that’s self-explanatory
  • building relationships of trust, respect and productive interactions
  • understanding ourselves and moderating our responses – being able to answer the questions: “why do I behave the way I do?”, “what might have caused my reaction?”, “How can I get less angry next time?”, and so on.

The “why” and the “how” of this talk

What I would want to do is not necessarily discuss each skill one-by-one. I’d rather want to share with you my observations gathered through the years that I’ve been working in tech companies, both as a project manager and an engineer. The social skills I mentioned hardly foster success if you learn to use just one.

I consider this an interesting topic when thinking of DevOps ideas since DevOps is not mostly about technologies. Furthermore, technology is more interdisciplinary than we might think at first and both social skills and knowledge transfer are essential parts of propagating it, whether you like it or not. Too often, we get stuck in the self-fulfilling prophecy of the anti-social tech-genius and we consider ourselves socially awkward and/or anti-social. But if you look at the traits I listed above, I am pretty confident that most of you could tick at least one off.

I’ll give you an example of this and tell you a bit about myself to make sure you understand what I mean: I have always considered myself a “socially awkward” person. I decided to analyze my behaviors when I started preparing for this talk. I realized that while I do have communication problems, they get quite selective and mostly appear in personal – and not professional – interactions (I talk fast and quiet, avoid eye contact, mumble to myself, and so on). Another thing I’ve noticed is that while I’ve heard from numerous people that I am a person they feel safe around and that they can trust, also a very empathetic person, at the same time I have problems with moderating my responses – I get angry easily and I can get really stubborn really quickly – a reaction which clouds my empathy. To sum it up, I believe that people that consider themselves “socially awkward” do not “fail”, so to speak, on the social skills front altogether, but that they probably have problems adopting only a few of them.

I am mentioning all this because figuring which of our coworkers are good at which social skills is critical in effectively communicating, cooperating and sharing knowledge.

Communication is crucial

I understand this is a cliche and some of you might even be cringing inside when they read it. Of course, we need to communicate effectively when we work in teams, when we approach clients, when we do client support, and so on. I have also – as probably most of you – always understood that.

Or have I?

I’ve worked in Acrolinx for over five months now. In preparation for this talk, I decided to ask some people on my team the following questions:

  • what would they consider the biggest obstacles to popularizing DevOps ideas and implementing DevOps-inspired practices in our company,
  • what was the most important thing they learned recently (this year); did they learn more by collaborating or by themselves, and
  • what’s their approach to internal documentation.

I approached each person individually to discourage a potential discussion about who’s right, and I think a one-on-one made people a bit more open. I also did not want to talk, I just wanted to listen and ask follow-up questions. After I finished interviewing four of my colleagues, I was shocked. You might ask: what shocking truth did I discover? Well, it wasn’t the information I gathered in particular. It was the fact that each person had a different answer to all of those questions!

I made the mistake of assuming we were on the same page about priorities in reorganization and knowledge flow. I realized it was something I should have asked just after I started working there to better know where my team mates are coming from. Don’t get me wrong, we’ve done a lot of good work since I joined, but having a better understanding could have helped me prepare for handling differences in opinions and approaches.

I’d like to share some of their answers with you.

First question: things that make implementing DevOps in your organization difficult

  • Mindset: people don’t want to be bothered. They have their own tasks that they find interesting and important and they don’t want to be dragged away from them.
  • Tools: in Acrolinx, we wrote or need to use specific tools that are cumbersome to maintain; for example, we still build MSI installers, we sign our executables on a Windows 2008 server VM, and so on. Spending a lot of time maintaining, troubleshooting and customizing leaves us no time for process changes.
  • No one wants to do the “boring” tasks, like write Python scripts, practically everyone prefers to work on the cloud because it’s new and exciting. We don’t do cloud anymore – but we still do Python.
  • Using outdated software/technologies (we are in the process of shutting down a server running an Ubuntu distribution from 2009… so you get the picture).
  • Lack of documentation on our outdated software/technologies, which means that no one has the full picture and some (ugly but necessary) hacks come into light only during troubleshooting and/or by accident.
  • Lack of resources: there is not one team member whose work is dedicated purely to cleaning up the mess (this changed after I joined). Client support and developing features get priority over DevOpsing.
  • No bigger picture: our IT manager and sysadmins don’t actually fully know which services are installed on our main servers, neither is there an overview of what software is on our VMs and Jenkins workers.
  • Other people: you send emails, ask people questions and they don’t respond.
  • Being kept in the dark: developers actually don’t know how much time they can contribute to improving collaboration.

Does any of these sound familiar to you? I don’t know about you, but a lot of these problems look solvable to me, and solutions have nothing to do with technology:

  • knowing who prefers to do what might improve task assignment (that actually isn’t that bad in our organization) – so communicating one’s needs and your organizational skills should work here. You might even already know who likes to do what if you pay attention during daily stand-ups.
  • fighting the rigid mindset: a good dose of empathy and effective communication could help. Try to understand what the other person is feeling. Be aware of the fact that there is a difference between a rigid mindset and an underdeveloped skill set. They both might make a person oppose change, but addressing each of them is different. You can either develop someone’s skill set so they feel more comfortable and confident, or you can try to figure out where their mindset comes from. I think the main options are:
    • “my work is more important”,
    • I don’t see the benefits at all – and it’s not my problem,
    • current tools work better, or at least as good: so the benefits of switching aren’t visible.

Once you know more about the person’s feelings, you can effectively address them.

  • outdated technologies, lots of tools, lack of a bigger picture: fixing this will be a slow and sometimes mundane process. People might feel stagnant and unmotivated because they don’t feel challenged. Sitting down together and building a long-term plan (leadership skills, anyone?) will spark enthusiasm and refresh motivation. People like to know their work has meaning. (Personal story: a significant part of my work at this moment consists of migrating svn repositories to git and complicated Jenkins jobs from one instance to another. I’m sometimes pretty boredom, but I know that this is temporary and that these tasks are part of something bigger that I actually haven’t done before. Also, when I communicated my concern to my senior colleagues, they very quickly came up with interesting side projects involving Raspberry Pi that I can work on to refresh my enthusiasm.)
  • other people aren’t responsive: Realize there can be different reasons for their lack of communication: they might not be answering because they’re really very busy, they have some problems with procrastination, they don’t understand the intent behind your questions or they don’t know the answer and don’t feel comfortable admitting it. Each of these will require a different reaction. If they don’t respond to emails for a long time, you probably need to get assertive and approach them at their desk and talk to them face to face (it might be scary, I know).If they are procrastinators, it doesn’t mean they are lazy, it just means they have difficulties with handling some tasks. Maybe you should consider breaking down your request into smaller ones that might be more manageable? Consider asking them the occasional short question during lunch. Or even better: if you used shared calendars, find a free spot in their calendar and arrange a short, 15-30 minute meeting so you can talk a few days in advance. This way the other person won’t be taken by surprise and will know in advance that you want something from them.

Second question: what was the most important thing you have learned this year:

This was a tough question and in retrospect I think I could have been more clear. Here’s what I got, though:

You need to know what you want to build before you actually build it.

So a few years ago, we bought our now-outdated server specifically for hosting SVN, but then started installing other software on it without a long term maintenance plan. We build a git-Transifex integration without having plans for a future Github integration in the future and now my colleague is writing not pretty hacks. Our Jenkins slaves are a mix of VMs, physical machines and a few dedicated users on our two main servers – that ecosystem grew without a longterm plan and now we have stuff to fix.

You should keep things simple and don’t engage in premature optimization.

Focus on maintainability and simplicity first, even at the cost of for example speed. Otherwise, you’ll try to outsmart yourself and regret it.

“I’ve learned a lot and it’s hard to pinpoint the most important things, but I’ve mostly learned by myself or pair programming”.

Hey, you’ve got confirmation that pair programming is useful in your organization.

Complaining is great. I love listening to people complaining, because good things come out of it sometimes. Remember that git-Transifex integration? Complaining about it was followed by spending 20 minutes talking about how it should be redesigned and written from the beginning, drawing a plan and choosing technologies. All of a sudden we had a concrete plan and a healthy, microservice design that we are excited about.

Conclusions: encourage people to complain and pair program. Seriously, complaining is awesome. Sidenote: keep in mind that people will trust you more when they feel heard.

Third question: what’s their approach to internal documentation.

I got different answers from different people.

  • We have so many tools and internal knowledge lying around that internal documentation is necessary.
  • Internal documentation is unnecessary: software changes and we have a low turnover rate, so it gets outdated quickly; additionally, someone has to maintain it.
  • Documentation is important, but no one wants to do it and having it is a team decision – not an individual one.

The differences in approach are both surprising and not. The people who’ve been in the company for the shortest time were the ones that were rooting for documentation more than their more senior colleagues. It makes sense – the longer you work in a group, the more knowledge you retain. This brings me to the next point: how do we make sure that we properly transfer knowledge and do it in the most effective way?

Gaining “tribal” knowledge

The company I work in has a fairly low turnover rate: it has a healthy work environment and promises interesting work; our linguists were paid for doing original research that ended up in peer-reviewed magazines, and so on. Now, when people work together for a long time, they start treating special knowledge as if it was common. When someone new joins, they don’t always get that knowledge until way later. Example: it took me three months to learn (by accident) that a few of our /etc folders were kept under version control using etckeeper.

One of the more important non-technical things I’ve learned as a junior software engineer at Three of Coins was: the more senior a person is, the bigger the risk that they might treat a lot of their knowledge as obvious. That’s something my boss told me numerous times when he was explaining tasks or new things to me and that’s something I also noticed among other people I’ve worked with.

You need to decide whether to spend a lot of time writing down docs and then drop them on the new hire or to sit down with them every day and show them something new. The first option will probably be better when you have a high turnover rate. By the way, you should also probably consider addressing the high turnover issue.

In organizations with low turnover rates and lots of internal knowledge, the importance of knowledge transfer can make itself known in difficult circumstances. To understand what I’m talking about, spend a few minutes thinking about each of your colleagues: what would happen if they suddenly disappeared? They went on an extended sick leave or decided to leave the company? How much would that influence everybody else’s work? Stress levels? Deadlines?

I hope you can realize that making sure no one is irreplaceable is a form of risk mitigation. Companies may have different approaches to how to solve that problem. My brother works as a system administrator and IT manager at a nationwide bookstore chain. One of their chief executives has a rule that if someone becomes irreplaceable… he fires them. That’s a pretty drastic way of encouraging people to cooperate. A less drastic way would be to persuade your team members that spending time on teaching others benefits everyone, including them: when I’m sick, I want my work to leave me alone – I don’t want people calling me on the phone and asking me work-related questions.

Not everyone will be enthusiastic about this approach, as they will feel their current work is more important or that it’s not their responsibility. Try to poke them and figure out where the resistance comes from.

Ask your colleagues how they learn.

The thing about knowledge sharing is: it’s both about what you know and about where you look. Explain to your colleagues that they don’t have to just sit down with less senior coworkers and introduce them to some old, cranky code. Rather, encourage them to talk about how they themselves acquire experience. Ask them about how they debug things. You might be surprised how enthusiastic some people might get when they talk about debugging and their approach to problem solving. This approach fosters sharing more universal, “recyclable” knowledge that will be technology-agnostic.

Wrapping up: practical tips

Don’t only talk with your colleagues about current tasks; ask your teammates about their ideas – ask sysadmins about their opinions on “developer stuff” and vice versa. Neither need to have in-depth knowledge of what the other teams are doing, but firstly, an overview will help a lot, and secondly, it’s good to know how much the other team knows.

Try to understand where everyone is coming from, especially when you don’t agree with them – that would be an exercise in empathy. Ask yourself: what makes that person think this way? Try to follow their thought process and recognize places where it could be shifted.

Make these your new favorite questions: “What is the problem you’re trying to solve?”, “What did you expect to happen and what happened instead?” If you want good answers, you need to ask good questions.

In case of hesitation or resistance, try to figure out if the problem lies in the skill set or the mindset and act accordingly.

Don’t make assumptions about what other people are thinking.

Ask, ask, ask.

When in doubt: ask.

Don’t go quiet when things go wrong or when you’re stuck: lack of communication is a bottleneck. Other people might be waiting for your work. Being stuck does not mean you’re unskilled, everyone gets stuck once in a while. Be clear that you’ve encountered a problem. The sooner you do that, the faster it’s solved.

Encourage knowledge-sharing: host a regular education lunch or a demo time. When your coworkers share their knowledge with others, they also deepen their own knowledge on the subject, as you need to – surprise – understand things in order to explain them.

Have your own science log: make notes on interesting problems that you fixed during your work, leave some space for jotting down ideas. That will be helpful when other people ask you about problems they know you’ve experienced a long time ago.

Practice non-violent communication: when you need to criticize, talk about behaviors, not people. When someone gets defensive, you won’t come across to them.

Consider doing regular brainstorming sessions – once per month, for example. Make sure to create an atmosphere where people feel heard, important, respected, as they will share their ideas more freely.

Mimicry. When you see it works for other people, try it yourself. Example: our IT manager – who is a fantastic and extremely kind person – has to turn off or restart some of our servers or VMs sometimes for maintenance. As you can imagine, developers can’t some services then. The problem was that he didn’t always inform everyone in advance. When other people started sending emails informing about breaks in availability of things they were responsible for, after a while he started sending warnings himself, sometimes creating calendar events for scheduled maintenance so that people don’t forget.

And lastly: take care to foster a healthy work environment, whether you’re in a leadership position or not. Be nice to your coworkers. Be patient. Ask how their day was. You don’t have to be best friends with your coworkers, but if you all feel welcome and comfortable in each other’s presence, everybody wins.

So that’s it. Shoot me a comment. I’d love to hear your input.

I'd like to thank Maciej Pasternacki from Three of Coins 
and my colleagues at Acrolinx 
for their valuable input.
Advertisements