Month: August 2017

The Case for Desired-State Configuration and YANG

I’ve been spending a while now using Solarwinds Orion configuration scripts to harmonize and update configurations on a large legacy network, so the significant limitations of the current model of network configuration are fresh on my mind.

Traditionally a lot of network equipment such as switches and routers are configured via a text file, where each line has configuration settings that get applied, with sometimes nested blocks of sub-configurations. For example, from Cisco:

hostname myfirstswitch
logging host
interface Ethernet0
 ip address
 ip access-group MyAccessList in
 duplex auto
 speed auto
 no shutdown

Here we set the host name, then configure the first Ethernet interface with an IP address, an access list (similar to a firewall or iptables), and set the duplex and speed values, and finally turn the interface on (on some Cisco models there’s a default for an interface to be off, so turn it on you have to turn it not-off.)

Traditional methods of configuring the switch are over a serial port or SSH, or possibly by loading configuration commands in as a file via SCP, TFTP and the like. While the exact details of the complications do change based on the method used, a lot of the basic problem remains.

In my particular case one issue is that there may be random old configuration left. References to DNS servers or logging servers that no longer exist.  It’s easy enough to add a server, but unless you know that there happens to an old entry (logging host the new configuration doesn’t remove the old. So now you’re stuck writing rules to look for configuration statements of that particular format that shouldn’t be there. Certainly doable, but it adds a lot of extra complexity.

Another issue is the order of operations. A traditional example is the above access list. It’s typically a set of “permit” statements followed by an implicit (or explicit) “deny” statement.  So it’s easy to either blank an access list that governs access to the switch and lock yourself out in the middle of the configuration, or apply an access list before it’s defined. Another common issue is re-addressing devices; you change the IP and subnet mask on one line, and the default gate way on another. But changing either may stop your ability to communicate until the other is applied. Once more, there are ways around it, but it still means a lot of extra complexity in planning and scripting. 

You can’t just tell the switch what configuration you want it or its components have. You have to figure out what state it is currently in, and then do a lot of conditional logic to determine how to get it to the state you want it to be. There are of course additional projects to try to abstract some of that complexity, but on some level they just add yet another level of proprietary components and a black box. Some other vendors, and even some Cisco models, allow configuration sessions with roll-backs, confirms, and the ability to do more atomic applications, but that’s still short of ideal.

One attempt to fix this state of affairs is YANG and NETCONF, IETF standards for representing the state in XML or JSON and transferring the state via an RPC mechanism. This approach isn’t perfect either, and isn’t well supported by vendors. One issue is that the capabilities and peculiarities of each platform differ so much that it’s difficult to abstract away. At the very least, though, it allows for a proper desired-state configuration, which would be a fantastic step forward.

It’ll be very interesting to see whether vendors will start supporting the IETF standards or other APIs, but it’s hard for me to see that going forward we wouldn’t quickly start adding APIs for configuration instead of the old SSH and line configurations. It’s equally hard for me to see that we’ll quickly get away from this problem, considering that typical life cycle of networking gear is 10+ years in enterprise networks.

Posted by Toivo Voll in Information Technology

128 Technology and Secure Vector Routing

Photo: Johannes Winger-Lang

I ran across an interesting new company today, and decided to walk through some of the technology.

You can catch the video here.

The basic idea, as far as I can tell, is that you replace or augment your existing routers with the company’s x86-based boxes. You’re not replacing the underlying Internet, despite what some of the claims might lead you to think — instead they have a proprietary encapsulation/tunneling technology. It’s a lot like a dynamic multi-point VPN, where your traffic moves from one node in your network to another over encrypted tunnels, except here the system builds “tunnels” based on sessions and flows rather than network nodes. What makes the technology really interesting, though, is that it seeks to combine many functions that you get when you maintain a lot of state and know more about your traffic and flows.

In addition to encrypting traffic from point A to point B, it allows you to do traffic engineering / optimization in the vein of SD-WAN — the presentation doesn’t go into full detail, but it’s easy to think of ways that you could optimize for cost and bandwidth, and if application-aware, send VoIP and media streams over low-latency, expensive links and bulk traffic over higher-latency but cheaper links, for example; or shift traffic patterns, allow for overflow peaking to metered links and so forth.

Simply offering an easy-to-manage multipoint VPN — which is currently a major headache that takes a lot of engineer hours to implement — and SD-WAN — which saves money — is a winner, but they aim higher.

If the system knows flows and applications, it’s an easy jump to add security functions to it — firewalls, possibly even IPS/IDS/DLP. Perhaps traffic shaping and policing as well.

There’s a lot of telemetry and visibility that is possible from a modern system that has flow and application-level visibility at every hop. It’s not that current routers couldn’t do this, but they’re badly hamstrung by lagging legacy management schemes such as SNMP.

Configuration of traffic patterns, routing, IP addressing etc. can be done centrally, in the vein of overlays and SDN.

No need to reconfigure anything on the underlying network. The idea that you don’t want to have to ask carriers for anything is pervasive, and it’s attractive for a reason as anyone who’s ever dealt with carriers can attest.

An x86-hardware agnostic approach might allow for a nice range from affordable to high-performance hardware to support many low-cost branches.

High-touch services on the routers? If Cisco is putting container support in their LAN access switches and routers, this may be the way to go.

Where’s the catch? Well, a lot of these things aren’t exactly new ideas, and the difference between wanting to do something and being able to do so is fundamental. Making firewalls is hard. Coming up with a way to route and prioritize traffic is hard even before you add more complex decision criteria to it. Troubleshooting underlying transport issues and how they present through this vector-routed mesh might be a challenge. A particular detail I’m curious about is whether the scheme requires either a transport MTU of more than 1500 bytes, or if it limits the TCP/UDP payload. It says it’s inband signaling and doesn’t have the complexity of MPLS, but it’s still an encapsulation with effectively another set of headers, unless they have a surefire way to compress every packet enough. How is the reliability, and how does it deal with outages of underlying networks?

With the advent of SD-WAN, NSX, ACI, and the already boringly old MPLS infrastructure the engineering and conceptual framework for something like this might be there, though. It does seem to me that if they can deliver on their promises, this would be the perfect time to offer any distributed businesses a simple, single-vendor solution that replaces dozens of expensive, complex, difficult-to-manage products with one centrally managed, software-defined networking stack.

Posted by Toivo Voll in Information Technology

Hidden Figures

Hidden Figures

During a recent flight I finally had a chance to watch Hidden Figures, the movie about the untold story of the black computers who were instrumental in NASA’s manned space flight program. They fought dehumanizing, demoralizing racism and even so managed to make key contributions and keep their pride and hope.

In short, the movie should be seen by everyone, and I’m glad to hear that it’s becoming part of school curricula. It touches on a lot of topics I care about. It shows the beauty and importance of mathematics, and dispenses with the idea that mathematics isn’t for everyone. I am slightly bothered by the genius-worship in the movie, but that’s a minor niggle. It talks about the incredible efforts that went into sending humans into space, and it finally recognizes the important role of people who had been written out of white-washed history.

The portrayal of racism was matter-of-fact which in many ways lessened the immediate emotional impact until you actually thought about what just happened. Not a ton of subtlety either, but some pointed and clever dialogue:

White woman: “You know I don’t have anything against you.”
Black woman: “I know. I know you believe that.”

The protagonists were lionized as perfect; it shouldn’t take that to be respected, and the relatively happy ending also seemed arguable in contrast to real history.

The protagonists were/are heroes, and they broke down barriers. I am immensely glad the movie celebrates this and gives them recognition. Yet I’m hoping nobody thinks that their accomplishments meant that others had the same opportunities shortly thereafter, or that the continuation of the very same fight isn’t happening today.

There has not been a magic turning point between then and now that has made everything better. Yes, things have gotten better, and they’ve gotten better because of the tenacity of people who refuse to sit in the back of the bus, refuse to give up their rights, and those who fight for equality.

I’ve chosen the still from the movie on purpose. This very same scene is still playing out in way too many meetings I have been part of in my own field generations later. The next time you find yourself at an IT trade show, or a training class, or a work meeting, look around and consider how many women of color there are. Then consider how much brilliance and contribution is going unused in a world where they can’t, or won’t, be part of our profession. Then consider what you can do to change that.

Posted by Toivo Voll in Information Technology

Ops, DevOps, and the Big Picture

DevOps is the way to IT nirvana, magically conjuring up the unicorns of agility, reliability, efficiency, engaged employees, and accelerated implementation schedules.

…Except we know the reality. It’s not about the tools or gimmicks – DevOps is a philosophy, and if an organization is built to, or transforms to follow the right principles those goals can be reached. 

I want to address one particular aspect of this philosophy that I have not seen discussed enough – the Big Picture. There are two sides to this.

The Goal

Back to basics! The point of all the DevOps magic is to achieve some goal in a more efficient manner. Unless the developers and operations teams know what this goal is, and how they can contribute to it,  the wrong initiatives and work get prioritized, and the work done is less meaningful in the end. Surprise, there’s nothing dev-opsy about this, it’s all about good, traditional communication and management. The need for clear direction, planning, and execution and communicating it down the organizational structure doesn’t vanish even if the engineers are wizards. Make sure development and operations engineers have a good idea what the organization is trying to do, and let them find ways to contribute. It means better focused work, and more meaningfully engaged employees.


Having a two-way communication between development and ops isn’t really a DevOps thing either; it’s clearly a part of a healthy ITIL, Site Reliability Engineering, Agile, DevOps or pick-your-methodology organization. 

It matters for multiple reasons.

First, if there is a division in labor between the operational tasks and the development-oriented tasks, communication helps foster better cooperation between the respective employees and groups.

Second, it makes for better solutions – having operations weigh in on what kinds of things make life easier and reduce unnecessary work and engineering on the front end of projects can be incredibly helpful. (*cough* Proper application-level HA. *cough*)

Third, it is a sign of a healthy organization and makes the role of an operations engineer more rewarding. If the operations engineers don’t have the time or the opportunity to get involved on the front-end of projects, it either means that they’re overworked, or that they’re getting the mushroom treatment and handed projects they have to magically make work without having been able to influence the design. Both of these are significant red flags.


Like so many parts of the DevOps fever, once you unpack the principles behind it, it turns out that there are some good, common-sense ideas at play. It’s not that DevOps is conceptually that different from the ITIL wheels of continuous improvement, it’s more about figuring out how to actually allow that ideal to be reached without falling into the process morass ITIL brought us. Alternatively, in places where strict controls and processes are unavoidable, there are still great lessons to be learned from DevOps and Agile methodologies. 

Posted by Toivo Voll in DevOps, Information Technology

The Goal and The Phoenix Project


I’m reviewing these two books together, since the Phoenix Project builds largely on The Goal by applying the Theory of Constraints to the IT environment.

The Goal by Eliyahu M. Goldratt is familiar to anyone who has studied management. It tells a fictional story, following a protagonist struggling with production problems at a manufacturing plant. By following the protagonist’s journey in understanding the problem definition, the mechanisms in action, and how to improve the situation the reader gains the same information, is guided through the logic and thought process, and the theory is applied to (fictional) practice. It’s not the most riveting piece of fiction ever written, and Goldratt spends too much time showing just how bad the problem is and the protagonist’s frustration before moving onto the enlightenment steps. That said, it’s certainly a much more pleasant and effective way to convey the concepts Goldratt wants to share than a traditional theory book would be; much like a a textbook it does require the reader to put it down here and there and think through what just happened and was suggested.

The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win by Gene Kim, Kevin Behr, and George Spafford follows the same method, but is objectively a much better book. It starts off with a dysfunctional IT organization within a company. Here it shines by painting a picture of archetypal IT staffers and situations with such skill that anyone who ever has worked in IT may be tempted to replace the characters with names from their own organization. The pain-points are also all too familiar. It moves along at a much faster pace while still succeeding in conveying the principles and theory it sets out to communicate.

The Phoenix Project in particular should be required reading for anyone in IT operations or development to get a better idea of organizations as a whole, especially management. Regardless, helping the reader understand how to be more efficient, and how to spot inefficiencies around them, it is helpful no matter the level of an employee. It additionally serves as a great source of references for more reading, such as Personal Kanban, The Five Dysfunctions of a Team, and The Goal to name a few.

The Phoenix Project I highly recommend; if you want to get into more nitty-gritty about the Theory of Constraints in still a very accessible work, The Goal is a good follow-up.



Posted by Toivo Voll in Book Review, Information Technology