August 2017

The Importance of Managing Complexity

Introduction

We write software to make our lives easier; to reduce the complexity and chaos. The need to reduce complexity is at the very center of software development. Unfortunately, we tend to mirror the complexity of the real world in our software and this leads to many problems. As Bruce Schneier in his essay A Plea for Simplicity: You can’t secure what you don’t understand said, “the worst enemy of software security is complexity.” In this blog post, we will talk about the importance of managing complexity and how to reduce complexity in software.

The Importance of Managing Complexity

Fred Brooks, in his paper “No Silver Bullet: Essence of Accidents of Software Engineering”, explains there are two types of issues that introduce complexity, the essential and the accidental properties. Essential properties are the properties that a thing must have to be that thing. Accidental properties are the properties a thing happens to have, but they are not really necessary to make the thing what it is. Accidental properties could also be called incidental, discretionary, or optional properties. Brooks argues that difficulties from accidental properties have been addressed with higher-level languages, the introduction of IDEs, hardware enhancements, etc.

This leaves the problem of essential difficulties and essential difficulties are harder to solve. You are now interfacing with the complex, chaotic real world where you must identify all the dependencies and edge cases for a problem. As software addresses larger and larger problems, interactions between entities become more complex and this increases the difficulty of creating new software solutions. The root of all essential difficulties is complexity.

Conceptual errors (specifications, design, and testing) are much more damaging than syntax errors. C.A.R Hoare, the inventor of the quicksort algorithm, stated in the talk The Emperor’s Old Clothes, “there are two ways of constructing a software design: one way is to make it so simple that there are obviously no deficiencies, and the other is to make it so complicated that there are not obvious deficiencies.” Steve McConnell in Code Complete states that software’s primary technical imperative is to manage complexity. When software projects fail for technical reasons, one reason is often uncontrolled complexity (i.e. things are so complex that no one really knows what is going on). Edsger Dijkstra pointed out that no one’s skull can contain a whole program at once; we should try to organize our programs in such a way that we can safely focus on one part at a time.

Brooks paints a pessimistic picture of the future stating there is no “silver bullet” in either technology or management technique that promises even an order of magnitude improvement in productivity, reliability, or simplicity. While there is no silver bullet, there are things that can be done to reduce complexity.

How to Reduce Complexity

When designs are inefficient and complicated they can be broken down into three categories:

  1. A complex solution to a simple problem
  2. A simple, incorrect solution to a complex problem
  3. An inappropriate, complex solution to a complex problem

As stated above, the most important technical goal is managing complexity. In terms of software architecture, complexity can be reduced by dividing a system into a series of subsystems. The goal is to break a complicated problem into simpler, easier to manage pieces. The less subsystems are dependent on each other the safer it is to focus on one area of complexity at a time. McConnell gives 14 recommendations on how to conquer complexity:

  1. Dividing a system into subsystems at the architecture level so your brain can focus on a smaller section of the system at one time
  2. Carefully defining class interfaces so you can ignore the internal workings of the class
  3. Preserving the abstraction represented by the class interface so your brain doesn’t have to remember arbitrary details
  4. Avoiding global data, because global data vastly increases the percentage of the code you need to juggle in your brain at any one time
  5. Avoiding deep inheritance hierarchies because they are intellectually demanding
  6. Avoiding deep nesting loops and conditionals because they can be replaced by simpler control structures.
  7. Avoiding gotos because they introduce nonlinearity that has been found to be difficult for most people to follow
  8. Carefully defining your approach to error handling rather than using an arbitrary number of different error-handling techniques
  9. Being systematic about the use of built-in exception mechanisms, which can become nonlinear control structures that are about as hard to understand as gotos if not used with discipline
  10. Not allowing classes to grow into monster classes that amount to whole programs in themselves (Lattix Architect has a number of metrics that can help with this)
  11. Keeping routines short
  12. Using clear, self-explanatory variable names so your brain doesn’t have to waste cycles remembering details like “i stands for the account index, and j stands for the customer index, or was it the other way around?”
  13. Minimizing the number of parameters passed to a routine, or, more importantly, passing only the parameters needed to preserve the routine interface’s abstractions
  14. Using conventions to spare your brain the challenge of remembering arbitrary accidental differences between different sections of code

Conclusion

The importance of managing complexity can be seen when we start looking at other people’s source code. Depending on the complexity of the software, this can be a daunting task. As stated in Code for the Maintainer, “always code and comment in such a way that if someone a few notches junior picks up the code, they will take pleasure in reading and learning from it.” Lattix Architect has helped hundreds of companies reduce the complexity of their source code by visualizing, optimizing, and controlling their software architecture.

How to Achieve High Velocity in Software Development

How do you achieve high velocity in software development? The main challenge to high velocity is tight coupling. Coupling is the degree of interdependence between software modules, or a measure of how closely connected two routines or modules are, or the strength of the relationships between modules. Let’s break down exactly what coupling is, the problems with tightly coupled software, and how to reduce coupling.

What is Tight Coupling

According to John Lakos in Large Scale C++ Software Design, there are two types of coupling: logical and physical. Physical coupling is putting classes, structs, etc. in the same component or creating dependencies from one component to another. Logical coupling is when types (classes, structs, etc.) that are used in one component are supplied by another component. Tight coupling is when a group of component or modules are highly dependent on one another.

tight coupling

Often this happens when components assume too many responsibilities. Large files are sometimes a quick way to find tightly coupled components.

Problems with Tightly Coupled Systems

Tight coupling is a challenge when you want to make changes to a software system. Coupling causes a ripple effect - like circles in a still pond.

coupling ripple effect

You make a small change and suddenly twenty other seemingly unrelated things need to be changed as well. While zero coupling is impossible, what level of coupling is right? The answer, according to Jeppe Cramon in his slideshare presentation “The Why, What, and How of Microservices,” highly depends on how likely the component/system/service is to change and what components need to change together because of the design.

Maintainability suffers in tightly coupled systems. One example from “Large Scale C++ Software Design” is creating interfaces that only contain primitive functionality. If you need non-primitive functionality it should be put into a separate class or operator without private access.

Blithe Rocher, in her talk “Avoiding testing headaches with tightly coupled services,” discusses how tightly coupled services are the most common mistake she sees in microservices. When services are tightly coupled they produce headaches all the way down the deployment stack. In production, if one service goes down, both (or all) services go down. In development, both services need to be running to develop either one. If one service is broken, you can’t work on the other service even if another team is responsible for that service. This adds to development time and makes working in the development environment harder. All of this results in a higher learning curve for new engineers to a project because a great deal of time is spent learning the development environment.

In automated testing, tight coupling means more tests need to be created and the environment needs to be set up for all the coupled services. Blithe’s advice is to not build tightly coupled services. A loosely coupled service or component can be developed, put into production, and tested independently of other services or components.

In the Harvard Business School paper "Exploring the Relationship between Architecture Coupling and Software Vulnerabilities: A Google Chrome Case,” the authors make the case that there is a strong relationship between security vulnerabilities and architecture coupling metrics. Coupling affects quality, productivity, maintenance costs, and security vulnerabilities. In the Google Chrome codebase, component metrics and coupling metrics were significantly correlated with vulnerabilities.

How to Reduce Coupling

Interfaces are a way to reduce tightly coupled components. By communicating through interfaces you break the dependency and any component can be on the other end of that communication. Steve McConnell, in Code Complete, has these guidelines:

  • Minimize accessibility of classes and members
  • Avoid friend classes, because they are tightly coupled
  • Make data private rather than protected in a base class to make derived classes less tightly coupled to the base class
  • Avoid exposing member data in a class’s public interface
  • Be wary of semantic violations of encapsulation
  • Observe the “Law of Demeter

Finally, John Lakos states you should keep both physical and logical coupling to a minimum. As an example, minimize the use of external types in an interface. This will make the component easier to use and to maintain.

Conclusion

Coupling is a measure of the dependencies between components. Tight coupling affects development velocity, quality, maintenance costs, and security. Lattix Architect analyzes your software dependencies and will help you reduce coupling. Click here if you would like to try it out.

Software Architecture and GDPR Compliance

The General Data Protection Regulation (GDPR) is an EU regulation on privacy protection that goes into effect in May 2018. GDPR applies not only to EU companies that process personal data on EU residents but also to companies not located in the EU. As Article 3 states, it is “applied to the processing of personal data of data subjects who are in the Union by a controller or processor not established in the Union.”

Software architecture is an important part of GDPR compliance. An architectural model of the software gives you a complete view of everything connected to the personal data in your system. The GDPR defines personal data as any information that has the potential, alone or paired with other information, to identify a person. You need to preserve the identity of an individual across different names and properties and be able to trace them across the system and disparate data points as stated in Article 30. You have to record what you do with personal data and define which applications use it.

The Automated Decision Making section of the GDPR states that any system which undertakes automated individual decision-making, including profiling (Article 22), is now contestable by law. This includes automation components such as calculation engines, scoring systems, or other processing of personal data. You need to be able to trace the personal information through these systems and demonstrate compliance. Article 5 states “the controller shall be responsible for, and be able to demonstrate compliance…”

Compliance Steps for GDPR

As part of ensuring compliance for GDPR, you will need a good overview of the personal data involved.

  1. Identity all data that the GDPR considers personal data. Lattix Architect will give you this information with its member level expansion feature that allows you to see all of the variables associated with personal data.

    software architecture (see our video on Member Level Expansion)

  2. Once you have identified the personal data, you need to analyze its use. Lattix Architect understands all of the dependencies in your software system, so it will know all of the dependencies on the personal data. Now you will be able to model the data flow and show which applications, processes, etc. use the personal data.
  3. Once you have modeled the data flow, you will be able to demonstrate compliance with GDPR by using the Impact Analysis Report in Lattix Architect. This report tells you all the dependencies on selected elements (in this case variables) and can be exported to Excel, csv, or XML formats.

    software architecture (see our video on Impact Analysis)

GDPR compliance is something you need to regularly revisit. You must go through the above steps frequently to ensure you remain compliant. This becomes part of your governance framework.

Summary

Non-compliance with GDPR can result in large fines. Penalties,as outlined in Article 38, include “fines up to 20,000,000 EUR or in the case of an undertaking, up to 4% of the total worldwide annual turnover.” There is personal damage that can be claimed by any individuals who are the data subjects and there is personal liability for directors and senior managers. This all makes it worthwhile for organizations to take these risks seriously.

Using Escalation to Solve Cyclic Dependencies

As we discussed in the previous blog post, cyclic dependencies degrade the quality of a software system, leaving it inflexible and difficult to manage. As the size of a software system grows, the importance of catching and fixing cyclic dependencies grows because the overall cost of developing and maintaining the system increases.

How Cyclic Dependencies are Created

Software systems usually start out with clean, well thought-out designs that don’t have bad dependencies (cyclic dependencies). But, as software development starts in earnest, custom enhancements or bug fixes are requested, usually with quick turnaround times. This can lead to situations where the design is compromised because of insufficient development time and can lead to architectural problems like cyclic dependencies.

As an example, here is some C++ code that has two similar classes in a software system that do similar things (hold similar info), Circle and Tire, both representing a circular shape. A Circle is a defined by a radius.

// circle.h
class Circle {
// ...
public:
    Circle(int Radius);
    // ...
    double getarea();
};

While a Tire is defined by a center point and a radius.

// tire.h
class Tire {
// ...
public:
    Tire(int xCenter,
           int yCenter,
           int radius);
     // ...
     double getCircumference() const;
};

Both provide similar functions and hold similar information but might also have custom functionality or different performance characteristics. What if we now have a client request to convert between these two types of circular shapes? One solution is to add a constructor which has one argument: a const reference to the other class. In this solution, the conversion is performed implicitly. Allowing two components to “know” about each other via #include directories implies a cyclic dependency.

// circle.h
#include "tire.h"

class Circle {
// ...
public:
    Circle(int Radius);
    Circle(const Time &T);
    // ....
    double getarea();
};
// tire.h
#include "circle.h"

class Tire {
// ...
public:
    Tire(int xCenter,
           int yCenter,
           int radius);
     Tire(const Circle &c);
     // ...
     double getCircumference();
};
This solution has two problems:
  • Performance degrades because you have to create a temporary object of the other type when you call the constructor.
  • You have introduced a cyclic dependency between two components. Now you can't compile, link, test, or use one class without the other.

How to Solve Cyclic Dependencies with Escalation

One solution is to move the functionality that is causing the cyclic dependency to a higher level. If peer components are cyclically dependent, it may be possible to escalate the interdependent functionality from each of these components to static members in a potentially new, higher level component that depends on each of the original components. We can create a utility class calledCircUtil that knows about the Circle and Tire classes and place the definitions in a separate component.
// circutil.h

class Circle;
class Tire;

struct CircUtil {
    static Tire toTire(const Circle &c);
    static Circle toCircle(const Tire &t);
};
// circle.h

class Circle {
    // ...
    public:
    // ...
};
// tire.h

class Tire {
    // ...
    public:
    // ...
};
Now you can use, test, etc. each class independently.

Summary

While this is a simple example, cyclic dependencies in larger systems have the potential to greatly increase the cost of developing and maintaining software systems. Escalating mutual dependencies to a higher level can fix cyclic dependencies. This reduces the maintenance costs of a system by getting rid of unnecessary dependencies among components at the same level. This also makes the system more flexible and reusable. If you are interested in identifying your bad dependencies, check out Lattix Architect.