Chapter 12

Structuring the Ill-Structured

Having examined the nature of software and some of the characteristics of the design process in the first part of this book, and some of the ways that we can describe these in the second part, in this third part we address the role of ‘design as a verb’ and the question of how software might be designed.

We might expect that, in the ideal, the design of software should be an opportunistic process, with design decisions being driven by the needs and characteristics of the required application. However, to do so well requires extensive experience and considerable ability from the designer(s). Given that software is often more likely to be designed by less experienced and more imperfect beings, various design strategies and guidelines have been devised to assist them and to help organise the design process.

This chapter introduces these, and then the following chapters (among other things) look at some of them in rather more detail. A core theme that runs throughout is that of how to exchange and share design knowledge, whether it be in terms of strategies for developing a design, or ways of organising the design elements. Indeed, given that providing knowledge transfer is one of the main goals of this book, this chapter can be considered as introducing some key elements of what this can involve.

12.1Challenges in creating a design

The difficulties involved in creating software applications and systems have long been recognised by software developers, if not always by others. While other technologies related to computing such as hardware design have raced along, rapidly gaining orders of magnitude in performance, while reducing size and cost, the techniques employed for designing software appear to have inched along in a series of relatively small steps. Much of this of course stems from the ill-structured nature of software design, reinforced by some of the characteristics of software itself. As observed by Brooks (1987) the challenging properties of software include the following.

Complexity. This is seen as being an essential property of software, in which no two parts are alike and an application may possess many states during execution. This complexity is also arbitrary, being dependent upon the designer (and the model they create), rather than the problem.
Conformity. Software, being ‘pliable’, is expected to conform to the standards imposed by other components, such as hardware, external bodies, or existing software.
Changeability. Software suffers a constant need for evolution, partly because of the apparent ease of making changes (and relatively poor techniques for costing them or recognising the technical debts they create).
Invisibility. As observed in the preceding chapters, any forms of representation used to describe software lacks any visual link that can provide easily-absorbed descriptions of the relationships involved. This not only constrains our ability to conceptualise the characteristics of an application, it also hinders communication among those involved in its development (including the customer)

An interesting question is whether or not these have altered significantly with time. Being intrinsic properties of software itself, both complexity and invisibility remain issues that are as challenging as they were when Brooks was writing his analysis. Systems remain complex and while design representations have evolved to help cope with different architectures, the lack of any visual parallels remains one of their characteristics. And if anything, the other two properties have come to pose even greater challenges. The connectivity of software systems (typified by the ‘internet of things’) has increased and software often needs to confirm to standards that themselves change and evolve. Indeed while complexity and invisibility are largely intrinsic properties of software, both conformity and changeability are more affected by extrinsic factors.

So these properties of software continue to form a set of constraints upon the activities of designers as well as upon the form of the outcome from the design process. And they will all influence the design process, regardless of the scale, purpose and form of an application. In Chapter 4 we looked at some of the knowledge that software designers may possess (or need to possess) as individuals. In this chapter (and the following chapters) we are concerned with the ways in which knowledge about designing software (knowledge schema) can be shared between experienced designers, and also transferred from experienced designers to those having less experience.

12.2Evolution of knowledge transfer mechanisms

Since the process of software design is an ISP, the very idea of teaching others to design software would appear to be a contradiction. If the design of every software application is both unique and also has many possible forms, we might reasonably ask how we can usefully learn anything other than rather general principles when it comes to acquiring design knowledge. However, while this may be true, it does ignore the fact that, at an abstract level software applications may also have many things in common. Our example of the CCC is focused upon booking the use of some form of resource, in this case cars. But there are many other applications that book resources, whether these be theatre seats, train travel, hotel rooms etc. Individually they will differ, but they will also possess many characteristics in common.

The same effect occurs when we consider architectural style. Here the commonality lies less in what the applications do, and more in the way that their operations are organised (call-and-return, pipe-and-filter etc.). As it happens, the earliest forms of knowledge transfer used with software design were essentially based upon this commonality of architectural form.

Developing a design (implicitly, developing a design model) can proceed in various ways. Early approaches assumed that the designer essentially began with a ‘clean sheet’ and a clear specification of what was needed. (We may smile at this last assumption, but of course, many early applications such as payroll systems and applications recording commercial transactions were relatively well-defined and involved little or no user interaction.) What we usually term plan-driven design strategies could be used to create these, essentially by treating the task of designing software as though it were almost a form of WSP, and early forms did so by organising the design process as a sequence of ‘steps’ that involved:

building a model of the requirements (essentially a ‘system model’) to provide a specification of what was required;
mapping this on to the preferred architectural form (typically call-and-return or pipe-and-filter) by allocating the tasks identified in the previous step to specific elements;
refining the details of the elements and their interactions (such as the use of parameters) to create the final design model.

We will look at such strategies, their strengths and limitations in Chapter 13. In terms of the properties of software identified above, they do provide support for addressing both complexity and invisibility, but are less readily able to incorporate ideas about conformity or changeability.

Early plan-driven approaches commonly employed a top-down strategy with a clear sequence of actions, starting with a description of the system as a whole and then developing a design model that describes the detailed design elements in a step by step manner. As noted above, this may well produce designs that lack flexibility and that can be difficult to adapt and modify. Over the years various forms employing a more ‘bottom-up’ strategy emerged, culminating in the concept of agile development approaches. These aim to provide better support for changeability (and to some extent for conformity) and are discussed in Chapter 14.

Any form of knowledge transfer tends to depend upon some element of commonality. Plan-driven thinking is largely centred upon commonality of architectural form (essentially implementation). If instead we seek to share ideas based upon commonality of function, then this leads to the next development in design knowledge transfer.

During the 1990s, as the use of objects and object-oriented thinking became widely used, there was much greater interest in the idea of reuse of design knowledge at different levels. Adelson & Soloway (1985) observed that designers reused past experiences (what they termed ‘labels for plans’) and as the corpus of software applications grew, the patterns concept emerged as a way of codifying useful forms. Whereas a designer was previously expected to begin with a ‘clean sheet’, basing key elements of a design upon well-established structures or ‘patterns’ made it possible to draw upon the experiences of others. Again, we look more fully at the ideas involved in Chapter 15, but one thing we can observe here is that one of the strengths of this approach is that it gives greater attention to changeability.

The adage that “there is no such thing as a free lunch” does certainly hold true for when designing software. While plan-driven forms may produce design solutions that are relatively constrained and prescriptive, they do provide quite a lot of detailed guidance about how to develop them. Later (and ‘lighter’) approaches may help produce more adaptable and innovative designs, but their use does require greater creativity and skill, as we will see.

12.3Designing with others

While the preceding sections (and chapters) have tended to emphasise the activities of ‘the designer’, software is something that we often develop in collaboration with others. Given the nature of the design process, it is hardly surprising that designing software as a team activity adds further complications, some of them in the form of added constraints upon the form of the process, and hence also upon the form of the resulting product. (Of course, one of the hazards of team design is that the number of major components of a system will end up being the same as the number of people in the design team!)

In (Brooks 1987), Fred Brooks Jr. observed that, at that point in time, many of the very successful and ‘exciting’ software systems had been the work of one or a few ‘great designers’. His examples may now look a little dated (and systems have got much larger, making it less likely for them to be the work of one great designer), but we might still usefully compare the ‘clean’ design of Unix and its derivatives with operating systems such as the Windows family.

Given that great designers are rarely available to help us, and that the task of design may have to be shared in some way, we need to consider two issues when designing as a team.

How to organise the tasks of the design task among the team, and how to determine how the different elements will interact.
How to integrate the individual contributions to the design, which may well require a process of negotiation between the members of the team to resolve different expectations about the overall design.

The first of these can be managed, at least in part, by ensuring that a suitably modular view is taken of the overall design structure. Both will also be influenced by the form of the design team, whether this be a hierarchical one (with someone being the ‘chief designer’) or a more collaborative one, as typified by Gerald Weinberg's concept of egoless programming (Weinberg 1971). The chief programmer model, proposed by Harlan Mills (1971) has the disadvantage of requiring someone with clear authority and knowledge as team leader, so not surprisingly, given the difficulty of finding such people, current ideas such as agile development have tended to put more emphasis upon collaboration and coordination among a set of peers forming a design team.

Researchers have also sought to understand and model the psychological factors that influence team behaviour when designing software (Curtis et al. 1988, Curtis & Walz 1990). Relevant factors they have identified include:

the size of a team—with some indication that a size of 10-12 members is probably an upper limit for productive working;
the large impact that may be exerted by a small subset of the team who possess superior application domain knowledge;
the influence of the way that an organisation operates—with particular emphasis upon the need to maintain a bridge between the developers and the customer (Bano & Zowghi 2015).

So, operating as a design team requires the use of knowledge schema drawn from the domain of group psychology as well as technical knowledge about software design. This is perhaps best summarised by the following quotation from Curtis & Walz (1990).

We will particularly return to this when we examine agile development ideas in Chapter 14.

12.4Empirical knowledge about design creation

Since much of the material of this chapter refers to empirical knowledge about how people and teams create designs, many of the relevant empirical studies have already been discussed.

One topic not really elaborated though is that of opportunistic design activities. There are of course good reasons for this. If a process is opportunistic, then its form is determined by the problem, and the way that designers begin to understand both the problem and ways in which it might be addressed. That doesn't offer much in the way of guidelines about how to employ an opportunistic strategy of course, as it doesn't lend itself to being generalised. (Unfortunately, this doesn't therefore fit well with van Aken's idea of design sciences that we encountered in Chapter 5, where the goal was to “develop valid and reliable knowledge to be used in designing solutions to problems” (van Aken 2004).)

To highlight this, we briefly examine some outcomes from the study of software design that formed the basis of the book edited by van der Hoek & Petre (2014). This study was based upon recorded observations of a number of pairs of software designers who were addressing a particular problem (creating a simulation package for city traffic management planning). The plot shown in 12.1 is based upon the qualitative analysis reported in Budgen (2014), and was obtained by analysing the recorded utterances of one of the teams of designers and interpreting these to determine which viewpoint was the focus of attention at that point. In all, the utterances of three teams were analysed, and all were very different.

Figure 12.1: Example of viewpoint changes during design

The main point is that over a period of nearly two hours, as the two designers in the team produced their design model, their modelling process ranged widely between all four viewpoints, as well as spending some time revisiting the original requirements specification. This was very much an opportunistic design session, with the designers exploring different aspects as their understanding unfolded, and helps to illustrate why such a process cannot readily be generalised.

Key take-home points about structuring ‘solutions’ to ISPs

Software design practices employ a range of strategies for addressing the ill-structured nature of software applications and need to cope with some important factors.

Challenging properties of software. In 1987 Fred Brooks Jr. identified four key challenges for the designer as arising from the complexity of software, the need for conformity with other elements of a system, the constant need for change and the invisibility of software. Complexity and invisibility are largely intrinsic and have remained unchanged, but conformity and changeability are probably much bigger influences (and constraints) than they were at the time that Brooks identified their effects upon the design process.
Knowledge transfer. Mechanisms for sharing knowledge about ‘what works’ in particular situations have evolved over the years and their evolution is quite closely linked to the evolution of software architectural styles. While plan-driven forms were (and still are) practicable for applications employing call-and-return and similar architectural styles, later forms of architecture require more complex design models and hence more complex processes are involved in developing these.
Design teams. Software is increasingly developed collaboratively within a team, and so the design process needs to provide opportunities for interaction and coordination among team members.