Architecture Conformance, Testing, Reconstruction, and Trade-off Analysis

This session covers four connected duties of a software architect: keeping a built system loyal to its agreed architecture, shaping the architecture so testing works, recovering the architecture of an old system, and evaluating architecture choices against business goals with stakeholders in the room. Each duty builds on the one before it.

8.1 Ensuring Conformance to Architecture

Conformance means the built system matches the agreed architecture — the one that was designed, committed, and signed off by the stakeholders. Two phrases from practice make this precise: the as-designed architecture is the architecture on paper, and the as-built architecture is the architecture the running code actually has. Conformance is the state where the two coincide.

Making sure the implementation stays loyal to the signed-off design is one of the most important roles of an architect. It is also a duty of the whole organization and of the development process that produces the application — not a private hobby of one person. An architecture that exists only on paper, while the code quietly walks away from it, protects nobody.

Why does this matter so much? Because the architecture is what carries the quality attribute promises — performance, security, availability, modifiability. Those promises live in the structure: which layer may talk to which, who owns the data, how components exchange messages. When the code departs from the structure, the promises quietly lapse even though every feature still seems to work. The reference text puts a number on how easily this happens: in one experimental study, students given UML designs for simple systems violated those designs over 70 percent of the time. Drift is not an exception; it is the default unless someone actively guards against it.

8.1.1 Architectural Drift

Architectural drift (also called architecture erosion) is what happens when the code slowly moves away from the agreed design. It rarely starts with a big decision. It starts with small overrides that each look harmless.

How drift actually begins: a war story. In earlier project work, weekly design meetings were held with the client. The client was represented by one key person who could take decisions on behalf of the whole client side. In each meeting, the developers and that representative froze certain parameters and wrote them down, and everybody understood the agreement. The very next day, the client would call an emergency meeting — one the architect was not invited to — and instruct the team that changes had to be made. The development team would then report back: the client said this, the client takes responsibility for those changes, do not bother. And sometimes you accept such things, because after all, the client pays the money. One month down the line, those very decisions created development hiccups. They did not let the project run, and they did not allow conformance. When that happened, the same people said the team should have taken care of it.

The professor's image for this trap is buttering bread. You wanted to butter the bread on both sides. If you buttered this side, you could not butter the other side. We agreed to butter this side, and the moment I left the room, you got the other side buttered — and now this side can never be buttered. The two choices were mutually exclusive; taking both breaks the project. Where the analogy bites: a frozen architectural decision usually closes a door. Accepting an override that reopens that door does not give you both options — it gives you a contradiction that the structure cannot carry.

A simpler technical example: bypassing the data access layer. Suppose you access data through a data access layer — a layer created just for talking to the database, sitting under the business layer. In less streamlined projects, especially informal setups of years past, a developer would make a quick direct connection to the database, connect up, and serve something on the screen. The purpose was solved, but this is basically architectural drift: the team had decided on a layered architecture, and the developer bypassed the layer, reaching past it to touch the layer below. That bypass is a type of violation of the architecture. Nothing crashed. The screen showed the data. But the layer boundary — the thing that made the database swappable and the business logic testable — was silently punctured.

The same pattern in publish-subscribe. A publish-subscribe model assures that all user interfaces get access to the same data, either at the same time or at least get a notification at the same time. Any change in the model information is published. Normally you have a publisher which holds a model that maintains the data, and subscribers fetch data from that model through the publisher. Drift sneaks in when someone decides that certain information can appear directly in a view without being updated in the model. It also sneaks in when a method for logging information just bypasses the publish path and writes somewhere on its own. These are typical examples of where people drift from architecture without ever announcing a decision to change it.

Notice the common shape of all three stories: each individual act looked locally reasonable — the client was urgent, the developer was fast, the logger was convenient. Drift is an emergent property of many locally-reasonable acts, which is exactly why it cannot be prevented by good intentions alone. The reference text identifies three recurring routes into drift: no constraints are imposed on coders at all; the architecture is abandoned under technical or schedule pressure; or, most commonly, after deployment the system keeps changing through code-only edits while the published architecture is never updated to guide or record them.

8.1.2 Techniques That Keep Code Conformant

Several techniques guard the design against drift, and they work best together:

1. Embedding design concepts in the code. The code itself makes it plain what design it belongs to. If the architecture says "layers," the code should visibly contain layers — not have its structure scattered across unrelated files.
2. Using frameworks. A framework is a half-built skeleton; everybody codes inside it, so the structure repeats itself correctly. You fill the slots the framework provides; the architectural interaction pattern comes for free.
3. Using code templates. A template shows exactly how the code will be put in, and every new piece follows it. Once the template is debugged, entire classes of errors disappear across the system, because nobody re-invents the tricky part.
4. Updating architecture documents. The document states the architectural intention, a reference to it is maintained, and any work done should conform with it. Mark outdated sections explicitly rather than letting the whole document rot into disbelief.
5. Architecturally evident coding style. Every piece of code carries a reference to where it belongs and which layer it belongs to. Naming conventions, package declarations, and header comments make the architecture visible inside each file.
6. Educating team members about the architecture, so they can recognize a violation before they commit one. A developer who understands why the data access layer exists will not route around it at 5 p.m. on a Friday.
7. Code review. Some teams work with a peer review process where two people work as partners, and each reviews the code written by the other. A second pair of eyes catches violations the author cannot see.
8. Code separation through folders. You create folders or packages while developing, one per component in the architecture, so conformance is enforced by where the code is allowed to live. If there is no folder for "talking straight to the database from the UI," that code has nowhere legitimate to go.

Architecturally evident coding deserves a closer look. If you use a publish-subscribe pattern, you decide up front what components will be coded in the publisher and what will be kept in the subscriber. If you use a message queue to help with buffering between components, you spell out — in the code and its placement — what activity is being conducted at the end of the queue. Someone reading the code can then see the architecture instead of guessing at it.

A useful way to organize these eight techniques is by when they act. Techniques 1–3 and 8 act before or during coding: they make the correct thing the easy thing. Techniques 4–6 act as knowledge: they make the correct thing the known thing. Technique 7 acts after coding: it catches whatever slipped through. Strong teams run several at once, because each covers another's blind spots. The reference text adds a process-level rule that binds them: mandate that changes to the system, whenever they occur, are vetted through the architecture first — synced at life-cycle milestones, at check-in time via automated rules, or (worst case) at crisis.

Real-world: teams in industry also share their own practices in sessions like this — experience sharing around conformance techniques is normal professional behavior, and hearing how other shops enforce layers and reviews is useful information for everyone.

8.1.3 Frameworks and the MVC Pattern

Frameworks deserve their own section because they are the strongest practical tool for conformity. A framework is a reusable set of libraries or classes forming a half-built application: it fixes the skeleton — the control flow, the component roles, the interaction protocol — and leaves slots for you to fill. Because the framework already embodies the architecture, you cannot easily code outside it; conformity stops being a review question and becomes a compile-time fact.

The family mentioned in the lecture:

• Spring — the standard choice for Java enterprise applications; it wires components together and enforces separation between business logic, data access, and presentation.
• UI frameworks — used at some stage for the user interface; they dictate how screens, events, and widgets relate.
• Hibernate — handles object-relational mapping (mapping objects in code to rows in a relational database), so data access goes through one disciplined place instead of ad-hoc SQL scattered everywhere.
• AUTOSAR (AUTomotive Open System ARchitecture) — a framework for automotive software, jointly developed by automobile manufacturers, suppliers, and tool developers. It shows the idea at its strongest: an entire industry agrees on one framework so that every supplier's code fits the same architecture.

Frameworks like these assure conformity because the framework already embodies the architecture: the structure repeats itself correctly in everybody's code.

MVC: the model-view-controller pattern. MVC is described here as a more mature realization of the observer/publisher idea from the previous subsection. Walk through its moving parts:

• The model maintains the data. Think of it as maybe a database plus the rules around it.
• The controller updates the model. Any user action is conveyed to the model through the controller.
• The view is the user interface. Look at the view as the screen the user sees.
• The model has a notification system which may not go through the controller — it may go directly to all the views. The model updates the views, informing them there is an update in the data.
• The view informs the controller when the user does something, and the controller passes that intent on to the model.

So the cycle is: user acts on the view → view informs the controller → controller updates the model → model publishes the change → views refresh. The controller sits as the first level of code behind the user interface, interacting with models and views.

Picture the three parts as a triangle. Along one edge, requests travel upward: view → controller → model. Along the other edge, notifications travel back down: model → views. No edge runs directly from user actions to the model, and none runs from the model back to the controller — that asymmetry is the pattern. It guarantees one data truth (the model) with many possible faces (views), and it explains why MVC is the framework behind countless web applications: the model can serve a browser view and a mobile view from the same data.

Beyond MVC, the family of frameworks mentioned includes publish-subscribe frameworks, GMF (the Eclipse Graphical Modeling Framework, which generates graphical editors from models), workflow frameworks, Salesforce, rule engines, and logging frameworks. All of them are types of templates: they indicate how the code will be put in, which keeps a hundred developers' code shaped like one architect's design.

8.1.4 Agile, Documentation, and the Architect's Judgment

Agile projects give regular opportunities to meet and realign, and that cadence matters for conformance. But agile has to be handled very delicately. Agile talks about listening to the user and being willing to change; it asks for less documentation and more usable code. These are not black-and-white rules — it is a question of varying the level. All agile really says is: do not be so obsessed with documentation that you forget the end product you are worried about. Concern yourself first with the end product, then with having a usable product and documentation around it.

To feel why, hear how documentation obsession worked in an earlier generation. For a financial accounting system, the documentation could run to seven or eight printed volumes with all sorts of details. A single change in code somewhere made documents redundant. There was a document controller who had to keep track of the documents, version updates were issued, and it was impossible for anybody to know where things stood. So "just right" documentation means: having just the right amount of documentation so everybody can be conscious of the design, plus just what is required to assure conformance. Nothing more, nothing less.

Agile also talks about early realization of architecture. Somebody objects: if you have to change the architecture because you are agile, it was poor architecture. That objection hides a deeper truth about the role. Architects have experience and knowledge, and on that basis they can tell what will be required before people even dream about it. In the other world you have fortune tellers; in IT you have architects. Fortune tellers have their own mechanisms — sometimes confidential, sometimes above our heads — but architects have something better: pattern knowledge from many systems. That is what makes the Steve Jobs of the world special: they know what you want before you even thought you wanted it. They delivered before you were hungry, and afterwards you wondered how life went on without those solutions. Great architecture means listening to people and to requirements, but while listening, hearing between the lines — catching needs even the speaker did not realize they had said.

And once you have decided, convince people. Take the stakeholders on board with you. You are not a dictator as an architect; you are working for a client and getting paid, so in various occasions you are the servant. But what a great servant — he or she knows what the master wants and delivers it. You have to understand agility well, understand its context in architecture, and understand what conformance means and how to ensure it.

8.2 Testing and Architecture

8.2.1 Testability Versus Testing

Keep two words separate. Testability is that ingredient of an architecture which makes it amenable to testing — a property you design in, like strength in a bridge. Testing is the activity that checks the finished application against expectations — the act of walking onto the bridge with load trucks. One is a designed-in property; the other is an activity performed later. A system can be tested heavily and still be poorly testable: every check fights the structure.

Earlier material covered testability: you must know how to achieve quality attributes using tactics. You must also know the management decisions and the checklists given at the end of each set of slides on quality attributes. Those checklists list the actions an architect takes to reach certain quality attributes — testability included.

Does architecture have a role in testing too? Yes, twice over:

1. Conformance testing. Once the architecture has been fixed, you should be able to test whether your application conforms to that architecture — this closes the loop with Section 8.1: drift is caught by checking the as-built against the as-designed.
2. Advance test design. A lot of tests can be developed and designed in advance once the architecture has been declared — the shape of the system tells you what needs checking before a single test is written.

The second point deserves unpacking, because it is where architecture saves real money. The reference text maps testing levels onto architectural views directly. Unit testing checks individual pieces in isolation; the units themselves are architectural elements from the module views, so the architecture defines what the units are and what responsibilities each carries. Integration testing checks that separately built pieces work together; the planned increments come from the architecture's uses view, and the interfaces between elements — part of the architecture — determine the integration tests. Acceptance testing, done by users under realistic loads and attacks, stresses quality attribute behavior; to bring down a house efficiently, consult the blueprint for which wall holds up the roof rather than swinging a sledgehammer at random walls. Finally, risk-based testing concentrates effort where risk is highest — and architecturally significant requirements are its natural candidates, because if an ASR fails, the system is unacceptable by definition.

8.2.2 Architecturally Significant Requirements Drive Tests

The architecture was developed based on the ASRs — the architecturally significant requirements, the requirements strong enough to shape structural decisions (you have worked with ASRs in the assignment). The logic is symmetrical: if ASRs are important enough to develop the architecture, they are also important for developing your tests. Remember that the client or the user will definitely look to make sure every ASR has been properly delivered. The tester is going to be ready — it could be a developer doing the testing, or dedicated testers, depending on your organization structure and how you handle code. So the architect has to make sure the application allows testability for the architecturally significant requirements. Enough room must be designed in.

Think of it as a chain of custody for promises. Each ASR is a promise made to a stakeholder. The architecture is the plan for keeping every promise. The tests are the proof that the promise was kept. If the architecture never left room to observe the relevant behavior — no way to inject load, no way to watch internal state, no seam where a component can be swapped — then the promise can be neither proven nor disproven, and the argument ends in opinion.

The utility tree connects here. In the utility tree you mark high, medium, low twice for every scenario: once for priority to the organization (business value) and once for ease or difficulty of implementation in the architecture (architectural realization). Both ratings matter, and tests should target the scenarios that score high on value — especially those that are hard to realize, because that is where the architecture is most likely to disappoint. A scenario marked (High, High) — vital to the business and difficult for the architecture — is precisely where test effort buys the most protection.

8.2.3 Designing the Architecture for Testability

The architecture tells us how various subsystems interact with each other, what subsystems exist, and what the interdependencies between subsystems are. That knowledge helps you identify the test cases, identify the needs of every module, and decide what type of test each part needs so they work harmoniously together.

Concrete design moves that make testing possible:

• Modularize so data swaps cleanly. Make it easy to plug in production data and plug out test data. A persistence boundary lets the whole database be replaced by a small fixture — or even an in-memory stand-in — without touching business logic.
• Plan rollback. Have mechanisms to roll back changes made while running test cases. You might like test cases running live, recording results, and then rolling everything back to a clean state, so yesterday's experiment never contaminates today's run.
• Design components to be plugged in, plugged out, or replaced. Subsystems are independently developed — by different developers, even different organizations — and some places keep alternatives which can be hot-swapped.
• Test replacements before trusting them. Testing should provide a facility to try any new component you plan to substitute, to find out whether it is a complete fit for your system.
• Simulate missing parts. You should be able to simulate components and use the simulations in your architecture. Then even before a component has been developed, the overall system can be tested to make sure it will conform to the specifications. This is the same idea as a wind tunnel: test the airplane model before the airplane exists.

All of these are achieved by having proper test cases designed against the architecture, not improvised afterwards.

8.3 Architectural Reconstruction

8.3.1 What Reconstruction Is and Is Not

Architectural reconstruction is determining the architecture of an existing system. Full stop. You have a program that is running; as a user you see one box and a screen — you type something, you get a response. What are the submodules? How are they distributed? What type of communication is taking place at the back? None of that is visible from outside. Finding it out means determining three things:

1. The components of the existing system (the module view),
2. The component-and-connectors picture of how components interact at runtime,
3. The allocation structure — which subsystem runs on which server, where the file systems live.

That triple is the deliverable of reconstruction. Anything else — migration plans, rebuilds, technology upgrades — comes later, if at all.

Three truths anchor the topic:

1. No documentation does not mean no architecture.
2. Bad architecture does not mean no architecture.
3. Every system — good, bad, ugly, documented, undocumented — has an architecture.

Why care? If you want to migrate from an old technology to a new one, it is like stepping up: you have got to have your step firmly grounded down there first. You must understand how the old system went about what it did, understand its components and functionalities, and only then can you move forward and redesign. There are also lots of components which are quite invisible as far as functionality is concerned: security pieces, performance-related tools, testability-related activity, statutory compliance in the form of logs. A reconstruction that only maps the visible features misses exactly these — and these invisible parts are often the ones regulators and auditors ask about.

One monolith note: a monolith is also an architecture. "Monolithic" is not the absence of architecture; it is one particular style whose internal components still need discovering.

8.3.2 Experience Reports from Industry

Hands rose when the session polled who had reconstructed the architecture of a legacy system — several people had. Four engineers shared how they uncovered old architectures, and each story teaches a different technique. Read them as four points on one scale: from no documentation at all to too much documentation.

Report 1 — the AI assistant rebuilt from code alone. A digital-level product: an AI-based assistant helping people automate parts of their job. Built around two years earlier, it was in pathetic shape — slow, not responding fast, unable to scale. A new team took over with a mandate to restructure completely. They studied pain points across user experience and the entire architecture itself, redesigned the NLU (natural language understanding, the part that interprets what the user meant) interaction, changed how components interact, and moved to a microservices architecture.

Technique to extract: static reading plus dynamic tracing. The code told them what could happen; the network traffic told them what actually happens at runtime. Neither alone suffices — code hides runtime binding, traces miss never-executed paths.

The assumption behind every reconstruction effort is exactly this: not enough is known about the legacy architecture. In spite of all documentation, what you need to know you don't know — and unless you know the legacy system, you cannot decide what changes to make to it.

Report 2 — the telecom planning suite going web. A telecom-domain network planning product had two application types. One covers planning: it simulates plans and does the heavy lifting of finding sites and link engineering against hardware. The other is network control and management, which talks to real devices out in the field. The desktop application was WPF-based on .NET (not .NET Core) — WPF being the Windows desktop UI framework, which is why the coupling mattered. The scene is changing — everyone is moving to software-as-a-service web applications. But the UX component was tightly coupled to the OS part. Before moving to the web, the team had to migrate to .NET Core and remove the tight WPF dependency, so the reusable logic could be extracted for the web application.

Technique to extract: interview the humans, then mine the lightweight records. Even thin wiki pages naming key decisions beat nothing, because they tell you which questions to ask next.

Report 3 — the Python-only monolith. A six-year-old system with reasonable documentation available, though in some cases the code still had to be read to understand how it was architected. The main driver for reconstruction: it was a monolithic structure where you could only add components in Python. Adding a component in a different language was impossible. Add six years of technology change, and a move to newer technology became necessary.

Then came the correction that this whole section exists for. The student began describing the technology migration and application rebuild plans, and the reply drew a hard line: we are talking about architectural reconstruction, not application reconstruction. On previous occasions, answer papers showed everybody talking about reconstruction of the application — that is the classic error. Architectural reconstruction has got nothing to do with reconstructing the system itself. We may or may not change the system afterwards; first we want to know the architecture. One of the main purposes of doing reconstruction is indeed to rebuild the application eventually — that is why we do it half the time — but what reconstruction is is determining the architecture, not the rebuilding.

Keep the boundary crisp: reconstruction ends where the as-built picture is complete and validated. Rebuilding, migrating, re-platforming — those are separate projects that may consume the picture as their starting point.

Report 4 — the banking payment processor. A payment processing product in the banking industry, twenty years old, technology outdated, and an acquired product rather than one built in-house. Documentation existed — a lot of it, so large that reading and understanding it would have taken six months. Around 600 services and screens needed migration from old technology to new. The procedure:

1. Identify the screens and services.
2. Write down the basic functionality of each service or screen.
3. The original creators had retired, so ask the latest members who worked on those screens and services what specifics they could answer, and correct each document accordingly.
4. Create a basic product (a prototype of the new system).
5. Run many hundreds of non-regression tests: execute the same case on the previous system and the same case on the new system. Identify the delta — the difference — and go back into the old C/C++-based code to find what was recorded there.
6. Bring that behavior up into the new Java-based code.

Total duration: one and a half years. Notice the loop: compare old and new on identical inputs, measure the delta, trace the delta back to old-code behavior, port the behavior, repeat until deltas vanish. Each delta is a place where the documentation lied or stayed silent — the old system's true behavior, recovered empirically.

The synthesis after the four stories: in some cases some documentation existed, in others none — and remember, no documentation does not mean no architecture. The banking deltas and non-regression comparisons between the running system and the test bench carry forward into agile too: the gap between the existing system and what is being developed shrinks gradually. They develop a prototype, and gradually, when the delta between the prototype and the running system reduces to zero, the prototype evolves into the new application. You identify components, create abstractions for the components, and build forward from the abstractions.

8.3.3 Where Architectural Information Hides

If you have running source code available, you are really lucky. Some experts can do a quick read of source code and tell you important aspects about the application. Beyond reading, several evidence sources feed reconstruction:

• Executable traces. Run the system with instrumentation and watch the processes interact — enough to draw a component-and-connector diagram from observed behavior rather than guessed behavior.
• Build scripts. Often still available, because every modification reuses the same build script to produce the deployment module. From them you learn what classes exist, what files are used, the dependency relationships between files, where the data lives, and what gets compiled into the shipped product.
• Files. The include relationships and contents give an idea of the functionalities in use.
• Variables. Give an idea of the data structures — who reads them, who writes them, whether a global data store exists or data flows through calls.
• Directories. Subdirectories suggest packages that were developed, or cohesive modules used together — they normally sit in one directory.
• File names. Give the type of functionalities that were intended.
• Functions. Knowing read/write access to variables and the call relationships lets you plug each function into a module.

Each piece of evidence moves your knowledge along a three-color scale. Black means nothing known. Gray means gray areas — partial knowledge. White means the architecture is completely transparent to you. Reconstruction is the disciplined walk from black to white. No single artifact turns the whole board white: traces whiten the runtime picture, build scripts whiten the module picture, and interviews whiten the intent behind both. Think of the opening image of the classic reference chapter — several blind scholars each touching one part of an elephant: the one at the side says "wall," the one at the tusk says "spear," the one at the tail says "rope." Each was partly right and all were wrong, because none fused their views. Reconstruction is the process of fusing the touches into the elephant.

For the record, the "4+1" name refers to Philippe Kruchten's view model: four views — logical, process, development, and physical — plus a fifth unifying view made of scenarios and use cases. It is a standard place to see how module, component-and-connector, and allocation-style pictures relate to one another.

Real-world: anyone in IT who is unaware of UML should look up the site UML Diagrams.org and spend time with good recorded UML teaching material. UML itself is taught deeply in separate object-oriented analysis and design courses, not in this one.

8.3.4 Tools and the Reconstruction Process

Tool support exists, and a published case study shows the full pipeline. The reconstruction environment discussed comes from the Software Engineering Institute (SEI) at Carnegie Mellon, whose workbench for this purpose is known as Dali. The case study also mentions Armin, the SEI's second reconstruction workbench — a complete rewrite and rethink of Dali. Both focus primarily on the module structures of an architecture. The case study diagrams are worth studying alongside the shared PDF.

Understand the goals, the process inputs, the techniques used, and the outputs, and then iterate deeper. The process has four repeating phases:

1. Raw view extraction. Pull facts out of code, execution traces, and build scripts. Parsers and abstract-syntax-tree analyzers read the code structurally; lexical analyzers match patterns; profilers and instrumentation capture what happens while the system runs.
2. Database construction. Convert the extraction results into a standard form — different tools each emit their own output format — and load them into a reconstruction database, so the facts can be queried and combined.
3. View fusion. Fuse the various views — the component-and-connector view and the module views — and see the fused views in relation to each other. Fusion fixes individual blindness: a static call view misses late-bound calls, a dynamic trace misses rarely-executed paths, and together they cover each other's gaps. This is also where human interpretation enters: an expert decides that a cluster of elements should be aggregated into a layer.
4. Present the architecture, then iterate again, going around the loop until the picture stabilizes. Fused views are hypotheses; analysis tests them, disproven hypotheses send you back to extraction.

One reported scenario breakdown lists execution points, debugging output, grouped extraction, and view fusion performed using CodeSonar and Sonar. Violations of the architecture were determined from the tool output — you declare the intended layering and constraints to the tool, and it flags every dependency that breaks them. The reference text shows exactly this violation-finding role with tools of the SonarJ class: define layers and allowed-to-use relations, then let the tool search its database for illegal arcs.

On cost: extraction is cheap to repeat once set up, but the tools carry a learning curve, no single tool speaks every language, and interpretation needs people who know the system. Budget for iteration — the first fused view is almost always partly wrong.

8.4 Architecture Trade-off Analysis Method

8.4.1 Why Evaluate and Business Goals First

This is the most important part of the session from the viewpoint of architecture. People senior in organizations — solution architects and similar roles — may have used this method or a variation; what we study are the fundamentals, not exact implementations. Two methods matter: the Architecture Trade-off Analysis Method (ATAM) and lightweight architecture evaluation, which is nothing but ATAM in a shortcut format — the full process, compressed.

Recall the background from the previous session on designer, peer, and outside reviews. Evaluation usually takes one of these three forms:

1. Evaluation by the designer within the design process — every time a key design decision is made, its alternatives are tested against analysis.
2. Evaluation by peers within the design process — reviewers fix a set of quality attribute scenarios that drive the review; the architect presents the portion of the architecture to be evaluated; for each scenario, the designer walks through the architecture and explains how the scenario is satisfied; potential problems are captured for follow-up.
3. Analysis by outsiders once the architecture has been designed — outside reviewers give a more objective view. The outsider has an objective eye and is often chosen for specialized knowledge the organization lacks internally. Problems uncovered by outsiders are very often taken more seriously: all the stakeholders listen carefully, because an expensive independent team has no hidden agenda.

Right at the beginning of any architectural evaluation, the most important factor is understanding the business goals.

The chain of logic runs: the architecture drives quality attributes; quality attributes make the system good; but who decides which quality attributes you need to satisfy? Not the architect. The business does. That is why every serious evaluation method starts by forcing the business goals into the open before anyone debates a single design choice.

8.4.2 Prioritizing Quality Attributes: The Coupon Story

Ask a stakeholder what they want, and with no cost attached they want everything — tea, coffee, and cold drink, all three, whichever one they happen to like at the moment. Why should they have a preference? Now change the game: hand the stakeholder a coupon that buys either a tea, a coffee, or a cold drink — just one. The stakeholder suddenly starts thinking, because the coupon forces a choice. Bring in three different stakeholders, give each the same coupon: one takes tea, one takes coffee, one takes fruit juice. Who is right?

As a chef manufacturing one item for everybody, which do you make? Tea, coffee, or crushed-fruit juice? Maybe you deliver differently to different people — but here we come out with a software solution and an architecture which has got to be one-piece-fits-all. Unless it is important enough to build separate solutions. Then the client pays for each solution: one high on performance, one high on availability, the third high on usability. Different audiences get served by different applications, maybe over the same back end.

Either way, there is financial priority involved. The organization has one kitty. It wants a mix of quality attributes, and it has got to determine how much of which, which gets priority, and what it will cost. Set cost aside for the moment and talk priorities. Priorities need evidence: somebody must be able to document the quality requirement properly. There have to be people capable of presenting a scenario — drawing a nice scenario that says: this is the source of stimulus, this is the stimulus, this is the artifact, this is the environment, this is the response, and this is how we will measure it. The measurement is going to cost money — money comes later. This commitment is achievable; that one is possible but may not be achievable. Ranking attributes honestly is the heart of the method.

The coupon works because scarcity reveals preference. Unlimited wish lists produce unlimited architectures; a forced choice produces a rank order that an architecture can actually be designed around.

8.4.3 Stakeholders You Must Include

Stakeholders come in many types, and all types must be present:

• actual users of the system;
• the network team;
• the team which develops the product and the team which supports it;
• people concerned with compliance with institutes and regulators;
• somebody in the organization concerned about the amount of money spent and the value realized;
• the marketing team, who may have to sell the product;
• a team interested in enhancing the business value of the company.

You cannot leave out any important stakeholder. A cautionary tale shows why: in one organization, all stakeholders were taken into account except the people who would actually operate the system. In those days usability, user experience, and user interface took a backseat — today usability is the number one quality attribute, with specialists devoted to interface and experience design. Leave the operators out and the quality they care about silently drops off the list. Nobody voted it down; it simply had no voice in the room.

8.4.4 Roles in the Evaluation Team

An ATAM evaluation team fills specific roles (typically three to five people; one person may hold several roles):

• Evaluation lead. Must be able to talk to people on both sides: the development people, the operations people, the client, the developing organization. The lead operates at a level of seniority where people will listen, coordinates them, and fixes up meetings. Send a lightweight messenger and it will not happen.
• Evaluation technical authority. The technical authority on architecture, able to dig into the experience of the architects on the team or draw on their own, and knows how to present a scenario.
• Scenario scribe. Jots down the scenarios as stakeholders talk about what they need, captures the agreed wording of each one, halting discussion until the exact wording is written, and understands exactly what the person is talking about.
• Proceedings scribe. Takes raw notes of everything, which can be cross-checked later against the evaluation need and the scenario scribe's record.
• Questioners. In any big meeting you plant one or two people who are willing to open their mouth — most people are a silent audience, and questioners provoke the audience into discussing, raising issues of architectural interest tied to their own quality attribute expertise.

Alongside the evaluation team sit the project decision makers — people empowered to speak for the project and mandate changes (project manager, customer representative, and always the architect; a cardinal rule is that the architect must willingly participate) — and the architecture stakeholders, whose job is to articulate the quality attribute goals the architecture must meet. A rule of thumb: expect to enlist roughly 12 to 15 stakeholders for evaluating a large enterprise-critical architecture.

8.4.5 What ATAM Produces

At the end of an ATAM exercise you get a presentation of the various scenarios: all stakeholders got an opportunity to put up what they require. Everyone understands the business goals in full — why are we going in for the system, why are we enhancing it, exactly what is the company going to get out of it, in business terms. Business terms normally means better market share, better profitability, better employee satisfaction, customer satisfaction — however you quantify it. The quality attributes get prioritized with ranks 1, 2, 3, 4, 5, 6, 7, 8, and all the stakeholders sign that yes, we jointly agree this is the priority. Things not likely to happen, things that would put a spoke in the development process, things unclear as to whether they will work — identified as risks: architectural decisions that may lead to undesirable consequences given the stated quality requirements. Areas not worth bothering about are also identified as nonrisks — decisions examined and found safe — so you don't waste time on them. Once risky themes are listed, the evaluation team works to determine exactly what outcome can be expected.

The complete output list, in one place:

1. A concise presentation of the architecture (prepared for the exercise, survives afterward);
2. An articulation of the business goals — often captured in writing for the first time;
3. Prioritized quality attribute requirements expressed as scenarios;
4. A set of risks and nonrisks;
5. A set of risk themes — overarching systemic weaknesses that threaten the business goals;
6. A mapping of architectural decisions to the quality requirements they help or hinder — a written rationale for each decision;
7. Identified sensitivity points and trade-off points.

Real-world: a risky area today is the new payment gateway — new methods of identification and paying. Companies run research on the cost of doing the billing versus losses from inadequate security scrutiny at shopping-cart checkout. One popular system: all item masters have weights fed in. At checkout you keep placing items in the basket area one by one; the item gets scanned by itself, or you point its barcode at the box scanner. The screen shows the picture of the item and verifies the weight differential — the differential weight should tally with the weight of the product. So a barcode scan, a weight-differential check, and a visual display run together. One supervisor stands far away, between maybe ten machines. At a glimpse, if what appears on the screen doesn't tally with what went into the basket, the supervisor raises an alert. A lot of mischief goes on — people remove a label from one product and stick it on another. Similar products escape the weight difference check but come out on visual inspection. Identification tags add another layer: the tag comes off only when billing is allowed, done by the same machine that bills. Read it architecturally: three independent checks (scan, weight, visual) plus a human supervisor trade cost and speed for security — a textbook sensitivity point, since weakening any single check moves the fraud rate sharply.

After analysis come the architecture decisions. Which quality requirements are the priority? What architectural decisions will handle them? And how do you explain to stakeholders why each decision — based on experience, records, and past exposure — helps the quality requirement it targets?

8.4.6 The Steps of ATAM and the Utility Tree

The phases wrap around the steps: prepare the stakeholders, select the stakeholders, form the team, run the evaluation with the team, then the architect with their team comes to the solution, generates a report, and presents it to all the stakeholders. Sometimes the entire iteration has to be done twice. In the reference text's shape: phase 0 is partnership and preparation (logistics, stakeholder list, studying the documentation), phases 1 and 2 are the evaluation itself (phase 1 with decision makers, phase 2 with the wider stakeholders), and phase 3 is follow-up — producing and delivering the written report.

The steps within the evaluation:

1. Present ATAM itself.
2. Present the business drivers — the goals and priorities.
3. Present the architecture to the stakeholders, so they see what we plan to do and why we intend to do it.
4. Identify the architectural solutions, list them, and present them to the users.
5. Generate the utility tree.
6. Analyze the architectural approaches.
7. Report the results.

(The reference text splits this into nine steps by separating the stakeholder brainstorming and re-analysis of approaches after the utility tree; the sequence above is the same arc compressed.)

The utility tree you built for the assignment returns here, with one instruction: notice that scenarios are assigned a rank of importance — high, medium, low — and not only importance to the business but also high, medium, low as to degree of difficulty for realization architecturally. In brief: you have various quality attributes — performance, availability, security — and under each, a few scenarios told by the users. Each scenario carries a pair of marks, (importance, difficulty), like (High, Medium) or (Low, High). From the marked tree you fix priorities 1, 2, 3, 4, 5, 6. Sometimes this ranking conflicts with the high-medium-low marks — it doesn't matter; the ranking comes out of voting. Note the perspective shift: in the assignment utility tree you bothered about importance from the user's perspective. ATAM adds the organization's business weighting on top.

Picture the tree: roots are quality attributes (performance, availability, security), branches are refinements ("performance → latency under load"), leaves are concrete scenarios, and every leaf wears its two-letter badge — (H,H) leaves are where analysis time goes first.

8.4.7 Voting, Sensitivity Analysis, and Trade-offs

How do you fix priorities among many voices? Brainstorm: allow the stakeholders to make a presentation of what they feel is important, then put it to a vote. The arithmetic: if you have 100 votes and 20 people, give each person 5 votes and ask them to place their individual votes on individual scenarios. Count the votes each scenario received, sort the scenarios, and present the result to the stakeholders. Let them know the importance of the scenarios that got voted out — then hold another round of voting. After the presentation, people begin to realize importance differently, and the second round lands nearer the truth. Then analyze the approach: examine the decisions taken, how they handle the scenarios, and present the result to the stakeholders.

(For scale: the reference text's standard allocation is votes equal to 30 percent of the number of scenarios, rounded up — 40 scenarios give each stakeholder 12 votes, placeable anywhere. Same mechanism, different budget.)

Sensitivity analysis has a proverb attached: the more sugar you put, the sweeter the product is going to be. So how much sugar is right? A sensitivity point is an architectural decision to which a quality response is markedly sensitive — more security machinery, more safety; but how much is enough, and where does extra sugar stop helping? Once sensitivity is mapped, you arrive at the trade-off point: the sweet spot, the best place to start off. A trade-off is basically between various scenarios — the decision that helps one scenario hurts another, and the art is locating the balance point. The classic paired example from the reference text: the frequency of a component's heartbeat signal determines how fast a failure is detected — higher frequency improves availability but consumes processing time and bandwidth, hurting performance. One dial, two qualities pulling in opposite directions: that is a trade-off point.

Some results are intangible — user satisfaction is the classic case. Delphi methods help: brainstorming of stakeholders, getting them into open communication, then voting. As the architect you try to work out experiments which will help determine the satisfaction level of that quality attribute; the test may be done later, but meanwhile the stakeholders' voting determines which intangibles matter most.

8.4.8 Lightweight Architecture Evaluation

Lightweight architecture evaluation is the same animal on a short leash. It is ATAM in a shortcut format: all those points are there, but less time is given to everything. Four to six hours, the whole thing finished within a day, and you come out with the result the same day. In the compressed agenda, the business-driver and architecture presentations shrink to minutes, existing scenarios are reused rather than regenerated, the separate brainstorming step may be dropped because the assembled internal stakeholders contribute scenarios directly, and there is no formal final report — a scribe records the risks, nonrisks, sensitivities, and trade-offs for distribution. Use it when a full ATAM's calendar cost cannot be justified but the architecture question cannot wait. The honest limitation: an internal team is rarely as objective, so expect fewer new ideas and fewer dissenting opinions than an outside review would surface.

8.4.9 Real-world Evaluations: Train Announcements

A closing experience report shows stakeholder-driven trade-off in action. The product: public announcements in trains. The design goal: a wireless crew device — via a mobile device, a conductor in the train should be able to make a public announcement over the loudspeakers. Something existing already ran on real-time streaming protocols. The comparison had to respect a business constraint and performance reality: the conductor cannot make a live announcement through this device, because it results in echo in the loudspeakers. So the flow becomes record-then-play: he records the announcement, then plays it in the train. Multiple solutions were explored — using the real-time streaming protocol versus using FTP-style file transfer: since recording is mandatory anyway, transfer the recorded file instead of streaming live.

Notice how the story mirrors the method: a business goal (crew announcements without new wiring), competing architectural approaches (live streaming vs record-and-transfer), a decisive constraint discovered by involving the actual user (echo makes live announcement unusable), and a trade-off resolved on performance and bandwidth. Nothing here was decided by the architect alone — every input came from a stakeholder in the room.

The lesson drawn from the story: the importance of involving stakeholders, and the importance of doing an evaluation at all. With that, the portion for the midterm examination closes; the next session reviews and then begins the comprehensive-examination portion.

Exam Guidance Summary

• Scope: Modules 1 to 5 form the midterm portion; Modules 1 to 10 form the comprehensive exam. The midterm portion is included in the comprehensive exam — absolutely no doubt about it. The next module (Module 6) is not part of the midterm.
• Answer discipline: Understand the question carefully and explain the answer in the context of the question and the given case study. Do not lecture beyond what has been asked for. Brilliant people have landed up with zeros because their (correct) answer belonged to a different question than the one asked.
• Diagrams: UML-perfect diagrams are not expected in a software architecture course; perfect UML syntax is not required. Always give a legend for the items you refer to.
• Architectural reconstruction: An elaborate question on this appeared in recent years. Questions don't usually repeat, but nobody stops a question from repeating. The classic error seen in answer papers: writing about application reconstruction when the question asks for architectural reconstruction — determining the architecture of an existing system, not changing it.
• Tools: Awareness-level knowledge only — know that reconstruction tools exist and what they are capable of, not advanced usage.
• Utility tree: Expect to rate scenarios twice — importance (business/user perspective) and difficulty of architectural realization.
• ATAM: Flagged as extremely important. Know the phases, team roles, outputs, steps, utility tree handling, voting mechanics, sensitivity points, and trade-offs, plus lightweight architecture evaluation as the compressed variant.

Key Industry Applications

• Frameworks as enforced architecture: Spring (Java), UI frameworks, and Hibernate embody architectural structure in code, assuring conformity; MVC organizes model, view, and controller responsibilities in countless applications; AUTOSAR does the same for automotive software across an entire industry.
• Framework families: publish-subscribe frameworks, workflow frameworks, Salesforce, rule engines, and logging frameworks act as enforced templates for how code is put in.
• Lightweight decision records: Confluence pages serve as lightweight stores of architectural decisions and high-level diagrams in agile shops (undo service design, sockets programming).
• Legacy rescue and re-platforming: microservices and NLU redesign rescued an unscaleable AI assistant product; WPF/.NET desktop products migrate to .NET Core and web/SaaS by first removing OS-coupled UX dependencies.
• Banking modernization at scale: banking payment modernization inventoried around 600 services and screens and reconciled old C/C++ behavior into Java using hundreds of non-regression delta tests over eighteen months.
• Retail loss prevention: smart-checkout systems combine barcode scanning, weight-differential verification, visual display, distant human supervision, and bill-released security tags to fight label-swap fraud.
• Stakeholder-driven trade-off in transport: train crew announcement devices chose record-then-transfer over live streaming to avoid loudspeaker echo, trading bandwidth against latency with conductors consulted as real users.
• Tool-supported reconstruction: static-analysis tools of the Sonar/SonarJ class detect violations of the declared layering during reconstruction; the Software Engineering Institute's Dali workbench (and its rewrite, Armin) fuses extracted views into a validated architecture picture.