Modeling Activity

Computational Design and the Opportunity for AI Applications

Thomas J. McLeish · Berkeley MDes · November 4, 2026.

Modeling is central to design.

Modeling has always been part of design.

McLeish, Modeling Activity, p. 6.

Computational design moves the physical world into the virtual world,
and the virtual world into the physical.

The virtual world is where the full reach of computation becomes available: to simulate, analyze, optimize, generate and explore, at speeds and scales the physical world does not allow.

McLeish, Computational Design, Fig. 1, p. 4.

Central to computational design is the construction and use of models.

McLeish, Computational Design, p. 1.

A model is a representation,
of selected aspects of the world,
constructed from a perspective,
and for a purpose.

It can take many forms: a drawing, a diagram, a graph, a set of rules, an equation, a physical mechanism, a learned computational representation, and others.

Every model embodies assumptions about what matters and how the world might work.

Its selectivity is not a flaw to be eliminated. Deciding what to represent is central to the work of modeling.

The distinction between a model and its use is important.

A model may be used to explain, to generate, to recognize or to predict, and these uses can overlap. What a model is good for depends on the purpose it was made for.

The following case studies show different ways to make a hypothesis about activity operational in a computational system.

Musicolour embodied a model that encouraged improvisation.

The machine listened to the performer and responded with light. The machine would get ‘bored’.

Gordon Pask and Robin McKinnon-Wood, from about 1953. Pask 1963; Pask 1971.

The machine was designed to entrain the performer, and to couple the performer into the system.

Given a repetitive input, the system ‘directs its attention’ to the potentially novel. The performer could train it by accenting properties of the music.

Pask 1971, pp. 78–80.

Colloquy of Mobiles modeled social interactions through the mechanisms of autonomous agents.

Gordon Pask showed it at Cybernetic Serendipity, at the ICA in London, in 1968. He called it “a socially orientated reactive and adaptive environment”.

Female forms by Yolanda Sonnabend. Pask 1971, p. 88.

Colloquy of Mobiles is an agent-based model.

Multiple agents, each with local states and rules, interact to produce patterns at the level of a group. The term came later; Pask wrote of a “community of mobiles”.

Pask 1971, p. 88.

The resulting activity is generated by the model rather than retrieved from observations.

Behavior can be explored by changing initial conditions, the rules of response, or the surrounding environment. Such a model should be evaluated against the purposes and hypotheses for which it is used.

A model specifies a representation and its mechanisms; a simulation is an execution of that model under particular conditions.

Repeated simulations can reveal sensitivities, unexpected interactions, or differences between what the designer assumed and what the mechanism actually produces.

A coherent simulation is not, by itself, evidence that the model describes real social behavior.

It may be internally consistent while remaining an imaginative, partial, or deliberately abstract account of the world.

At most, Pask was modeling
a self-organizing system with abstract models
of human drives and communication.
Do we see that when we watch it?

The displays show the drives, and the lights and beams carry the signals. The documented four-hour studies do not show continued growth in organization under their measure.

Pask 1971, pp. 88–89. Simulation Studies, Realizing the Colloquy of Mobiles.

Negroponte explored a model of design activity as a basis for computational conversation.

His books explored how a computer might participate in architectural design rather than merely carry out drawing or calculation.

Negroponte 1970; Negroponte 1975.

The machine would need some account of what a designer was doing, not simply a vocabulary of commands and outputs.

Design is a process of exploring possibilities, making judgments, responding to constraints, revisiting decisions, and interpreting the work as it unfolds.

The computational system can embody a model of the activity in which it is meant to participate.

Negroponte hired Pask as a consultant to the Architecture Machine Group, and invited him to introduce the chapter on machine intelligence in Soft Architecture Machines.

Fig. 6 · Pask’s drawing of a conversation between a designer and a machine, from that introduction. Pangaro 2017.

Sensors integrated into the architecture
provided evidence about activity.

PlaceLab was an instrumented living environment in which ordinary domestic life could be observed over time.

Its sensors recorded movement, object use, environmental conditions, and sequences of events. Sensing associated with people and objects added to that.

Fig. 8 · PlaceLab, layer by layer. Drawing by Thomas J. McLeish. Munguia Tapia, Intille and Larson 2004 (PDF)

An observation is not an activity model,
but you need one to determine observations.

A sensor may report that a door opened, a cup moved, or someone entered a room, but none of those events alone establishes what the person was doing.

People are unpredictable, so it can pay to overinstrument for edge cases: overlapping streams, retained to support multiple models.

Fig. 9 · PlaceLab’s video, wireless accelerometer and plan viewers, August 20, 2004. MIT House_n and TIAX.

A model relates observations to an account of activity.

Recognition and prediction are possible uses of a model, not proof that its account of the activity is complete.

Iota developed specific behavioral models through experiments tied to real-world questions.

Iota Partners, later continued within SapientNitro, took this experimental sensibility into workplaces, products and practical business questions.

The work could resemble an experiment in physics.

Data does not arrive already labeled with the meaning of an activity.

The selection of an instrument, the definition of an event, the alignment of observations in time, and the testing of an inference are all choices that shape the model.

People, places and things provided several ways into the same problem: activity unfolding in relationships among them.

METIS connects concepts to proposed measurable components so that relationships can be investigated.

“A tool for defining measurable components of a high level, abstract concept.” A 2013 Iota methods document.

The Iota studies ran from 2010 to 2015, anchored on people, places and things.

Fig. 10 · The 15 studies on a timeline

Instrumented People, Places and Things

Contemporary vocabulary groups quite different computational representations under the idea of a digital counterpart.

Kritzinger et al. 2018; Digital Twin Consortium; NIST; Google DeepMind, Generative Ghosts.

These terms suggest different kinds of accountability.

What counterpart is represented, which variables are synchronized, how often they change, and what degree of fidelity is warranted?

A simulation of the mechanisms of Colloquy is not a live twin of the original installation.

The defining relationship to an ongoing, particular counterpart is different.

The interesting question is what hypothesis a representation embodies, what evidence supports it, and what it can legitimately be used to do.

It is not whether the representation appears lifelike.

Computational design relates the world, models,
possibilities and changed conditions.

Fig. 13 · A recurring process rather than a linear sequence with a final outcome. McLeish, Computational Design.

Language models, multimodal AI, context-aware computing and physical AI expand how applications can interpret evidence and produce responses.

An AI model’s general capability and an application’s model of the particular activity remain different things.

The application still has to settle what the activity is, which hypotheses it embodies, and what would count as a meaningful interpretation.

Context means the information and representations of a situation that inform an AI system’s completion, conversation, reasoning or action.

A multimodal stream can supply evidence, but more data is not necessarily more understanding.

Designing an AI application includes the model of the world it embodies, not only the selection of a powerful foundation model.

The apparent fluency of an AI system is not evidence that it has an adequate model of the situation.

A plausible account of why a meeting worked is not the same as evidence that people collaborated. Collaboration cannot be sensed directly: equal participation, for instance, had to become turn-taking in conversation.

Computational Design may offer an important contribution to AI application development by giving sustained attention to the modeling of activity and behavior.

How the model is conceived, how it relates to a hypothesis about the world, what observations or experiments support it, and how its operation shapes the application.

Modeling activity is one of the ways to make the hypotheses underlying AI applications explicit, examinable and consequential.

As designers, you have models, whether you articulate them or not.

Think about articulating them carefully, and about how your concept maps become computationally activated. They represent what you care about, what you don’t, and your values and ethics.

These questions can guide a model of your own:
a model as a hypothesis.

1. What are we trying to understand, change, or make possible, and why?
2. What hypotheses or propositions do we want to investigate?
3. What observations would support, refute, or help distinguish them?
4. How can models and computational methods help?
5. How would the findings inform design, decisions, or action?
6. What remains uncertain, or outside the model?

The appendix holds the kitchen’s results and the references.

The kitchen’s exploratory evaluation uses one participant’s 54 labeled episodes.

Instrumented Homes Explorable.

This talk rests on these sources.