I doubt that anyone who’s read my blogs even somewhat regularly will know I’m very interested in the notion of “digital twins” in their broadest sense. Largely because of this, and at the same time feeding my interest, I’ve gotten many comments from enterprises and even vendors who share my interest. Even some LinkedIn comments have contributed, and so I want to lay out what I think is going on in that “broadest sense”, and how it might relate (or should relate) to LLMs and AI overall.
A digital twin is a replica of something, obviously. That something could either be a real physical system, which we could call a “world model”, it might be a totally imaginary system like a computer game, or if might be a mixture of the two, an artificial framework that is “inhabited” by some real things or people. This last concept, to me, relates to Meta’s “metaverse”. The point is that a digital twin is a tool that generates output as a system, based on events that may be external or internal.
As a software architect, I’m naturally interested in the way that something like this could be created. To me, the solution has to lie in the fundamental rules that would define the system the digital twin represents. If the twin creates a fully imaginary framework to represent, then it has to set those rules completely. If it represents a real-world system, it has to be based on the rules of that system, and be synchronized with its state or it’s no longer a twin. The latter is clearly a sensor/event problem, but the former isn’t so clear-cut. We need to think about how those rules could be inferred.
Arguably, a collection of people is the most complex real-world system. Every element of the collection is autonomous, so do you have to model every aspect of human behavior for each element, and then every possible interaction? I submit you do not. A collection of random people isn’t a “system” in the sense of IT targeting; for that, you need some common link. Work, process automation, provides this sort of focus, and not surprisingly most digital twin work has been done in this space. I’ll look at the rest below, but I’ll start with process automation missions.
The nice thing about process automation twinning is that the process frames the IT structure. There are steps in almost every process, and the steps proceed in a sequence determined by conditions. Each step in the process is itself a kind of black-box micro-process, and the complexity of the process as a whole is hidden by the fact that the step complexity isn’t visible outside the step.
If you think about it, this mirrors the way biological organisms seem to work. We “see”, “hear”, “walk” and so forth, without conscious awareness of the details. Some of these things seem to be hard-wired, or at least assisted by some built-in functionality, and it’s fair to call these “instincts”. A digital twin used for process automation is, in a sense, a set of instincts linked in a way that’s similar functionally to how they’d be linked in a brain.
When you introduce human workers into this, you don’t change the high-level process much; the worker has a greater breadth of adaptation than a device would, and so can respond to a broader range of conditions, many of which won’t require a specific definition/action to be preplanned. “Place a broken item in the bin” relies on human ability to detect an abnormality that’s out of range, without cataloging in advance every property that makes it so.
Machine learning could, in theory, be used to “train” a micro-process implementation, providing that the properties that made something “broken” can be detected at the compute/ML level and reported to the model. What enterprises tell me is that when you try to replace workers in roles like “inspector”, you quickly find that it’s probably smart to rely on the AI/ML equivalent of eyesight, because what the worker is using is that sense. It’s not only more logical to assume the machine analog of eyesight is essential in training AI/ML to perform human-like functions, it’s certainly easier to visualize how you’d train the model.
What I think this leads to might be broadly important. If you want to use AI in any form to perform or manage human activity, you have to consider how humans themselves make decisions and take actions. Whether artificial general intelligence (AGI) is a goal or not, whether AI could achieve human-like consciousness or not, humans and other critters have evolved to deal with complexity at least in part by dissecting “living” into tasks that can be executed autonomously, without conscious involvement. Some of this is accomplished by hard-wiring the brain, by “instincts”, and some can be trained in through repetition, which I’ll call a “habit”. This is why I favor the notion of “distributed” AI over the notion of a centralized all-knowing model.
Now let’s look at the non-process-automation applications, the best example of which might be the “metaverse” or a computer game. You have avatars that represent real beings, including real people, and you have spaces that the avatars inhabit. The people behind the “real” avatars see, on their device, the metaspace that visualizes the metaverse area their avatar occupies. Since the goal is to make the virtual-reality experience realistic, we would want the metaverse experience to mimic reality, though we could argue that a metaverse and a gamespace would differ depending on just how close to objective reality the experience was supposed to be. Meta proposed imposing rules to prevent metaverse avatars from doing things that their real human equivalents would not be able, or permitted, to do.
The rules that collectively govern the virtual-reality experience, that decide what can be “seen” and “done”, could be established by software/model constraints, or trained in perhaps by having a model analyze video. The latter approach is what seems to be emerging as a means of controlling robots that are supposed to perform at least a credible subset of human tasks.
Perhaps the most obvious, and most important, question all this raises is how it might impact LLMs and what we almost universally see as “AI” today. An LLM does not think, nor does it learn in the way people do. It builds word sequences based on your prompt as a “seed”, and on models of sentences learned from its training, often from the Internet. Could any of this be improved by introducing some of my points above?
I’d looked at “knowledge extraction” myself over the years, and one thing I found was that it seemed important to classify sentences in order to extract information. You had, for example, “identity” sentences that asserted what something is, was, isn’t, wasn’t. You had “descriptive” sentences that qualified something by stating properties like color, size, or shape, and you could apply both these to objects (“The car is red”) or actions, (“Running is fun”) You could also classify sources based on credibility and intent. A dictionary is an authoritative source of definitions, Wikipedia is a more credible source than a randomly selected site. Also, in general, sentences are structured in a specific way, which tells us what to expect for terms within them. How much do LLMs accommodate this? I don’t know; some of my contacts who do say that it varies.
What I really wonder is whether the training process mirrors how we learn, which would seem important if artificial intelligence is our goal. We learn through observation, through tutoring, through reading or listening. We are exposed to these stimuli not all-types-at-once, but successively over time, which suggests that we probably focus on sensory classifications first, then on parsing language in spoken or textual form. Would an LLM learn differently, better, if it tried to mimic that? Would it learn better if, instead of learning “a bird can fly” as a word sequence, if first had a solid view of what a bird was, and what flying was? Does learning order matter? We probably learn through the creation of what I’ve called “habits”, which are sequences of thought that have become almost automatic. When we see a bird fly, we’re not conscious of how we identify it, or how it’s flying, but we can learn that, and if we do, we’re not conscious of how we apply the lesson. Could AI mimic that?
I think there’s a need to look at AI more through the lens of digital twins and ML, to refine models and training to align with the way we learn and think. Otherwise, we don’t have artificial intelligence, we just have artificial.
