From Spatial to Reality Computing
The magical mashability of Reality Computing...
The Reality Computer is a new category of computer whose operating system delivers “magical mashability” for all of our digital property.
Reality Computing drops the apps and programs of traditional operating systems and frees all of the 2D digital property once held by them into a 3D virtual space.
The virtual spaces on all of our individual computers are then merged over the internet through Web-like links, to form a single, seamless, shared universe: the Reality Network.
I’ll carry this on as a dialogue with an imaginary reader, just for a change of style...
So we’re on the topic of “magical realities”, now?
Stepping back and looking at how far we’ve come in just the last half century, it’s mind blowing what we can do today through computers and the internet. Or, in fact, it would be mind blowing to someone from half a century ago, but here and now, we’re getting quite used to it! We already live a significant fraction of our lives via the internet without even being conscious of that fact.
Further, Virtual and Augmented Reality (VR, AR or “XR”) and the Internet of Things (IoT) will go on to make that even more immersive and transparent in the near future. XR and IoT represent two facets of a gradual merging of the physical and virtual or digital.
This, I believe, is what computers and the internet were always about from the start, in other words: “making the world more magical”. There’s a phrase from Arthur C. Clarke that goes: “any sufficiently advanced technology is indistinguishable from magic”, and there’s no technology this applies to more than computers on the internet.
I sense a “However, …” coming up!
However (!) the way computers and the internet have evolved so rapidly has almost baked in decisions that were made for historical and cultural reasons and that could needlessly constrain future progress - the smooth access by all of us in the human race to this magical potential.
I call these magic-constraining decisions the “app trap”.
My vision of “Reality Computing” boldly turns this “app trap” - that exists within all current computing technology - inside-out, in order to free the magic for everyone!
This is all very abstract - I need an example!
“Spatial Computing” is a narrow slice of the potential of AR and VR. However it’s already showing us a tentative glimpse of a future 3D immersive and “magical” computer experience:
In Spatial Computing, users can wear a headset that gives them the illusion that many application windows are arrayed around them in the full space of the room, instead of being squashed into a small computer monitor screen. 3D objects can also appear right on your desk.
Recently, Apple put Spatial Computing back onto the tech agenda with their Vision Pro XR headset, several years after Microsoft’s Hololens first showed the way. And Meta have recently announced they’ll be adding Spatial Computing aspects to Quest in collaboration with Microsoft.
While Spatial Computing takes a tentative step towards a potentially much richer, “magical” future, it also nicely shows us what the app trap looks like, when we use one of these Spatial Computing devices!
OK, show me Spatial Computing’s app trap!
What do these three images of Spatial Computing have in common?
Hololens:
Vision Pro:
Windows plus Meta Quest:
Hmm, something around being a bit awkward?
Almost: the 3D objects (the globe, that odd pink thing, the model of a gaming controller) are each floating freely in 3D space. However, the 2D objects (emails, calendar events, paragraphs, images, messages, contacts, threads) are bound firmly into their application windows.
This is literally what the app trap looks like, in full 3D!
Spatial Computing clearly exposes the app trap, when you can see all of your 2D digital property trapped in their app or program windows arrayed around you alongside perfectly free 3D objects.
We’ve evolved online and computer technology into a design where all of our digital property is chopped up into these separate apps and programs.
What’s wrong with 3D in 3D space and 2D in 2D windows?
Well, “2D in 2D windows” or “2D within applications” is actually completely un-intuitive. It actually has a feel of the “horseless carriage” about it, especially when seen this way in 3D.
3D objects in 3D space is how we all live our daily lives and how we’ve evolved cognitively to perceive and interact in the universe.
But “applications” (or apps or programs) are a very recent, artificial and constraining concept in the context of human evolution and intuition. Apps trap all of our 2D content quite arbitrarily and un-intuitively, preventing all of the digital content they imprison from being interacted with and intermingled in the same familiar, 3D ways.
We should be able to do anything that our real universe enables or inspires us to imagine in the virtual digital realm, and be able to do that in much the same way.
Instead we have to flip between intuitive “real thinking” where everything is intermingled within a single universe, and “virtual thinking”, where we have these unnatural barriers enforced on us by apps and programs that fragment our virtual property and act as gatekeepers to them.
The way computers currently do things - both in Spatial Computing and in normal everyday desktop PC and mobile operating systems - works against our human intuition and expectations.
Well you can share and export between apps can’t you?
Of course, but our actual reality doesn’t have anything that looks like apps that make you clumsily share and export stuff, so there’s absolutely no reason we should be made to do that in the virtual digital reality.
So: you’d pop out the 2D stuff from their apps into the 3D space?
Exactly:
Instead of a calendar app, why can’t we just look up on a virtual wall to see our calendar?
We should be able to rip off a task from our to-do list that’s hanging on that virtual wall, next to a pinboard covered in photos and notes, and pin it right onto that calendar.
We should be able to paperclip a friend’s contact info to a task on that to-do list.
We need to build our computers to create a digital “Reality Network” that works just like our actual physical reality.
The “Reality Network” is quite a dramatic name for this 3D space on a PC!
Ah! But this concept can expand beyond a single computer and into the internet:
Instead of a chat app, why can’t we just meet and chat?
Instead of a gallery app, why can’t we just visit and wander around a gallery?
Instead of game apps, why can’t we build a place where people play games?
That’s the vision of “Reality Computing”.
An app-free 3D space that spans the planet.
So no apps in Reality Computing?
Right: we need to build an operating system for our PCs, mobiles, etc., that simply doesn’t have any apps or programs!
In this illustration, I show a simple example of this, with our digital property freed from the apps that held them apart, then dropped into a common 3D space.
The decision to free our digital property by flipping everything inside-out like this is such a significant choice with such far-reaching consequences, that I call it “The Inversion”!
This 3D space on a single device then goes on to become the “Reality Network”: a single, seamless, shared, virtual digital universe running across all of our devices around the globe.
So the 3D space takes in every diverse type of application data?
Not at all: the powerful advantage of doing this is that we can distil and reduce all the diverse and complex app data and digital content down to their elements: small simpler pieces that behave the same everywhere - their pure or generic forms and their interactions. This is ideally based on any related and familiar objects in our physical reality.
We can then build it all back up again into more complex structures of world objects in the Reality Network, however we like (not however apps like!).
For example, when we free our digital property out of: an email app, a web browser, a document editor and a messaging app, we are given the opportunity to break all of that apparently diverse content down into, well, just paragraphs and images! (Maybe some more bits like lists, but you get the idea).
We then need a way to do the building back up again, to start to aggregate and associate related objects in our 3D spaces.
I’ll call that “magical mashability”!
“Magical mashability” sounds fun, but is it really magical?
We’re recreating reality and supporting our intuitions about how things hang together there, so, just like in reality, we should be able to work with, refer to, pick up, drop, replace and share world objects all the way up from a small paragraph (or leaf!) to a huge library (or forest!).
We can imagine picking up or cutting out snippets of 2D documents rendered in 3D and dropping or pasting them somewhere. We don’t need to work only with complete documents any more, we can play with individual paragraphs, images, lists and sequences of smaller stuff.
It should be easy to assemble paragraphs and images into sequences to form messages, documents or books, messages into chat groups and on into global communities, then books into bookshelves, bookshelves into libraries, photos up to galleries, a library and a gallery into a street, streets into cities, leaves into forests, cities and forests into lands; a working drawbridge up to a fairy castle, a chess or skittles game up to a theme park, lands into worlds, all in one huge universe!
That kind of extreme mashability would seem quite “magical”!
Wow! But isn’t all this mashing up of objects hard to manage?
Not if you have a Secret Magic Power! Links…
Our world objects are sewn, pinned and mashed into the fabric of this single, shared virtual universe through the pervasive use of links.
Links form handles or hooks onto all of our world objects. They work a bit like the Web’s URLs in the way they point to things, where-ever they may be.
However, these are “upgraded Web URLs” that can work in a 3D virtual world.
So we can grab, wire and share direct links to and between any world object owned by anyone, anywhere.
Every single object we create or see has a link - notes, documents, contact cards, calendars, messages and photos; flowers and petals, vases, tables, rooms, houses, galleries, libraries, streets, cities!
Everything can be pointed to, picked up, re-used and saved via these links. Then everything can be tied, snapped and mashed together with everything else via these links.
Every world object we make is in a single unbounded, linked-up universe, so we can re-use, re-mash and re-purpose anything, and join all of our spaces, buildings and games together in new ways, sharing materials or entire constructions between our creations.
I can link to a city you've built from my virtual garden, which lets me see it and just walk across into it, or you can link to a leaf I made, and use it yourself.
Paragraphs are linked into messages or posts, messages into feeds, feeds into collections of friends and families. In this universe, your “social network” is literally a visibly-linked network of world objects.
Petals link to flowers, vases to tables, rooms within houses, streets of cities.
Links are at the heart of the extreme and pervasive mashability of this Reality Net. Links empower us to create anything we like, then to join it all up into a huge landscape. And then to go on to build a huge universe together, mixing and mashing, sharing, re-using and re-purposing anything with anything else as we like.
Wow, so nothing as mundane as files or folders in this OS?!
Ha, no. Not only do we not need apps and programs, but we also don’t need the rigid hierarchies of files and folders any more!
Links take on all of that, breaking down files and reassembling them into much more than folders. In fact, locally on your device it’s more like a database (overall a “distributed graph database”, for techies following along).
If you’re feeling nostalgic, in the real world there were literal physical “files” and “folders” before computers came along - so we could simply have 3D versions of those if we liked, stored in virtual 3D file cabinets!
But these “upgraded Web URLs” don’t just point at files either?
Right: unlike the Web, our links work at a much finer level of detail, pointing to live or self-animated 3D scene elements or world objects, that may be hosted symmetrically on each other’s PCs, sewing everything into a seamless 3D fabric.
Those three facets - self-animated, symmetric and seamless - are effectively a 3D triple-upgrade to the current 2D Web!
So, does “self-animated” replace app or program functionality?
Yes, exactly: the “Inversion” doesn’t just spill all of our 2D digital property out into the 3D space, it also packs some application functionality and behaviour inside those 2D world objects in the form of live self-animation.
Reality has no apps, just world objects, items, tools, etc., with behaviours and internal animations, that are usually predictable from looking at what the object is, and that drive interactions in the space, both between world objects and between those objects and us. Well, technically, we’re also world objects - or avatars!
Having said that, for the relatively few cases where the behaviours and animations of objects need to be customised, there will be a simple and powerful way to program more complex internal animations and object interactions, that can be picked up by anyone who can use a spreadsheet.
How do you get from our single 3D spaces to a shared universe?
We need to build the “Reality operating system”, for XR headsets, PCs, tablets, smartphones and online servers, with their local 3D spaces, to manifest this virtual universe.
Then, simply using our links, we can all sew and pin together our virtual objects and places, hosted by these devices and their operating systems, into the single vast universe.
Our Reality operating systems, running on all of our Reality Computers (and friendly servers) work together to manifest this single, seamless virtual universe, owned by each and all of us.
In this single universe, each of our operating systems shows us all of our own digital stuff on our own device, seamlessly linked and intermingled with the live digital stuff of everyone else’s (including their avatars!) shared from all of our devices across the planet. Of course, we’re in full control of the read and write permissions of our own locally-hosted world objects.
How does this actually look on the screen or in VR?
Just like in physical reality, all of the 2D stuff that we associate with our current programs, applications, computers and operating systems - scrolling documents, maps, calendars, etc. - are treated just like any other world object, except now they’re rendered in full, natural 3D. A contact can look like a tangible contact card, and when you drop it, it falls to the floor!
This picture illustrates how some of this may look: a 3D space with us in it, in 3D rooms. There are flat 2D objects on the walls, the floor, and floating in space. Some objects are embedded in other objects and render just as well like that.
We can have notes, messages and documents made up of paragraphs and images, to-dos, events, contacts, etc., all linking to and from each other in collections and sequences, perhaps nested within one another, or stuck to the walls, on the table top or even just lying on the floor.
This could be experienced on a PC, tablet or mobile screen, in Virtual Reality, or in Augmented Reality - where virtual rooms and objects map onto physical ones.
Here’s a conception of how things may look in a more photo-realistic style:
Source: https://dribbble.com/shots/6112246-floating-UI
And this from Unity:
Both show 2D panels with thickness for very simple objects that have a real sense of being part of the 3D space.
Wouldn’t this Reality Net be better on VR headsets not PCs?
Of course, but until AR and VR and the IoT are immersive and transparent, having something like a PC, or better still a tablet lying around, is the next best way to make this Reality Net as accessible as possible from actual reality:
Imagine a family tablet that you can just pick up, for ambient and casual engageability with this globally-shared Reality Net. This is an open, intuitive and humane space that focuses on the pervasive “magical mashability” of every object you interact with, through powerful links.
The Surface Go 2, that can become a Reality Computer
The computer’s operating system is always running your contribution to this shared universe, and puts you right back to where you were before, every time you pick it up.
Can you give me examples of daily life in the Reality Net?
There’s a very comprehensive list of examples of how we’d interact and co-create in the document “Mashing it all up“, published on this site. There’s another very detailed scenario described in the document “Spatial Computing with links not apps”. Both documents are also well worth reading all the way through, especially the latter.
Our Reality Net lives will be similar to our physical reality lives. We’d go there to meet, chat, share, build, exchange, sketch, create, think, work and play together. We will interact through 3D forms of 2D content like social media, documents, messages, etc.
And since we’re in a 3D virtual world, we can build realms of cities and landscapes, build our dream homes and our perfect communities. We can play games together, create art and do science.
What challenges are there in building this Reality OS?
Well, the app trap is baked into all of the traditional underlying operating systems that run our computers today, so the main thing is that we have to migrate away from those.
Fortunately, though, we can still free our actual digital property from their apps and programs; it should all still be accessible (and if not, then you really need to fix that already today!)
It can then be imported into our Reality Net, and exported back out again if you want.
An app or program that will be one of the first to free its 2D content into 3D is the Web browser. Our 3D “upgraded Web” can easily subsume at least the static parts of the 2D Web. It's actually quite easy to start with importing all of a static site like Wikipedia. Dynamic, interactive sites will, however, require them to be open to us connecting-in to whatever functions they offer, through internet “pipes” or “plugs and sockets” (APIs).
This import/export is much more important than worrying about keeping hold of all the latest apps and programs and services, especially once you see how they are trapping our digital property.
It does mean building this Reality Network, that will welcome all of our rich content, from the operating system up, and into the internet.
Of course, it’s all being built as free and open software, with open internet protocols. Keep following this site for updates on progress!
Has no-one ever done anything like this before?
Not as far as I know, beyond the Spatial Computing offerings described above, and perhaps Anarchy Arcade. There are various app-free operating system research projects, including the Itemized OS and Dynamicland, which have some related concepts.
Maybe some would describe it as a “Metaverse computer” (or “like an operating system that only runs Second Life”!) But it’s not just creating The Metaverse, it’s creating the “Universal Metaverse”, from the operating system level up, to empower us in every aspect of our digital lives, including both 2D and 3D digital property.
Summary
The Reality Computer is a new category of computer. Unlike Spatial Computing and traditional operating systems, Reality Computing has no apps or programs.
Instead it frees all of the 2D digital property once held by them into the single, seamless, shared, 3D Reality Network, manifest via the internet from all of our individual computers.
It does this through the familiar and powerful mechanism of the link. Links are pervasive in this shared universe and work across all types of object, across all scales and across and between all of our computers.
Magical Mashability
It's a magical, mashable universe!
One day this universe could simply be called “the internet”, subsuming all documents, messages, social media, shopping, currencies and the rest of our 2D digital content, as well as supporting all of our online 3D games and experiences.
It’s a computer that you can just fire up to immediately access this Reality Net, where everything works just like it does in our normal reality, except that the Reality Computer allows us to do much more, much more easily and more intuitively than either current computers or physical reality itself - it really could feel like magic!
What do you think? Would you pick up a Reality Computer tablet along with your family or friends? Drop your thoughts into the comments below.
The “app trap” is fully described in the other articles on this site.