An Augmented Reality Experience to Complement a Vintage Pinball Machine

My 1948 Gottlieb Cinderella pinball machine won 1st Place Best Antique at Texas Pinball Festival 2024! As one of the earliest games made with powered flippers, I worked tirelessly for weeks before the show to resolve any problems that would spoil the gameplay experience, such as mechanical malfunctioning and leveling inserts on the playfield so the ball doesn’t get stuck in spots. However, such an old game lacks many of the interesting aspects of modern games, so it might be overlooked by casual pinball players without some kind of augmented experience. Enter the Cinderella Augmented Experience!

This augmented-reality (AR) app aims to provide its users a glimpse into what goes on inside the game as they play it. Games from the 1940s are much different inside than even other electromechanical machines from the 1960s and 1970s. For instance, before score reels were invented, there was a single score stepper that had rivets representing every possible score in the game. Aspects like this would appeal to anyone remotely interested in how these antiques work.

Setting Up the Project

Ahead of time, I had to devise requirements for the application, such as:

It must be a Web-based experience; I don’t want to require people to download an app
The AR framework must be:

Marker-based, so as to recognize images of the playfield and portray media on top of them
Easy for a developer to use

Everything must be small, because connectivity at Texas Pinball Festival can be unreliable, particularly in a room with over 400 games, each with dozens of electromagnets

Augmented reality has been around for a long time. Many of my patents (130 at the time of this writing) relate to AR techniques that my teammates and I championed back in 2017-18. As such, I had to wade through tons of outdated articles on the topic, and focus in on what is modern and relevant.

I found a framework called MindAR https://hiukim.github.io/mind-ar-js-doc/ that nicely supports image tracking and face tracking, and easily integrates with AFrame and three.js. Using MindAR requires three steps, only one of which involves coding:

Make the image descriptors. These are contained in a file that MindAR uses to recognize the markers, i.e. elements of the playfield, backglass, or cabinet you wish to augment. MindAR provides the tool to build the file.
Make the media (pictures or video with captions) that gets projected on top of the markers.
Describe in the HTML document the AR scene, the visual assets, and each marker as an entity.

For instance, to make an app that reacts to the backglass to show the score mechanism, all I have to do is:

Take a picture of the backglass, and run it through MindAR’s Image Targets Compiler: https://hiukim.github.io/mind-ar-js-doc/tools/compile
Edit a photo or a video of the score mechanism to add captions and callouts; one can do this easily on a mobile device using apps like Phonto, InShot, or even TikTok or Instagram
Write the following HTML:

<a-scene mindar-image="imageTargetSrc: targets.mind;" color-space="sRGB" renderer="colorManagement: true, physicallyCorrectLights" vr-mode-ui="enabled: false" device-orientation-permission-ui="enabled: false">

<a-assets>

</a-assets>

<a-camera position="0 0 0" look-controls="enabled: false"></a-camera>

<a-entity id="score-mech-target" mindar-image-target="targetIndex: 6">

<a-plane src="#score-mech" position="0 0 0" height="1.78" width="1" rotation="0 0 0"></a-plane>

</a-entity>

</a-scene>

Notice how the above succinctly describes the entity and assets used within the scene, and even sets up the camera without requiring lots of work.

Augment the App with Artificial Intelligence

The augmented reality aspect requires specific knowledge, plus high-resolution graphics of the item(s) in question. I spent quite a bit of time working on the assets for the markers and what to project on the markers. But there were still two aspects of the application worth developing: a gamification system to convey what markers actually exist (i.e. where the user should point the camera to see anything happen), and analytics to help me understand how people are engaging with the app at the show.

As you know, CSS can be fickle and produce unexpected results if you are not an expert at it. It has also been around for a while, meaning ChatGPT is well-suited at generating code for me just so I can save the time of poring over documentation and wasting time on guess-and-check just to build a few specific things, and focus my time building more useful things only I specifically have. I asked ChatGPT for these specific features:

HTML table listing the markers I built: “bumpers”, “bonus”, “coin slider”, etc.
The table must be responsive; it must show marker names in a comfortable, reasonable font size scaled for the display, and the number of names shown in a row must flow with the display size so the table looks natural
Unicode checkmark that appears next to marker names upon completion of a task (such as a click, to keep it simple)
An API in PHP to interface with a database of my own schema so I can track anonymized usage and what each user was able to scan in the AR world (yes, I still use old-school shared hosting for personal stuff in many cases)
Calls to the PHP API from my JavaScript front end using XMLHTTPRequest

Granted, I still had to fiddle around with these things quite a bit, as I did not tell ChatGPT 100% of what I truly intended to build. In fact, I even changed the database schema after I had ChatGPT make the PHP API. As such, I still had to do some shaping, documentation reading, and testing of my own before everything worked 100% perfectly. But at least I didn’t have to start these from scratch.

Gamification and User Studies

I wanted to inform the users exactly what markers are available to scan, so I provided a table that could be checked off (as described above). This allows for a sense of accomplishment and perhaps a surprise once everything has been achieved. For this, I wrote a simple event listener for each marker, an example of which is below:

const bonusTarget = document.querySelector('#bonus-target');

bonusTarget.addEventListener("targetFound", event => {

document.querySelector("#bonus-status").classList.add("checked");

checkAccomplishment('bonus');

sendXHR({sessionId, activity: "bonus"});

});

When the user has found the marker relating to the bonus target, it:

Adds the `checked` class to the “bonus-status” class list, so it gets the Unicode checkmark
Stores the accomplishment in JavaScript, and checks if the rest are finished so as to unlock the surprise
Sends information to the API regarding what marker was found

Not only that, but to anonymously track users’ progression through the app, I made each user a nonce (random number) identifier upon opening the application, and associated that nonce with a series of UTM parameters consisting of:

UTM campaign = the event (such as `tpf2024`)
UTM medium = who referred them (namely, a QR code I posted next to the game)
UTM source = which game they’re interacting with (`cinderella` is the only game with this feature)

This way, as the system grows, I can track engagement per game and find out what game is more compelling, where there are deficiencies or weak spots with my images, etc.

The application involves two database tables. As users first load the app, their random session ID is stored in the `sessions` table, along with the load timestamp and the three UTM parameters. Then, interactions with markers are stored in the `activity` table that records which session ID interacted with what marker at what timestamp. This gives me a full picture of who does what with my system.

Adding Assets

There are several steps involved with adding more markers to the application. Fortunately, they are all short, and some only require one line of code, or simply editing media:

Take a picture of the marker
Run it, in conjunction with the other markers, through the Image Targets Compiler mentioned above - note the order in which you load the images
Make the event listener that fires when the marker is found
Notate the marker name in the table, with a parent div belonging to the “item” class nesting divs with a “status” class and a “task” class
Add the asset projected upon finding the marker into the <a-assets> object
Add the <a-entity> representing the marker, noting its mindar-image-target="targetIndex: 3" (where 3 represents it was the 3rd picture you added to the Image Targets Compiler earlier - so don’t forget the order you used above, and don’t forget to mind this setting!)

You can see what the code looks like for all these steps, where applicable, by referencing my repository.

What Didn’t Work?

Some things didn’t function properly by the time I decided to leave the app alone for the remainder of the show. In particular, touching or clicking anywhere on the screen did not function properly, probably because the A-Frame library is usurping the touch or click elements. This became rather annoying when I was trying to refresh the page after updating the server, as I couldn’t just swipe down; I had to tap the URL and hit Enter to reload the page. This also had the sad side effect that no one could tap the link to see my ending surprise video, which was a blooper video of me having way too much fun with the ball serve.

For some reason, the video assets never played. I would have to try the <video> tag to see if it actually works properly under normal circumstances. All it would do was show the first frame of my video, rather than playing the video.

Finally, the ceiling lights caused a problem with scanning the markers, because it messes up the algorithm that performs computer vision routines to match critical points coming from the camera imagery with those learned from the sample of the markers, normally taken under ideal conditions. I think this one could be solved if the Image Targets Compiler allowed for multiple images to be used for a single target. In fact, under the hood, the Image Targets Compiler actually performs various manipulations, such as stretches, skews, rotations, and adjusting brightness and contrast, to produce the marker data. As such, it should not be that much of a stretch to incorporate other pictures featuring some of the non-ideal lighting conditions.

Why not just use AR.js?

I’m not sure. This seemed to be quite simple, but then AR.js seems to be relatively simple too. However, many of the contemporary examples provided by AR.js seem to focus on VR, not AR. Also, MindAR can portray not just flat objects, but also 3D objects coming from A-Frame’s primitives, three.js, or even assets rendered in Blender or Unity. It’s possible the other frameworks can achieve this too.

Nevertheless, portraying 3D models of actual parts would be a cool follow-on project. Imagine having a cool 3D model of one of these stepper units, the score motor, or other features that even those who maintain these games regularly can find confounding and tedious. (Myself included; in fact, before the show, I tried to fix the game’s play counter, and traced the circuit from the front of the game all the way through the head unit and then down another path to the front again. Eventually I gave up and figured out how to trigger it with a rather short bodge wire, so I did that instead.)

Search This Blog

GOSHtastic - Game shows, Options, Software, & Hardware!