Thursday, February 1, 2018

How Old Am I, According to Spotify

I'll explain this down below.

Spotify facilitates very convenient access to lots of great music on many different types of devices.  However, being a standalone application that manages its own music in its own way means you can't interact with the songs as you could when they were in MP3 format or on vinyl.  For developers, the Spotify APIs can change the game.

A Brief History of My Music Listening


I'm a bit of an odd bird in my musical tastes, which were influenced by my mom as we listened to a lot of oldies stations driving around to various events and activities.  Where most oldies stations typically play music that is 30-40 years old, one station in Dallas (KAAM 770) plays music that is around 50 years old.  Thus, back in the 1990s, they were playing lots of big band and swing standards.  (Now they have caught up to where traditional oldies stations were back then, but playing music more along the lines of The Carpenters rather than classic rock.)  I developed an affinity for Nat King Cole, and was soon introduced by family to the music of Frank Sinatra.  Then, playing in various school bands, I took a liking to orchestral/instrumental scores with rich musical and technical content, including game show music, as I was becoming fascinated with all aspects of game show production.  Fast forward to 2007, when I saw The Jersey Boys in concert, and then Frankie Valli in person shortly thereafter, and I have been seeking my fix of 60s & 70s pop ever since.

It fits, along with my game show collection and seeking exact airdates for all my episodes, that I would like to know when all the music in my collection was recorded and by who exactly.

Curating a Collection, and Odd Ways To Listen


In the days of file sharing, I would come across MP3 files that often had incomplete or incorrect metadata as to their origins, especially the year but sometimes even the artist, often attributing the work of a much lesser-known artist, but sharing a similar style, to the greater-known artist.  I would do research on these songs and correct the information in programs such as JRiver Media Center that not only offered a powerful suite of music and playlist management capabilities, but also the best in digital signal processing to make your music really shine on your speakers.  Sometimes, to heighten the emotional content for me, I would pitch-bend the song, usually 1/4 to 1/2 step flat, but occasionally 1/2 step sharp.  (Too much beyond that and the timbre of voices and instruments becomes distorted, ruining the believable effect.)  On vinyl, an adjustable turntable could provide a similar effect, except JRiver was smarter in that it could adjust only the pitch, not the tempo if you didn't desire.

Gone are the days where people would freely trade MP3s on the Internet.  It was a temporary fix for a music industry that refused to adapt to the times, but now it is so inexpensive and convenient to maintain a subscription to a service such as Spotify.  The only problem is you are typically constrained to using only their media player which lacks much in the way of audiophile DSP capabilities (notably not even an equalizer for the desktop version), and the geeky curator will miss the ability to see see so much data at once (and edit it) as JRiver shows.

Nevertheless, to account for the lacking feature (and satiate my curating curiosity and desire for data), I thought that, as a developer, there might be a way to gain further insight into what I've been missing.  Lo and behold, they offer an API for developers that can get you all this information, and possibly even a gateway into playing music through your own DSP tools (if Web-enabled).  Since I have had Spotify since the end of 2014 and haven't really paid much attention to my old music files since then, there are a great many new (to me) songs in my list that I don't know much about except for when I search for them on Google.

On a Quest For Information


Spotify usually provides a lot of interesting, insightful analytics as long as you have not blocked their emails.  (They really don't email you much, so if you're blocking them, you're missing out.)  One of my favorite emails from them included the following tidbit:

Spotify must think I'm an African-American at least 60 years of age.  (Or maybe a white person of around 50, since once they offered me a deal on Chicago + Doobie Brothers tickets.  Of course, I took them up on it.)

This is very cool and only the tip of the iceberg of data you could collect about yourself.  However, things like this often lead to more questions than answers, such as "What's another genre I'd like that doesn't relate except for only some esoteric way a computer would understand?" or "What's the song I really liked in my Discover Weekly but forgot to bookmark?" or "How have my listening preferences changed over time?"

Nevertheless, one thing I've understood since about 2007 is that my musical tastes fall in line chronologically with "doo-wop to disco", meaning core 1962-1978 with fringes between 1956-82 (gotta include the classic rock & roll).  Of course, this doesn't include the works of Frank Sinatra and Nat King Cole, who were often still covering songs written in the 1930s and 40s all the way into the 1960s and beyond.  But because their new recordings were often completely different interpretations and instrumentations, I am not so concerned with the song's original release date as I am with the date of those particular recordings.

As such, I was expecting to obtain data from Spotify and present it graphically in such a way that would display a big hump around 1962-78, with a short tail backward, and a long tail onward into the present.  However, this is what I got:

Not much love for the 1980s; just 60s & 70s, and mostly rehashes of the same from later years.

However, Spotify doesn't concern itself with the original release date of a song, but with the release date of an entire album.  This means that looking back on, for instance, The Voice album by Frank Sinatra, which was in 1955 already a compilation of his much earlier works (and released once again in 1999), any song on that album (originally recorded between 1943-52) would show up as 1955 (or possibly 1999, one year after Sinatra passed away) on Spotify.  I know for a fact there are songs in my library recorded in 1958 and 1961, but the graph above doesn't show anything prior to 1962.  I haven't experimented with how it handles local files I have curated because, quite frankly, I don't want them intermingling, so this data pertains to only what I stream from their service.

So, How Old Am I, According to Spotify?


To be totally honest, I don't know, but if I had to guess, it would think I was as old as about 70, with maybe a 50/50 chance of being white, and formative tastes appearing in the late 1950s, and staying decidedly hip until around 30 when everyone got sick of disco.  Well, I'm around 30 now, and my tastes have never been decidedly hip.

Once upon a time, Spotify made playlists for their users, consisting of music from the 1970s, 80s, and etc. that reflected music you would have listened to back then based on your current tastes.  It struggled to do much for me, but I have to give it credit for giving me a "1970s playlist" consisting of many compilation albums from the likes of Glenn Miller and other famous big bands dating back to the 1940s.

How Old Are You, According to Spotify?


If you are a developer, here is how you tell.  It's definitely cutting-edge JavaScript code (if you traveled back to 2012 to read this), but not too difficult to get working.  All you need is:

  1. A Spotify account where you have signed into the developer portal and registered an app.
  2. The example Spotify Web API app, downloadable from https://github.com/spotify/web-api-auth-examples
  3. Your app in Spotify configured with the correct redirect URI.  If running it locally, it is http://localhost:8888/callback/ because this is already established in app.js for you.
  4. Your app.js file configured with the corrseponding client ID and secret provided by the developer portal, and the redirect URI you specified earlier.

Once you have these things, incorporate the following changes into the code you just checked out from Github (because I'm too lazy to fork it and incorporate the changes for you to download conveniently).  Then, kick off the Node server, open your browser, navigate to your server, open your developer tools, log in, and watch the console.log statements reveal your data.  Hopefully running all these calls for your whole library won't DDoS Spotify; with only about 200 songs in my case, I didn't bother putting in any pauses.  The next step would be seeing how the age of the music you are listening to has evolved over time (will I finally like 1980s music within 10 years?), but I guess that would only be interesting for me, since most people probably only listen to current music.

$ git diff
diff --git a/authorization_code/app.js b/authorization_code/app.js
index b37f9c5..1e51945 100644
--- a/authorization_code/app.js
+++ b/authorization_code/app.js
@@ -12,9 +12,9 @@ var request = require('request'); // "Request" library
 var querystring = require('querystring');
 var cookieParser = require('cookie-parser');

-var client_id = 'CLIENT_ID'; // Your client id
-var client_secret = 'CLIENT_SECRET'; // Your secret
-var redirect_uri = 'REDIRECT_URI'; // Your redirect uri
+var client_id = 'something'; // Your client id
+var client_secret = 'something else'; // Your secret
+var redirect_uri = 'http://localhost:8888/callback/'; // Your redirect uri

 /**
  * Generates a random string containing numbers and letters
@@ -44,7 +44,7 @@ app.get('/login', function(req, res) {
   res.cookie(stateKey, state);

   // your application requests authorization
-  var scope = 'user-read-private user-read-email';
+  var scope = 'user-read-private user-read-email user-library-read';
   res.redirect('https://accounts.spotify.com/authorize?' +
     querystring.stringify({
       response_type: 'code',
diff --git a/authorization_code/public/index.html b/authorization_code/public/index.html
index 9c57f1c..9544ba8 100644
--- a/authorization_code/public/index.html
+++ b/authorization_code/public/index.html
@@ -64,6 +64,7 @@
     <script>
       (function() {

+           var years = {};
         /**
          * Obtains parameters from the hash of the URL
          * @return Object
@@ -78,6 +79,50 @@
           return hashParams;
         }

+               function getAlbumInfo(albumIds, next) {
+                       $.ajax({
+                url: 'https://api.spotify.com/v1/albums/?ids=' + albumIds.join(","),
+                headers: {
+                  'Authorization': 'Bearer ' + access_token
+                },
+                success: function(response) {
+                                 var json = eval(response);
+                                 console.log(json);
+                                 for (var a in json.albums) {
+                                       var album = json.albums[a];
+                                       var year = album.release_date.substring(0, 4);
+                                       if (years[year] === undefined) {
+                                         years[year] = 0;
+                                       }
+                                       years[year]++;
+                                 }
+                                 console.log(years);
+                                 if (next !== undefined) {
+                                       getPlaylists(next);
+                                 }
+                }
+            });
+               }
+
+               function getPlaylists(next) {
+                       $.ajax({
+                url: next,
+                headers: {
+                  'Authorization': 'Bearer ' + access_token
+                },
+                success: function(response) {
+                                 var json = eval(response);
+                                 console.log(json);
+                                 albumIds = [];
+                                 for (var song in json.items) {
+                                       albumIds.push(json.items[song].track.album.id);
+                                 }
+                                 console.log(albumIds);
+                                 getAlbumInfo(albumIds, json.next);
+                }
+            });
+               }
+
         var userProfileSource = document.getElementById('user-profile-template').innerHTML,
             userProfileTemplate = Handlebars.compile(userProfileSource),
             userProfilePlaceholder = document.getElementById('user-profile');
@@ -109,9 +154,9 @@
                 },
                 success: function(response) {
                   userProfilePlaceholder.innerHTML = userProfileTemplate(response);
-
                   $('#login').hide();
                   $('#loggedin').show();
+                                 getPlaylists('https://api.spotify.com/v1/me/tracks');
                 }
             });
           } else {
(END)

Epilogue


I didn't take fully after my mother in my musical preferences.  While I tend to enjoy disco hits with funky grooves or complex chords and progressions, her feeling on the genre is summarized by her shirt with iron-on letters reading "Disco Sucks."  I'll post a picture of it if she can find it.

Thursday, January 18, 2018

A Cold Wind From the USSR - Part 2, Keyboard Emulator for a PDP-11

To read Part 1 of this series, click here.

The Russian Elektronika DVK-3 PDP-11 clone is a very fascinating computer to me, and I want to share it with the world.  Part of that endeavor involves allowing people to actually interact with it and play games on it remotely, not just see videos of me using it.  To get this to happen, I need to actually allow people to input into the system via the keyboard.  I could build a ridiculous robotic action to hit the keys on behalf of remote users, but this would likely introduce a great deal of latency into the system and would not be good for those critically-timed Tetris block rotations.  The best bet is to spoof the keyboard by basically building a whole new one, but instead of keys, use an Internet connection and microcontroller to generate the scan codes.

The Elektronika MS7004 keyboard is based on the DEC LK201 keyboard interface, standard for PDP-11 minicomputers.  It utilizes the RS423 serial communication standard at 4800 baud.  I (painstakingly) (well, really Stacy did it after I gave up in an epic fit of rage) fabricated a 5-pin DIN connection "tapping" system that allowed me to insert a breadboard in between the keyboard and computer so that I could measure the signals generated by the keyboard, and eventually drive signals with a different device.  First, the DVK-3 provides +12V and -9V on two pins of the DIN connector.  Another two pins are used for communication from the computer to the keyboard, and vice versa.  These pins do not operate at TTL, but instead, RS423 defines these pins as conveying a "High" at between +4 to +6V, and "Low" at -4 to -6V.  Finally, the fifth pin in the DIN connection is Ground.  It seems silly to me that they would not use the outer ring of the DIN connection as Ground and just go with a 4-pin DIN connection.  However, I was not in the room, nor even born yet, when they designed it originally, nor have I even been to that side of the planet to this day, so there is no way I could have suggested it to them.

Nevertheless, these odd non-TTL voltages mean that I can't just drive the scan codes straight from a microcontroller.  I'll need to put some other devices in front of the computer's DIN input so that it will recognize the scan codes correctly.


Approach #1 - Two Voltage Dividers & NPN Transistor


My first thought on creating the correct RS423 voltage for the keyboard was to drive down +12V to +6V and to drive up -9V to -6V with voltage dividers.  To divide voltage in half, you simply need two resistors of the same value in series, with one end of the series hooked up to +12V and the other to GND, and then tap your +6V from where the two resistors are connected to each other.  To cut voltage by 1/3, resistor 1 (from -9V) must have half the resistance of resistor 2 (to ground).

LTSPICE circuit simulation showing the original input voltage divided by 2/3.

My original idea was to use an Arduino to drive the scan codes, and source the power from the Arduino from the divided -6V output.  I would wire -6V to the Arduino's ground, and the computer's ground to the Arduino's VIN.  From there, I could use an NPN transistor with its emitter hooked up to the same -6V to expose the the scan code pin to either -6V or the divided +6V depending on its activation.  The circuit worked fine in theory, but I ran into some practical hurdles.  The original resistors I picked for the voltage divider allowed for just a small amount of current at those voltages, and with the demands of the Arduino, soon my voltage dividers were reading nowhere near their intended values.  I ended up with just over a volt showing up across the +5V and GND pins on the Arduino, not even enough to power it on.

My next thought was to lower the resistance values on the voltage dividers.  I thought about Ohm's Law and how much power would be required for higher current, and grabbed from my 1/2-watt resistors and put the 1/4-watt resistors away.  This time, I went with an order of magnitude less resistance, but the Arduino still only showed a little bit over 2 volts.  Using the unregulated power pins didn't seem to help the situation either.  You might be wondering "Why not just go with really low-value, high-wattage resistors?"  Well, the electronics store was closed, plus there's no reason I should ever use anything more than a 1/2-watt resistor unless someone is paying me to do something specific.

Finally, I decided to hook up the Arduino not to the divided -6V and GND from the computer, but to the divided +6V and the divided -6V, so that hopefully the regulator would have a better shot at regulating down to +5V.  However, this still caused the Arduino to read just 3.7V or so across the 5V pin, and it smashed my high-side divider (which was supposed to cut +12 to +6) down to just 0.8V.  Clearly, even if I could get the Arduino working, it wouldn't generate the needed voltage for the scan code!  And besides all this, the NPN transistor was somehow transferring its +0.7V base bias to somewhere else in the circuit, as I could see values differ by 0.7V when it was installed versus not.

(Mind you, I'm testing all this using a proper bench power supply, not from the computer itself at this point, so there is no risk of damaging the Soviet goods.)

Approach #2 - Two 7805 Voltage Regulators plus NPN BJT or N-channel MOSFET


The 7805 voltage regulator is a TO-220 packaged device that simply takes an input voltage, and a ground voltage, and can supply some amperage at +5V from ground.  No more messing around with specific currents coming from a voltage divider; I wanted the voltage regulator to handle everything for me.  The idea here was to use one to regulate +12 down to +5, and to regulate -9 (as if it's the ground input into the 7805) up to -4V.  I was still trying to drive the logic from the low side (-4V) to use an NPN transistor (or even a MOSFET to properly switch on voltage at the gate rather than current at the base).  (Also note that my bench can't supply -9V; it only supplies -12V, so really my regulated line is sitting at -7V for testing.)

Polluting the Groundwater


With all this wired up, the voltage regulators would show a spread of about 12V between their output pins.  However, by inserting the Arduino on the regulated low side between -7V and GND once again, the circuit wasn't able to provide enough juice to it.  I had the Arduino regulate itself from both the regulated lines at +5/-7, and once I wired it up, what would normally show a 12V spread was now showing something more like a 9V spread.  At least I managed to get the circuit to power up.  However, the voltage of the switching side of my NPN transistor (and even a 2N7000 N-channel MOSFET) relative to computer ground would only swing between +5 and roughly -0.5, still not enough to show the computer a proper Low signal for the keyboard scan codes.

Something must be polluting the groundwater, as the voltage between the low-side voltage regulator and the computer ground seems to vary slightly, and is much higher than it should be.  I can imagine, since the 7805 consists of various transistors and diodes, that it must not like providing an output voltage to something that is actually using it as a ground rather than as the high-voltage source.  I might have to go back to a voltage divider once again...

Approach #3 - One 7805, One PNP, and Something for the Negative Side


I thought to myself that it must be necessary to regulate the logic from the high side rather than the low side, since the 7805 probably doesn't like when its output is used as a ground rather than as the high-side input.  I decided to use the 7805 on the +12V side to both regulate the High RS423 signal and provide +5V to the Arduino.  This means that when the Arduino outputs +5V on its serial line, then the serial line can be driven up to the same value.  However, when it outputs 0V, then the serial line will need to be driven down to -5V, which could be provided by the low-side 7805 or by a plain voltage divider once again.  I needed to find a way to make the Arduino pin behave as if it would output either +5V or high Z, since with high Z, I could use a pull-down resistor to bring it to -5V.

As it turns out, for a PNP transistor, the emitter must be at a higher voltage than the collector, thus the emitter is hooked up to the voltage source rather than to ground as it is when an NPN transistor.  Thus, the base is always negative with respect to the emitter, and once you drop the base at least 0.7V from the emitter, current will start flowing through the base and the output of interest will rise to the 5V provided from the regulator.  However, when the Arduino pushes 5V to the PNP's base, then no current will flow through the transistor since the base voltage will be the same as the emitter voltage, thus the voltage at the output of interest will drop to the output from our negative-side voltage regulator or divider that is supposed to be providing -5V.

Here, it is shown that as the base (in blue) swings from 5V to 0V, the output of interest at the PNP's collector (in green) will swing from -5V to 5V.  The red line indicates the current through the base.

And if I need the RS423 signal polarity reversed, I can invert the Arduino's serial line with another transistor or an inverter IC.  Luckily, the signal did not need to be inverted; the output of the PNP transistor is normally low and pulses high, which is the same type of signal produced by the MS7004 keyboard.


Ironing Out Last-Minute Details


In practice, the -9V line was showing -8V with the LM7805 voltage regulator sitting on it, and its regulation treating 0V as high and -8V as ground means the regulated voltage was only sitting at -3V relative to ground.  This would not be enough to drive the RS423 signal.  Rather than grabbing a variable voltage regulator, I decided to simply build a plain voltage divider and connect its output to the collector of the PNP transistor.  At first, I decided to use a 4700-ohm resistor and a 6800-ohm resistor for the voltage divider, but the high-side input seemed to be getting squished.  I found that a 330 and 470-ohm resistor did the trick, but then realized I read the measurements originally from the wrong spot.  With the original resistors and the oscilloscope probing from the correct location, I found out that both these resistor configurations are suitable for the job.  (As for which one keeps -9V more true to -9V... I'm not sure, but I'd wager the higher-value resistors would do the job better.)

Before wiring up my Arduino, I needed it to send the correct values to the system in order for any key presses to be registered.  There are two things to consider here: first, the keyboard introduces itself to the computer (and indicates the absence of errors) by sending the bytes 0x01 0x00 0x00 0x00 at once following power-up.  Secondly, the ODT (Octal Debugging Terminal) of the PDP-11 only accepts a few keys from the keyboard, such as the numbers 0-7, the letter B (followed by 3 letters indicating a device driver location to load), and the letter R (indicating a CPU register number).  To avoid pulling my hair out debugging something that's not a problem, I referred to a graphic of scan codes for the LK201 PDP-11 keyboard interface and programmed the Arduino to spit out the number 4 (0xD0) once every second following the 4-byte handshake explained earlier, which itself is sent one second following power-up.


The schematic of my final keyboard emulator design.  Note the red stylized "LT" letters are a trademark of Linear Technology, just in case you are not familiar.  The LM7805 is a common part.

It still took me a few power cycles of the computer before I finally saw the desired output, especially making sure that RS423-compatible voltages would be output by the circuit, but eventually I saw a line of "4"s growing across the screen every second, corresponding to the serial pulse waveform being monitored by the oscilloscope.


The March of the Fours begins...

The Arduino and circuit involved to make keyboard emulation happen

The waveform of the number 4 as an LK201 scan code.  I just noticed that the low side only goes down to about -2V according to the oscilloscope, rather than the -4 to -6V expected.  Oh well, it worked regardless.

Thursday, January 11, 2018

Tensorflow, from Scratch to the Cloud

Having used plenty of pre-built machine learning models using Tensorflow and GCP APIs, and having gone through the pains of setting up Tensorflow on plain vanilla Amazon EC2 AMIs (not even the pre-configured ones with all the goodies installed already) and getting it to run classifications through the GPU and on Tensorflow Serving, I thought it was high time I try coding my own machine learning model in Tensorflow.

Of course, the thing most folks aspire to do with Tensorflow when starting out is to build a neural network.  I wanted to model my basic neural network based on the MNIST examples just to get my feet wet, but use a dataset different than MNIST.  There are many datasets on Kaggle to choose from that could fit the bill, but I decided to use one of my own from a while back.  It consists of skin tones found in pictures, cropped carefully and aggregated into a dozen BMP files.  Don’t question where I got these skin tone pixels from, but rest assured that a wide variety of skin colors were covered, captured by mostly nice cameras under ideal lighting conditions.  To be honest, it’s a little bit less interesting than MNIST, because instead of 10 distinct classes, there are only two: skin and not-skin.

Performance from All Angles


Previous research shows that converting the pixels from the RGB space into the HSV space leads to better neural network performance.  Luckily, this is easy to do using the Python Imaging Library.  However, performance of neural net computation (i.e. the speed of the training phase) has improved dramatically.  Just seven years ago, I made a neural net in Weka to train on skin tones.

Back in late 2010, it was fancy if you had a system with, say, 200 CUDA cores.  Back then, my Lenovo W510 laptop shipped with 48 CUDA cores.  Now, I have a roughly 18-month old Aorus X3 laptop that boasts 1280 CUDA cores.  Note from above that there are orders of magnitude more nodes in the Tensorflow neural net, and that the Tensorflow net is also considering all 3 values in the HSV space (quantized 0-99, thus 1 million possible values), and not just the best 2 out of 3 as in the previous research (quantized 0-255, thus 65,536 possible values).  The performance comparison is as such:

Model Type Time
Tensorflow (GPU 256) 90.9
Weka (CPU 5) 17
Weka (CPU 256) 750


Building the Network


Ultimately, the trickiest part of building the network was formatting the data in a way that would be meaningful to train on, and setting up the data types of the training data and variables to match up altogether and keep the training functions happy.

Ultimately, I ran into four pitfalls, one operational, one in code, one with clashing resources, and one out of stupidity or blindness:

  1. Can’t pass in a tensor of (1000000, ?) into a variable expecting (1000000, ?).  This one eluded me for the longest time because I was positive the correct tensor was being fed into the training function.  I finally saw the light when I decided to split up the training data so that training is not run on the entire dataset at once.  In doing so, I decided to have it train on 2,000 examples at a time, rather than the whole million.  Then, the error message changed, and it immediately became apparent that the error was being caused by the computation to calculate how many HSV values have [0, 1, 2, …] appearances in the dataset, not the calculation of the neural network itself.  How frustrating it is that such an amount of time was wasted trying to look into an issue that was ultimately caused by something merely calculating metrics that I had already discovered earlier and had since moved on from.
  2. Can’t allocate 0B of memory to the cross_entropy or optimizer or whatever it was.  My Tensorflow is only compiled to run on the GPU, I’m pretty sure, and doesn’t handle CPU computations.  I had a separate terminal window open with the Python <shell> to quickly run some experiments before I put them in the real training code.  However, having this open seemed to tie up resources needed by the program, so closing the extra terminal eliminated my memory allocation error.
  3. Uninitialized variable <something> for the computations of precision & accuracy.  For this, I had to run init_local_variables right before <either defining them in the graph or running them>.  It was not good enough to run this right after the line “with tf.Session() as sess”.
  4. Weird classifications.  I was expecting a binary classification to behave like a logistic regression and simply output “0” if it’s not skin and “1” if it’s skin.  Unfortunately, the neural network tended to return results more like a linear regression, and was giving me nonsensical values like 1.335 and -40.8987 for the single class.  I ended up changing the output layer of the neural network to reflect what I had originally, which called for two classes of “skin” and “not-skin”.  When I made this change, it was possible that the calculated values when evaluating any given valid HSV value could still be totally out of line with the “0” and “1” I would expect (and also not a cumulative probability adding up to 1), but at least by taking np.argmax() of the output layer, I can turn it into the outcome I was expecting.  And, it actually works very well, with precision and recall exceeding 97 or 98% with not a whole lot of effort.

All This Effort Just To Find a Better Way


Now, after having written my model with low-level Tensorflow routines, and trained it to the point where I thought it would return really good results, it was time to try to put it on the cloud.  For this, the output from Saver (a “checkpoint”) basically needs to be converted into a SavedModel.  One can do this by writing a whole lot of code defining the exact behavior of the neural network in the first place, but it seems to be much more efficient (with the end goal being to exporting the model to the cloud) to use an Estimator object to define the neural network.

And this doesn’t even take into account what kind of goodness lies before me if I simply decide to use Keras…

However, I wanted to see how feasible it was to write a converter from checkpoint to SavedModel using the standard Tensorflow APIs.  Scouring for examples and tutorials, and diving in to find out what exactly all this code was doing to see how I could shorten it into the most concise possible example, I realized there were new APIs that these examples weren't leveraging.

Basically, the two main principles involve:

tf.saved_model.signature
_def_utils.predict_signature_def(
inputs,
outputs
)

And:

builder.add_meta_graph_and_variables(...)

There are several signature_defs you can choose from:

predict_signature_def(...)
regression_signature_def(...)
classification_signature_def(...)

It makes the most sense to me to use the "predict" when you're looking to perform inference on a pre-trained example using your SavedModel, use the "regression" when want to solve an equation based on inputs and outputs you provide directly into the function (and not through a SavedModel), and use the "classification" for learning and inferring the class in which something belongs to, once again, by using examples directly into this function and not through the SavedModel.  Now, I haven't quite used the latter two functions yet, so these are purely assumptions, so I'll update this if I find it to be false later.

As for the meta-graph and variables, all you need to do to make a prediction-flavored SavedModel working on the cloud is to populate these three positional variables with the correct terms (not to mention your Tensorflow session variable):

 tags: just set to tf.saved_model.tag_constants.SERVING.

signature_def_map: This is an object conisisting of mappings between one or more Tensorflow tag constants representing various entry points and the signature defined above.  If you assigned the output of predict_signature_def(...) to prediction_signature, then you would probably want to use this:

{
 tf.saved_model.signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY:
 prediction_signature
},


Finally, define in your SavedModelBuilder an operation to reinitialize all graphs.  It is not a problem to run this one if you are mobing around a lot.

main_op=tf.saved_model.main_op.main_op())

From here on out, you just need to actually execute the exact Python scripts to train the model and build it into the correct format.  For the specific code I used to build this skin tone detector and convert it to a format usable for Google Cloud ML Engine, see my GitHub repo at https://github.com/mrcity/mlworkshop/tree/master/google-apis/cloud-ml.

Making Inferences With Your Model


Assuming you have gone through the steps of uploading the contents of your entire SavedModel directory to a bucket in Google Cloud, and you have gone through the steps of initializing an ML Cloud Engine "Model" pointing to this bucket, you now need to create a script that can call Cloud ML Engine to make predictions.  The Python code to do this is also very simple.  The main gist is that, unlike for other GCP API accesses from Python, you must configure the path to your service account using an environment variable rather than with the Python libraries.

Once you have done this, there is some very simple code from Google that demonstrates making an inference with your cloud model.  However, one thing they don't make clear is how to make multiple inferences at once.  If you do this incorrectly, you may run into an error such as Online prediction is not a matrix.  If this happens to you, study how the instances variable you pass into predict_json() must be an array of dictionaries:

instances = [{"hsv": [5., 49., 79.]},
     {"hsv": [4., 59., 63.]},
     {"hsv": [21., 67., 2.]},
     {"hsv": [99., 99., 99.]},
     {"hsv": [5., 35., 83.]},
     {"hsv": [5., 29., 71.]}]

One dictionary key must be defined for each expected input into the graph.  Recall that you defined the expected inputs in the inputs field of the call to predict_signature_def() earlier, and the keys required will be the same as the keys you defined in this field.  One bit of good news about the Cloud ML Engine is that it seems to only count one usage even if you specify multiple input values to the API (i.e. multiple elements in the instances array) in a single request.

Again, here is the pointer to my GitHub repo where you can see how all this fits together: https://github.com/mrcity/mlworkshop/tree/master/google-apis/cloud-ml

Thursday, December 28, 2017

A Cold Wind From the USSR - Part 1

In my retrocomputing adventures, I have sought things not purely based on style alone, but based on combinations of processor, wide 3rd-party adoption and support, and "interesting" factor.  Having picked up at least one system from each common type of processor (8088, x86, Zilog Z80, 6502, 68xx, 68xxx) and the schemes popular in the USA (IBM & compatibles, Apple, Commodore, Amiga, Atari, Tandy), I thought it was time to try for something from overseas.  Plenty of computers were made for the UK market, such as the Amstrad, Acorn, and BBC Micro, and Japan had many interesting varieties of computers offered by NEC alone, not to mention their other manufacturers such as Sharp and Fujitsu.

However, really not much is known (in English) about computers from behind the Iron Curtain.


A Brief History of Why This Is a Thing


One thing that is for sure: in the 1970s, the Soviet Union, in an effort to keep up with rapidly-evolving Western technology, decided to put an end to all the custom hardware implementations (and poor reliability that comes along with small-run manufacturing) and simply pirate designs from the West.  By studying patents for, say, the IBM/360 mainframe, the Soviets (and even domestic competitors to IBM) could begin to design similar computers.  Furthermore, President Nixon's detente saw relaxed export restrictions on computer hardware in 1974.  For what couldn't be imported or smuggled into the Eastern bloc physically (Zilog Z80-based systems were prevalent, such as a ZX Spectrum clone, as well as the "Pravetz" line of famously unreliable 6502 Apple clones), they would copy and manufacture their own designs of systems, especially involving DEC's PDP-11 systems and Intel's 8080 chips, and they would devote time to copying, rewriting, or reverse-engineering the popular software that went along with them as well.

Amidst the copying, which ran rampant in the Eastern bloc among both hardware and software, one interesting innovation was the advent of DEC's various PDP-11 architectures being shrunken down into a single chip, such as the K1801BM1, for which the Soviets were designing "microcomputers" by 1981.  At that very moment in history, IBM was taking the western world by storm with the introduction of their PC 5150, and totally caught DEC off-guard.  DEC scrambled to answer IBM, but the production of the DEC Professional line of microcomputers was too little, too late.  As such, most people remember PDP-11 systems as mini-fridge (or larger)-sized minicomputers, rather than desktop micros.  However, many Soviet microcomputers were based on these PDP-11 clones, including the "Elektronika 60", the computer which Tetris was originally programmed on.

Fast-forward to mid-2017, as I am having lunch at Google I/O, and a thought exploded in my head.  Instead of going with common Western European computers or Japanese computers which are popular among hard-core gaming enthusiasts and hardware collectors, why not go for something out of the Eastern bloc?  We never hear anything about those machines.


The Three-Month Sales Cycle


Shortly thereafter, I was paging through the For Sale section of the vcfed forums and found this interesting post offering three Soviet DVK-3 PDP-11 clone computers -- well, looks like I could turn my thought into a reality!  Now, at the time, I didn't know the first thing about Russian computers, not even that their power system runs at 240V rather than American 120V.  I discussed various issues with the seller for the next 3 months or so:

  • Does it work? - Among the three computers, one fails memory tests, two don't even come up.  Oh, wait; one works, one fails memory tests, and one just prints random dots on the screen.
  • How much does it cost? - It is not easy to search for such things unless you know Russian and/or can type it into Google, and then you are relegated to looking at online forums because their primary auction website is closed.  Some common low-end computers and peripherals are offered on eBay fairly consistently, though.
  • What will it take to ship it here? - About $1,050 was the original estimate from CDEK, the shipping company.  As such, this is a serious piece of computer, not just the little home-user hobbyist kit you can find on eBay.  Luckily, CDEK ended up charging less than that, but the savings were spent on other additions & protections noted later.
  • What kind of power does it take? - Well, 240V of course, but also at 50Hz, not 60Hz.  He offered me some solutions for a step-up transformer, and of course, the local makerspace has a fancy transformer and frequency converter I ended up using to run tests and find out exactly what I needed.
  • What kind of media will it come with? - Floppy drives & hard drive.  The floppy disks did not end up making it through customs.
  • Does the hard drive work? - The hard drive would not boot the OS, and the HDD controller seems to cause a memory error for that matter.
  • How do I interface with the hard drive? - The HDD is basically an ST-412 clone, 10MB MFM.
  • Does it have documentation? - There are various booklets that come with it.  The document called МАТЕМАТИЧЕСКОЕ ОБЕСЛЕЧЕНИЕ ЭВМ, which Google Translate said was "Mathematical Destruction of Computers," was not included, but I got a .DOC file of it which is evidently a PDP-11 assembly instruction reference.  I ended up receiving a "Passport" (technical reference) for the MC1201.03 mainboard and two programming references, one seemingly centered around DEC's RT-11 OS, errors, tests, and assembly language, and the other possibly discussing ФОДОС (FODOS) which seems to be a bit more custom.
  • What do I do if it needs to be fixed? - Donate it to a museum.  The rest of the world makes DIP-socket chips with pin spacings of 0.1" (2.54mm), but Soviets only made chips to that standard for export only.  Most of their serious-grade stuff was made at a "metric inch" of just 2.5mm, meaning that for all but the smallest chips, replacing any failing chip with an equivalent American version would be difficult. 
  • How can it be packed for such a long trip and not break tragically? - The seller built a large crate for me that exactly enclosed the original box and Styrofoam packaging that came with the system to begin with.
  • Which shipper will be the gentlest on it? - Well, I hope UPS won't bust it when they take it from CDEK.
  • Can we get shipping insurance? - No.  And also, CDEK wouldn't provide a crate for international use, thus why the seller had to build one.
And then, once I told him to take my money and go forward with shipping, besides the hope he wouldn't simply cut and run with such a large amount of money (particular just for the shipping cost alone), there were issues with getting such a thing shipped internationally and through customs.  The seller also went on a two-week vacation, which I was hoping he would not disappear nor perish from, but luckily for me, he came back and sent the package to me right away.  It was weird, because they needed a number of documents from him, which required things like his birthdate.  It's not like he's trying to get a passport, he's just trying to send out a computer!


And Now, the Fun Part...


Fortunately for me, on September 5, a large Russian crate was placed on the corner of the retaining wall on my front yard.  (I was hoping to catch the UPS guy to tell him to deliver it around back; what ended up happening led to a 90-minute struggle to lug this large crate up the rest of the hill, over the front stairs, and around side of the house.)  Once I got it into the garage, I unpacked it and made sure it was in one piece -- indeed it was! -- but was immediately overwhelmed with some sort of stench and got a runny nose quickly after opening the box.  There was a lot left to do before I would feel comfortable powering it on for the first time, so I put it all back in the crate and went back to work.

I must say the Russians have done a great job of communicating and sharing information about this system.  There are plenty of grainy photocopies of old books and schematics to be found online, and numerous forum posts that translate into broken English.  There is apparently even a community of people making new hardware for these systems, just as people make new hardware for old Commodores, Apples, and so on.  And, as one Imgur comment says, "In Moscow, these are just computers."  Not retro, but still things people use every day, so one would hope they'd be well-documented!


A large crate appeared atop my front steps.  The hydraulic lift cart is waiting patiently for me to get some bricks to elevate the 150-lb. wooden crate to a level where I can easily push it on.

The computer, with its original packing material

Stay tuned for Part 2, where I talk about what all it took to get this system checked out and powered on for the first time making a keyboard emulator for it so it can be remotely controlled.  (It's kind of hard to let someone over the Internet take control of its regular keyboard.)  I might write up the inspection process later on.

Thursday, December 7, 2017

Journey to a Fully Custom Pinball Machine - Part 2

From walking the show floor at Texas Pinball Fest 2016, I couldn't help but get the vibe that something novel and big would be in store for TPF 2017 -- something beyond the big but also typical/expected releases of commercial games such as The Big Lebowski and Ghostbusters (more on those later), but in fact the ushering in of a new era: totally home-brew and open-source pinball.  As the re-themed games became more impressive from 2015 to 2016, and with easy access to leaning about hardware, fabrication techniques to develop new things and restore/renew/improve on old things, and a rejuvenated fascination with pinball in general, it was not surprising to me in the least that we would see someone totally knock it out of the park like Scott Danesi did at TPF 2017 with Total Nuclear Annihilation.

However, just in case Scott wasn't there with his amazing game (for which I placed one of the pre-orders slated to ship sometime in 2018), I wanted to produce some work as well in order to show what could be done in this realm by just two people working hard together over a short period of time.  Unfortunately, while this article accounting my activities around the Wylie 1-Flip custom pinball machine is long overdue and probably should have been published way back in May, something big transpired that really made me put it off for a long time.  The basis for the electronics in Wylie 1-Flip was the Intel Edison development kit, since it was a convenient mix of an x86-based chip running Linux combined with an interface supporting Arduino sketches without having to wait as long as a Raspberry Pi does to boot.  However, as you may know, Intel decided to discontinue much of its 2017 hobbyist IoT line, leaving me lamenting the significant time invested into learning a dead platform and lots of memory-hogging tabs open in Chrome for my research.  (Well, I'm not really lamenting the time; after all, I did study Latin, a famously dead language, and continue to tinker with retro-computers that haven't been manufactured nor supported in decades.  However, using a discontinued platform doesn't exactly usher the art of pinball into the cutting edge.)


Where We Left Off


In case you missed Part 1 of this series, there was another goal besides making an awesome custom game to go along with the trend I predicted for TPF 2017: it was also to impress my coworkers and continue producing mind-blowing projects to show off alongside their top creative talent at various internal and external events.  You got a slight peek at the CAD design process of the game, and the frustration around installing the various mechanisms that go on top and below the playfield, but then also learned at a high level the enhancements and innovations that went into it.  Here is where I start describing the innovations at a lower level.

So I can finally close those Chrome tabs...


Despite that Intel Edison is no longer a thing, I wanted to still describe for you the stumbling blocks in working with the Edison platform that cost me so much time and trouble.  Granted, there's always a learning curve with anything, but here I was biting off a whole lot at once by trying to basically hand-route all the electronics for the game and write controlling logic for it using a platform I hadn't explored too deeply for its hardware capabilities before in the two weeks or so I had left between finishing the cabinet and actually taking the game to shows.  Yes, it was pretty insane, given that a "Makers Gonna Make" event was to be held on 3/2, followed quickly by TPF 2017 starting on 3/24.  However, Stacy decided to take a buyout package from her employer at the time and took a couple months off work, and believe it or not, she spent a great deal of her time off dealing with artwork and 3D modeling the various parts for this machine.

As the Edison supported a couple different modes of development (one involving the Arduino IDE and another in standard C++ with gnu/gcc through MRAA), I had to choose which one would suit me the best.  It looked like, at first, the Arduino approach would be simple because it was a familiar programming style and way less verbose than the C++ constructs of MRAA.  My first approach was to utilize interrupts to watch for changes in state on any of the sensors, but if I recall correctly, it was really only feasible to set up a whole bunch of rising-edge and falling-edge interrupts using gnu C++.  I did experiments for a long time in just trying to get reading a pin to work, but it is confusing how the pin numbers are laid out between the GPIO numbering scheme on the board, the Arduino IDE's view of the 20 standard I/O pins, and what the GPIO "files" are named on the file system.

// Arduino | Edison | MRAA
//       0 | 26     | 130
//       1 | 35     | 131
//       2 | 13     | 128
//       3 | 20     | 12
//       4 | 25     | 129
//       5 | 14     | 13
//       6 | 0      | 182
//       7 | 33     | 48
//       8 | 47     | 49
//       9 | ???    | ???
//      10 | 51     | 41
//      11 | 38     | 43
//      12 | 50     | 42
//      13 | 37     | 40
//      14 | 31     | 44
//      15 | 45     | 45
//      16 | 32     | 46
//      17 | 46     | 47
//      18 | 36     | 14

//      19 | 15     | 165
Sheer quackery.

The next big annoyance was that the event loop didn't even work properly when there were rising- or falling-edge interrupts triggered.  The basic premise here is simple; when an interrupt is triggered, raise a flag.  Then, when the event loop runs a condition to check if the flag has been set, run the desired action (e.g. score points, flash an animation, increment the ball counter...) and clear the flag.  By using rising- and falling-edge interrupts, I can monitor for the side of the button press I really care about -- the actuation, rather than the release.  However, by using such interrupts on the Edison, it would for some reason only pick up on the very first pin being monitored -- the left lane rollover switch.  At the time, I was only trying to wire up the three rollover lanes on top, and coded it up to read from these switches in this manner, but I obviously didn't proceed like that with the rest of the switches because functions for each rising edge on each specific I/O pin are not named explicitly in the rest of the code.  Instead, I resorted to pin change interrupts, monitoring all the I/O pins for any change whatsoever.  At least this way, it'll tell me which pin changed as a function argument which can get passed directly into an array, saving me from explicitly naming each pin.  The downside was that I had to get serious about my debouncing code, since interrupts were being triggered on the actuation and the release of the switch, and if you know anything about switches, it's possible there were 2 or 3 such toggle cycles registered by the I/O pin before the ball moved away from the area.

I figured that there's no point in using pin change interrupts; I might as well just read all the switches at once during the event loop, setting all the flags at once before they each get analyzed one at a time (acting accordingly for whoever is pressed).  It's not quite as pretty as using interrupts, but:

  • My early understanding of the disassembled code for Gottlieb's Gold Wings (1986) indicates they only use interrupts for countdown & event timers, and that they read pin statuses at some point in the event loop like this anyway
  • MRAA interrupt frequency is only about 100Hz anyway due to the complexity of what's involved in checking for interrupts on the Edison, so if my event loop runs faster than 100 times per second, I'm able to react faster than the interrupts anyway

In the table above, you might have noticed those Edison pin numbers, and especially the MRAA pin numbers, get pretty high.  This is because there are a whole bunch of other GPIO pins available on the system to be configured.  I spent a great deal of time, energy, and effort trying to figure out how to tap into all these extra pins, but was ultimately disappointed that all these extra GPIO pins were only there to feed into various multiplexers to change the purpose of the 20 standard Arduino I/O pins.  Because the processor inside the Edison wasn't engineered with exactly the same types of I/O registers as, say, the ATmega328, functionality such as serial UART, PWM, SPI, and even setting up pull-up or pull-down resistors in front of the I/O pins.  The ATmega chips handle this all internally, but the Intel processor had to externalize this into a ton of extra GPIO pins I thought I could hack to read from more sensors, but alas not without compromising functionality I need in order to keep the rest of the system behaving as expected.  To see what all the extra GPIO pins control and where the table above is codified, read this codethis article, and this thorough writeup.

In short, given that:
  • It's unfeasible to access GPIO pins outside of the Arduino realm for your own uses
  • The gnu C++ coding style requires a whole lot more variables to be created, casting to be performed, and just longer lines of code to be written than the Arduino C++ style
  • Despite the documentation here and even from Intel's own site, attempts at making an input pin also utilize an internal pull-up resistor through MRAA code (and possibly the initial line states if I recall, for that matter, meaning solenoids might randomly fire upon starting the system) never seemed to work, leading me to have to solder on my own bank of resistors to the board by hand and possibly compromise electrical reliability of the system
  • Evidently I was trying to do something with timer interrupts or just pure waiting around for some amount of time that didn't work in gnu C++ either, whereas in Arduino I could use a very simple delay() function
I ended up porting my pinball code back to Arduino C++ after doing all this work in gnu C++.

Then came the next pitfall: Edison Arduino C++ code can't send serial data, despite the best advice from here and here.  As I was using BriteBlox LED displays as my DMD of choice (also not a great idea for quality purposes, as they tend to flake out at times, probably due to voltage fluctuations in the presence of unstable power, which is largely but not 100% helped by attaching a huge capacitor between power & ground), they must be driven by serial signals in order to show anything meaningful.  I already had lots of experience writing Arduino serial routines to deal with BriteBlox as that's their native environment, but the Arduino implementation of Serial.write() on Edison just wasn't having anything to do with me.  This means I had to go back to gnu C++ once again (just for the graphics & serial routines), write a routine in there to parse the .BMP graphics files I utilized for DMD artwork, and then promptly send this over serial.  I ended up finding a way from within Arduino C++ to execute binaries with arguments, so each time I needed something put on the DMD (whether it's graphics or just a simple score change), I'd use something akin to this, explained here:

String hi = "/home/root/dmd score ";
hi.concat(ballInPlay);
hi.concat(" ");
hi.concat(score[player]);
hi.concat(" &");
system(hi.buffer);
Updating the score on Wylie 1-Flip.

Where do we go from here?


The next endeavor would likely have been to launch the Wylie 1-Flip game software upon powering up the Edison.  (Right now, you have to reflash the Arduino side of the processor with the program in order for it to start.)  However, considering that:
  • Intel Edison is discontinued
  • There are still electrical gremlins in the system causing random switches to appear toggled when nothing in the game is happening, meaning the pop bumper constantly goes off, the score & flipper changes at will, and the ball in play counter moves up on its own until your game is terminated
I'm keen on switching this project to the Android Things framework and hope that it'll bring about a less buggy, more electrically isolated hardware platform where I can write all my code in one place without so many confusing or deceiving constructs.

Nevertheless, here's what I have so far:


Unfortunately, based on the few times I've gotten to play the game thus far, it doesn't really seem all that fun anyway.  There are still some issues with the ball getting stuck and the shooter lane not working well that really hamper it (not to mention by far the most annoying electrical issues mentioned earlier), but maybe once I solve those issues, it would actually be something I would play.  As you can see, the legs are built in a special way so that the machine can really be expertly nudged, because while play-testing it in Visual Pinball, the game was much more fun if you pushed on the cabinet.


That's already a lot of hand-cut wiring, and there's probably still a ways to go! (At least judging by how the leg plates hadn't been put on yet, so there was probably still a lot being worked on)

I don't anticipate you'll be seeing a Part 3 of this series anytime soon -- maybe after TPF 2018 in March at the earliest, if I manage to switch successfully to Android Things and happen to solve problems in a noteworthy fashion.

Epilogue - And what of Ghostbusters or The Big Lebowski?


As for those two pins mentioned at the top of this article, neither has fared well: Dutch Pinball has been facing many difficulties shipping TBL to those who pre-ordered it, despite the passage of many years since the initial hype, and the value of Ghostbusters and many other games designed by John Trudeau has taken a hit (if only temporarily) since he was arrested for possessing child pornography outside Chicago in August just as Hurricane Harvey was rolling into the Texas coast.  Meanwhile, if anyone needs to just drop their Ghostbusters LE edition quickly, you know how to get a hold of me... ;) Sorry, I managed to find a Pro edition for cheap, and it's holding me over just fine.