Thursday, May 22, 2014

Getting your significant other's attention in a large house

The physics of sound in our house lend well to Stacy being able to shout up to me from downstairs, but not so well for me shouting back downstairs.  It makes communication inefficient, and often times she's not exactly watching for messages on GChat either.  I don't like going halfway downstairs when I'm trying to hold very skinny wires in a special arrangement when poking around for defects in LEDgoes, much less just to get a Yes/No answer on a simple question.

Recently, we picked up a Yamaha HTR-4065 stereo receiver for our living room.  Stacy was not happy with the sound quality of the original receiver, and hearing how the Yamaha can fill up the room while not vibrating every tiny appendage on your body with obnoxious bass, it was definitely money well-spent.  Hopefully it will add some more years to our natural hearing capability before we have to get hearing aids or whatever magical, mystical implant they develop by the time we're that old.  However, there is something else that piqued my interest about this receiver in particular that can solve the problem I proposed in the preceding paragraph.

You'd think the communication problem would be simple to solve; just install an intercom, right?  Sure, our house was built in the '90s, but the folks who built it weren't that wealthy.  Thus, it's not wired for an intercom.  We are both licensed ham radio operators, but neither of us listen for messages from each other unless we have previously agreed to, and usually the 2m or 70cm simplex frequency we use doesn't get a great range -- after all, we are two great big bags of water that excel at attenuating RF signals.  There are other aspects of our life, though, that elevate the Yamaha receiver into an excellent, effective means of communication: you can change its input source over WiFi.

Most of the time, the Yamaha is blasting music or TV when she's home (which also describes why I have such a hard time getting messages downstairs); thus it's always on.  All of our entertainment devices are routed through that receiver -- our turntable, "cable" (fiber?) box, DVR, Google TV, and PlayStations are switched at the receiver so now there's only one HDMI cable going into the TV.  We also hoard single-board computers such as the Raspberry Pi, Beaglebone Black, Udoo, pcDuino, and even some exotic builds of the Panda board that were never released to the public.  Most of the time, these boards sit idle, to be honest; now one of them is about to be put to work for me.

How will this be done?

Our devices are all Internet-connected, making them perfect targets for IoT/M2M protocols such as MQTT or AT&T's M2X platform.  Gone are the days when you need to use socket programming and open up ports on your router to have external devices connect to them; now I can run an application on my Android smartphone without bothering to turn its WiFi radio on, and through the magic of these M2M protocols, I can use it to control these devices in my house (or out of my house, or someone else's devices, or what have you).

The mobile app simply takes a recording from the microphone.  This recording is stuffed into a JSON object by storing the binary data as two letters representing each byte's hexadecimal value (done efficiently thanks to this StackOverflow post).  Hence, we can send the message in plain text and not have to worry about any special UTF-8 or Base64 encoding involved with sending binary values that would break the JSON object's structure.  (Not only that, but I wasn't able to figure out what string encoding they were using, and trying to guess & check in Android Java was becoming painful).  The string representing the audio recording gets submitted as the payload to the 2lemetry API.  The specific API call I use


ultimately publishes the message to their MQTT service, so that anyone subscribed to the prescribed {domain}/{stuff}/{thing} on their MQTT endpoint will receive the message.  Their service doesn't have a nice blob-storage capability (like MySQL has) that could straight-up store a binary file (but it does support JSON nicely), hence the need for a bit of additional legwork.  AT&T's M2X service should support blob storage by the end of 2014, at which time it might be worth re-evaluating the services.

An MQTT client, written in Python, is left to run perpetually on the Udoo single-board computer.  It subscribes to 2lemetry's MQTT server (or broker, in proper terms) to listen for new messages on the desired channel & pathway.  Once the mobile app finishes uploading the voice data onto the broker, the Udoo client will receive an asynchronous message and commence downloading the JSON-encoded data from the broker.  The JSON data (particularly the value containing the audio data) will be converted from "hex string" format back to a list of byte-sized numbers (0 to 255), and the series of numbers will be saved into an audio file.

The Yamaha HTR-4065 can be controlled via their AV Controller mobile app, but several eager developers have already taken the time to discover the ins & outs of the Yamaha stereo's API.  A great list of many (if not all) of the possible commands is here:  After the Udoo has converted my JSON-encoded message back into a binary file, it will be responsible for switching the HDMI input in such a way as to get the precise timing of when the message can start playing, then play the message, and finally switch the receiver back to the original HDMI input when the message is done playing.  This is achieved using a couple of POST requests, described in the previous link, to the stereo's API.  Sadly, it also means that the video will also go black (or switch to the Udoo's desktop) temporarily while the message is playing.

Do we expect this to be effective?

Stacy doesn't tend to look at the TV screen very much; she runs shows just to have background noise.  Nevertheless, the sudden change in the environment should, from a psychological perspective, wake up sensory neurons and put them on full alert, especially if what's currently playing involves loud music or an intense action scene: to suddenly hear silence will be startling.  Even if the scene is quiet, it'll be shocking to hear my voice coming through the speakers -- how did I just work my way into the show?  Since the receiver is totally remote-controllable, I can even power it on if it's off, and the message will still get through!  The only way for it not to work is if the power goes out, in which case my shouting will probably be more effective than usual.

This seems like the equivalent of a modern-day Rube Goldberg device!

Damn straight, and that's the way I like it.  It could have been worse, though: I might have had to use TCP sockets, or perhaps develop a robot to attach to the front of the stereo receiver in order to select the right inputs, plus use a camera with optical character recognition to determine what HDMI input to switch back to after the message has finished playing.  Even though it may seem to fulfill a niche market now, having to design a robot to press buttons would be the epitome of over-engineering and over-thinking.  My implementation should easily apply to several different applications, devices, and "user stories" as-is or with little modification.

Other things worth noting

  • 2lemetry now requires that the "password" of a user looking to authenticate themselves on the MQTT broker be hashed with MD5 before being sent to the server.  This threw me off for a couple days.
  • In Python, to convert the "hex string" back into a list of numeric values between 0 and 255, I used an interesting array operation to convert each pair of letters to integers, then each integer into a byte:intArr = [int(i, 16) for i in hexArr[0:len(hexArr):2]]
    audioByteArray = bytearray(intArr)

    It evidently wasn't good enough to write the integer array straight to the file: some of the data was getting misinterpreted and corrupted, so I needed to use the approach above.  There is a competing approach I could potentially use straight into the output file writer:

    from binascii import unhexlify
    output.write(unhexlify(''.join(format(i[2:], '>02s') for i in b)))

    However, its use of "format" leads me to believe it could be a bit less efficient.  I might do a benchmark to see which method runs faster.
  • I managed to debug the Android side of this application by downloading a hex editor to see if the file it saved locally matched what the MQTT broker had stored as the last message it had received in my particular topic pertaining to audio messages.
  • The length of the audio file doesn't need to be calculated.  I looked into MP4 headers for a little bit before realizing I can just spin up an instance of VLC in its own thread (with some particular arguments so that VLC only plays the audio file once and then quits), then write thread.join() so my Python client will resume regular operation when VLC is done.  The interesting question is what happens if a message comes in while VLC is playing audio already; will the MQTT client hold the message, or discard it?
  • Before using the MQTT client in Python, make sure you've run

    pip install paho-mqtt

    to install the required Python library.
  • The Android app is mostly a mashup of two previous applications I wrote: Street Beats and SpeechPipe.  This is why I could get it done in two weeks with so many other distractions, even amidst all the new learnings.  The features I did for this application will be rolled into SpeechPipe.
  • The Udoo allows several choices of Linux distributions to be installed on the device by means of SD card.  I chose Linaro Ubuntu 12.04 but had terrible luck getting any sort of audio to come out of it, even with VLC.  VLC was even seg-faulting on me!  I then tried xbmc media player, but didn't feel like going through the trouble of writing a plugin (even though it shouldn't take long).  Luckily, I already had an SD card in the Udoo formatted with some flavor of Ubuntu already, and that one seemed to work fine.  The MQTT client I wrote in Python had no trouble running on Linux.
  • The audio quality from whatever compression Android is using sounds terrible.  I need to keep playing with codecs & compression rates.  Nevertheless, a few seconds of audio is only taking up 8KB of space right now!
  • Sorry I'm not showing very much of my code in this post.  I need to clean it up and get it production-ready before I am comfortable open-sourcing it.
  • Every time we switch the configuration of our entertainment devices, it takes me about a week to figure out the new setup regarding the remote controls.  Now, to watch regular TV, I have to:
    • turn on the receiver with the Yamaha remote,
    • turn on the TV with the Samsung or FiOS remote,
    • switch to regular TV mode with the Google TV remote, and
    • wake up the FiOS receiver with the FiOS remote.

Thursday, May 15, 2014

The Options Scalper

The Options Scalper was another project that had a lot of potential during its time, but would have required some startup capital (VC attention) and more care and nurturing in order to really get off the ground.  Now I feel like the guys at Quantopian are quite on the right track, but they too have recognized that designing an intuitive interface for stock options analysis will be difficult... plus there's just so much data!!!  Nevertheless, Dan Dunn at Quantopian seemed to like the interface I provided, but they're designing the interface I ultimately wanted to provide so I'm just going to let them finish it up. ;)  (Plus, they know a little more Python than I do, which seems a great programming language choice for this task.)

The scalper was designed as a means to learn and study option decay in a graphical manner.  As weekly options were relatively new back in 2011, I decided those would be a great target because there's not a lot of data you need to store for weekly options before they expire, plus as being new instruments, perhaps one could pick up on potential arbitrage from market inefficiencies.  It was also to show the motion of the options themselves, or an entire spread, in a manner that brings it up at the same level as a stock's price.  Graphing tools I was familiar with back then typically would put the option way down on the Y axis compared to the stock's current price, and thus it's hard to draw comparisons when you're trying to study the dynamics of a cheap out-of-the-money call or a butterfly spread on a very expensive stock.  This solves the problem by smartly transposing the option's X axis ($0 price level) up to its strike price on the Y axis of the stock's graph.  At the time, there were 12 stocks whose weekly options I cared about; nowadays, I'd probably switch some of them out for more trendy or volatile securities.

The fundamental backbone of the options scalper is the options data.  A great source for options data is Yahoo! Finance, but writing a crawler & scraper for each of those pages could prove unreliable as everyone knows how many times sites change their markup, and I didn't want to have to deal that much with maintenance.  Luckily I happened to walk down the sidewalk at school once during my masters with another fellow who was an intern at Yahoo! complaining about how he perceived very few people were using the Yahoo! Query Language (YQL), so I ended up exploring this and finding out you can obtain stock and in fact options data using YQL queries.  Great!

Here's the exact query I used to fetch the prices of the stocks tracked on the Scalper:^DJX%22%2C%22BIDU%22%2C%22GS%22%2C%22C%22%2C%22PCLN%22%2C%22AAPL%22%2C%22SLV%22)&

This query simply returns the last trade price, and I store this information along with a timestamp into a MySQL database in order to present a graph of these stocks over the desired time frame.  Here's the query used to collect the options prices for AAPL in particular:*'AAPL'&

The options query collects only the data for the current month (weekly series & monthly series if unexpired), but gets much more information than the stock query.  It fetches the options's symbol, whether it's a put or call, its strike price, last traded price, change from yesterday's close, bid & ask price, volume, and open interest.  In general, I ran the collection cron job every 15 minutes.  It was a compromise between smooth data points & exceeding the storage limits on my server. :-P

The other interesting facet about the Scalper was the user didn't directly have access to the underlying data, much like the model Quantopian has adopted.  In fact, Dan told me that dealers of the historical stock & options data are likely to work with you for much cheaper if you don't directly redistribute the data, so simply being able to query & visualize the data without the user needing to know the value of every single point is important.  This also means you need a really good front-end, which I whipped up using JavaScript and Flot.js charts.

The boxy elements on the front page were meant to summarize the stock price and the 4 nearest to at-the-money contracts in the front-week expiration.  The different elements would highlight in red or green depending on what had moved up or down.  With no new data in years, they all pretty much show N/A.  I had begun experimenting with some different graphical elements on before I abandoned the project.  The top bar, listing the ticker symbols tracked, is the gateway to all the historical information hosted on  From here, you can see a graph of the stock, and select an expiration day to play with the options for that particular week.  Once you select an expiration week, it will load the option chain on the bottom left.  You can click on any strike price to see the motion of that option graphed on top of the stock's motion in a graph right below the main stock graph.

By far, the most interesting feature is the big white graph at the bottom.  This is where you can click any of the +1/-1 fields in the option chain to simulate going long or short on any of the puts or calls in the option chain.  You can put together spreads on the fly including verticals and lopsided butterflies, or whatever you desire (as long as it doesn't involve shares of the underlying security), and the total value of that spread over the course of the whole week will be represented in the white graph.

I ended up using some of this data in a project for my master's where we tried to predict which iron condor spreads would be most ideal to sell on a given week, assuming low market volatility.  None of the visualizations were incredibly useful for it, but we were able to churn through a lot of data real fast and feed it into some pattern analyzers and machine learning algorithms.  As it turned out, the classification data worked very well on the real-world example just in time for our final presentation, but the following week, the market dynamics changed and our algorithm would have lost us a bunch of money!  There must have been an earnings surprise we didn't account for before picking.

Thursday, May 1, 2014

Three Obscure Pricing Games On "The Price Is Right"

Everyone has their favorite pricing game on The Price Is Right, whether it be Plinko, Lucky $even, Hole In One, Clock Game, or whatever the case may be.  Some may fancy retired games such as Hurdles (1976-83) or Super Ball!!! (1981-98).  Long-time producer Roger Dobkowitz once said "every pricing game is somebody's favorite," and with over 100 pricing games played over the show's long history, it's been sufficient to keep millions of viewers still watching loyally every morning.

For as many memorable games as the show has, they have certainly had some, shall we say, "experiments" that did not go so well.  Some games were dull; others simply did not mesh well with the spirit of Price is Right (like Professor Price or Double Bullseye).  Still others have grown old (like Barker's Bargain Bar), were killed off due to inflation (like Walk of Fame), were plagued with mechanical issues (like Hurdles), or became too confusing for today's contestants to win with regularity (like Hit Me).  And it's not for a lack of effort on the part of the staff; some of these involved detailed game props or an interesting game play, and as such, they still make for exciting fun to watch--if not for the gameplay, then for the "vintage factor" or even the "kitsch factor" (though it is hard to find kitsch on Price is Right if you ask me!).

Most of these retired games can be viewed by the public by means of tape trading or on YouTube.  As such, the game where two people bid on a car (Double Bullseye), and many other games which you may have never heard of, can be seen and enjoyed by die-hard fans.  And, this month, thanks to venerable emcee Wink Martindale (who I'm surprised is this into Price Is Right and its fans to have done this!), three very short-lived games from 1978 have now been recovered from the vaults to be mused over by fans.

For 35 years, these three games had remained unseen since they originally aired because Game Show Network did not air them for one reason or another and no original broadcast tapes of these episodes had appeared on the trading circuit.  These games include Shower Game, Telephone Game, and Finish Line.  (By the way, Wink recently posted another video detailing the original rules for Punch-A-Bunch.)

Tell me more!

If you plan to watch these videos with all the curiosity of someone whose imagination has intrigued them about these games for years, you'd better not read on.  However, I'm guessing most of you would go either way, and perhaps some of you wouldn't even make it through the whole video. :-P  So, here I shall describe my initial reactions to these videos.

First, I started with Shower Game.  This game was only played ten times, all airing between September 4 and November 30, 1978.  The man who played Shower Game on its premiere sported the quintessential '70s style: poofy hair, mustache, and tinted glasses indoors.  His shirt, with big pointy collars, was already unbuttoned quite far, and became even moreso once Bob told him he would be playing the Shower Game.  The game is played for a car, and its prop is quite large.  There is one price listed above each of six shower stalls, and the contestant must enter the stall and pull the cord to "start the shower."  The shower could consist of confetti (in 3 stalls), 100 $1 bills (in 2 stalls), or a large prop resembling a car key in one stall.  The game continues if the contestant draws confetti; once they win an actual prize, they stop.  As for the key, it doesn't fall all the way down; it's on a short hook so it simply swings out.  It was amusing to see the camera swing from price to price over this huge prop in a "Jeopardy" style, as if it were revealing the categories in the round.

Anyway, it took Bob Barker longer to explain the rules of the game than it took the contestant to jump in a stall and win the car.  You would expect more from a prize reveal, and as it's up to the contestant to pull the rope, there's really no way Bob could have built it up in such the way as he was masterful at doing.  Not only that, but what if the contestant could look up and see what prize was there?  I'd have to imagine the big car key was hard to hide.  Nevertheless, it was great to see the contestant win, but it would have (hopefully) been more amusing if he'd have gotten some confetti on him first.  The game was obviously based on sheer dumb luck, with no strategy involved; not to mention, the shower stalls evoked imagery of the Holocaust among some viewers.  This would explain its quick demise.

I hastily proceeded to watch Finish Line, which had been posted next.  Finish Line was played 16 times, seen from February 21 through September 25, 1978.  Among the trio of obscure games I'm presenting, this was the longest-running one.  This game definitely had flair, and would have been a great replacement for Give Or Keep (seeing as how the rules were identical), had it not suffered from mechanical problems and just taken too darn long to play.  Give Or Keep itself was not a flashy game by any means, but the essence of the game in Give or Keep was not lost in all kinds of superfluous activities such as moving a yellow bar slowly down the track to indicate the Finish Line (i.e. how far the horse has to go, which is based on the sum of the prizes the contestant thinks has the lowest price) and then moving a tacky plastic horse slowly down this track until it either crosses the finish line... or doesn't.  Give Or Keep revealed this information right away; instead, on Finish Line, it takes a long time to learn the outcome.

On the surface, both games have quite uninteresting gameplay compared to some of the more modern games, which boast more challenges than simply picking the cheapest prize among three pairs of prizes.  Nevertheless, it's still a solid concept, totally fits the Price is Right theme, and is a quintessential game.  They might have luck bringing Finish Line back, in particular, if someone would rework the prop to use solid-state electronics and stepper motors instead of analog electromechanical devices (an entire wall full of relays, for instance) which are more prone to failure and inaccuracy.  (And, they'd need to pick a theme with a broader appeal than horse racing; plus I can see how in later years, as Bob became more of an outspoken animal rights activist, he'd want nothing to do with horse racing anyway.)  As for Give or Keep, that one's pretty much toast.  The staff tried for years to get it retired, probably because it was so simplistic, and ended up finally getting their way after 18 years of it in the Price is Right rotation.  At least one other game (Trader Bob) was built around the same concept, but it too only managed to last for a couple years.

Finally, I viewed the game among these three I was most excited to see: Telephone Game.  Even though it was only played three times, all airing within November 1978 (and played more times than only Professor Price), it has the most novel gameplay among these games.  Several elements contributed to this factor.  In a cue from Five Price Tags, you had to win at least one choice of prices among the three that could be for the car; except on this game, you had to buy two grocery items from among four with a budget of only $1 and save at least 10 cents to make a phone call on the payphone.  Then, in a twist unlike any other game I can think of, Bob would present the contestant with a phone book containing the price of the car in dollars, and the price of the other two items in dollars and cents (prizes less than $100, as new cars cost only about $4,000 or $5,000 back in 1978).  The contestant would then dial the price they chose on a rotary phone, and after a painful amount of waiting around, a telephone would ring next to the prize that had the price the contestant picked, and the model standing next to that prize would answer the phone.

The game started out solid, with picking out the grocery items, but then quickly went downhill from there.  Bob giving the contestant the dime seemed hokey, and then he had to tell the contestant to actually put the dime in the phone!  That seemed superfluous because it doesn't actually need to happen (unlike Master Key, where the contestant needs the key to find out what prize they've won).  Finally, in another stroke of "dumb luck," the contestant managed to win the car, only after arbitrarily picking the right price and waiting quite a while to see if he was right.  It was reminiscent of waiting for a cell phone call to go through when the network is really busy; sometimes it takes a while for the phone to start ringing!  Maybe they only strung this game along through three playings to see if anyone would actually win it, because after this game was finally won (on the third try), it was retired and never to be seen or heard from again until clips of its third playing finally resurfaced in 2014.  This one could possibly be revived if they could somehow remove the painful elements of watching the game unfold; perhaps this means taking away the whole "telephone" theme and playing through it just for what it is.  However, then it'd become a little too close to Five Price Tags except that you would win something on the first try as opposed to continuing to try for the car (or getting nothing).

Hey, that was cool!  May I have another?  How about that Punch-A-Bunch?

Since Wink Martindale has posted additional rarities beyond these three obscure games, I'd like to go into one more as a bonus.  The first game played on The Price is Right to offer a big cash prize had remarkably different rules in its first 11 playings than it does now.  The original rules for Punch-A-Bunch throughout 1978 had a higher "suspense beta" (where "beta" is a stock-market term for volatility) than it does now.  While Punch-A-Bunch is (and has been for a long time) a very exciting game, it could have been originally super-suspenseful or super-boring.  The contestant gets to choose which one of the four prizes they would like to attempt to earn a punch with, and then before punching, they must pick a letter on the top row among "PUNCHBOARD."  Each letter contains a number from 1-10, so if the contestant picks a disappointingly low number, it becomes lame (kind of like in the mid to late '80s when someone wins the $6,000 showcase instead of the $20,000 showcase).  This number will then be multiplied by their punch from among the 50 slots -- where nowadays there are values between $100 to $25,000, back then there were "DOLLARS", "HUNDRED", and "THOUSAND."  Thus the "suspense beta" is pretty high if someone were to pick the number 10 from among PUNCHBOARD, then pick "DOLLARS" (and as such, would be a huge disappointment!), but less so if the contestant picks the number 1 and then goes on to pick THOUSAND.

The other thing that makes the original Punch-A-Bunch rules wildly different is you don't know how many punches you will get ahead of time!  Since you picked prizes one at a time, you pulled your punch after each prize.  If you wanted to give your cash winnings back to try for another punch, you ran the risk of losing the other prizes and not winning another punch, thus walking away with nothing.  That was a nifty element of the gameplay, but ultimately slows it down and adds to tedium while watching it.  However, it does introduce a bit of strategy most contestants probably didn't pick up on: you would want to pick the prize you are least confident about first so you'd be less likely to give up your cash winnings and then not get another chance. Punch-A-Bunch started 1979 with the format we all know and love today, which definitely plays through quicker yet still provides plenty of entertainment and suspense.  The rule change was wise, as the game has been a favorite among fans ever since.