Adding Stock Quotes to a BriteBlox Display

Background

People interested in the stock market might be inclined to have a cool stock ticker in their room or office.  While it's not quite as useful as a whole bunch of monitors displaying entire panels of quotes and graphs, it does provide at least a mesmerizing distraction, not to mention your room now looks at least a little bit more like a real trading floor in a big office.  This was the original use case for BriteBlox, the modular LED marquee designed by #DoesItPew and myself.  Not only is the marquee itself modular (i.e. you can configure it in various lengths, heights, and shapes), but ideally the control software is modular, leading to a design where any coder can develop plugins to drive the board from any data source they can convert into animated pixels.

Unfortunately, we haven't gotten the software quite to the point where it is incredibly easy for third-party developers to write plugins; right now, they would actually have to integrate the feature into the program with an understanding of what needs to be called where in the UI thread and how it ties in with the display thread.  But, as we hope to have shipping for our Kickstarter backers wound down by the end of this week (finally!), there should be more time to flesh out this modular software design.

That being said, there was another challenge: despite actually developing this marquee with the idea of displaying stock quotes, there was the problem of actually finding a legitimate source of quotes that's free and not on a delay.  For those who haven't done this before and are searching Google for the first time, there are many money-hungry demons thickly spamming Google's search results pages with false promises of a useful product for my purpose.  And, despite that I have several brokerage accounts with various institutions, it's hard to find one that actually provides an API for stock quotes unless you meet the $25,000 minimum equity requirement usually required for day trading.  You might get lucky and find one that interfaces with DDE for ancient versions of Microsoft Office or Visual Basic for Applications, but it's been a very long time since I've ever touched any of these and don't want a service that requires a whole lot of other dependencies for the user to install.  The most general approach to take seems to be to provide an RSS feed reader.

The Best Solution For General Cases

Really Simple Syndication (RSS) is useful for quickly scanning sites for updates.  Most of the time, it is provided for news sites or sites whose content changes frequently.  Of course, stock quotes also change frequently; since RSS offers "pull style" updates (it doesn't get updated until you refresh), it works well with our protocol since there's no need to manage what particular symbol appears where and what to do with an updated price.  Sometimes, on days when one particular stock is trading volume an order of magnitude above others, you'll see tickers dominated with quotes from that stock.  Our mechanism won't do that because each ticker symbol is represented on each update, and updates occur each time the marquee is done scrolling all messages.

Python can easily parse RSS feeds by means of the feedparser module which you can install with pip.  Once you download the feed, you need to parse through its XML with xml.etree.ElementTree.  This is all pretty easy, but if you're using the NASDAQ feed in particular, you'll notice the quotes are embedded in ugly, unwieldy HTML.  It is difficult to parse because they do not provide unique identifiers as to what information is contained in what table cell, so you have to do a little bit of exploration ahead of time in order to see which cells contain what you want.  Here is how I'm currently handling the parsing, from end to end:

                try:
                    d = feedparser.parse(self.feedURL)
                except:
                    console.cwrite("There was an error in fetching the requested RSS document.")
                    self.active = False
                    continue
                info = []
                feed = "<body>%s</body>" % d.entries[0].summary_detail.value.replace("&nbsp;", "")
                tree = ET.ElementTree(ET.fromstring(feed))
                root = tree.getroot()
                # Find the last updated time
                last = root.find(".//table[@width='180']/tr/td")
                info.append("%s%s  %s" % ((setColor % yellow), last.text.strip(), endColor))
                # Find all the quotes
                counter = 0
                for elem in root.findall(".//table[@width='200']"):
                    for elem2 in elem.findall(".//td"):
                        for text in elem2.itertext():
                            idx = counter % 13
                            if idx == 0:  # Ticker symbol
                                info.append("%s%s " % ((setColor % yellow), text.strip()))
                            if idx == 3:  # Last trade
                                info.append("%s %s" % (text.strip(), endColor))
                            if idx == 5:  # Change sign
                                sign = text.strip()
                                info.append("%s%s" % ((setColor % (green if sign == "+" else red)), sign))
                            if idx == 6:  # Change amount
                                info.append("%s %s" % (text.strip(), endColor))
                            counter += 1
                # We're done parsing, so join everything together
                newMessage = globals.html % ''.join(info)
                # FIXME: For now, this will be Message #1
                globals.richMsgs[0] = newMessage

 Now, the next challenge was to come up with the means to integrate stock quotes with the usual message display thread.  Twitter is a unique case; since its API updates my app via push notifications, it can tell my app to save the incoming message into the next available message slot in the queue, and then when it's time for the serial output thread to refresh what goes out onto the matrix, any messages currently in the queue get shown.  Ideally, it'd be nice to find a stock API that behaved in a similar manner, despite that it'd expose us to possibly showing multiple quotes for the same stock in one run through the message queue -- there are ways we could work around this if needed.  However, since this is a pull-based notification, I needed a way for the serial output thread to signal pull-based threads to refresh their data.

There were, in fact, two ways I debated on while trying to develop this feature: 

• Create an array of flags that the serial update thread raises before each text update; then, each respective class instance who registered a flag in this array will update its message in the appropriate slot
• Tie a reference to a class instance or flag in with the RawTextItem objects (derived from Qt's QListWidgetItem) initialized whenever you make a new message in Raw Text mode; this would be empty for raw text and push-based notifications but would be populated for text requiring pull-based notifications, and would require the serial input thread to iterate over these items typically stored in the UI class instance

Ultimately, I settled on the first design.  A plugin developer would be required to know where to register the flag in either case, and I thought it'd be better to make that an array defined in the LEDgoesGlobals module rather than requiring people to pull in the UI class instance just to have access to that.  Also, they're not having to add extra data to something that gets displayed on the UI thread.  As you can imagine, my biggest pain points were simply refactoring & debugging all the little changes made throughout mostly the data structures that are used to pass the bits from the computer onto the marquee.

In the process of writing this big update to support pull updates between refreshes of the matrix, I also cleaned up code in the serial update thread that was iterating through the XML tree twice for no good reason other than just to put the letter(s) into the appropriate color data structure (i.e. red or green).  I also started to make this modular by defining colors in LEDgoesGlobals, but there are still many parts of other code that treats colors individually by name rather than agnostically (by simply sending a particular color data structure to a particular set of chip addresses).

As with most things, there is still some work left on this code before it's "perfect," but it's available on our GitHub right now if you are capable of running the BriteBlox PC Interface from source and would like to check it out.