Thursday, June 28, 2018

Validating Pre-Made Tensorflow Estimators Mid-Stream

In Francois Chollet’s book Deep Learning with Python, he stresses the importance of utilizing a separate validation set of data while training a machine learning model in order to test periodically (say after every epoch) that the accuracy on something else besides strictly the training data (e.g. this validation set) is in fact improving.

Machine learning models are subject to learn relationships that have nothing to do with the problem at hand.  For instance, a model tasked with trying to determine which way a military tank is facing might end up making assumptions based on whether it is day or night.  This is often a result of trying to eke out the model’s maximum performance, say by optimizing for the smallest value of a loss function.  However, what ends up happening is that the model overfits on the training data, which means it loses its generalization — its ability to predict the correct outcome of new samples or examples that we as humans would intend for it to classify.

One way to compensate for this is to withhold a separate set of data, known as a validation set, so that we monitor for not just the loss we are trying to optimize for, but also the performance of the model on a separate set of data it hasn’t seen.  While the loss function may be decreasing, thus insinuating the model is getting more accurate, you might in fact see that the performance on the validation dataset stops improving, or in fact reverses course and gets worse.  By evaluating a validation dataset throughout training, we can figure out when to terminate training for the best results.

Pre-made (Canned) Tensorflow Estimators

As each day goes by, there are more and more benefits to using canned estimators.  For instance:
  • They manage for you which parts are best run distributed vs. a single machine
  • They can stand on the shoulders of Tensorflow Hub modules, allowing you to focus on adding just one or more instances of a simple type of layer to much more complicated models pre-filled with highly trained and optimized parameters
  • You focus on the configuration of the model as a whole, not configuring so many details of every specific layer

However, pre-made Estimators do not offer many functions other than train, evaluate, predict, and exporting as a SavedModel.  And, looking for extensibility of pre-trained Estimators can be tricky: neither the train function nor a pre-trained Estimator object offer a direct, apparent way to run evaluation of a validation dataset mid-stream during training.

But Chollet said we need to validate frequently!

As it turns out, Tensorflow allows you to run side functions (invoked by callbacks) during train, evaluate, and predict.  These can serve any purpose, such as hooking up to the Twitter API to post memes to Twitter at certain intervals during the underlying operation (at either specified intervals of time or steps), or as we are most interested in doing, running an evaluation alongside training.

The feature is specified in these three functions by the hooks parameter.  Hooks are instances of the SessionRunHook class that allow you to define custom code that runs at the start and end of each Session, as well as before and after each step.  (There are built-in functions that will help you count steps or time so you’re not executing your desired action at every step.)

Let’s take a look at a complete SessionRunHook example for a pre-made Estimator.

class ValidationHook(tf.train.SessionRunHook):
    def __init__(self, parent_estimator, input_fn,
                 every_n_secs=None, every_n_steps=None):
        print("ValidationHook was initialized")
        self._parent_estimator = parent_estimator
        self._input_fn = input_fn
        self._iter_count = 0
        self._timer = tf.train.SecondOrStepTimer(every_n_secs, every_n_steps)
        self._should_trigger = False

    def begin(self):
        self._iter_count = 0

    def before_run(self, run_context):
        self._should_trigger = self._timer.should_trigger_for_step(self._iter_count)

    def after_run(self, run_context, run_values):
        if self._should_trigger:
            print("Hook is running")
            validation_eval_accuracy = self._parent_estimator.evaluate(input_fn=self._input_fn)
            print("Hook is done running. Training set accuracy: {accuracy}".format(**validation_eval_accuracy))
        self._iter_count += 1

Our purpose above is to run evaluate on the Estimator during training after a predefined number of seconds (every_n_secs) or steps (every_n_steps) elapses.  To this end, we can actually pass in the Estimator itself as an instantiation argument to our ValidationHook object, rather than trying to create a new Estimator in this scope.  You will see in begin() that some variables get initialized just at session runtime.  The before_run() function defers to a Timer initialized in the constructor that dictates whether or not to run our desired operation (the validation evaluation) in the after_run() function.  Without asking the timer if it’s time, the evaluation in after_run() would run after each step, and that would waste a substantial amount of time.

Now, actually running the Estimator with intermediate validation steps is a simple matter of defining the correct parameters in train().

settings = tf.estimator.RunConfig(keep_checkpoint_max=2, save_checkpoints_steps=STEPS_PER_EPOCH, save_checkpoints_secs=None)

estimator = tf.estimator.DNNClassifier(

    hooks=[ValidationHook(estimator, validation_input_fn, None, STEPS_PER_EPOCH)],
    steps=STEPS_PER_EPOCH * 40

What’s happening here is that I’m writing a RunConfig seeking to reduce space consumed on the hard drive by keeping a low number of recent checkpoints on hand.  Then, I configure it to save checkpoints at every STEPS_PER_EPOCH steps and make sure to explicitly disable saving checkpoints periodically, since defining both parameters is disallowed and the default is to save a checkpoint every 600 seconds.  Then, in train(), just make sure to specify the hook you built and the config you defined.

NOTA BENE: If you do not specify the RunConfig to save a checkpoint at your desired interval, the weights will not be updated when you run evaluate() inside the hook, and thus your validation performance will not appear to change until another checkpoint is written.

Visualizing Validation Performance with TensorBoard

TensorBoard can easily graph the loss over time as the model gets trained.  This is output by default into the events file that TensorBoard monitors.  However, there are a couple things you might wonder about:
  • How can I show other metrics, such as accuracy, or precision and recall, over time?
  • If I’m showing loss every 100 steps, and TensorBoard is picking this up to graph it, how do I get it to only convey my desired performance metrics at the times when they’re actually calculated?
As it turns out, the validation accuracy should already be available for you in TensorBoard.  Under your main model's output directory (defined by model_dir), there will be another directory called eval where the validation accuracy metric consumed by TensorBoard will be placed.  You can overlay the validation accuracy with the graph of loss, and/or with any other such tf.metrics collected in other log directories.

But if you want additional metrics, especially the ones defined in tf.metrics such as precision and recall at top k, there is a function called add_metrics() that actually comes from the tf.contrib.estimator module (rather than tf.estimator).  This allows you to define a metrics function returning a dictionary of your calculated metric results.  The good news is this function returns a brand new instance of Estimator, so you don’t have to worry about your original Estimator in training trying to constantly run these evaluation functions.  But the best part is even though this is now a separate Estimator object, all the parameters of the original Estimator are conveyed to this new one as they are updated by the checkpoint writing process.

To add additional metrics to TensorBoard, add a function to your code that complies with metric_fn as such:

# This is the function that meets the specs of metric_fn
def evaluation_metrics(labels, predictions):
    probabilities = predictions['probabilities']
    return {'auc': tf.metrics.auc(labels, probabilities)}

# And note the darkened modifications below:

class ValidationHook(tf.train.SessionRunHook):
    def __init__(self, parent_estimator, input_fn,
                 every_n_secs=None, every_n_steps=None):
        print("ValidationHook was initialized")
        self._estimator = tf.contrib.estimator.add_metrics(
        self._input_fn = input_fn

    def after_run(self, run_context, run_values):
        if self._should_trigger:
            validation_eval_accuracy = self._estimator.evaluate(input_fn=self._input_fn)
            print("Hook is done running. Training set accuracy: {accuracy}".format(**validation_eval_accuracy))

NOTA BENE: While it is convenient that you only need to add a couple lines in your subclass of SessionRunHook, don't forget to add the model_dir to the parameters you use to initialize the parent Estimator object, or else TensorBoard might not be able to pick up on any of your metrics at all for both training and evaluation.

What about tf.estimator.train_and_evaluate() ?

This is a function provided in the estimator module itself, and is not exposed in pre-made Estimators.  However, it does take your Estimator as an argument.  And so it is, with very few lines of code, that you can run interleaved training and evaluation.  After defining your Estimator (omitting the hooks parameter this time), don't bother writing any hooks at all.  Just do this:

estimator = tf.contrib.estimator.add_metrics(estimator, evaluation_metrics)

train_spec = tf.estimator.TrainSpec(input_fn=train_input_fn, max_steps=5000)
eval_spec = tf.estimator.EvalSpec(input_fn=validation_input_fn, start_delay_secs=60, throttle_secs=60)

tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)

evaluation_metrics is the same function as defined earlier.  start_delay_secs defines the delay from the very instant your model begins training.  If you put even 1 here as the value, training probably won't have even made it beyond the first step before an evaluation is performed.  And, throttle_secs defines the minimum delay between evaluations.  If you put 1 here, chances are you will perform just a single step of training in between evaluations.  The default values are 600 and 120, respectively.

Unfortunately, the implementation to date of train_and_evaluate() seems incapable of counting steps rather than time, so it will take some empirical measurement to find out exactly how much time an entire epoch takes to run if you care to line up evaluations with epochs.  However, this approach should be very scalable onto distributed systems, for those looking for very fast training.

One other thing that seemed a bit odd to me is that, in the course of running all these experiments, I would often times delete the log directory and reuse the same name.  Most times, with the hooks method, TensorBoard would pick up on the new run and work just fine.  However, using this feature just above, TensorBoard only seems to pick up on the new run about half the time.


And, of course, the Tensorflow documentation

Thursday, May 31, 2018

15 Days in California: Google I/O and IoT World 2018

Earlier this month, I got a chance to attend two conferences in the Bay Area: Google I/O and IoT World.  They are two very different venues, as Google has a lot for developers and the media to consume, experiment, discuss, and play with, and IoT World has some good content for developers but also helpful strategies for executives to understand where things are going.

Of course, from the Google I/O keynote, everyone has been the most impressed with the Google Assistant handling phone calls on behalf of users.  This is something that has held me back in the past from getting things done; there’s a level of “activation energy” I have to want to talk to someone; given my ability to be such an easy push-over and not either quick enough or persistent enough to try to contradict with or argue with someone, so I often put off phone calls or worry about how the conversation will go on.  Having someone do the talking for me will free up some of my brain cells from worrying about this.

However, most people brought up the example where the Google Assistant calls a hairdresser “on behalf of a client” in order to schedule a hair appointment.  That to me seemed rather pedestrian and mainstream.  Yet the very next example they showed to us at the keynote, which I have not heard as many people talking about, was where the Google Assistant tried to book reservations at some sort of Asian restaurant with a non-native English speaker with a very thick accent on the phone.  The human on the line didn’t speak perfect English, and also couldn’t remember conversational context to save their own life.  It would have been frustrating to deal with, to try to keep the conversation on track, but the Google Assistant handled the situation adroitly, correcting the restaurant phone answerer whenever they incorrectly stated details of or misunderstood what the user wanted.  Of course, speculation has run wild about the potentials to abuse this technology — one thing I can think of right away is the ability to design complex social engineering schemes, especially when the Google Assistant is resistant to responding with emotion, stuttering, hesitation, or anything to tell you whether they’re being honest or not.

There were several other cool things mentioned at the I/O Keynote, but one thing that seemed pretty silly to me was spending about 5 minutes explaining the use of machine learning to adjust the brightness of your phone’s screen.  I have no trouble spending 3 or 4 seconds to adjust it manually, and rarely have to adjust it anyway unless I’m in a very, very dark room where even the minimum brightness is usually still too bright.  Don’t get me wrong, though; in general, the explanation of Android P was so compelling that I wanted to download it onto my primary phone right away, despite the advice against that by Stacy, an Android GDE.  Nevertheless, I saw quite a few people who appeared to be running Android P already throughout the rest of the conference.  Perhaps they downloaded it ahead of time, or after the first day of Google I/O ended.  Overall, though, the keynote exuded to me way less of this vibe that Google was trying to automate even the tasks of software engineers and especially data scientists.  It was way more human-centered, focusing not on tech

From there, I attended two talks, one on Tenworflow TPUs and one on Spectre & Meltdown as it pertains to Web pages, before using the rest of the conference to hit up the product demonstration booths and schmooze with Googlers who work on these things.  And even if I’m not so fluent in some of their work as I’d like to be, I still hoped to be inspired and moved to pursue new ideas as well as get various questions answered regarding anything from potential use cases or expansions up to specific pointed questions about problems I’m facing.

Demos That Struck My Fancy

The thing I saw at Google I/O that I wanted to try most was in the Machine Learning tent where they had two “autonomous vehicles” running a figure eight course.  Rather than using regular line-following sensors or even reinforcement learning, the machines were trained on behavioral mimicking, where they try to tell based on previous input from humans what to do in given situations.  And it was obvious they were trained like this, because as the sun began to set, reflections of it shined right into the vehicles’ cameras and confused them because it hadn’t seen that type of input before.

I also spent a lot of time talking with people in the Web technology booth, particularly about front-end performance testing (since that still seems like a dark art to me, as opposed to API performance testing which is well-understood) including PageSpeed and Puppeteer, not to mention Lighthouse which has been out for a little while now.   Web Bluetooth and WebXR also seemed cool, and I finally got in touch with some smart people behind Web Bluetooth that I tried to meet last year but missed.

There were also on display incredible APIs you can access through scripts you can link to in Google Sheets that can basically act as quick & dirty real-time dashboards, as well as the ability to get all sorts of analytics on, for instance, Gmail, which is of special interest to me because I really want to know who has sent me the most unread mail over the last 10 years or so I’ve had my favorite account.

Firebase always brings fun games to show interactive abilities of their apps, as well as other interactive ways to demonstrate the ever-expanding capabilities of their product.  And, of course, there was Android Auto which I have had app ideas in my head for a while now but can’t be written for the normal projected version of Android Auto.  However, as manufacturers look to integrate Android directly, rather than the old-school Blackberry QNX or even Automotive-grade Linux (AGL), my dreams might come to fruition, plus it might help eliminate the huge kludginess (Web people would call it “jank”) that I see when trying to use the interfaces provided by my current car, which can sometimes be very laggy for no reason.

And after a weekend gallivanting around San Francisco, meeting nice people at Haight-Ashbury, and taking accelerometer measurements on the Golden Gate Bridge to post on Kaggle…

The second major thing I did out in California was to attend the IoT World conference in Santa Clara, which brought together some very important movers and shakers in the world of the Internet of Things.  One cool thing that went on during the Monday of the conference was the Eclipse IoT Day, where they showed off many of the open-source tools one can use to write code, manage IoT deployments and digital twins, and even some real-life examples on doing hardware development from the ground up for a contest.  All these talks were recorded, so I spent part of the time in another very brisk overview of machine learning that wasn’t being recorded just to catch any interesting tidbits that I might not know about.

I would say overall that the coolest people to talk to at IoT World were the CIOs from various local governments who are interested in smart city initiatives.  They generally don’t introduce themselves and then have a frank conversation with you before trying to sell you something all the sudden.  They like to talk about their cities, their problems, ideation, and cool solutions, and that’s the type of stuff I like to hear — what people are working on for themselves, whether it’s at a high level or geeking out about the technical details.

However, as I think about where I’m at in my career, and correlating that to the big message I saw from especially Days 2 and 3 of the conference, I can’t be the end-to-end solution provider.  To put it another way, I’m a developer who would really rather you spare the marketing mumbo-jumbo and just let me dive straight into playing with the product.  I don’t really like being sold to, partly due to my defective BS detector and my lifelong disdain and mocking of commercials, but now I realize I have to adjust to it and become receptive to it in order to magnify my own capabilities.

Even big companies who I would think have perfectly capable engineering teams leverage partnerships with other companies in order to get things done faster and build better ideas.  It’s difficult for one person to be fluent in al angles of IoT, from all the different wireless spectrums and protocols coming available to where to run your compute (edge or cloud) and all the different types of equipment you’re looking to monitor or control/communicate with, not to mention failure prediction using sensor data (and the plethora of sensors available), and especially something that was brought up at least once in every talk — security.  (Speaking of which, based on what IBM showed me at the Bay Area Maker Faire, I think security is a false assumption or assertion these days anyway, as quantum computing is probably breaking all of it as we speak.)

Bonus Content

The Intel Day Zero event is a lively party right before Google I/O starts.  It’s a showcase of various technologies, usually makery in nature, that are driven by Intel products.  I redid Wylie 1-Flip to use an Intel UpSquared system on a module, rather than the original Intel Edison inside, which I was hoping would drive the game more reliably.  Intel sponsored part of my trip in order for me to exhibit the Wylie 1-Flip game at the Devil’s Canyon Brewery in San Carlow, but alas I think something happened to it on the journey over, as the solenoids would lock on immediately after plugging in the game.  That’s not good; they’re designed to be too powerful to be left on for more than a brief time, so they will burn themselves out if left on.  Also, they’ll emit a very distinct odor that electronics and pinball aficionados are familiar with, so it’s often easy to spot before there’s a total meltdown as long as someone knowledgeable notices the problem.  Unfortunately, in scrambling to get the game working, I didn’t really get a chance to chat with many of the other Innovators at the event or see their projects, but I did meet a guy who worked on Hyperledger Sawtooth, and I did describe Wylie 1-Flip to at least 400 attendees (since I gave out that many tickets to pave people’s way to swag bags after collecting enough).  I was told it was a very engaging demo, and it could lead to future engagements for me and Wylie 1-Flip.  The first thing I need to do, though, is make it lighter-weight so I don’t get charged for its weight and its size ($325 to take it on as checked baggage on American).

Maker Faire was right after IoT World.  I had to go Friday only because I was doing something else that weekend on Saturday and Sunday.  For $75, I got to sprint through Maker Faire for four hours and try to soak in as much of it as I possibly could.  And sprint I did, since at that price and those hours, you’re not tripping over busloads of children and slow-moving families.  I didn’t really get to see a lot of the outside booths and people’s personal projects, but pretty much made a beeline to the vendor area to see what I should be paying attention to for the next year.  It seemed like there were a lot less vendors than in years past, or else there were so few people there that it just seemed way emptier anyway.  Quite a few vendors were 3D printing related.  I felt like there were less CNC-related vendors, certainly not the Chinese CNC manufacturers that were selling ridiculously cheap mills last year.  However, Dremel had a huge presence there showing off various CNC machines and 3D printers, all with an Apple-like aesthetic (and probably price tag to boot).  There were a couple goodies I took home with me from the floor, though, most notably a Walabot RF & image sensor for 50% off (which impressively even my handyman not-so-techie step-father-in-law is aware of and pining over).

The Golden State Pinball Festival… or I should say the way to the festival… was a long Uber ride stuck in traffic and a very nerve-racking train ride away from the Bay Area Maker Faire.  The ACE train is a nice commuter train, but I was frazzled from being literally the last person on the train, trying to get there on-time with my very heavy Wylie 1-Flip game, and having forgotten a small bag of mine in the Uber car on the way to the train.  Nevertheless, I managed to let go by looking out the window, watching the windmills and rolling hills, contemplating the wind (and my ground speed), and talking to other people on board.  As a commuter train, so many people are buried in their phones or their devices because they see these scenes day after day and take them for granted.  As it was all new to me, I wanted to look out the window as much as possible, and in fact some of the ride was in very remote parts of California with no good cellular coverage.

As for the festival though, and not just the way there, I got a chance to play definitely some different games than what comes around to Texas Pinball Festival.  And, as I was camping out in a trailer on the Lodi Grape Festival campgrounds, I met some nice people from all around California and we all had at least one thing in common.  As it turns out, though, it’s not hard to find techies in California, so beyond talking about pinball, we’d often talk about what we did when we worked at Hewlett-Packard (several HP alums besides myself were there), software languages we’ve built or written in, or what we’re dreaming of adding to the Internet of Things next.  Unfortunately I had some further technical difficulties with Wylie 1-Flip, so I didn’t get to say Hi to as many people as I’d hoped, and could have used to play at least a little bit more pinball.  Nevertheless, I got to play my uncle on Flash Gordon, a machine he used to own back when it was new and he worked as an arcade tech.

One thing that astounded me is that the mild, mostly easily predictable late Spring weather in Wine Country led many people to actually leave their games outside and run them.  In Texas, no one would ever do that due to heat, humidity, and the random squall that might come through on a moment’s notice (which is what happened the Friday after I returned from this long trip — a forecast of low humidity and a high pressure ridge over Mexico yielded to a brief yet very strong thunderstorm just a few hours after I heard the forecast).

There is a growing trend I’m seeing at pinball shows — people starting games and then abandoning them.  I had several situations where I would walk up to a game and hit the Start button out of hair, but then I in fact just “coined in” a second player because the first player was still on their first ball.  It’s frustrating, because I don’t want to play the game for that long, no one is standing behind me who wants to be the second player, and for some reason these Californians tended to have the power switches for their games hidden so I wasn’t able to restart them (besides power-cycling a game can put a lot of stress on its electronics, so it’s not something I like doing anyway).  The good news is I ran into a fellow from Iron Transfer who is developing a product called PinRemote, which allows for really fine-grained control of various pinball machines including Williams WPC-era games (throughout most of the 1990s) and modern Stern SPIKE/SPIKE 2 systems (and maybe the not-too-old Stern SAM systems too; I’m not sure).  In particular, the PinRemote has a switch that can toggle the slam switch on a game so that if an operator notices a four-player game has been abandoned on the first ball, it can be reset without having to perform a risky power cycle.  It can also adjust the game volume, add credits, manage which type of user profile can add what amount of credits to machines they can log in on, and so forth.  It all works over Bluetooth Low Energy, so there’s no Internet connection required to control a game.


I’m still getting unburied from all the stuff sprawled everywhere as I unpack everything I took with me (and not to mention shipped to myself).  My stuff did take some damage during the trip; I lost my glasses at I/O, damaged my sunglasses in the final hour or so of the pinball festival, and my acrylic “playfield glass” for Wylie 1-Flip got broken in return shipping.  Not to mention, I bet I spent something like $500 just in Uber & Lyft rides.  The good news (especially for my wallet) is that my flight and all but two nights of my lodging were paid for by others, and I’m still filing for reimbursements on some of my other expenses.

Also, while shopping at Excess Solutions (which is probably my new favorite electronics store since Weird Stuff closed) I ran into a YouTuber named Rinoa Super-Genius who also happened to be shopping, and we chit-chatted (I was being recorded during this as well) while wandering the store, and as such, I might in fact be in one of their videos.  I’ll have to check!  Meanwhile, I'll also have to make a quick response video once I track it down so I can show them the pieces of my vintage computer collection I mentioned that they were unfamiliar with.

Thursday, April 26, 2018

Making a new Transaction Family on Hyperledger Sawtooth with Docker

Are you excited to see what all the buzz is about regarding Hyperledger Sawtooth?  Are you a fan of using Docker containers to conveniently test Sawtooth in an isolated and easily-deployable environment?  Then it probably won't be long before you want to run your own smart contract code beyond the basic examples provided to you.

At the time of this writing, the Docker containers for Sawtooth only provide examples of smart contracts written in Python 3 -- no JavaScript nor Go containers are available from Sawtooth Lake.  As far as transaction processors, which are the entities that actually run the smart contract code, the only ones available as Docker containers from Sawtooth Lake are:
  • sawtooth-settings-tp
  • sawtooth-intkey-tp-python
  • sawtooth-xo-tp-python
"Settings" is required in any Sawtooth deployment.  "Intkey" is an extremely simplistic transaction processor that simply stores integers on the blockchain, and "xo" allows you to play tic-tac-toe on the blockchain.  However, these are enough to help you write and deploy your own transaction processor, as long as you're comfortable with Python.

Installing and Testing Hyperledger Sawtooth on Docker

Follow the instructions on the Sawtooth documentation about downloading the default YAML file, sawtooth-default.yaml.  Then run docker-compose <YAML file> up.  Setting up any computer program doesn't get much simpler than that!  (Unless you're behind a corporate proxy, like me, but hopefully you can finagle your way around it.)

At this point, you can play with a fully-functioning Sawtooth system.  The first thing I wanted to try was the XO transaction family, since I thought it would be fun to play a game with it.  Note that to use the xo client, you must log into the sawtooth-shell-default Docker container:

docker exec -it sawtooth-shell-default bash 

However, there are some deficiencies in their quick & dirty example:
  • The game is instantiated by a creator, but the creator can get locked out of their own game if two other players quickly jump in
  • It doesn’t check to see if Player 1 == Player 2, thus a player can play against themselves
Also as I was looking to test the mechanism around sending events,  I needed the ability to modify the code running on the Docker container.  However, running docker-compose up & down would nuke the changes to my container’s filesystem.

Why? docker-compose is like docker run, where it refreshes the instance back to its initial image state.  With docker exec, you are running commands on an existing instance that is building up uncommitted changes to its file system.  You can commit these with docker commit.  Also, note that docker-compose down removes the created container instances, meaning that docker-compose up remakes everything from scratch.

Make a copy of the XO container for modifications

The docker commit command takes two parameters: the name of the running container, and the name of the desired image output.  You’re welcome to make changes to the XO transaction processor before running docker commit, as once you run docker-compose down & up, they’ll be nuked from the original image anyway.  But just to start with a clean template, I ran commit on an unmodified XO transaction processor container.

The command I used looked like this:

docker commit sawtooth-xo-tp-python-default hyperledger/sawtooth-xo-tp-python-mod:0.1

Now if you look at docker images, you’ll see your new image that matches the format of the original Sawtooth images.

There’s two things left to do:
  1. incorporate your new image into the docker-compose file so it will be brought up with the rest of the network upon your command, and
  2. incorporate an external volume so you don’t have to continue writing docker commit each time you make a tiny change to your files.

Instantiate the image as a container with docker-compose

To do this, I simply duplicated the xo-tp-python section in the YAML file, and tweaked a couple things, as such:

    image: hyperledger/sawtooth-xo-tp-python-mod:0.1
    container_name: sawtooth-xo-tp-python-with-events
      - validator
    entrypoint: xo-tp-python -vv -C tcp://validator:4004

Adding an external volume

The things that make the transaction processor tick should live within a volume you attach to your Docker container so that you can modify it at will from within the host (if you like GUI text editors) or from within the container (if you like masochism and wasting time).  If you don’t do this, you will lose all your changes, or you will have to run docker commit each time in order to save them into the image.

To do this, make a directory on your system where your volume will live.  I called mine:


Note I named it after my container so it would be easy to recall what the directory is for later.  Then, add it as a volume to your YAML file:

    image: hyperledger/sawtooth-xo-tp-python-mod:0.1
    container_name: sawtooth-xo-tp-python-with-events
      - "Source/Sawtooth/sawtooth-xo-tp-python-with-events:/root/tp-store"
      - validator
    entrypoint: xo-tp-python -vv -C tcp://validator:4004

At this point, consider commenting out the original XO transaction processor service in the YAML file to positively test your new service and avoid any interference from the original service.

Now the directory I created on the host will reside on /root/tp-store in the container.  Through the container, move all the guts of the XO transaction processor into the tp-store directory.  You should consider finding the root directory of all the code for the transaction processor so that you can copy its contents into tp-store and then delete that root (for my containers, this is shown below).  Then, save the state of the Docker container once again by running docker commit on your container.  Bring down your Sawtooth instances, and then modify your YAML file one last time to mount your host’s directory containing the transaction processor into the directory where the container will expect it once you bring the system back up. Finally, run docker-compose up and await the moment of truth!

NOTA BENE:  Getting an error as follows: ERROR: repository hyperledger/sawtooth-xo-tp-python-mod not found: does not exist or no pull access

Or even one like this (because you’re behind a proxy that blocks Docker) ERROR: Service 'xo-tp-with-events-python' failed to build: Get net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Make sure you have your repository name and tag typed in just exactly how you specified it in the docker commit command!  This eluded me for a while, as I had typed “1.0” in the YAML file rather than 0.1.

Anyway, a quick search led me to the Python modules existing at /usr/lib/python3/dist-packages/sawtooth_xo .  I took this folder, moved its contents to /root/tp-store, and then committed the image before writing docker-compose down.

tp-mod$ cd /root/tp-store/
tp-mod$ mv /usr/lib/python3/dist-packages/sawtooth_xo/* .
tp-mod$ rmdir /usr/lib/python3/dist-packages/sawtooth_xo

host$ docker commit sawtooth-xo-tp-python-with-events hyperledger/sawtooth-xo-tp-python-mod:0.1
host$ <Ctrl+C on the process running docker-compose up>
host$ docker-compose -f sawtooth-default.yaml down

Now, don’t forget to update the path where the Python modules need to be remounted to in order for the application to run correctly:

      - "Source/Sawtooth/sawtooth-xo-tp-python-with-events:/usr/lib/python3/dist-packages/sawtooth_xo"

While you’re at it, consider making some modifications to the transaction language in order to see your changes executed, such as replacing “X” and “O” with “A” and “B”, before starting the system.

Finally, run docker-compose -f sawtooth-default.yaml up to restart your Sawtooth environment.  To test out the game quickly, enter sawtooth-shell-default with docker exec, then write:

sawtooth keygen Alice
sawtooth keygen Bob
xo -v create firstgame --username Alice --url='http://rest-api:8008'
# (Note: It’s very important not to have the trailing slash after the port number in the URL, or else the command will fail.)
xo take firstgame 5 --url='http://rest-api:8008' --username Alice
xo take firstgame 4 --url='http://rest-api:8008' --username Bob
# Etc…
xo list --url='http://rest-api:8008'

Once you run xo list, you will see the state of the board reflect your desired player symbols rather than the standard “X” and “O” if you changed them.

Deploying Live Changes to the Transaction Processor

Unfortunately, it seems as though there is no easy way to roll out changes to your transaction processor code.  The servers must be cycled with docker-compose down & up in order to load any new code.  This means it'll be tedious to debug any code relying on a particular state in the blockchain that takes a long time to set up.

Attempt #1: I tried starting up another instance of the xo-tp-python process, which showed the results locally but transactions on the chain showed as PENDING rather than COMPLETED.

Attempt #2: I tried writing “kill -INT 1” in order to restart the entry point process, but this simply stopped the Docker container altogether.  Upon restarting it, transactions seemed to no longer be validated at all, so I cycled docker-compose.

Attempt #3: I tried writing a Bash script that would allow me to send a signal, which would run a function to stop and restart the transaction processor.  However, in experimenting with this, I was unsatisfied with the way interrupts were handled and with Bash's process management, as kill would spuriously fail to find the process ID of the process I spawned through the shell script.  Furthermore, it would tend to work reliably after I sent a signal to the script first, and then to the spawned process, which is quite a hassle.

Attempt #4: Having modified the entrypoint setting in the YAML file not to run xo-tp-python directly, but instead to call it through a separate script, at least xo-tp-python was not running as PID #1 anymore.  This means I could safely send signals to it without stopping the container.  But, since the Bash script didn't do what I had hoped, I wrote the following Python code:

import signal
import subprocess
import sys

cmd = "xo-tp-python -vv -C tcp://validator:4004"
killCmd = ["pkill", "xo-tp-python"]

def interrupt_handler(sig, frame):
    global p
    p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

def kill_handler(sig, frame):

signal.signal(signal.SIGINT, interrupt_handler)
signal.signal(signal.SIGTERM, kill_handler)

p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

while True:
#    for line in iter(p.stdout.readline, ''):
#        print line,

Basically, this script restarts a process upon receiving a signal interrupt.  I experimented with it on my host machine by using the ping command rather than xo-tp-python and it worked to my satisfaction.  After installing this into my Docker image and setting up the following:

    entrypoint: /usr/bin/python3

The script (shown above) manages interrupts in order to manage the state of the transaction processor (so that my script doesn't simply stop upon a signal, and so that the system won't come down because the transaction processor wants to restart).  Upon configuring the YAML file as above, I was able to send signals to it, and see that it restarted the transaction processor, but the main console output shown by docker-compose up showed that in fact no transactions were processed once I killed the initial transaction processor.  As such, more research is required on this front.

NOTA BENE: When you write “exit” to leave your Docker container, all the instructions you wrote in its terminal will be saved to the history if you write “docker commit”.  However, if you don’t write “exit”, then these commands are not saved to the history.  You can leverage this to save useful commands to the history, but beware of it so that sensitive information you write on the terminal does not get stored.

Epilogue & Sources

If I end up doing anything with sending Sawtooth events in a transaction and then subscribing to them with a separate state delta subscriber, I might write about it here.  However, I was too focused on trying to figure out how to deploy changes live, and didn't get any time to research eventing.  If you know a way to refresh a transaction processor immediately, let me know in the comments!