Thursday, April 26, 2018

Making a new Transaction Family on Hyperledger Sawtooth with Docker

Are you excited to see what all the buzz is about regarding Hyperledger Sawtooth?  Are you a fan of using Docker containers to conveniently test Sawtooth in an isolated and easily-deployable environment?  Then it probably won't be long before you want to run your own smart contract code beyond the basic examples provided to you.

At the time of this writing, the Docker containers for Sawtooth only provide examples of smart contracts written in Python 3 -- no JavaScript nor Go containers are available from Sawtooth Lake.  As far as transaction processors, which are the entities that actually run the smart contract code, the only ones available as Docker containers from Sawtooth Lake are:
  • sawtooth-settings-tp
  • sawtooth-intkey-tp-python
  • sawtooth-xo-tp-python
"Settings" is required in any Sawtooth deployment.  "Intkey" is an extremely simplistic transaction processor that simply stores integers on the blockchain, and "xo" allows you to play tic-tac-toe on the blockchain.  However, these are enough to help you write and deploy your own transaction processor, as long as you're comfortable with Python.

Installing and Testing Hyperledger Sawtooth on Docker

Follow the instructions on the Sawtooth documentation about downloading the default YAML file, sawtooth-default.yaml.  Then run docker-compose <YAML file> up.  Setting up any computer program doesn't get much simpler than that!  (Unless you're behind a corporate proxy, like me, but hopefully you can finagle your way around it.)

At this point, you can play with a fully-functioning Sawtooth system.  The first thing I wanted to try was the XO transaction family, since I thought it would be fun to play a game with it.  Note that to use the xo client, you must log into the sawtooth-shell-default Docker container:

docker exec -it sawtooth-shell-default bash 

However, there are some deficiencies in their quick & dirty example:
  • The game is instantiated by a creator, but the creator can get locked out of their own game if two other players quickly jump in
  • It doesn’t check to see if Player 1 == Player 2, thus a player can play against themselves
Also as I was looking to test the mechanism around sending events,  I needed the ability to modify the code running on the Docker container.  However, running docker-compose up & down would nuke the changes to my container’s filesystem.

Why? docker-compose is like docker run, where it refreshes the instance back to its initial image state.  With docker exec, you are running commands on an existing instance that is building up uncommitted changes to its file system.  You can commit these with docker commit.  Also, note that docker-compose down removes the created container instances, meaning that docker-compose up remakes everything from scratch.

Make a copy of the XO container for modifications

The docker commit command takes two parameters: the name of the running container, and the name of the desired image output.  You’re welcome to make changes to the XO transaction processor before running docker commit, as once you run docker-compose down & up, they’ll be nuked from the original image anyway.  But just to start with a clean template, I ran commit on an unmodified XO transaction processor container.

The command I used looked like this:

docker commit sawtooth-xo-tp-python-default hyperledger/sawtooth-xo-tp-python-mod:0.1

Now if you look at docker images, you’ll see your new image that matches the format of the original Sawtooth images.

There’s two things left to do:
  1. incorporate your new image into the docker-compose file so it will be brought up with the rest of the network upon your command, and
  2. incorporate an external volume so you don’t have to continue writing docker commit each time you make a tiny change to your files.

Instantiate the image as a container with docker-compose

To do this, I simply duplicated the xo-tp-python section in the YAML file, and tweaked a couple things, as such:

    image: hyperledger/sawtooth-xo-tp-python-mod:0.1
    container_name: sawtooth-xo-tp-python-with-events
      - validator
    entrypoint: xo-tp-python -vv -C tcp://validator:4004

Adding an external volume

The things that make the transaction processor tick should live within a volume you attach to your Docker container so that you can modify it at will from within the host (if you like GUI text editors) or from within the container (if you like masochism and wasting time).  If you don’t do this, you will lose all your changes, or you will have to run docker commit each time in order to save them into the image.

To do this, make a directory on your system where your volume will live.  I called mine:


Note I named it after my container so it would be easy to recall what the directory is for later.  Then, add it as a volume to your YAML file:

    image: hyperledger/sawtooth-xo-tp-python-mod:0.1
    container_name: sawtooth-xo-tp-python-with-events
      - "Source/Sawtooth/sawtooth-xo-tp-python-with-events:/root/tp-store"
      - validator
    entrypoint: xo-tp-python -vv -C tcp://validator:4004

At this point, consider commenting out the original XO transaction processor service in the YAML file to positively test your new service and avoid any interference from the original service.

Now the directory I created on the host will reside on /root/tp-store in the container.  Through the container, move all the guts of the XO transaction processor into the tp-store directory.  You should consider finding the root directory of all the code for the transaction processor so that you can copy its contents into tp-store and then delete that root (for my containers, this is shown below).  Then, save the state of the Docker container once again by running docker commit on your container.  Bring down your Sawtooth instances, and then modify your YAML file one last time to mount your host’s directory containing the transaction processor into the directory where the container will expect it once you bring the system back up. Finally, run docker-compose up and await the moment of truth!

NOTA BENE:  Getting an error as follows: ERROR: repository hyperledger/sawtooth-xo-tp-python-mod not found: does not exist or no pull access

Or even one like this (because you’re behind a proxy that blocks Docker) ERROR: Service 'xo-tp-with-events-python' failed to build: Get net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Make sure you have your repository name and tag typed in just exactly how you specified it in the docker commit command!  This eluded me for a while, as I had typed “1.0” in the YAML file rather than 0.1.

Anyway, a quick search led me to the Python modules existing at /usr/lib/python3/dist-packages/sawtooth_xo .  I took this folder, moved its contents to /root/tp-store, and then committed the image before writing docker-compose down.

tp-mod$ cd /root/tp-store/
tp-mod$ mv /usr/lib/python3/dist-packages/sawtooth_xo/* .
tp-mod$ rmdir /usr/lib/python3/dist-packages/sawtooth_xo

host$ docker commit sawtooth-xo-tp-python-with-events hyperledger/sawtooth-xo-tp-python-mod:0.1
host$ <Ctrl+C on the process running docker-compose up>
host$ docker-compose -f sawtooth-default.yaml down

Now, don’t forget to update the path where the Python modules need to be remounted to in order for the application to run correctly:

      - "Source/Sawtooth/sawtooth-xo-tp-python-with-events:/usr/lib/python3/dist-packages/sawtooth_xo"

While you’re at it, consider making some modifications to the transaction language in order to see your changes executed, such as replacing “X” and “O” with “A” and “B”, before starting the system.

Finally, run docker-compose -f sawtooth-default.yaml up to restart your Sawtooth environment.  To test out the game quickly, enter sawtooth-shell-default with docker exec, then write:

sawtooth keygen Alice
sawtooth keygen Bob
xo -v create firstgame --username Alice --url='http://rest-api:8008'
# (Note: It’s very important not to have the trailing slash after the port number in the URL, or else the command will fail.)
xo take firstgame 5 --url='http://rest-api:8008' --username Alice
xo take firstgame 4 --url='http://rest-api:8008' --username Bob
# Etc…
xo list --url='http://rest-api:8008'

Once you run xo list, you will see the state of the board reflect your desired player symbols rather than the standard “X” and “O” if you changed them.

Deploying Live Changes to the Transaction Processor

Unfortunately, it seems as though there is no easy way to roll out changes to your transaction processor code.  The servers must be cycled with docker-compose down & up in order to load any new code.  This means it'll be tedious to debug any code relying on a particular state in the blockchain that takes a long time to set up.

Attempt #1: I tried starting up another instance of the xo-tp-python process, which showed the results locally but transactions on the chain showed as PENDING rather than COMPLETED.

Attempt #2: I tried writing “kill -INT 1” in order to restart the entry point process, but this simply stopped the Docker container altogether.  Upon restarting it, transactions seemed to no longer be validated at all, so I cycled docker-compose.

Attempt #3: I tried writing a Bash script that would allow me to send a signal, which would run a function to stop and restart the transaction processor.  However, in experimenting with this, I was unsatisfied with the way interrupts were handled and with Bash's process management, as kill would spuriously fail to find the process ID of the process I spawned through the shell script.  Furthermore, it would tend to work reliably after I sent a signal to the script first, and then to the spawned process, which is quite a hassle.

Attempt #4: Having modified the entrypoint setting in the YAML file not to run xo-tp-python directly, but instead to call it through a separate script, at least xo-tp-python was not running as PID #1 anymore.  This means I could safely send signals to it without stopping the container.  But, since the Bash script didn't do what I had hoped, I wrote the following Python code:

import signal
import subprocess
import sys

cmd = "xo-tp-python -vv -C tcp://validator:4004"
killCmd = ["pkill", "xo-tp-python"]

def interrupt_handler(sig, frame):
    global p
    p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

def kill_handler(sig, frame):

signal.signal(signal.SIGINT, interrupt_handler)
signal.signal(signal.SIGTERM, kill_handler)

p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

while True:
#    for line in iter(p.stdout.readline, ''):
#        print line,

Basically, this script restarts a process upon receiving a signal interrupt.  I experimented with it on my host machine by using the ping command rather than xo-tp-python and it worked to my satisfaction.  After installing this into my Docker image and setting up the following:

    entrypoint: /usr/bin/python3

The script (shown above) manages interrupts in order to manage the state of the transaction processor (so that my script doesn't simply stop upon a signal, and so that the system won't come down because the transaction processor wants to restart).  Upon configuring the YAML file as above, I was able to send signals to it, and see that it restarted the transaction processor, but the main console output shown by docker-compose up showed that in fact no transactions were processed once I killed the initial transaction processor.  As such, more research is required on this front.

NOTA BENE: When you write “exit” to leave your Docker container, all the instructions you wrote in its terminal will be saved to the history if you write “docker commit”.  However, if you don’t write “exit”, then these commands are not saved to the history.  You can leverage this to save useful commands to the history, but beware of it so that sensitive information you write on the terminal does not get stored.

Epilogue & Sources

If I end up doing anything with sending Sawtooth events in a transaction and then subscribing to them with a separate state delta subscriber, I might write about it here.  However, I was too focused on trying to figure out how to deploy changes live, and didn't get any time to research eventing.  If you know a way to refresh a transaction processor immediately, let me know in the comments!