Posts

Showing posts with the label software

Start Azure Pipeline from another pipeline with ADO CLI & PowerShell

Image
Are you looking to find the one simple command that can kick off a pipeline from another pipeline in Azure DevOps? You may have found a lot of annoying restrictions using straight YAML because so many things have to be known ahead-of-time (i.e. set using compile-time variables). This prevents you from doing such things as conditionally running pipelines, utilizing arrays defined at runtime as parameters into many ADO tasks/commands, or performing manipulations on runtime data for use in subsequent commands. You can't even kick off a pipeline without the previous one finishing (and invoking a pipeline trigger ). Follow along as I explore a scripted solution to solve my problem. My Use Case I want to provide fresh Databricks images on a private Azure Container Registry (ACR). This way, data scientists are not confounded by changes to Databricks runtimes when their clusters start and stop, since changes can prevent their package & library installation scripts from running successf...

Extensible Database Tester in Python, with Permutations

Here is some Python code that allows you to generate all combinations of options by choosing one item from an arbitrary number of different lists.  Not only that, but each option is actually a data structure that also includes lots of metadata about the ramifications of the option, in addition to the option value itself. Let's say you're looking to run performance testing on different types of tables and underlying data arrangements in Amazon Redshift Spectrum.  You can devise a template SQL query that can be configurable in multiple ways.  For instance, you could choose between multiple tables or views from which to query data (say, if comparing Redshift to Redshift Spectrum performance), and you also want to see the performance of count(1) vs. count(*) to see if one uses more data or works quicker on Redshift vs. Spectrum.  Thus, you already have two lists of options, and need to come up with the four permutations of these options. The Groundwork Basically...

Touring with Turo - The Pros and Cons

Driving to more remote yet scenic places as a side trip to a big city can reveal places and ways of life unknown to folks who don’t get out of the downtown bubble.   Not everything a city dweller enjoys is in the city, so part of the experience of a culture is to get to know where folks go to take a break.   For instance, when visiting Washington, DC, why not hit Virginia Beach too?   Maybe next time you’re in Boston, think of taking a couple hours to go to Cape Cod.   Maybe if you’re in New York City, take a visit to Long Island (though better plan that one in advance).   But if you find yourself in Detroit, it might be better to go to Windsor just across the Canadian border and stay put. :-P  As such, for once, I will regale you with experiences not from software I'm writing, but that I'm using as a consumer. Background - What brought me here? I was in quite a conundrum during a recent trip, visiting a city where several famous Ivy Leagu...

Unit Test Database Mocking in Golang is Killing Me!

For some reason, writing a particular set of unit tests in Golang proved to be an extreme hassle.  In this case, I was trying to add a column to a table and then write a unit test to verify this new functionality.   No matter how much I looked around for where to correctly specify how to build this table with the new column, the Cassandra database kept complaining the column didn’t exist.   Imagine how frustrating it is to specify a relational database schema that seems to be ignored by your unit tests. Background Our system consists of a Cassandra database plus a Go backend server.   The Go unit tests require Cassandra to be running locally on a Docker container, but do not seem to actually utilize the existing databases, opting to make its own that is out of reach from anything I can see from TablePlus.   The Go code itself utilizes these objects through some level of indirection by a manager, a harness, and a couple modules dealing in the ac...

Making a new Transaction Family on Hyperledger Sawtooth with Docker

(Important Note: Conversational capabilities have been disabled on this article.  If you need to ask a question or comment, please find me elsewhere.) Are you excited to see what all the buzz is about regarding Hyperledger Sawtooth ?  Are you a fan of using Docker containers to conveniently test Sawtooth in an isolated and easily-deployable environment?  Then it probably won't be long before you want to run your own smart contract code beyond the basic examples provided to you. At the time of this writing, the Docker containers for Sawtooth only provide examples of smart contracts written in Python 3 -- no JavaScript nor Go containers are available from Sawtooth Lake.  As far as transaction processors , which are the entities that actually run the smart contract code, the only ones available as Docker containers from Sawtooth Lake are: sawtooth-settings-tp sawtooth-intkey-tp-python sawtooth-xo-tp-python "Settings" is required in any Sawtooth dep...