Thursday, December 26, 2013

nmake: Installing PyQt In Windows 7 with Visual Studio .NET 2010

For those of you interested in pursuing Python development involving a front-end GUI made from Qt, and don't know how to get started on Windows 7, this is what you need to do.  This could generalize to any software you might wish to install with the "nmake" tool, which allows you to "configure", "make", and "make install" different software packages in much the same way you would on a Linux box.

My setup:

  • Windows 7 machine
  • Windows 7.0 APIs installed (C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A)
  • Microsoft Visual C++ 2010 Ultimate (will hopefully work with Express too)
First, for our specific purpose, download and install Qt if you haven't done that already.  It'll take so long that you ought to have plenty of time to run the next step (building SIP) in parallel and still have time to play at least a couple rounds of solitaire.

In general, before you build from any Makefile in Windows, download and unzip your sources into a known location.  If you plan to be building from source frequently, you may wish to simply add these as environment variables in your "System" settings before you open up a Command Prompt.  Otherwise, just add them after you start your Command Prompt session.  Here's what you need to do:
  • set INCLUDE=C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\include;C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Include
  • set LIB=C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\lib;C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Lib
  • Add to your PATH:
    C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin
    for MT.exe file location
If you chose the latter approach, you can just run those first two bullets as-is in the Command Prompt.  Then, the third one would be:

set PATH=C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin;%PATH%

This is it.  Then, cd into the directory where you downloaded your source to, and run the usual three commands, with the noted twists:
  • python
    Note this one takes some options that pertain to your development environment.  It didn't pick it up correctly for me, so I had to specify it manually with the flag -p win32-msvc2010. You can see all options by using the flag --show-platforms.
  • nmake
  • nmake install
The second step among these three is where you'd run into potential problems.  Usually they''re correctable by ensuring the paths are indeed correct.  The others should go smoothly.

For PyQt in particular, you need to install SIP first, and then PyQt (using the means described above).  Both of these can be found on the Riverbank Computing website.  To get the PyQt installation to work (and avoid the errors about not having a working Qmake present), make sure you have Qt installed, and then add this to your PATH:


Or whatever your equivalent path is, assuming you plan to use the MS VC 2010 toolchain as I've described before.  Several other toolchains are supported here, and this will help the system find qmake properly and proceed with installation.

As for using PyQt with Qt Designer to build a user interface, I might save that for a later date.  I hope you had a Merry Christmas (or maybe s/Christmas/$yourFavoriteHoliday/g), and have a Happy New Year since this will probably be my last post of 2013.  See you next year! 

Thursday, December 19, 2013

Livin' La Vida SSD

The other day, I upgraded my laptop's hard drive from a 500GB 7200rpm drive to a 500GB Samsung 840 EVO solid-state drive.  I purchased the SSD from for $290.00 (which is probably the most I've paid for a drive since 2007), and after taking about a week to get packaged by their warehouse and UPS taking their sweet time to get it over here, it finally came time for me to clone the contents of my HDD over to my SSD.

The performance of my HDD had been dismal, especially as it became more full of Heaven knows what.  (This is still an issue, hence why I recently downloaded Mark Russinovich's "du" program for Windows, so I could analyze which directories contained the biggest wastes of space.)  As you'll see if you watch the video, it took over seven minutes for my computer to become responsive enough to be useful after a cold boot.  Windows itself took a real long time to spin up, then it took even longer for my desktop icons to come up so I could actually click on and run anything.  As such, I would reboot maybe once every month or two, and opt to put my computer to sleep instead.  Hibernation wasn't a very good deal either, since I have 12GB of RAM in here and it would take a few minutes to do that too.

After cloning my HDD to the SSD with EaseUS Todo Backup Free Edition, I attempted to boot my SSD straight-up without any modifications afterward.  Unfortunately, Windows couldn't handle itself when it's been cloned to different hardware, so I had to straighten it out with some techniques.  I dusted off my Windows 7 installation disc and proceeded to run several of the automatic repair methods to no avail.  Eventually I ran across these instructions on fixing your boot record from Tom's Hardware.  Turns out I got myself pretty close, but just needed to select the "Command Prompt" option from the Windows 7 Repair/Recovery menu, and type:

bootrec.exe /FixMbr

This is, luckily, all it took to restore my Windows 7 to working order.  If you don't have such good luck, the Tom's Hardware article I just linked to has some more suggestions.  Once I got that squared away, my computer rebooted and, from POST to "Scan your fingerprint now", my computer booted up in around 50 seconds, as compared with about two and a half minutes with the HDD.  In under two minutes since hitting the power button, I was able to open, click around in, and close two programs very smoothly.  The SSD has made a substantial difference in my 3-year-old laptop's performance.

I currently have another SSD running Windows 8 in my desktop, and gave one to my wife for  use in her HP Envy laptop.  Between these and my new NAS box, I plan to rid myself of all the old IDE drives lying around collecting dust, plus any SATA HDD under 1TB.  My nasty old plastic bin of hard drives will, someday soon, be recycled itself.

Wait, I missed something... where's this video you spoke of?  I haven't had a chance to edit it yet, so I'll update this post later on and hopefully embed it.

Thursday, November 21, 2013

I'm a Loser... (of Data on Hard Drives!)

I started getting good at building computers in 9th grade, and throughout high school, I had lots of fun constantly upgrading and improving my "big rig" to have more hard drive space than anyone else in the neighborhood.  By 2005, I was well on my way to 2 terabytes of space, and I now have approximately 20 hard drives ranging from 100GB to 2TB in size, and with both SATA & PATA (IDE) interfaces.  (This doesn't count all the really old <5GB drives I used to play with while I was busting my chops.)

Anyone who's taken a probability class with a focus on electrical engineering knows of a little statistic that works against me and anyone else who owns a large number of hard drives: this statistic is called "Mean Time Between Failure" (MTBF).  In short, the more hard drives you own, the more likely any one of them will fail on a given day.  Well, over the last 16 years, I've had a fair share of failures:

  • 1997 - Laptop hard drive
  • 2006 - Power surge on my "server", prompting me to get real Web hosting - $2200 recovery
  • 2008 - External hard drive failure
  • 2011 - 1TB hard drive partition table became corrupt after computer showed "CPU Over-temperature Error", which was ironic as it was only 6 degrees Fahrenheit outside that day :-P
  • 2013 - Really old 250GB SATA drive going slower, showing numerous SMART errors increasing daily
This doesn't count the times in which user error has been the primary reason for a failure, like the time I borked a disk using PartitionMagic and lost some really cool videos, or the time recently when I was using dd to make a disk clone, but had just rebooted and didn't realize all the hard drive sda_ assignments had just changed, so I was writing to the wrong disk!  Turns out I was making a disk clone of the failing 250 onto my real live game show hard drive instead of my blank backup drive!  Good thing I was maintaining a clone of that drive -- originally it was part of a RAID 1 group, but that's a whole different story.  So let's say I wasn't extremely pissed off that day, which was probably a first for a data loss event. :-P

I learned about ddrescue in Linux from a coworker after mentioning the story about my 1TB drive, but after it made a disk image that seemed to have absolutely nothing on it, I abandoned that and haven't looked back.  I had great luck with the testdisk utility which works in Windows & Linux, and have gotten most of the stuff I'm interested in off that drive already.

The only problem with my clone-keeping of the game show drives was sometimes I did stuff on one hard drive and didn't do it to the other, such as extracting VOBs from ISOs.  This caused me to become interested in the differences prior to wiping my partially-overwritten drive to make it part of a NAS box.  Well, due to the way I corrupted the hard drive with dd, testdisk told me the filesystem was damaged and couldn't read anything for me like it had so handily done with my 1TB drive.  Looking for another solution, I stumbled upon GetDataBack for NTFS.  I haven't had too much luck with Windows data recovery solutions in the past, so I was willing to give something besides R-Studio and EASEUS Data Recovery a shot.  One of the reviews noted how GetDataBack would even give you a preview of the data it's recovering by showing you real files open in real applications, instead of just showing you the names of the files it recovered and having you pay $79 only to realize whatever you're trying to recover is actually empty.  That'd be a bummer.  I feel sorry for anyone who has paid for software on false promises it'd recover your data.

GetDataBack scanned my hard drive for files overnight, and it found everything I wanted!  I painstakingly scanned and compared the contents with my good game show drive and found around 100 files on the corrupted drive that weren't on the good drive.  Conveniently, you can save your hard drive's layout in 2 or 3 different steps so you can quit the program and reboot without losing any of your progress; I took advantage of that feature several times.  Now it was time to pony up $79 for a full license to GetDataBack in order to recover my files... OR NOT!  One of the first things I did after installation was to adjust the application settings.  One such setting was the location of the temporary folder the application used (for what reason it needed one was unbeknownst to me at the time), and since I was curious what kind of data it would be getting on my hard drive, I changed it to something easily accessible to me.  Originally I was poking through GetDataBack's data files to find out where it had detected all these files started and ended on my hard drive, and was planning to use dd for Windows in order to just clone the data myself.  After seeing how that didn't exactly work on the first try for a JPEG image, I decided to use a hex editor to find out if I was even close.  I opened up what I saved in dd in Hexplorer, then picked Hexplorer to open up the original file from within GetDataBack.  There, Hexplorer uncovered the weakness of GetDataBack: it actually saves the files to that temporary directory before you preview them, then it opens the file or asks you which application you want to open it!

Well, now I knew what to do.  I needed to recover files of up to 4.5GB in size, and I really didn't need another application to try and "open" them on the spot.  I used GetDataBack to "Open with..." all these files, and then hit Esc to cancel out of the popup menu once it came up (which happened after quite a while).  Then I would simply rename the file it produced into the temporary folder, and after doing several files, I'd shuff them off into a completely different directory so GetDataBack wouldn't wipe them away when it cleared my temporary directory.  However, there was one pesky file that wouldn't move because it was being used by "System" for Heaven knows what reason -- you know how Windows works, and the way to cure most any problem is a reboot.  I don't care much for rebooting because while my Windows 8 boots up in about 2 seconds from the SSD, the rest of my motherboard takes about a minute to come up & POST, initialize all the hard drive controllers, and do all the rest of the stuff an old, slow BIOS does.  (Yes, for someone who has written uEFI executables that can run in the uEFI Shell, and worked with testing various facets of it for four years, I still don't have a uEFI-enabled computer myself!)  Nevertheless, after my reboot, the one remaining file I couldn't copy was still in its place.  GetDataBack doesn't even delete the contents of its temp folder when you close the application.  If anyone from that company reads this blog, I would implore that you add this feature so people's hard disks don't become cluttered with files they'll never know exist.

Now, the rest of the story is that I'm trying to populate my NAS box with all sorts of data because I'm sick of having hard drives scattered throughout various computers or sitting idle in a bin, and tired of having to set up all sorts of slow network shares to access data from wherever else I am besides at that computer's keyboard.  That's been a struggle so far too, since I purchased a cheap NAS box (on a closeout, no less) that has mixed reviews at best.  I got a terrible transfer rate (1.25-1.75 MB/s) using Windows with Wi-Fi.  Once I had my installer come in and finish the patch panel, I was able to wire the NAS box straight into my gigabit switch; then I was able to see a sustained data rate of almost 23 MB/s over about 8 hours.  Not all good things are meant to last, though; the transfer was interrupted once again for some unknown reason, and now I have to figure out which file it was trying to write to and what directories haven't made it in yet.  Not only that, but the dying 1TB drive's warranty expires next month and I need to make sure I recover everything good from it before I RMA it.  And I'm still working on delivering products to my Kickstarter backers... lots of stuff going on!

Thursday, October 24, 2013

Ben Stein wasn't the first...

Those of you who have followed Win Ben Stein's Money at all may recall that time back on July 1, 1998 when a contestant finally managed to defeat Ben Stein and get a perfect 10 score on the bonus round, thus prompting Ben & Jimmy to drop their trousers.  However, you may not have heard about another instance over 20 years prior when the pants-dropping bug bit Pat Sajak on Wheel of Fortune!

While I've been meaning to post this clip for a very long time (and it appears someone did in fact beat me to the punch with a very brief 7-second clip of it), a Facebook thread prompted me to finally take action, dust off the ol' DVD collection, and spin this one up to rekindle its glory and a piece of the Internet's ever-wandering attention.

The year was 1987, Black Monday was about to hit the stock market, Microsoft released Windows 2.0, and to start its fifth season in syndication, the producers of Wheel decided to lose the shopping round format and have the players spin strictly for prize wedges directly on the wheel, or for straight-up cash.  The show reached its pinnacle of popularity as Vanna-mania swept the nation, and if you care about what else happened on WOF in 1987, you can read more here.  Anyway, on one episode during the Big Bonanza of Cash, Pat mentions at the top of the show how Bob Murphy, then president of Merv Griffin Enterprises, interrupted him in his dressing room, thus causing him to forget his belt before coming out on stage.  "If my pants fall down during the show... write to Murphy!"

Well, needless to say, I'm sure Pat was anxiously awaiting the end of the show, looking for an excuse to do something silly while talking with Vanna before the credits rolled.  Here it is - the big moment!  The audience goes nuts, and announcer Jack Clark laughed his way through the fee plugs.  However, as Jack's health was starting to fail (he passed away in 1988), Pat and Vanna started doing many of the fee plugs (you can hear Pat's VO work in this clip too, as likely this recording is from a summer repeat) before the show brought in a temporary announcer, M.G. Kelly, for most of the '88-89 season.

But wait; there's more!

Also in 1987, the show's producers made a very different ruling than they would probably do nowadays.  There has been some chatter on the Interwebs about a contestant who recently started to say the answer to the bonus puzzle and then stopped while the buzzer sounded, thus forfeiting her chance to win the bonus prize.  It's possible that the producers might have ruled her out of time had she finished saying it anyway, but I bring you, from 26 years ago, a very different outcome...

Dean, who had dominated the whole show, had the chance to guess the bonus puzzle for a new Mercedes-Benz 190E.  (A later 190, in fact, was my first car!)  Given only an 8-letter puzzle, the hint "TITLE", and 15 seconds to solve the puzzle, he used up every last nanosecond of that 15 seconds before blurting out the answer just as the buzzer started to sound.  Pat asked for a ruling, producer Nancy Jones is heard off-screen saying "Yes," and boy, does Dean's face light up!  Good for Dean, winner of $74,834 on Wheel of Fortune back in 1987.  That's a heck of a lot of money to win on WOF, even by today's standards, but with the show being as popular as it was at the time, they had the budget to let that happen constantly, and it sure did.  (Oh, and if you watch the clip of Dean, be sure listen closely at around 4:49 to hear the comment to his wife.  Hopefully they at least shared the cash.  :-P)

Thursday, October 3, 2013

Presenting at the Perot Museum

Honestly I've been forgetting to post here lately; had a lot of company at the house keeping me distracted!  Plus, the work on the LEDgoes project & some other extracurriculars have been keeping me rather busy.  One such extracurricular is a series of interactive demonstrations displayed in conjunction with the Dallas Makerspace at the Perot Museum in downtown Dallas tomorrow, 10/4/2013.

I'm showing off the DecorreLab project (that I wrote about here previously) at the museum, and am hoping to inspire and engage a bunch of people with some really neat psychoacoustic effects in person.  (I might want to study up on some of the psychoacoustic theories in the meantime!)  Some of the other DMS members are also working on displays driven by a Makey Makey, allowing museum-goers to tap on various fruits and conductive non-Newtonian fluids in order to get a laptop to play sounds.  The third display (that lots of DMS members took videos of while we were building it) is non-Newtonian fluids placed in a bowl on top of a 12" 1KW subwoofer.  As the sub plays different frequencies, the non-Newtonian fluid (basically corn starch mixed with water to get a specific consistency) will morph into different patterns and shapes at the bottom of the bowl (as seen on The Big Bang Theory), and even form "towers" that will eventually glob off and fly everywhere. :-P  I also invited artist & circuit-bender Darcy Neal to come out, and she is running a Soldering 101 course & letting people build theremin kits too in order to play around with electronic music.  If you're in Dallas, you should... well actually, the museum event is already sold out, so you'll have to ask someone who was able to get in how cool it was!

Thursday, August 29, 2013

Creepy Facebook Ads: How Do They Work?

First, it's been a long time since I've been over here!  The LEDgoes Kickstarter has been keeping me quite busy, and I've been focusing most of my attention on that the past several weeks.  We raised our funding goal in about 9 hours, and currently stand at almost 800% funded.  However, it takes me at least a couple hours just to write these blog posts, and usually by the time I sit down on Thursdays to start writing them, I'd really rather be doing something else.  (I don't know how influencers balance their time doing vs. their time writing... maybe they have staff?)

And just another brief aside: today, my samples from Texas Instruments arrived!  I ordered a CC3000 WiFi chip and several USB transceivers including the TUSB1105.  Clearly I thought it'd be fun to build my very own mobile device from the ground up. :-P  As much as these chips cost me nothing, they're also ridiculously small and hard to solder since they're in a QFN package -- tiny chips with *no* leads hanging off the edges.  You'd basically need to reflow solder these suckers onto a board, but I'm going to try to come up with a pressure mount for them.  Plus, yesterday I received my actual order of a sub-1GHz RF spectrum analyzer, meaning I'm about to turn my garage door opener into an RFID tag.  Let the tinkering continue!

Thursday, August 1, 2013

My New Kickstarter: LEDgoes, a Modular Scrolling LED Matrix

Well, this is big news!  LEDgoes, my modular scrolling LED matrix system, is finally being put on Kickstarter.  Check out the product here and learn more!  (The Kickstarter isn't live yet because it's still awaiting approval; I'll post the link here when it's ready.)

Why is this cool?

  • Most LED matrices are fixed-width panels; ours is made from smaller panel modules you can push together to make your own desired size, up to 64 panels long.
  • Because you snap the modules together, you can make a bendy LED display.
  • This kit does 3 colors; most DIY kits can only show 1 color.
  • It has an open, simplistic serial interface implemented in firmware, and very versatile software that can drive the firmware to scroll a message from various sources or even show animations.  The software could even help you break the 64-panel limit.
  • The software can control a display consisting of multiple rows of panels in order to display bigger, taller letters.  In this case, each row could have up to 64 LED panels.
  • We got our retail price down to $22/module.  Most DIY kits cost between $20-$50/module and don't come with supporting firmware or software.  Many 8-character commercial DIY kits cost at least $35/character and are difficult to program or repair if something goes wrong.
  • Many LED panels have identical shape & size to the ones we developed this with, including some super-bright panels.
  • The chips on the boards are independently programmable, so you could in fact run whatever program you want to and your application would have 24 GPIO pins per module you use.
  • The LEDgoes modules can be completely manufactured in Texas.
  • Most, if not all, of this work will be open-sourced so we can tap into the power of the community for enhancements.
  • Because my wife & I have real jobs, we're not looking to send our kids to college with this product, so we can sell it to you for cheaper.

I started working on this project roughly October 2011 when attending a DPRG event at Tanner Electronics, and spotted dual-color 5x7 LED matrices in their LED section.  I'd always been fascinated with having a stock ticker in my room, and felt like these modules would be the perfect thing to build it with.  After brainstorming with my wife, we decided to drive each panel using two ATmega168 microcontrollers, and designed the protocol in such a way that lends well to modular flexibility.

The project was started shortly before my "final push" to prepare Owtsee/Head On Out for the Grand Challenge of the Apps for Metro Chicago contest.  I made a little headway on it before I finished Owtsee enough for a 3rd place prize and $3,500 in the hackathon.  Then I worked on it some more, finishing up the flexible firmware that allows software to drive any letter or shape on the matrix.  I wired up one LED panel to two DIP-socket ATmega328s on a breadboard, and drove messages to it using a Netduino microcontroller listening for input from a COM port.  After another diversion getting Owtsee ready for multiple cities, I added a second LED panel to my breadboard and could now show enough of the message for it to make sense. ;)

Once the whole "modular" concept was proven, when two LED boards were shown to work side by side, it was time to move it off the breadboard.  (There wouldn't be enough room for a 3rd panel on the breadboard anyway.)  Stacy began doing board designs in February 2012, and we presented the working display panel, supporting firmware, and board draft at the Dallas .NET Micro meetup on 2/29 (yes, that's how I spent Leap Day :-P).  I then attempted to learn and use the PCB mill at the Dallas Makerspace in order to fabricate the boards myself, but that didn't work out because the work surface is uneven and the mill just doesn't produce the accuracy needed -- it surprisingly gets close, but I broke way more mill bits than I'd care to mention, so I simply gave up on milling the boards myself.  After several design iterations -- 5 or 6 months of double-checking, triple-checking, quadruple-checking, ..., everything, I finally sent my board design to seeedstudio in China for production.  Concurrently, I bought some TQFP-packaged ATmega328 chips from Kineteka.  After about a month, I received my boards in the mail and started going to town making a real live prototype in time for my wedding.

This is a picture of the output from the PCB mill.  Close, but no cigar.

Meanwhile, I showed the breadboard version to videographers with Red Bull Creation at the Dallas Makerspace on 8/5 (almost exactly 1 year ago!) and debuted the new PC controller software, including the "Scroll Live Tweets" feature.  Not only can you send messages for the board to display in plain text, but you can also enter a Twitter hashtag (or handle or search string) and it'll show all related messages in order of arrival, as long as the rate of messages doesn't overflow our queue (then it'll drop some of them),  We used this capability at our wedding in October 2012 so our guests could tweet to #wyliewedding and see their message on our board.  (What ended up happening in practice was we proved the technical ineptitude of some of our guests, who tried to use that hashtag on other social media services. :-P)

Fully-loaded breadboard with both LED panels, as the Red Bull people saw.

Rev 1 from China worked out to about $1/board for 10 boards, and for that price, it was pretty nice quality.  There were, however, a couple design flaws: for one, the output pins of the chips were not routed to the same spots on the board that our firmware was expecting, so I had to tweak the firmware a little bit.  The boards were almost incompatible with the matrix panels due to a big difference in the panel size vs. where the pins are, but luckily this egregious mistake was caught just before production.  However, the spacing on the right-angle header pins to snap the boards together didn't take into account the size of the headers themselves, so there would have been a gap between characters if not for the clever use of the leftover leads from the many through-hole resistors we soldered onto these boards.  I produced a working 3-panel matrix in time for the wedding, but these leads didn't produce a very good connection between the boards and even just looking at it funny will cause it to go all out of whack.  Also, some of the 328 chips had this bad habit of producing really noxious smoke when powered (though they still appeared to work), so this further limited the amount of boards I was willing to produce.

After the wedding, I focused on some contract work and house-hunting.  Stacy found some time to work on Board Rev 2, which fixed the issues with the header pin spacing and added new pads to support both through-hole and SMD resistors.  The pads are designed for 0805 resistors, but 0604 and 1206 sizes should also work in these spots (I've used both 0805 and 1206 myself successfully.  I might leave it to Stacy's steadier hands to try 0604s).  We elected to keep the "errors" in the pin mappings as our Rev 2 firmware had now been thoroughly tested and was doing very well.  I ordered this round of boards from Dorkbot's OSH Park on 2/4/13, and it came out to about $4/board for 12 boards.  However, I got them in 10 days while China was celebrating its New Year, so ordering from China would have put me way behind.  For this run, we went with ATmega168A chips from Mouser because the 168s are much cheaper, especially when you buy them in bulk.  However, even on the Rev 1 boards, I had the darndest time trying to program firmware onto the chips once they were installed on the board.  Either I couldn't find a compatible bootloader, or I couldn't get the actual sketch loaded on once I installed a bootloader.  (You can see how I solved this problem in my previous blog post.)  I tried off & on unsuccessfully for a couple months until I finally found the winning combo (also described in that post).

LED Matrix boards, Rev 1 versus Rev 2.  Theoretically, these are compatible with each other.

Once I found the secret formula, I started going to town & using every last ATmega168 I had in stock to produce as many boards as possible.  I cranked out 8 working boards, and 2 more with dead chips (hopefully getting drag soldering down pat and/or using a Pick & Place machine will help with my yield).  Stacy registered the domain and began writing the materials for the site and the Kickstarter while I shot videos of the boards in action and continued making interesting enhancements to the software suite to support stacking boards vertically to display larger characters, as well as the use of custom fonts. I wrote the Netduino out of the equation, opting to use the PC itself to drive the COM port connection by means of an FTDI breakout board, any Bluetooth serial module (e.g. Firefly or Sparkfun), or even a *Duino running with an open serial port.  If you have a TTL serial port on your device, you can use it control the modules.  Stacy is currently out at a well-known convention in Las Vegas trying to hawk as many of these as possible while I continue to look ahead at what these boards can do next.

Some other related future projects include:

  • Variable-width characters
  • Self-addressing modules
  • Controller software for Mac, Linux, Android, iOS, and other interesting platforms
  • Making plastic, bendable, snap-together housings for each module so it can live in a nice "frame" instead of just hanging out
I look forward to the future of LEDgoes!

Thursday, July 25, 2013

Trials & Tribulations with the ATmega168

I created a board design for some hardware that involves two ATmega chips in TQFP form.  TQFP stands for Thin Quad Flat Package, and is basically on the larger end of the surface-mount chips (but still way smaller than chips in DIP form).  Whether you use ATmega328 or ATmega168 chips on this board is irrelevant until you start talking about large quantities, in which the 168 will spare a great deal of production costs.  What this hardware does is irrelevant for now (though you will find out soon enough), but I wanted to share my experiences trying to get this board and its chips to work in exactly the way I intended.  It was very difficult, for some reason, to get the 168s to cooperate with me, while the 328s worked with no trouble at all.

My Board Design
My Board Design, with ATmega168s and SMD resistors (all soldered on by hand by Yours Truly, who doesn't even have all that much soldering experience :-P)

The Problem?

One of my product requirements is that users can upgrade the firmware on the chips using a simple serial connection, such as in the example at the bottom of this page (where you use an Arduino Duemilanove without a chip installed in order to load a sketch onto a chip in a breadboard).  In order to do this, there must be a bootloader present on the chip to begin with.   Barring that, you will need to write the sketch directly to the ATmega chip using an ISP programmer (like a USBtiny or an Arduino wired up in a different manner) and set the fuses on the ATmega a little differently so it starts with your sketch instead of the non-existent bootloader.

Truth #1.  Long story short: You need a bootloader to install sketches the easy way, or you need special equipment to install sketches without a bootloader.

Well in any case, I purchased twenty ATmega328s from Mike down the street at Kineteka and went about soldering these onto my early prototype boards.  Afterward, I wired up my Arduino to program the bootloader onto each of these chips, and since Arduinos have been shipping with ATmega328s for a long time, this was a piece of cake.  From the Arduino IDE, I just had to select "ATmega328 on a breadboard", then "Burn Bootloader", and the deed was done.  The sketch could easily be installed onto the chip using a much cheaper FTDI module from Sparkfun.

Then, to go with my next batch of boards, I bought twenty ATmega168s from Mouser (also not very far away).  Problems befell me here because the bootloader specified when you choose that "ATmega328 on a breadboard" happens to be just a bit too big for the 168, so it won't let you install it.  I didn't have time to futz around with this since this was just before my wedding and I really wanted to show off these prototypes at the DJ's booth, so I went ahead and made more boards with the 328s and put the 168s in the drawer.

Fast-forward a few months -- I'm settled into a new house and the initial wild & crazy first 6 months of marriage have elapsed (and some contract work was finished).  I have time to look into this project again, and have since acquired the USBtinyISP to help program the bootloaders without needing to spend a whole bunch of time tediously setting up wires from the Arduino to the various pins on my boards.  I also got emboldened with thinking I could make life a little easier by using the USBtinyISP for programming the bootloader and the sketch all in one go.  However, after loading the bootloader onto the chip, the USBtinyISP simply could not write anything at all to the flash section of the chip.

To work through this failure, I dug up my wicked old IBM Thinkpad T42 laptop complete with a parallel port, and dug up an old parallel ribbon cable that went on the inside of some sort of PC slot from way back in the day and featured a nice parallel to 10-pin header connection.  I opened up the parallel port side and was able to rearrange the pins and insert various resistors where the tutorials told me to do so.  I used this parallel cable to attempt to write the program as well, and while it seemed to have success writing the bits, it still didn't really work.  The parallel cable did help me restore many chips though, ones that had fuses corrupted by communication problems and were stuck desiring some kind of external clock non-existent on my board, or any number of other weird states these things can get themselves into.  I even spent an entire weekend building the entire AVR-GCC toolchain from scratch, only to find out:

  • It would end up missing key configuration files that the Arduino IDE has,
  • Truth #2. The Arduino IDE actually has this entire toolchain built-in already, and
  • I didn't even need this toolchain to begin with because
  • the latest Arduino IDE comes with a bootloader that, well I won't spoil it...
From this, you can draw Truth #3: If you build the AVR-GCC toolchain yourself, it won't include the special Arduino configuration "secret sauce."  Don't expect things to work properly.

It's amazing I still have stuff like this lying around.

The reason I went down that whole rabbit-hole of building the toolchain was so I could compile my own bootloader.  I wanted a bootloader that was small and would work with the ATmega168 where the others fell short.  If anything, there are plenty of custom bootloaders available from AVRfreaks, but usually they just give you the source and you have to compile them yourself.  After trying a few of these bootloaders and still short of anything that looked like successful serial communication once any of them were installed, I decided to nix that entire toolchain and go with the latest & greatest Arduino IDE instead.  (This is when I discovered the truth to most of those bullet points above.)

I started screwing around with the predefined bootloaders provided with the Arduino IDE and digging around through the boards.txt file in order to tweak the fuses and the clock speeds that get configured when you install it.  At one point, I noticed the "Lilypad Arduino w/ATmega168" entry existed and happened to invoke pretty much everything I needed -- only it had the clock speed set to 16MHz instead of 8MHz, so I went ahead and changed that in the boards.txt file and restarted the IDE.  I then tried to install my sketch for about the 756th time.

And, finally, I see serial communication happening!  Only it's a bunch of garbledy-goop.  After exhaustively trying numerous combinations of baud rate, 3.3 vs. 5V, and TX<->TX and TX<->RX, I put it down for another week or two.  During this bit of rest & repose, I collected enough of my brain cells off the floor to re-discover something I learned when I was doing this with the ATmega328 chips many months ago:

Truth #4: When many ATmega chips are sharing the same TX/RX lines, and they are powered, they will screw each other up when you try to program just one of them.  When you program one, you must hold the others in reset.

This is very specific to my board, but my protocol calls for each chip to share a common RX line in order to receive instructions from a "heart controller."  That is the main chip that converts basic ASCII text from the client application (PC, mobile app, HTTP request, or whatever you can imagine) and converts it into instructions each ATmega chip is listening for in order to carry out its task.  The TX line is currently unused, but it's still connected to every chip in the same fashion in order to either support something in the future or give me major grief and headaches now.

This is true when you program the bootloader on the second chip, and it's true when you are flashing the sketch onto either chip.  Hold the other chip in reset (which is Active Low for ATmega168s).

The Solution

Let me condense all the above story & findings into some distinct bullet points.  Once I:
  • Downloaded & used the latest Arduino IDE (1.0.5 as of now),
  • Selected the "Lilypad Arduino w/ATmega168" along with some light modifications,
  • And held the other ATmega chips in reset during programming,
I was able to install the bootloader with my USBtinyISP and the sketch using Arduino serial in the manner I linked to above.  I don't know why all my other attempts at bootloading failed when the bootloader for this very old Arduino product actually did the trick for me.  I guess I'll have to look at the difference between the sources, and it'll be educational.

Ahh, success!

Thursday, July 18, 2013

DFW Restaurant Week 2013 Listing in Open Data Format

Residents of the Dallas-Fort Worth area can, as of this past Monday, begin making reservations at any of a number of area restaurants participating in DFW Restaurant Week 2013.  The choices are astounding; there are 131 restaurants to choose from between Weatherford and Rockwall, and from Denton down to Balch Springs.

However, given the way the data is presented on most of these websites, how are you supposed to make heads or tails of it?  The restaurants are often listed in tabular form, complete with obscure pictures & color codes to tell you what days these restaurants will be offering what kinds of services and specials.  Not only that, but not everyone knows exactly where every single avenue and alley these restaurants are on, so it might be easy for you to miss out on restaurants near you.

To rectify this, I am opening up access to a JSON dataset of all the restaurants participating in DFW Restaurant Week 2013.  You can download the dataset at or simply embed that link into your website to make sure you always have the latest version.  I know there are a lot of creative types around here -- I just met some of you at the Digital DUMBO - Digital Dallas launch party yesterday!  As much as I'm going to be creating my own visualizations for this data at, I want to see what you can come up with using this data as well.  The more ideas & implementations we have out there, the more accessible, helpful, and visually stimulating we can make this data in the future.


Thursday, July 11, 2013

VGA Monitor Output from an Arduino

One of the coolest things about programming on a platform for the first time is making something graphical or somehow otherwise tangible to the eye.  Terminal apps generally aren't very exciting because they're just text, but making an application with its own window & graphics is much more exciting for the beginner.  If it's a microcontroller, making that LED blink is a first big step toward future fun.  However, there is a lot more to do when it comes to making graphical output from microcontrollers -- output on 7-segment displays, an LED matrix, LCD touchscreens, or the pinnacle & mainstay of computer displays -- the monitor.

Having taken a class in FPGA design and becoming fascinated with video output without needing an operating system, I decided to look into what it'd take to use the Arduino to produce VGA output.  It turns out the hardware requirements can be somewhat hard to come by these days:
  • A nice, old, tolerant CRT monitor -- my LCD monitors won't support this
  • A breadboard
  • Several resistors of various values; pick your own after reading below

Once you have the prerequisite hardware, you need to acquire some esoteric knowledge about the way Atmels and many other microcontrollers work.  This definitely isn't beginner-level stuff; it took a little while to pore over the Atmega328 datasheet to figure out just what settings are required to make the analog* signals required to drive a VGA monitor.  Instead of making you repeat all this research, I'll just give it to you.  I've attempted to make explanations in the comments, but if anything is unclear, please refer to the datasheet.

The schematic

Note the voltage dividers put in place to change the output of the Arduino from 5 volts down to the maximum 0.7 volts allowed by VGA hardware.  Arduino pins 13 & 12 (sync) go out to one 220-ohm resistor each, and 7, 4, & 2 (the RGB pins) each go to their own voltage divider with R1 = 1200 ohms and R2 = 180 ohms.  Some further notes:

  • Pin 9 on the D-SUB connector should really be wired to ground. However, my circuit managed to work without doing this, so it's not in the schematic.
  • I can't remember the rhyme or reason for putting a resistance on the sync lines of specifically 220 ohms.
  • The 1200 ohm and 180 ohm resistors are in place as a voltage divider, since the maximum voltage the RGB pins in a VGA monitor can take is 0.7V. These resistor choices lead to (180 / (1200 + 180)) * 5V ~= 0.65V. You can use any other combination as long as it leads to the correct voltage.

Arduino firmware

The firmware is designed to consolidate the Red, Green, and Blue values for each "pixel" into 3 consecutive bits.  While this prevents any interesting variations in color (i.e. only eight colors are possible), and isn't truly "analog", it helps speed up processing time because you can cram 2 "pixels" into the same byte.  The next pixel can be shown simply by shifting the register left by 4 bits, which takes fewer clock cycles than loading the value from the flash ROM.  (This kind of causes some pixels to be slightly narrower than others.)  Speaking of pixels and resolution, this code generates an image with a resolution of approximately 376x282 @ 60 Hz.  Here are a few places I got guidance from on my quest to write this firmware:

  • - My code is based off theirs, but heavily modified.
  • - A GTF spreadsheet you can use for finding out what frequency and timings to use given the desired VGA resolution and refresh rate you specify. If this spreadsheet doesn't work out for you, the one I used originally has been recovered and I can share it.  I don't know where it came from originally.
  • Lots of Google searches for "vga pinout" and "vga timing specification", etc.
#include <avr/interrupt.h>
#include <avr/pgmspace.h>     // for PROGMEM

// horizontal line counter (negative is during vertical blanking)
int hzCount = -10;
int vCount = 0;
int timingCount = -10;
int base = 0;

unsigned short int OCR1AMEM = 0;
unsigned int pixRow = 0;
prog_uchar* curPix = 0;

unsigned int displayInt;
int k;    // counter variable
char myChar; 

// ==============
// Pinout:
#define HSYNC 0b00100000;   // 13: HSYNC (Port B5)
#define VSYNC 0b00010000;   // 12: VSYNC (Port B4)

#define BLACK   0b00000000;   // Black                0x00
#define BLUE    0b01000100;   // 6:  Blue  (Port D6)  0x40, 0x04, 0x44
#define GREEN   0b00100010;   // 5:  Green (Port D5)  0x20, 0x02, 0x22
#define RED     0b00010001;   // 4:  Red   (Port D4)  0x10, 0x01
#define CYAN    0b01000100;   // Cyan                 0x60
#define YELLOW  0b00110011;   // Yellow               0x30
#define MAGENTA 0b01010101;   // Magenta              0x50
#define WHITE   0b01110111;   // White                0x70

// prog_uchar
// prog_uint32_t signMessage[] PROGMEM  = {0x04108000, 0x00000410, 0x00808000, 0x00000080}; // , 0x92009292, 0x92000092, 0x00009292, 0x00920029};
prog_uchar signMessage[] PROGMEM  = {/*this is where a whole lot of hex numbers go for your custom image*/};


// Others go to GND
// =======

// hurry up & wait
#define NOM    asm("nop");
#define NOM5   asm("nop\nnop\nnop\nnop\nnop");
#define NOM10  asm("nop\nnop\nnop\nnop\nnop\nnop\nnop\nnop\nnop\nnop");
#define NOM20  asm("nop\nnop\nnop\nnop\nnop\nnop\nnop\nnop\nnop\nnop\nnop\nnop\nnop\nnop\nnop\nnop\nnop\nnop\nnop\nnop");

void hLine(void)
  curPix = signMessage + base;
  PORTD = pgm_read_byte_near(curPix);
  curPix++; PORTD = PORTD << 4;
  PORTD = pgm_read_byte_near(curPix);
  curPix++; PORTD = PORTD << 4;
  PORTD = pgm_read_byte_near(curPix);
  curPix++; PORTD = PORTD << 4;
  PORTD = pgm_read_byte_near(curPix);
  curPix++;PORTD = PORTD << 4;
  PORTD = pgm_read_byte_near(curPix);
  curPix++;PORTD = PORTD << 4;
  PORTD = pgm_read_byte_near(curPix);
  curPix++; PORTD = PORTD << 4;
  PORTD = pgm_read_byte_near(curPix);
  curPix++; PORTD = PORTD << 4;
  PORTD = pgm_read_byte_near(curPix);
  curPix++; PORTD = PORTD << 4;
  PORTD = pgm_read_byte_near(curPix);
  curPix++; PORTD = PORTD << 4;
  PORTD = pgm_read_byte_near(curPix);
  curPix++; PORTD = PORTD << 4;
  // pix 21-
  PORTD = pgm_read_byte_near(curPix);
  curPix++; PORTD = PORTD << 4;
  PORTD = pgm_read_byte_near(curPix);
  curPix++; PORTD = PORTD << 4;
  PORTD = pgm_read_byte_near(curPix);
  curPix++; PORTD = PORTD << 4;
  PORTD = pgm_read_byte_near(curPix);
  curPix++; PORTD = PORTD << 4;
  PORTD = pgm_read_byte_near(curPix);
  curPix++; PORTD = PORTD << 4;
  PORTD = pgm_read_byte_near(curPix);
  curPix++; PORTD = PORTD << 4;
  PORTD = pgm_read_byte_near(curPix);
  curPix++; PORTD = PORTD << 4;
  PORTD = pgm_read_byte_near(curPix);
  curPix++; PORTD = PORTD << 4;
  PORTD = pgm_read_byte_near(curPix);
  curPix++; PORTD = PORTD << 4;

  // pix 40-

  delayMicroseconds(1);   // wait for back porch time - 1

  PORTB = PORTB ^ HSYNC;         // invert HSYNC - negative polarity
  delayMicroseconds(3);    // wait for "sync length" time - 1
  // 127?
  // NOM20; NOM20; NOM20; NOM20; NOM20; NOM20; NOM5;
  PORTB = PORTB ^ HSYNC;          // invert HSYNC again
  // NOM20; NOM20; NOM20; NOM20; NOM20; NOM20;
  // NOM20; NOM20; NOM20; NOM20; NOM20; NOM20; NOM10; NOM; NOM; NOM;
  // NOM; NOM; NOM; NOM;
  // NOM; NOM; NOM; NOM;


void hSyncSig(void)
  // PORTD = PORTD ^ PIN2;
  PORTB = PORTB ^ HSYNC;         // invert HSYNC - negative polarity
  NOM5; NOM5;
  delayMicroseconds(3);    // wait for "sync length" time - 1
  PORTB = PORTB ^ HSYNC;          // invert HSYNC again

  // do hSyncSig throughout blank interval, hLine for pixel data
  if (timingCount < 0) {
    // 14 + 1 NOM was good for 38 pix; what will 40 take?
    NOM5; NOM5;
    // do vsync with positive polarity
    if (timingCount == -10) PORTB = PORTB ^ VSYNC;
    if (timingCount == -5) PORTB = PORTB ^ VSYNC;
  } else {
  // was 159, why?
  // Wait for front porch

  if (vCount == 4) {
    vCount = 0;
    base = hzCount * 19;
  if (hzCount > 71) {
    hzCount = 0;
    timingCount = -10;
    base = 0;

void setup(void)
  // pins
  pinMode(4, OUTPUT);
  pinMode(5, OUTPUT);
  pinMode(6, OUTPUT);
  pinMode(12, OUTPUT);
  pinMode(13, OUTPUT);
  digitalWrite(4, LOW);  // start red low
  digitalWrite(5, LOW);  // start green low
  digitalWrite(6, LOW);  // start blue low
  digitalWrite(12, HIGH);  // start VSYNC high
  digitalWrite(13, LOW);   // start HSYNC low

  // external interrupt timer, see ATmega328 datasheet for more details
  // enables real-time change of the count-up register
  // temporarily disable interrupts
  // EICRA: External Interrupt Control Register A
  // ISC11, ISC10 = 1: with both set, rising edge of INT1 causes interrupt
  // EICRA = 1 << ISC11 | 1 << ISC10;
  // EIMSK: External interrupt Mask Register
  // INT0 = 1: when enabled, Interrupt Request 1 is enabled
  // EIMSK = 1 << INT0;
  // PCICR: Pin Change Interrupt Control Register
  // PCIE2 = 1: when enabled, fires an interrupt when PCINT23:16 pins are changed
  // PCICR = 1 << PCIE2;
  // PCINT19 = Arduino pin 3

  // hsync timer, see ATmega328 datasheet for more details

  // TIMSK0: Timer/Counter1 Interrupt Mask Register
  // TOIE0 = 1: when set, enables the Timer/Counter1 Overflow interrupt
  // This enables everything already set EXCEPT for TOIE0
  TIMSK0 &= !(1 << TOIE0);
  // Timer/Counter Control Registers
  // Timer/Counter1 Control Register A
  // DEPRECATED: Ignore any functionality provided by this Register
  // WGM10 = 1; combined with WGM12 in TCCR1B, enables count to TOP
  TCCR1A = 0;
  // Timer/Counter1 Control Register B
  // WGM22 = 8: Set timer to CTC mode with TOP = OCR1A (WGM12 may also work?)
  // CS10 = 1: Do not prescale the clock
  TCCR1B = 1 << WGM12 | 1 << CS10;
  // Sets the TOP (highest) value the Counter will count to
  // F_ocr1a = 16MHz / (2 * prescale * OCR1A);
  // F = 1/192 MHz: usu. 0x5FF; F = 1/57 MHz; usu. 1C7 F = 1/82 MHz; usu. 28F; 1/56: usu. 1BF
  OCR1A = 0x01BF;
  // TIMSK1: Timer/Counter1 Interrupt Mask Register
  // OCIE1A = 2: when set, enables the Timer1 Output Compare A Match interrupt
  TIMSK1 = 1 << OCIE1A;
  // re-enable interrupts

void loop(){
  //twiddle thumbs


Now this isn't the greatest firmware in the world.  One thing it gets hung up on is all the conditional statements in here.  To make this code run more efficiently, you should use smarter branching by assuming the code path that runs most frequently is the one that is going to run, then using "return"s instead of additional "if"s.

Make a picture, make a picture!

Finally, this code isn't very interesting unless you can display something cool (or at least easily, without having to think in hex to make your bitmap).  Below is a program written in Visual Basic .NET that will load a Windows 24-bit bitmap file and convert that to the special 3-bit format I use.  It'll output the hex that you can put directly into your firmware in the spot specified.  (Feel free to convert this code to the language of your choice.  I realize VB.NET isn't really open-source friendly, but it's an old project and this is what I wrote in back then.)

Imports System.IO
Imports System.Drawing.Imaging
Imports System.Text.RegularExpressions

Public Class Form1

    Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        ' ************************
        ' * 1. Variable Setup    *
        ' ************************
        Dim b As New Bitmap("C:\Users\Stephen\Pictures\arduino pic 1.bmp")
        Dim ms As New MemoryStream
        Dim stri As String = ""
        Dim hexNum As String
        Dim colors(63) As String
        Dim arduinoColors() = {&H0, &H1, &H2, &H4, &H6, &H5, &H3, &H7} ' KRGBCMYW
        ' now make every possible combination of 2 colors (black + ___ is already done! :)
        b.Save(ms, ImageFormat.Bmp)
        Dim abyt(ms.Length - 1) As Byte
        ms.Seek(0, SeekOrigin.Begin)
        ms.Read(abyt, 0, ms.Length)
        Dim mx As New MemoryStream(abyt)
        Me.BackgroundImage = Bitmap.FromStream(mx)
        ' go through and hex the whole bitmap
        For a = 0 To abyt.Length - 1
            hexNum = Hex(abyt(a))
            If hexNum.Length = 1 Then hexNum = "0" & hexNum
            stri = stri & ",0x" & hexNum
        ' look for the color palette
        hexNum = Regex.Match(stri, "0xFF,0xFF,0xFF,(0x00,0x00,0x00,.{105})").Groups(1).ToString
        Dim i As Integer = 0
        ' gather our colors
        For Each Color As Match In Regex.Matches(hexNum, "0x[0-9A-F][0-9A-F],0x[0-9A-F][0-9A-F],0x[0-9A-F][0-9A-F]")
            colors(i) = "C" & Color.ToString
            i += 1

        ' **************************************
        ' * 2. First-Pass Pixel Replacement    *
        ' **************************************
        ' delete bitmap headers & palette -- first pixel better not be white!
        ' If there are row bounds, use bottom two lines; if not, use top
        ' ---
        stri = Regex.Match(stri, "0xFF,0xFF,0xFF,0xFF,(?!0xFF)(.*)").Groups(1).ToString
        ' stri = Regex.Match(stri, "0xD4,0x2D,(.*)").Groups(1).ToString
        ' stri = Regex.Replace(stri, "0xD4,0x2D,", "")
        ' ---

        ' demarcate pixel bounds to faciliate replacement regexes
        stri = Regex.Replace(stri, "(0x[0-9A-F][0-9A-F],0x[0-9A-F][0-9A-F],0x[0-9A-F][0-9A-F])", "C$1")

        ' run the replacement of palette colors to Arduino colors
        For i = 0 To 7
            stri = Regex.Replace(stri, colors(i), "0x" & Hex(arduinoColors(i)))
        TextBox1.Text = stri

        ' ************************
        ' * 3. Image Rotation    *
        ' ************************
        Dim myBytes() As String = stri.Split(",")
        stri = ""
        For i = 0 To 71     ' # of pixel columns is hard-coded for now
            For j = 0 To 37 ' # of pixel rows is hard-coded for now
                    stri = stri & myBytes((j * 72) + i) & ", "
                Catch ex As Exception
                    stri = stri & "0xGG, "  ' something invalid is obvious, something valid keeps a picture
                End Try
        TextBox1.Text = stri

        ' *********************
        ' * 4. Pixel Merge    *
        ' *********************
        stri = Regex.Replace(stri, "(0x[0-9A-F]), 0x([0-9A-F]),", "$1$2,")
        TextBox1.Text = stri

    End Sub

End Class

Now I haven't done this in a really long time, but as I recall, you actually have to submit the picture sideways, rotated 90 degrees to the left of upright.  The very last row of pixels on the bottom needs to be your color palette; you can draw the image using whatever colors you want in Paint, but you have to define which color corresponds to black, red, green, blue, cyan, magenta, yellow, and white.  Just one pixel per color is enough; the rest in that row can be white.  Unfortunately, I don't have a sample bitmap at this time (I guess I haven't rescued it from the failed hard drive yet) but if I find it, I'll post it here.