Testing a home automation system is hard. In my experience, many professional shops don't do it well, leaving customers unhappy and sapping profits from the project. Here is what I've observed in the field over my 12 years in the audio, video and controls business:
- Tiered testing isn't done
- There isn't a written testing plan
- Programmer does the majority of the testing instead of an independent third party
- The testing that is done isn't methodical or detailed enough
- Regression testing isn't done after changes are made
Let's look at each of these challenges.
Tiered testing
Testing isn't just about finding programming bugs in an automation system. It's also about ensuring the project addresses the needs of the customer. The first level of testing a new system is to validate that the system satisfies the clients' goals.
If the customers' primary requirements are for the automation system to save energy and improve safety in their home, and the automation system does a great job providing whole-house audio without integrating with the HVAC system or the security system, there is a major issue.
The second tier of testing is to make sure the automation system is easy to use. Just because it makes sense to the programmer doesn't mean it is going to make sense to the customer.
Unfortunately, my own experience has been that programmers tend to make things too complicated. A good example is Crestron's implementation of Alexa voice control.
With the Crestron skill if you say, “Alexa, turn on the great room light,” and there are multiple controllable lights in the great room, the skill will come back and ask: “Which light do you mean?”
The skill will do this even if you have defined one of the light loads in that room with the name light because that is the default light that you want to turn on/off 99% of the time.
The engineering mind didn't think like a user that just wants a very simple command to turn the basic light on in a room when they walk in and it is dark. Instead they force the user to either use a longer name of a specific load or start to trip over furniture while they get into a conversation with the automation system over which light should be turned on.
What makes sense to a bunch of programmers doesn't necessarily make sense to the non-technical people who will use the system.
Finally, the third tier of testing is to make sure the functionality developed actually works — and that means each and every function, tested
Written test plans
A good testing plan is not easy to write. It should be in the form of a checklist and cover every button press available to the user, with a description of the programmed outcome of pressing each button.
If you expect a function — like turning on the TV to watch cable — to be accessible from a touchpanel, iPad, iPhone, handheld remote control, and voice commands, then the test plan needs to cover every one of those options.
Getting down to this level of detail is time-consuming and costly. Fortunately, if you are a professional, and are reusing code on multiple projects then you can also reuse your written test plan; which will save you a great deal of time and save your customers money.
An added benefit of a written test plan is that the final version, with all items on the document checked off as having passed the test, can be submitted to the customer as proof of the work that was done to assure the quality of the system being delivered to them. The test plan should also be dated and include the signature of the individual that did the testing.
Who should test the system?
Programmers shouldn't be the only ones to perform detailed testing. Being the one who programmed the system gives the programmer too much insight into how a system works, and they make too many assumptions as they test. It is all too easy, for example, for a programmer to think that if the play button works on a touchpanel to control a Blu-ray player, then the pause, stop, rewind, and fast forward buttons will also work.
This isn't to say that programmers don't have a role in testing. They should test the system as they write it to the best of their abilities; however, they shouldn't be the only ones to do the final testing.
A very large integrator could employ a dedicated quality assurance staff that does nothing but write test plans and test systems. Of course, that approach isn't practical for the majority of organizations.
Instead the project manager or sales person who sold the job is a great person to carry out testing. Their customer-focused viewpoint makes them ideal for testing. It also forces them to learn all the details of how the system functions so they can help train the customer in how to use the system.
Testing must be detailed
The test plan needs to cover every aspect of the system. This includes the details of every button on the touchpanel and every different functional area of the system. Without this level of detail the customer is going to be the one that ultimately tests the system and they can quickly become frustrated with the number of issues they uncover.
Regression testing
Unfortunately, fixing a bug or adding additional functionality to an automation system can have unforeseen consequences. Any time the code is changed there is a chance that you have created new bugs and not just in the area where the change was made.
For example, the customer decides that in addition to wanting a cable TV box in their home theater they also want to watch satellite TV. The code is then modified to provide control of the new satellite TV box.
Unfortunately, the portion of the code used to turn on all the audio/video equipment in the theater system was also modified and the order was changed. This results in the amplifier being turned on before the satellite TV box. That results in a loud pop being generated that may damage the speakers.
If cost didn't matter, you would go back and re-test the entire system after code changes have been made. Typically, that isn't practical so you at least need to reevaluate the functional areas of the system and re-test the functional areas that have the highest probabilities of being impacted by the changes.
High-quality testing isn't cheap and isn't easy; however, an investment in testing will provide you with happier customers, a stronger reputation, and a better bottom line. If you are a homeowner you will benefit from a much more reliable system and a happier family life.
Jay Basen has been a home automation hobbyist for over 25 years and has worked professionally in the industry for 12 years. With a master's degree in engineering, Jay has been writing software professionally for almost 40 years.
