Introduction
The Celtic Engineer is a weekly newsletter produced by Celtic Engineering Solutions. We hope you enjoy it. If you have any suggestions for topics, would like to give feedback or want your email added to the distribution list please send an email to [email protected].
Too much information is a dangerous thing
When you are faced with a problem, as an engineer, a salesman, a business owner, or a parent – as a leader, what you really have to do is to make a decision. Sometimes the decisions are easy. For example, if you have a flat tire, you need to put on the spare tire. If someone spills milk on the counter, you clean it up and buy more milk.
Those are not the times when you earn your money. When things get interesting is when you have a problem where the answer is not obvious. Everything looks like it is working as desired, but the car won’t start. This is when you can experience Engineer Lockup. It happens when you have thought through and eliminated all of the possibilities. The problem is that you still have a problem – the car won’t start. As engineers we are great at doing things, once the problem has been defined. But if the cause of our woes alludes us, then sometimes, we just don’t know what to do. We can’t do nothing, because doing nothing is failure, and we all know that, “Failure is not an option.”
Phial of Galadriel
When you are stuck, the best way to stay stuck is to do nothing. A course of action must be determined. I like to start out by making a list of the things we know and the things we don’t know. When we are lucky, the act of making that list will sometimes point us to the most likely cause of the problem.
If that does not work, I like to have one engineer go though, in some detail, and explain how the circuit works, even though we all know how it works. The reason is, what we all really know is how we would like it to work. When we design something, we make a model of how that thing should work in our mind. The exercise of debugging a circuit is merely comparing how the circuit behaves with the model we have created. When we get to the, “That’s Strange,” moment, we have found the clue that will allow us to diagnose and eventually fix the problem. By going though and explaining how a circuit should work, we are refreshing the model in our minds, as well as sharing that model with everyone else in the room. If, in our description, we say something incorrect or miss an important point, often others will be able to point that out to us.
The worst problems will elude discovery even at that level of scrutiny. At that point I have found the best thing to do is to challenge your assumptions. I don’t mean asking if Ohms Law is still true, but that you should list out the things that you need to make the particular problem you are facing work, then rank them in order of importance. Finally, you should do an experiment or take a measurement to ensure that what you think is happening, is really happening.
Examples are illuminating
Some time ago we were working on a circuit that contained sensors, two microcontrollers, a Bluetooth radio, some regulators, you know normal stuff. Code was written and executed. But we could not turn on the radio. We checked the schematics and nothing stood out. We checked the design against the datasheets. We cut the on off switch to the radio and powered it directly and the radio worked. We checked the on off switch and it worked. We verified that the code exercised the on and off switch.
We all agreed something was not working. The circuit was different from the model of the circuit that we had created in our minds. We decided to turn on the radio and then have the MCU announce that the radio was on. We were testing our model of how the circuit worked. If it and our model were the same, we should get a message that the radio was now on. This was not all that different from what we had done before. We had turned the radio on and it did not work. What was different was that we were asking for verification that the code had executed.
We never got the message. The MCU was still running but it had not sent the message. We added another message, “About to turn on radio,” followed by the turn on radio code, followed by, “Radio is on.” We go the first message but not the second. That’s strange – we had a clue. This is not how it was supposed to work. We knew that with the radio disconnected that the switch would turn on and off as desired. We put a scope on the input power and found a sharp dip occurred when we turned on the radio.
The problem was becoming clear. The startup current of the radio was starving the voltage line to the MCU which was resetting. What made this odd was there was not just a decoupling cap on the radio, but a large cap that should have sourced enough charge to keep it from pulling down then main voltage line. And that is when the difference between our model and the circuit became clear. The radio circuit was designed and validated before the main board was designed. So, when the on off switch was placed, it turned on a working/validated radio. The problem was created by placing the large cap on the downstream side of the switch. So, it was at zero volts when the switch was turned on. Not only did it not provide the needed charge but it made matters worse by requiring that the cap be charged when the switch was thrown. Moving the large cap to the other side of the switch, but leaving the decoupling cap where it was for noise suppression, solved the problem.
A seasoned engineer once told me that the problem, when found, will be easy. It’s finding it that will take most of the time.
Final thoughts
At Celtic Engineering Solutions we help our customers solve problems. Our goal is to make sure they are successful. We don’t believe in high pressure sales. We are here to help you. Why not start a conversation about what we can do for you?
This newsletter is sponsored by Celtic Engineering Solutions LLC, a design engineering firm based out of West Jordan and Murray, Utah, which can be found on the web at: www.celticengineeringsolutions.com. We are also on Facebook, Instagram and Twitter. If we can ever help you with your engineering needs please contact us. You can find the newsletter on the company blog, LinkedIn or in your inbox by subscribing. Send your emails to The Celtic Engineer at: [email protected], with the subject line SUBSCRIBE.
Do you know someone who would enjoy the newsletter? Forward them a copy or let them know where to find it.
Sean O’Leary is sometimes known as the Celtic Engineer. He was involved in putting two missions on the space shuttle. He has worked at the Smelter’s Biproducts department of Kennecott Utah Copper. Has helped design ballistic guidance systems for Northrop Grumman. Worked on various DARPA projects, an anti-RPG system known as Iron-Curtain and has been involved with the downhole oil and gas industry. He currently is the owner of Celtic Engineering Solutions a consulting Engineering Company in West Jordan and Murray Utah.
