Comparative technical support with tortuous tricky technical problems

Not for the first time, our own inimitable Max the Magnificent has provided a segue into one of my blogs. In his “Tortuously Tricky Technical Problems” (TTTP) he concluded:

“These are the sorts of problems that can keep you up at night. They are also the ones that, when you finally figure them out, cause you to leap into the air punching the sky and hooting and hollering with a great big smile on your face. Speaking of which, do you have any weird and wonderful experiences of this kind that you would care to share with the rest of us?”

Well I am sharing – blame Max! Often when I am faced with a TTTP no one really knows the answer, but it helps when there is someone with sufficient insight who can ask the right questions that prompts an investigation that leads to the answer. My employer is a small organization and so there aren’t many people to ask. In addition I am of an age that I am that person others ask for insight. When I have a problem I sometimes have to turn to technical support of the chip manufacturers and their responses can sometimes be helpful and sometimes, not so much.

Let me set the problem for you. I had designed a project and when it came back from manufacture I passed the hardware to a neophyte engineer along with a specification and asked him to design the firmware and then complete the design.

I love Maxim parts – Maxim is a company with innovative, inspirational and reliable parts and really good data sheets. My record also shows that I am an ardent PSoC supporter. My overall hardware design was a crib of a successful previous one with a slightly different I/O that used a part from both Maxim and Cypress. I prefer external watchdogs in microcontroller designs and I had designed in a Maxim MAX824L, an integrated circuit I have used successfully several times before. It is coupled with a Cypress PSoC4 microcontroller. The nub of the design as it affects the impending problem is shown in Figure 1.

click for larger image

Figure 1- Portion of the circuit pertinent to my reset problem. My apologies – the symbol for the MAX824 was created in PCAD. Altium didn’t do a great job in translation. (Source: Author)

Because of the interaction of the microcontroller debug/programming connection (plugged in on P2) the reset output of the watchdog is separated from the reset input of the micro by a jumper P3. Connection across jumper P3 is done towards the end of the development when it is possible to disconnect the debugger and test the watchdog operation. Let me add that according to the PSoC data sheet, the XRES input has a 5K pullup resistor that is permanently enabled.

The watchdog and micro both worked without the jumper installed, but when P3 was shorted the micro stayed permanently reset. My colleague initially reported to me that the XRES line of the micro (which is the /RST line on the watchdog) was continuously low. This was surprising to me because the source project seemed to work fine.

The tests I suggested and then executed as the problem persisted were manifold and here are only some of them. First we replaced the MAX824L with a MAX824R. The “L” version has a threshold of 4.63V, whilst the “R” version has a threshold of 2.63V. This was not inspirational – we thought the original device had been damaged and the “R” it was the only stock we had. This worked, but when we looked at other modules with the “L” part the problem persisted. Worse, the original project exhibited the same symptoms when the micro was blank.

At the risk of perhaps insulting your intelligence, let me explain. If the micro is unprogrammed then the /RST line of the watchdog should oscillate at about 1 Hz. In our case it was in a (reported by my colleague) continuous reset state. This seemed to suggest that the micro was somehow draining enough current from the /RST pin to appear to the MAX824 that the supply voltage was below the threshold (since the 2.63V part worked).

The MAX824 has a push-pull output so it can source current as well as sink it. I placed a 10K series resistor between /RST and XRES first to decouple and then to monitor the current. The MAX824 oscillated as it was supposed to. There was no significant current when /RST was high and when low the resistance divider created with the PSoC4 internal resistor confirmed that it was an internal 5K resistor. When I changed the series resistor to 5K (with resultant voltage of 2.5V on XRES) the PSoC remained held in the reset state. I also wondered if there was a current source in the PSoC and tried a Schottky diode in place of the external resistor in either direction. In addition I tried pullup and pulldown resistors in parallel with the XRES input. Again no conclusion – there went that theory.

I though there may be some other faulty connection on the board and so I disconnected pin 32 (XRES) of the micro. Another dead end.

Figure 2- Internals of the MAX824. (Source Maxim Integrated Products)

That evening in the shower I had an epiphany. The MAX824 has a feature that if the WDI input is left floating, the watchdog function is disabled and although the PSoC4 will supposedly set the output to a predetermined value on reset, I thought maybe whilst the reset was active the output wasn’t being defined leaving the watchdog inoperative in a perpetual motion feedback loop. You can see in Figure 2 the feedback buffer on the WDI input, which I thought may be influencing the setup. I woke up at 4AM thinking that it didn’t quite explain the circumstances, but I went ahead and removed the WDI input from the board and connected a pullup to it. No difference to the problem. Epiphanies aren’t always right! Which begs the question, if an epiphany isn’t right, is it really an epiphany? That’s enough philosophy; let’s get back to the problem at hand.

I then tried a MAX1232 (note the “1” before the “232”. This is not the famous serial port part, but a newer version of the Dallas Semiconductor (now Maxim) DS1232) as a watchdog instead. I have also used this part extensively and had stock to experiment with. The MAX1232 has an open collector output, just to take a different approach. It also monitored the XRES line for myself and noticed a very short 34us pulse every ~1 sec. At that point I decided that the PSoC was feeding back a reset signal to the watchdog and was about to insert a logic gate to prevent it affecting the reset from the MAX824. Conveniently a tech support provided the answer at that point, but before I get to the solution let me contrast the tech support I received.

In order to shorten this blog, I was going to break here for a month to leave you, dear reader, to see if you could think what the problem might be, but the solution is so far outside the details I have described, that it wouldn’t be fair. Also the powers-that-be disapprove of two parters. Still, if you want to give it a shot, go get a cup of coffee and think about what you might do next.

I contacted both Maxim and Cypress tech support since I was long convinced that it was some interaction between the micro and the watchdog and I knew negotiating between two organizations could prove problematic. Maxim replied quickly, but I am afraid there is not much good I can say after that. Their opening gambit was that the push-pull output of the MAX824 could not drive a pullup resistor and I should switch to an open collector device. Seriously? After that I should have known better than to continue. After verifying that we indeed had genuine Maxim parts, the applications engineer implied that I wasn’t using the /RST and RST as outputs and that they are only outputs. More fool him – the data sheet explicitly states that the reset pins are bidirectional. He also kept circling round and asking questions that I had provided details on already. He also said there couldn’t be a problem because “The MAX824L is available for more than ten years and customers have very good experience with the MAX824L . ” He ended by saying that since the MAX824 works when not connected, the problem had to be with the PSoC. I felt frustrated and that I was wasting time going round in circles, so I abandoned any further interaction.

Let’s contrast Maxim’s support with Cypress’. Cypress took a little longer to respond, pending the assignation of an appropriate applications engineer. I provided the same information to them. They suggested several tests and after some interaction they found the solution. Before I get to that I should point at the attitude – I felt quite impressed to see this as the signoff in one of the emails “Assuring you that the issue will be resolved soon .”  Whilst recovering from a medical procedure (you do get a regular colonoscopy don’t you?- see el Magnifico’s “The worst is behind you…”) I saw in an email from Cypress – “Also please note that the recommended decoupling capacitor on the VCCD pin is 1uF. But in your schematic we are seeing 10uF (C11) capacitor on VCCD. Please make a change. ” Fat chance, I thought. I have seen many discussions about the correct values for decoupling capacitors using values from 0.1uF to 0.01uF and then adding a tantalum for bulk storage, and with the advent of microfarad values of ceramic capacitors it is even possible to merge the two requirements into a single capacitor. The value seems to be largely irrelevant. However, the PSoC data sheet is quite specific and in earlier designs I had used a 1uF at this node, but somehow I had changed it to 10uF. As you can guess from my tone, it was the solution, but why?

This is my interpretation. “Decoupling” is perhaps a bad descriptor to use for this capacitor. VCCD is in fact the internal regulated 1.8V supply. Cypress tells me that the value is critical because “If this value changed, it effects stability of internal regulator and causes unexpected behavior of the device.” Nowhere in Cypress’ documents does it say that XRES is only an input – jumping to conclusions is the only exercise I get.

It seems to me that the unexpected behavior is based in the time taken to reach voltage stability. It also seems to me that perhaps the voltage startup is initiated at the end of an input reset signal. When the watchdog /RST of the watchdog goes inactive the signal goes positive and the PSoC starts its internal voltage generation. After 34us it sees a failure because the capacitor is too large, delaying the voltage startup. As a result it pulls the XRES line low, re-initiating the watchdog. The lower threshold voltage of the MAX234R would allow the PSoC more time for the VCCD to stabilize, and hence the discrepancy in the way the different threshold voltages work. On reading this a second time, there are some issues with this interpretation vis-à-vis some of my observations, but Cypress’ words “unexpected behavior ” probably mean that I am never going to get closer to a complete answer.

At the end of this I didn’t feel like “hooting and hollering” (as His Majesty suggests in his original blog), and there definitely were several sleepless nights. Rather, in addition to relief in solving the problem (maybe understanding and not having to modify the PCB to boot), I felt inadequate in that I hadn’t solved the problem and sheepish since it was self-inflicted. But worse was the feeling of frustration in dealing with tech support and getting nowhere. It is a pleasure when some companies get support right. Thank you, Cypress!

Full disclosure: My employer is a Cypress Design Partner. I blog, sometimes about the PSoC, and as a long time user of Cypress tech support, I may be recognized when I submit a query.

2 thoughts on “Comparative technical support with tortuous tricky technical problems

  1. “Unfortunately Cypress has just changed their tech support model. They have adopted the Texas Instruments model where you post your question on a user forum and hope that someone at Cypress will assume ownership of your problem. Hope it works out well alth

    Log in to Reply
  2. “Hi, Aubrey, if it works like TI, it will be fine. TI folks are very responsive most of the time. The problem is if it works like Analog's forum. 🙁 (Linear Tech folks tend to be more responsive though).”

    Log in to Reply

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.