True Story No8

 

Konstant trouble?

In the early 70s we had a problem with a multiplexer.

Nowadays such a piece of equipment would fit in a shoebox.

Then, it stood six feet high in a substantial 19" metal rack.

There were shelves carrying beefy power supplies, shelves carrying rows of plug-in printed circuit boards and a large shelf carrying special interface cards supplied by Telefunken who made the main-frame computers to which we were to be connected.

At the top there was a panel containing a row of switches and dials and at the bottom a row of powerful fans for cooling it all.

We had had about a year to design it all, get it made in the factory and to get it working. As I recall there were ten small processors, each built onto a printed circuit card, and these had to intercommunicate with each other and six main frame computers connected to their interface cards, each of which ran at the breathtaking speed of 20MHz.

Within each processor was a program written by members of the engineering team and set into read only memory.

There were two versions of the program; one handling input and one handling output.

Connected to the multiplexer were 512 ports, each designed to carry asynchronous data, at 2400 bits per second, originating at Air Traffic Control consoles.

During testing, data was supplied by a special test set, constructed by more engineers, capable of generating 512 outputs running at 2400 bits per second. The test set was plugged into the multiplexer which was also connected, via the six main-frame computer interface cards, to another special test set supplied by Telefunken.

All went well except every so often an alarm lamp came on and the system stopped working.

It transpired that it usually, but not always went wrong about 8:30am. That is to say, it always went wrong at about 8:30am but it also went wrong at other times as well.

After much scratching of heads and puzzling it turned out it was the goods lift just behind the wall at the back of the laboratory.

Some engineers turning up for work in the morning used the goods lift rather than the stairs, and clearly electrical noise in the mains supply affected the multiplexer.

A large isolation transformer cured the problem: however we were still plagued by intermittent faults… not as often and seemingly completely random.

An exasperated engineer, with good hearing, sitting watching the flashing lamps in the small hours one night noticed that just as the alarm lamp came on he had heard a slight click.

For the next hour or so, in the silence of the lab, he walked round listening intently.

Sure enough, every so often, he could hear a click.

It didn't always result in an alarm but maybe once in ten times it did.

By the time the rest of us turned up (via the stairs) he had it cracked.

"It's been running since 3am he declared".

We were amazed, as the problem had seemed insoluble and we had looked certain to miss our Factory Acceptance date which was now only days away.

"How did you fix it" someone asked.

"Easy", he said "I turned off the soldering iron".

We used Weller irons, which used an electro-mechanical form of thermostatic control, and the click he'd heard had been the temperature sensor operating.

For some reason, despite our isolation transformer, the iron had generated enough interference to get into the multiplexer and cause a program to go wrong.

A few days later, the customer's Project Manager arrived, was given a demonstration, and expressed his satisfaction by signing the Acceptance Certificate.

Two production multiplexers duly arrived in the lab and the old prototype was shipped off to somewhere in France, complete with a couple of engineers, to help with test-bed integration.

Both new machines worked well and were in turn sent off, this time to Southern Germany.

The two engineers came back from France, picked up drawings etc and a clean pair of socks and soon after departed for Konstantz.

After they'd got settled in off they went to the Prime Contractor's factory and helped unpack and connect up the two new multiplexers.

Now, a whole system was being run; main-frame computers; our two multiplexers and lots of radar consoles.

"How are things going", I asked a week later.

"Terrible" was the response, "The multiplexers keep falling over".

"YES we're using isolation transformers, and before you ask, NO there aren't any Weller irons around".

I went out to see them.

Sure enough, after about five minutes the alarm lamp came on.

"Check everything" I said.

All the boards came out and spares tried.

All the wiring was checked.

All was perfect.

The main problem now was the next Acceptance Test.

This was supposed to be a formality but a large stage payment depended on it.

"What are we going to do?" I said.

"Leave it to us" Jim said, winking at me.

Next day was the BIG Acceptance Test.

Problem number one.

Where was Jim and his colleague, co-incidentally, also a "Jim"?

The Customer's representatives were everywhere but, fortunately, we were scheduled last in the proceedings.

Soon it was our turn.

"After a coffee", was the consensus.

Sigh of relief; the two Jims appeared looking rather sheepish.

It turned out they'd been celebrating the night before and tagged onto a German couple who'd offered them a midnight trip on their motor boat on the nearby lake.

Jim said he remembers hanging spread-eagled over the front bit of the boat and whizzing up and down Lake Konstantz in a semi-paralytic state.

When he woke up that morning he was in a tent on the wrong side of the Lake with the other Jim snoring beside him.

They had only just managed to get back.

Coffee break was now over and a small group gathered round the multiplexers.

Somewhere consoles were fired up and data began to flow.

About fifty feet away a large line printer hummed into life and started clattering.

Jim leaned nonchalantly on the Multiplexer under test.

People went hither and thither poking at keyboards, peering at printouts and generally looking busy.

A few times the steady clattering from the printer seemed to hesitate as if the machine was catching its breath.

Finally, after about half an hour, silence reigned and a heap of printouts was gathered up.

"How did it go", I asked the young Belgian chap in charge.

He smiled, "Fine he said; maybe there's a few queries but all went OK".

I looked at Jim.

He was still standing in the same position he'd adopted half an hour earlier.

I looked more closely.

His hand was over the alarm lamp and his thumb was next to the RESET button.

"You didn't", I said.

"You don't want to know", he said.

The next morning, in the Hotel lobby, after a memorable evening wining and dining "the Customer", I asked about the young Belgian.

"He's not very well" was the response.

"We're due to fly back later this morning", I said, "and I need the certificate so I can claim a stage payment".

"Don't worry he's sharing your minibus to the airport you can get him to sign it on the way".

He did, and that was the first time I'd ever seen anyone with a green face before, as the bus wended its way up and down the mountain roads between Konstantz and Zurich.

Later, Jim was talking to an engineer from a French Company who were also involved in the project.

They compared notes.

THEY had just received a communication from Signetics, a big player in integrated circuits.

"Throw away your 82S06 chips and fit 82S106 chips", I think was the gist.

"The old ROMs are fusible link types and we've found that some of the fused links grow back again causing programs to be intermittent".

That was the answer!

We threw away all the rogue chips (dozens and dozens of them!) and programmed the newer devices.

We never had any more trouble after that.

To this day the two Multiplexers are probably still there, not in Konstantz but at the Air Traffic Control Centre in ……..I even got £30 for patenting them! (see Story 32)

Return to Stories Page