Status of Assembled Hit Finder Boards

Last updated on $Date: 2000/07/17 00:43:12 $ (UTC)

• Executive summary:

• 49 good, 0 bad total rev B boards, out of 42 needed (including L00), leaving 7 hot and 0 cold spares

• 2 rev B boards good after SMT rework

• 27 rev B boards good first try

• 16 rev B boards good after we fixed minor problems

• 4 rev B boards good after Twin Hunter replaced BGA chips

• 0 rev B board still not working

• 2 rev A boards working, with some kludges, different firmware, etc.

• Board 01 - March 1999 prototype

• Assembled by ADCO; arrived 1999-03-24

• Has broken DAD pin; requires "pinswap" version of DAD program as a work-around

• Otherwise, no known problems

• Originally, hf_vme_addr bit 11 tracked bit 14; traced problem down to HF (not boot) side of HCT244 buffer (U106), presumably pins 9,16; was never able to demonstrate problem with ohm meter, but WJA fixed 1999-03-29 by removing and resoldering U106

• Originally, grt_data bit 94 (highest data bit of stream 9) was stuck at 0; did not see any DC short with ohm meter; problem between P3-D47 and U92-230; broke pin U92-230 while trying to unsolder it; replaced with U92-240 as a kludge (WJA 1999-06-04)

• Currently at Fermilab as Vertical Slice Test data source

• Board 02 - March 1999 prototype

• Assembled by ADCO; arrived 1999-03-24

• No known problems

• Originally, grt_data bit 98 (clock input for G-link #2) was stuck; presumably bad connection on U92-211; Mircea fixed 1999-04-14 by cleaning U92 with alcohol

• Currently at Chicago as Track Fitter data source

• Board 03 - Jan 2000 pre-production

• Assembled by SMT; arrived 2000-01-26; reworked twice (arrived 2000-04-20 and 2000-05-03); installed at Fermilab 2000-05-05

• Summary: after two passes of SMT rework, board functions perfectly, and is installed at Fermilab

• Test log:

• No power shorts; power-up; power LED OK

• Both EPC2 and Boot chip seen by JTAG programmer

• Boot chip programmed directly via JTAG

• Load mode LED on; error LED flashing to indicate unconfigured Flex chain, OK

• Able to read brain register via VME; DTACK flashes; change mode to load/run/test; see LEDs

• Configure EPC2; EPC2 correctly programs Boot chip

• Load FRAM OK; Boot chip programs Flex chain OK; program Merger chip via JTAG OK

• Temporary confusion due to pilot error, described below

• Can W/R random contents to CRAM, ISPY, PedRAM, misc registers, OK

• Can send test patterns from GSTM through to ISPY; all data bits are OK for all streams from P3 to DAD to HitMan to ISPY

• FIFO W/R detects problems with stream 4 (bit 9), stream 8 (FIFO is always empty), stream 9 (bits 0-9) (out of 18 total bits/stream, 10 streams) (problems on streams 8 and 9 are now fixed; problem on stream 4 remains)

• OSPY W/R detects that bit 16 is stuck off, and bit 7 is floating

• problem with bit 7 seems to be between MOP and OSPY, as bit 7 works for I/O to other MOP registers

• problem with bit 16 seems to be between MOP and HF_VME_Data bus, as other VME reads from MOP have same problem, but VME reads from e.g. ISPY do not show problems with this bit

• Except for the bad bits, cluster data are correctly processed and merged, until stream 8's permanently empty FIFO stalls the pipeline

• 2000-02-04: On stream 8 FIFO (U65), clock/control signals were seen on both sides of FIFO with scope; pin 19 (wclk) was bent; several pins appeared not to be soldered; Mircea resoldered; problem fixed

• On stream 9 FIFO (U59), many pins in a row were not soldered; Mircea resoldered; problem fixed

• Stream 4 FIFO input bit 9 is floating, because HM4 (U37) pin L22 (fifodata9) appears not to be soldered to the BGA; we verified that trace is connected to FIFO; scope does not see activity on trace when BGA output is driven; JTAG scan does not see change in state when voltage is applied to trace

• MOP VME data bit 16 appears to be a missing solder connection on MOP (U30) pin P22, as JTAG does not see stimuli applied to this bit via TC9-19 (but does see stimuli on TC9-18, e.g.)

• MOP OSPY data bit 7 appears to be a missing solder connection on MOP (U30) pin J22, as JTAG does not see stimuli applied to this trace, but trace is clearly connected to OSPY (U7) pin 59.

• Went through list of PCB changes (prototype vs production) on 2000-02-01, and found everything to be OK

• EPC2 clearly works

• All JTAG chains have been used to detect/identify/configure their respective chips

• Flex configuration works, so pullup resistors on CONF_DONE and nSTATUS are evidently doing their job

• GSTM data move successfully into ISPY buffers, so HitMan dadclock input clearly works; hf_init was shown to be working by looking at FIFO reset signal on a scope; vmeaddr0-2 and vmeclock functionality demonstrated by W/R test to pedestal RAM (load entire memory with random constants and read it back)

• FRAM contents OK after 5V power on with 3V fuse removed, which was reason for pulldown on RP* line; check with ohm-meter confirms presence of pull-up/down resistors (on RP* and VPEN); voltages have desired values when 3V fuse is removed

• Able to program FRAM, so its VPEN voltage is OK now

• Power LED now requires both 3.3V and 5V power to be present; checked by removing each fuse in turn

• LEDs (e.g. HOLD light) now light when they should, not when they shouldn't; in particular HOLD LEDs behave properly

• Front panel fits comfortably, so LED mount positions are fine

• Pilot error:

• Signals sent from MOP & Boot chips for DS, HOLD, CdfError, SvtError LEDs were inverted, because that was what had been required to make them work with kludges on prototypes--now fixed in firmware

• Initially loaded old (prototype) version of HitMan program, which caused many temporary problems that I chose not to describe above--correct version now loaded

• back from SMT rework 2000-04-20

• powers up, boots

• undid change that made hitman E2,F2 dummy input pins (didn't affect this board anyway)

• data path from all HM chips to FIFOs to Merger to MOP is fine; problem with stream 4 FIFO is fixed

• MOP connections to OSPY and VME bus are fine; OSPY random fill/read now works

• looping 50-event burst/check test using ISPY as data source works (650 loops OK and counting); but doesn't work from GSTM (new problem) for streams 1 and 5

• I see that HM1,5 (U35,U36) think that data valid is always asserted; also true coming out of DAD (TC3-15); not true when I unconfigure the chips; need to track down with boundary-scan

• News flash! Mircea discovers that about 40 pins on U92 (a DAD chip) are not soldered to the board!

• back from second SMT rework 2000-05-03

• Passes all tests

• Board 04 - Jan 2000 pre-production

• Assembled by SMT; arrived 2000-01-26; reworked board arrived 2000-04-20; installed at Fermilab 2000-05-02

• Summary: after SMT rework, board functions perfectly, and is installed at Fermilab

• Test log:

• Initially, +3.3V was shorted to GND; Mircea and Harold traced the problem to a solder-blob short beneath the upper-right corner of U33 (HM3) and cleaned out the solder blob

• SMT agreed to use a better technology (different flux, e.g.) when assembling remaining boards

• Power-up, power LED OK

• Merger chip programmed OK

• EPC2 and Boot chip seen via JTAG; EPC2 programmed OK; EPC2 loads Boot OK; VME I/O to Boot chip OK; program FRAM OK

• Auto-boot of 13-chip Flex chain fails, even though FRAM contents are correct; need to use JTAG to diagnose, narrow down to a particular chip (now fixed, see below)

• 2000-02-02: With scope, I see that CONF_DONE is coming up about 14 clock cycles before end of second dad.ttf (end of Flex chain), indicating successful configuration, but that about 5 usec thereafter, nCONFIG is pulled back down for 150 nsec, after which the chips return to their unconfigured state

• Hypothesis is that one of the 13 chips in the Flex chain (probably a BGA) has an output shorted to nCONFIG

• By loading "empty" 10K50 firmware (no logic; all pins are inputs) into MOP and HitMen, I was able to get board to configure, consistent with hypothesis above

• Narrowed problem down to HM8; can load normal firmware into other twelve chips in Flex chain

• Now time for a few tests on the chips that are working:

• misctest OK; ospytest OK; ispytest OK; pramtest OK; cramtest OK; fifotest OK except stream 7 bit 14; board ID switches OK; dadtest OK

• 2000-02-03: Both ohm-meter and JTAG boundary scan indicate that HM8 (U38) has a short between pin D2 (nCONFIG) and pin E2 (an unused pin, marked RESERVED in the compiler report file)

• First use of boundary scan was misleading, because I got the sign wrong and was looking at chip two instead of chip four in a five-chip chain

• Changing pin E2 to a dummy INPUT pin is a successful work-around for the short beneath HM8

• But stream 7 FIFO (U61) bit 14 still stuck at GND (now fixed, see below)

• Four possible locations: output of HM7, input of FIFO, output of FIFO, input of Merger; last is ruled out by other working streams

• See activity on scope on both input (pin 2) and output (pin 48) of FIFO; notice that "fifotest" on stream 7 succeeds when scope probe is on pin 48

• Problem is fixed by heating pin 48 with iron

• Board now processes data successfully, 48 events at a time, in burst/check mode, for ~ 5000 iterations (1.5 hours)

• Connected each output connector in turn to TF; checked data in TF ISPY; no problems

• Ran random test (burst/check) overnight for ~ 15 hours, 40K loops of 48 events, checking both HF OSPY and TF ISPY; no problems

• 2000-02-06: Am pumping data continuously from GSTM, polling OSPY with C code on crate CPU

• GSTM loops over 50 events; 12686 words in each FIFO, of which 63% have DAV asserted

• 12686 words / 55 MHz = 230 usec per loop, or 4350 loops/sec, or 217K events/sec

• HF output is 5706 ospy words/loop, 24M words/sec

• polling program checks about 64K words about 7.5 times/sec, or 490K words/sec, or 2% of total HF throughput

• this is much more throughput than HF would see in real beam data

• 100KB of SIXD (seems like a plausible upper bound; do we have a hard number?) shared between 36 hit finders at a 50 kHz event rate would put DAV up ~ 25% of the time, compared with 63% in the test

• Guram's "B10" svtsim sample gave an estimate of 36 clusters per HF per event, which would be about 2M words/sec at 50 kHz event rate, compared with 24M words/sec in the test

• in ~ 21 hours, I checked 36.8 x 10^9 ospy words, no errors

• back from SMT rework 2000-04-20

• powers up, boots, processes data in burst/check mode from GSTM

• undid change that made hitman E2,F2 dummy input pins--still OK!

• processed 1700 loops of 50-event burst/check test--OK!

• I think the board is fine

• Board 05 - Apr 2000 pre-production ("first item")

• Assembled by Twin Hunter; arrived 2000-04-28; installed at Fermilab 2000-05-02

• Summary: Board is perfect, and is installed at Fermilab

• Test log:

• Mircea checked for shorts, powered up, configured EPC2

• VME I/O then worked right away; Bill loaded FRAM while Mircea configured Merger chip

• All single-board VME tests passed

• Burst/check test from GSTM has run for ~ 21 hours without failing: 73K loops, 3.5M total events

• I see that CDF error and SVT error LEDs have on/off inverted: problem was that older version of boot program had been loaded (fix was put in 2000-01-29); now correct program is loaded

• Interesting note: it seems that boot program compiled with Altera v9.5 does not correctly configure 13-chip Flex chain, while boot program compiled with altera v9.21 (what we were using before) is fine

• Both FP outputs OK to TF; all LEDs OK

• Continuous test from GSTM, polling OSPY, checked 6.54 x 10^10 OSPY words over weekend, no errors.

• Board 06 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Standard set of tests is this:

• configure boot, merger; configure FRAM; check ID prom

• run python vme tests

• run ispy data flow test 100 loops, then 20 loops to TF per connector; check LEDs

• run gstm burst 100 loops; run gstm continuous 5 minutes (150M ospy words) on crate cpu

• Weird things occasionally seen:

• ispy test mode can be screwed up at some low rate by presence of vtm? (so I use non-vtm slot for this test)

• boards sometimes need reseating in slots

• Board 07 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• bfifoio bit 15 was stuck on; fixed solder bridge at U63-50/49 (Bill)

• Board 08 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Board 09 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Board 10 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Board 11 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• delayed_modsel_ OK; delayed_DS_ OK; AM1_ OK; configuration via backup port works (can talk to single register on each chip), but after I turn "backup" mode off, chips become unconfigured

• removed solder bridge LED3-1/3 (Mircea)

• Clearing FRAM lock bit before FRAM erase fixes write problem

• GSTM test passes (1500 loops) if I disable stream 1 in test

• Back from Twin Hunter rework (replaced HM1); board did not respond to VME i/o, until delay line U81 was replaced. Now passes all GSTM tests.

• Board 12 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Board 13 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Fixed solder bridge on U92-200/201 (Mircea)

• Board 14 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Board 15 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Board 16 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Board 17 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• hm_data(117), connecting U92-87 to U35-A15, is shorted to ground (1 ohm with meter); suspect beneath BGA, where pin A14 is ground; removing debris beneath U35 (Mircea) fixed problem

• Board 18 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• hf_vme_data, bits 13 and 15, are driven low, apparently by U39 (HM2); no short found with meter

• after Twin Hunter rework (replaced HM2), board passes all tests

• Board 19 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Board 20 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Board 21 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• wrong LED (G/G) on lower hold/DS replaced with R/G (Mark)

• Board 22 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• removed a number of hairline solder bridges (Bill) from top row of U91 (even caused ispy burst tests to fail occasionally!)

• Board 23 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Board 24 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Fixed solder bridge on U92-65/66 (Mircea)

• Board 25 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• replaced U89 (Mark) because output did not change in response to input on VME address lines 18-21

• fixed solder bridge U17-31/32 (Bill); replaced U105 (Bill), which unfortunately resulted in a ruined pad, requiring a jumper

• U22 is upside down; fixed by Mark

• Board 26 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Board 27 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Board 28 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Board 29 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Board 30 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Replaced U107 (Mark); fixed solder bridge U13-31/32 (Mircea); fixed cold solder joints U68-18/20 (Mircea)

• Board 31 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Board 32 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Board 33 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Board 34 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• ispy 0 bit 9 was stuck at 0; fixed unsoldered pin U18-8 (Bill)

• Board 35 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• wrong LED (G/G) on upper hold/DS replaced with R/G (Mark)

• Board 36 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Board 37 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Fixed solder bridge U61-94/95 (Mircea)

• Board 38 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Board 39 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Board 40 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Fixed solder bridge on U13-45/44 (Bill)

• Board 41 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Broken trace hm_vme_data(10), just left of U37, jumpered (Mircea/Mark); solder bridge U92-41/42, removed (Mircea)

• Board 42 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Fixed two solder bridges on U91, at approximately pin 95 (Bill)

• Board 43 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• suspect HM3 is bad, but not at all certain

• disabling stream 3 brings ~ 100% failure rate on GSTM test down to ~ 0.03% (2/6000), now on stream 8 (disabling 8 also gives 0 errors after 11600 loops)

• low speed gstm test works now for ~ 200 loops, but high speed test fails

• second rework (replaced HM8) done 2000-07-13; GSTM tests now pass!

• Board 44 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• all non-GSTM tests pass, GSTM tests fail, always on stream 5, but ispy data look fine; HM5 dadclock looks fine

• Replaced U81 (Mark), since delayed_modsel_ was dead (fixes bus error on block transfer); clearing FRAM lock bit before FRAM erase fixes write problem

• With stream 5 disabled, GSTM test passes (1650 loops)

• Back from Twin Hunter rework; now passes GSTM tests! (June 23)

• Board 45 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Fixed U61-2,4,5 unsoldered pins (Bill)

• Board 46 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Board 47 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Replaced capacitors C16,17 (Mircea), but it was a red herring; fixed short U91-179/180, which was making boot fail (and changing effective capacitance); removed solder bridges (Mircea) U91-225/.../230

• Board 48 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Board 49 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Board 50 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests

• Board 51 - May 2000 production by Twin Hunter (arrived 2000-05-17)

• Summary: passes all tests