《microtcacern》由会员分享，可在线阅读，更多相关《microtcacern（25页珍藏版）》请在金锄头文库上搜索。
1、Modular Electronicsthe past, the present and the futureMarkus JoosCERNThe past: NIMThe present: PCI and PCIeSHB ExpressThe future: Serial interconnectsVXSATCATCANIM basicsInitially: NIM = Nuclear Instrument ModulesBut it was used outside of nuclear scienceTherefore: NIM = National Instrument Modules
2、But is was used outside of the USATherefore: NIM stands for NIM1st NIM standard: July 19641st commercial module: November 1964Module dimensions: 34 x 221 x 246 mmNIM logic levels:0 = 0A (0V)1 = -12 to -32 (typical -16) mA at 50 (-0.8V)NIM connector42 pins in total11 pins used for power2 logic pins (
3、reset & gate)pin 1 & 2 = +/- 3V (claimed by Wikipedia)29 reserved / spare pins1983 NIM digital bus (IEEE 488 GPIB)Rarely usedNIM the next generationNIM is still very aliveGeneral purpose NIM module with programmable logic (LabView, USB)16 Channel 12 bit 5 GS/s Flash ADC Waveform Digitizer with USB 2
4、.0 read-outPCI basicsPCI (Peripheral Component Interconnect): first standardized in 1991Initially intended for PC cardsLater spin-offs: CompactPCI, PXI, PMCMain featuresSynchronous timingBut wait cycles possibleClock ratesInitially 33 MHz. Later: 66 MHz, (PCI-X: 100 and 133 MHz)Bus widthInitially 32
5、 bit. Later: 64 bitSignaling voltageInitially 5 V. Later 3.3 V (-slot keying)TerminologyA data transfer takes place between an INITIATOR (master) and a TARGET (slave)Bus topology1 to 8 (depending on clock rate) slots per busBusses can be connected to form a treeAddress and data as well as most proto
6、col lines are shared by all devices; The lines used for arbitration are connected point-to-point; The routing of the interrupt request lines is more complicatedA system can consist of several Initiators and Targets but only one Initiator can receive interruptsInitiator & arbiterTargetTargetInitiator
7、bussed linesarbitrationPCI basics - 2Address spacesConfiguration spaceStandardized registers for the dynamic configuration of the H/W (plug-and play)I/O spaceFor device specific registersMEM spaceGeneral purpose space for registers and memoryCycle types (encoded in the C/BE3:0# lines)Single cyclesRe
8、ad / write of all 3 address spacesBursts MEM read / write (with special features for cache handling)(Typical) performanceSingle cycle: 2 (3 for read) - 10 clock cycles33 MHz / 32 bit: 66 MB/s - 10 MB/s64 MHz / 64 bit: 264 MB/s - 40 MB/sBursts:33 MHz / 32 bit: Max. 132 MB/s 64 MHz / 64 bit: Max. 528
9、MB/s PCI-X 133 MHz: 1.06 GB/sPCI-PCI bridges add additional delaysPCI devices under LinuxThe command “lspci” displays information about the PCI devices of a computerShow PCI tree: lspci -t -vShow device details: lspci -v -s 02:03.0 -0000:00-+-00.0 Intel Corporation E7520 Memory Controller Hub +-00.1
10、 Intel Corporation E7525/E7520 Error Reporting Registers +-01.0 Intel Corporation E7520 DMA Controller +-02.0-0000:01-03-+-00.0-0000:02-03.0 CERN/ECP/EDU Unknown device 0144 | +-00.1 Intel Corporation 6700/6702PXH I/OxAPIC Interrupt Controller A | +-00.2-0000:03-01.0 CERN/ECP/EDU Unknown device 0144
11、 | -00.3 Intel Corporation 6700PXH I/OxAPIC Interrupt Controller B +-04.0-0000:04-00.0 Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI Express +-05.0-0000:05-00.0 Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI Express +-06.0-0000:06-08-00.0-0000:07-08-+-04.0 Broadcom Corpora
12、tion NetXtreme BCM5714 Gigabit Ethernet | +-04.1 Broadcom Corporation NetXtreme BCM5714 Gigabit Ethernet | -08.0-0000:08-+-06.0 Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet | -06.1 Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet +-07.0-0000:09-0b-+-00.0-0000:0a-02.0 CERN/ECP/EDU U
13、nknown device 0144 | +-00.1 Intel Corporation 6700/6702PXH I/OxAPIC Interrupt Controller A | +-00.2-0000:0b-01.0 CERN/ECP/EDU Unknown device 0144 | -00.3 Intel Corporation 6700PXH I/OxAPIC Interrupt Controller B +-1d.0 Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #1 +-1d.1 Intel Cor
14、poration 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #2 +-1d.2 Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #3 +-1d.3 Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #4 +-1d.7 Intel Corporation 82801EB/ER (ICH5/ICH5R) USB2 EHCI Controller +-1e.0-0000:0c-01.0 ATI Techno
15、logies Inc Rage XL +-1f.0 Intel Corporation 82801EB/ER (ICH5/ICH5R) LPC Interface Bridge -1f.3 Intel Corporation 82801EB/ER (ICH5/ICH5R) SMBus Controller02:03.0 Co-processor: CERN/ECP/EDU Unknown device 0144 (rev ac) Subsystem: Unknown device 2151:1087 Flags: bus master, 66MHz, medium devsel, latenc
16、y 32, IRQ 209 Memory at d7200000 (32-bit, non-prefetchable) size=512 I/O ports at 2000 size=256 Memory at d8000000 (32-bit, non-prefetchable) size=16M Capabilities: PCI protocol Example: Single cycle read“#” indicates active low signals1)Assertion of FRAME starts cycle2)Initiator puts address and co
17、mmand (cycle type) on the bus3)The Initiator signals that it is ready to receive data4)The initiator uses the C/BE lines to define which bytes it wants to read5)Target looks at the Address and drives DEVSEL if it was addressed. If no target drives DEVSEL after at most 6 clock the Initiator will abor
18、t the cycle 6)The ownership of the AD lines changes from Initiator to target (only for read cycles). This requires one clock cycleCLOCKFRAME#A/DC/BE#IRDY#TRDY#DEVSEL#AddressDataBUS CMDBE#s1234567Address phaseData phase91087)The Target does not yet drive TRDY (it may need time to prepare the data) bu
19、t asks the Initiator to wait8)The Target has the data ready on the AD lines. The Initiator fetches the data in the same clock cycle9)By de-asserting FRAME the Initiator tells to the Target that it does not want additional data after the next data word10)The cycle is over and the protocol lines get r
20、eleased Color code: Initiator Target - sharedPCIe (aka PCI Express)Not a bus any more but a point-to-point linkData not transferred on parallel lines but on one or several serial lanesLane: One pair of LVDS lines per direction (4 wires)Clock rate: 2.5 GHz (PCIe2.0: 5 GHz, PCIe 3.0: 10 GHz)8b/10b enc
21、oding250 MB/s (PCIe 1.0) raw transfer rate per laneDevices can support up to 32 lanesProtocol at the link layer has nothing to do with protocol of parallel PCIFully transparent at the S/W layerPC motherboard with PCI slots32 bit slot with 5V key64bit slot with 3.3V key6U CompactPCI chassis and cardP
22、XI systemPMC card and carrier (VMEbus)Some examples of PCI H/WParallel bus - Serial linkParallel Buses Are Dead! (RT magazine, 2006)What is wrong about “parallel”?You need lots of pins on the chips and wires on the PCBsThe skew on the data lines limits the maximum speedSpeed is a function of the len
23、gth (impedance) of the linesWhat is wrong about “bus”?Communication is limited to one master/slave pair at a time (no scalability)The handshake may slow down the maximum speedAll parallel buses are dead. All? No!There is lots of legacy equipmentVMEbus is still used heavily (military / research)PCs s
24、till support parallel PCI (but this will change)What next?Switched serial interconnectsseriallinkSerial linksStandards (just the most important)PCIe1 / 10 GB EthernetSerial RapidIOInfinibandSerial ATAFiberChannel.CommonalitiesSignal rate: 2.5 10 GHzPacket switchingTopologyStar: Device connects to a
25、fabric switchDual Star: Device connects to two fabric switches (redundancy)Mesh: All devices have direct links to all othersDifferencesSupport for interruptsSupport for programmed I/OQuality of service (guaranteed bandwidth)SwitchDev.Dev.Dev.Dev.StarDev.Dev.Dev.Dev.MeshPICMG 1.3 SHB Express SHB Expr
26、ess = System Host Board standard for PCIeStandardized in 2005Defined in the standardSHB board mechanics (two board formats) Interface between SHB and backplaneAdditional I/O (SATA, USB, Ethernet, etc.) that may be routed to the backplaneBackplane design rulesSystems consist of:One SHB One backplaneO
27、ne or several PCIe, PCI-X or PCI cardsThe SHBTwo (A & B) or 4 (A, B, C & D) connectorsConnector A: PCIe(1 x16) or (2 x8) or (1 x8 + 2 x4) or (4 x4)Connector B: PCIe(1 x4) or (4 x1)Connector C:Additional I/OConnector D:1 32bit PCI(-X)ABCDSHB the backplanesThe backplane has to match the PCIe configura
28、tion of the SHBx16 on connector A: graphics class2 x8 on Connector A: server classSHB slotPCIe slotsPCI(-X) slotsSegmented backplane with 4 SHB and 12 PCIe slots for a 19” 4U chassisA complete 4U systemPCIe switchPCI bridgeVITA41: VXSPICMG 3.x: ATCA (Advanced Telecommunications Computing Architectur
29、e)PICMG MTCA.x: MicroTCA/uTCAPICMG AMC.x: Advanced Mezzanine Card (for ATCA and uTCA)Not covered in this talk:VITA46: VPXPICMG 2.x: Compact PCI (cPCI)PICMG EXP.0: PCIe for cPCIPCIMG CPLUS.0: A bit like EXP.0PICMG ATCA300.0: ATCA for 300mm deep systems (no rear I/O)And many moreThe next generationWha
30、t new standards are available?VXS (VITA 41, 100 pages)Essentially 6U (but 9U not excluded) VMEbus with a new P0 connectorTwo types of cardsPayloadSwitch (one card required, second for redundancy)Network topology: (dual) starConnectivity for payload cards16 differential pairs (10 GHz) defined by the
31、standard (and routed to switch cards)31 reserved pins available on P0Sub-standards41.1: Infiniband41.2: Serial RapidIO41.3 (draft): IEEE Std 802.3 (1000 Mb/s Ethernet)41.4 (draft): PCIeHot Swap: According to VITA 1.4System management based on I2C / IPMI but only formulated as recommendationAdvanced
32、TCA (650 pages + IPMI)More of a system than a board standardStarted in 2001 by 100 companiesOne form factorFront: 8U x 280 mm x 30.48 mm (14 slots per 19” crate)Rear: 8U x 60 mm (5W)Supply voltage: -48 V (- DC-DC conversion each on-board)Power limit: 200 W (400 W) per cardConnectorsZone 1: One conne
33、ctor for power & system managementZone 2: One to five ZD connectors for data transferZone 3: User defined connector for rear I/OConnectivityUp to 200 differential pairs4 groups64 pairs for Base Interface (usually Eth., star topology)120 pairs for Fabric Interface (star or full mesh)Ethernet, PCIe, I
34、nfiniband, serial RapidIO, StarFabric6 pairs for Clock Synchronization10 pairs for Update ChannelSystem management based on IPMI, I2C and FRU dataZone1Zone3Zone2ATCA HA features (applies also largely to TCA)RedundancyPower Supply modulesVentilatorsShelf managersSwitch bladesElectronic KeyingBased on
35、 FRU information payload cards may be accepted / rejected in a given slotHot swapPayload board will only receive (payload) power if the shelf manager can guaranty for the availability of the required resources (power, cooling, signal connections) Monitoring Low level: IPMI on I2CHigh level: SNMP (Si
36、mple Network Management Protocol) and other protocols on top of TCP/IP System event logsCoolingDynamically controlled fans and several alarm levelsSome ATCA equipmentShelvesShelf manager(s)AMC CarriersCutaway carrier (for full-size AMCs)Hot-swap fansRTMBackplane:Dual starAMCOriginally intended as ho
37、t-swappable mezzanine standard for ATCA but soon used as the basis for the TCA standard6 form factors:74 or 149 mm wide13, 18 or 28 mm high180 mm deepPower supply: 80W (max) on +12V (and 0.5W on 3.3V management power)Connector: 85 pin (single sided) or 170 pin (double sided) edge connectorConnectivi
38、tyUp to 12.5 Gb/s20+20 LVDS signal pairs for data transfer (Eth, PCIe, SAS/SATA, RapidIO)Clock interface, JTAG, I2C (IPMI)TCAA system standard based on the AMC, standardized in 2006Min. signaling speed: 3.125 GHzConnectivity: 4 AMC LVDS pairs defined as “Common Options” (2 Eth. & 2 SAS ports) and co
39、nnect to 1 or 2 MCH boards which provide the switching8 AMC LVDS pairs defined as (extended) fat pipes (1 or 10 G Eth, PCIe, RapidI/O). Connection to MCH not standardizedRemaining 8 LVDS pairs not defined (can be used for rear I/O (but rear I/O not foreseen in uTCA standard) System management based
40、on IPMI / I2CHot-swap support for PSU & coolingRedundant MCH (TCA Controller Hub)The MCH connector supports up to 84 differential pairs. Therefore only 7 pairs per AMC (based on a 12-slot backplane) can be routed to the switch.Some uTCA products6 mid size (single or double width) AMCsAC or DC PSUSin
41、gle star backplane19” rack mountable8 full size and 4 compact size AMC slotsFor 3rd party power supply modules2U / 19” chassisSlots for up to 12 AMCsCooling dimensioned for 40W per slot19” rack mountableDual star backplaneUp to 10 AMCsExternal AC-DC PSU requiredMCH Fat-pipe mezzanines for:PCIe10GB-E
42、thSerial RapidIOClocksATCACommunication protocol(s) on the fabric channels Routing of the fabric channels on the backplane (network topology)Connection between front board and RTMDegree of redundancyPower supply at shelf level (230 VAC or -48 VDC)AMCCard height (13, 18 & 28 mm)Card width (74 & 149 m
43、m)Communication protocols (currently 4 options)Number of pins on the connector (85 or 170)JTAG supportuTCAAMC height & widthDegree of redundancy (MCH, PSU, cooling)Routing of the fabric channels on the backplaneJTAG supportConnectivity of MCH to backplane (1 to 4 tongues) and type of communication p
44、rotocol on the fat pipesxTCA degrees of freedom (not necessarily a complete list)xTCA issuesThe operation of an xTCA system requires a complex, standard compliant S/W infrastructureEfforts to provide open source management S/W for xTCA: OpenSAF, SAForumAs many features of the standard(s) are optiona
45、l, products from different vendors may not be compatible Efforts to insure interoperability of xTCA products: CP-TA, SCOPE allianceMany vendors seem to be in favour of “profiles” that limit the number of options given by the standardsATCA profile for telecommunicationProposal for a “physics profile”
46、 for xTCA under the umbrella of PICMG-USThe market does not yet provide lots of front end modules for physics DAQThere is little information available about the system performance (end to end H/W performance and S/W overhead) of the data transfer linksWhat is the right standard for me?This obviously
47、 depends on your requirementsBandwidth & latencyAvailability of commercial products (front end)Existing infrastructure (S/W and H/W) and expertise in your experimentStart and duration of the experimentScalability requirements Trends in HEPLHC & experiments CERN: VMEbus & PCI basedCMS: Calorimeter tr
48、igger prototype in TCAATLAS: lAr ROD prototype in ATCALHC: prototype time distribution system in TCA COMPASS CERN: VXS derivativeXFEL DESY: control system in TCAInfinibandDeveloped by Compaq, IBM, Hewlett-Packard, Intel, Microsoft and Sun from 1999 onwardsCharacteristicsBi-directional serial linkAgg
49、regation of links (4x, 12x possible)Link speed: 2.5, 5, 10 GHzSpecial featuresData transfer performed without involvement of OS (latency 2 s)Remote DMA (fetch data from the memory of a remote system)Main purposeServer and storage interconnect for high performance computingRelevance for DAQLimited fo
50、r a lack of DAQ products Serial Rapid I/ODeveloped by Mercury Computer Systems and Motorola from 1997 onwardsCharacteristicsBi-directional serial linkAggregation of links (2x, 4x, 8x, 16x possible)Link speed: 1.25, 2.5, 3.125, 5, 6.25 GHzSpecial featuresQuality of Service (transfer requests can be prioritized)MulticastMain purposeChip/chip and board/board communicationRelevance for DAQLimited for a lack of DAQ products but some AMC/ATCA products
