《NPACIRocksTutorial》由会员分享，可在线阅读，更多相关《NPACIRocksTutorial（96页珍藏版）》请在金锄头文库上搜索。
1、NPACI Rocks TutorialNPACI All Hands MeetingMarch 18, 2003www.rocksclusters.orgSchedulenRocks 101 (45 mins)Introduction to RocksCluster Architecture (HW & SW)Rocks software overviewnAdvanced Rocks (45 mins)Modifying the default configurationCluster Monitoring (UCBs Ganglia)nHands on Labs (90 mins)Bre
2、akout into groups and build an x86 clusterRun an MPI jobCustomization (packages, monitoring)Compare to IA64 clusterCopyright 2003 UC RegentsRocks 101Make clusters easynEnable application scientists to build and manage their own resourcesHardware cost is not the problemSystem Administrators cost mone
3、y, and do not scaleSoftware can replace much of the day-to-day grind of system administrationnTrain the next generation of users on loosely coupled parallel machinesCurrent price-performance leader for HPCUsers will be ready to “step up” to NPACI (or other) resources when needednRocks scales to Top5
4、00 sized resourcesExperiment on small clustersBuild your own supercomputer with the same software!Copyright 2003 UC RegentsPastnRocks 1.0 released at SC2000Good start on automating cluster buildingIncluded early prototype (written in Perl) of GangliaResult of collaboration between SDSC and UCBs Mill
5、ennium group.nRocks 2.xFully automated installation (frontend and compute nodes)Programmable software configurationSCS (Singapore) first external group to contribute software patchesCopyright 2003 UC RegentsPresentnRocks 2.3.2 (todays lab is part of our beta testing)First simultaneous x86 and IA64 r
6、eleasenProven Scalability#233 on current Top500 list287-node production cluster at StanfordYou can also build small clustersnImpactRocks clusters on 6 continentsnNo Antarctica yet.4 large NSF ITRs using Rocks as core software infrastructureCopyright 2003 UC RegentsRocks Registration Page (5 days old
7、)http:/www.rocksclusters.org/rocks-registerRocks in the commercial worldnRocks Cluster VendorsCrayDellPromicro SystemsnSee page 79 of Aprils Linux JournalSCS (in Singapore)nContributed PVFS, SGE to RocksnActive on the Rocks mailing listnTraining and SupportIntel is training customers on RocksCallide
8、nt is offering support servicesCopyright 2003 UC RegentsPromicro SystemsCopyright 2003 UC RegentsAlternative Cluster ToolkitsnCplantDOE clustering softwarenOpenMOSIXSingle System ImagenOpenSCEKasetsart University, ThailandCompatible with RocksnOSCARAka cluster-in-a-box, MSC.Linux“Best-practices of c
9、lustering”Packaging effort, not a cluster distributionnScyldBproc based system (SSI process space)CommercialnSCoreOldest clustering effortTouches on all aspects of clustering software Copyright 2003 UC RegentsWhat sets us apartFully automated cluster deployment1.Get and burn ISO CD (DVD for IA64) im
10、age from http:/www.rocksclusters.org2.Boot frontend with CD/DVD3.Fill out 7 configuration screens (mostly Red Hat)4.Reboot frontend machine5.Integrate compute nodes with insert-ethers6.Ready to go!Complete out of the box solution with rational default settingsIdentical environment for x86 or IA64Cop
11、yright 2003 UC RegentsTestimonials and ComplaintsFrom the Rocks-discuss mail list and other sourcesNew User on Rocks Mail List“I managed to install Rocks with five nodes. The nodes have a small HD 2.5 GB each, the cluster is in my home on a private network behind a Linux box firewall. And it looks l
12、ike everything is working fine. I can see all the nodes and the front-end in the ganglia web interface. I built it so I can learn more about clusters.And to tell the truth I have no idea on what to do with it, I mean where to start, how to use it, what to use it for.”Copyright 2003 UC RegentsPower U
13、sersnResponse to previous poster“Its sometimes scary how easy it is to install Rocks.”This coined the phrase “Rocks, scary technology”nComment from a cluster architect“You guys are the masters of the bleedin obvious.”This one flattered usCopyright 2003 UC RegentsAnother New User“Ive set up a Rocks C
14、luster thanks to the bone simple installation. Thanks for making it so easy. The drawback, because it was so easy, I didnt learn much about clustering.”Copyright 2003 UC RegentsIndependent Survey of Clustering Softwarenhttp:/heppc11.ft.uam.es/Clusters/DocCompares Rocks, OSCAR and others“NPACI Rocks
15、is the easiest solution for the installation and management of a cluster under Linux.”“To install and manage a Linux cluster under OSCAR is more difficult than with Rocks.”n“With OSCAR it is necessary to have some experience in Linux system administration, and some knowledge of cluster architecture”
16、nOur goal is to “make clusters easy”Automate the system administration wherever possibleEnable non-cluster experts to build clustersCopyright 2003 UC RegentsAnd Finally a weekly messagen“Your documentation sucks.”nGuilty, but improvingRocks now installs users guide on every new clusterMailing list h
17、as several extremely helpful usersCopyright 2003 UC RegentsHardwareBasic System ArchitectureCopyright 2003 UC RegentsWhy x86 clusters?Source: Dave Pierce, SIOCopyright 2003 UC RegentsMinimum Componentsx86 / IA64 serverLocal HardDrivePowerEthernetCopyright 2003 UC RegentsOptional ComponentsnMyrinet h
18、igh-performance networknNetwork-addressable power distribution unitnkeyboard/video/mouse network not requiredNon-commodityHow do you manage your management network?Crash carts have a lower TCOCopyright 2003 UC RegentsSoftwareCluster Software StackCopyright 2003 UC RegentsCommon to Any ClusterCopyrig
19、ht 2003 UC RegentsRed HatnStock Red Hat 7.3 w/ updates (AW 2.1 for IA64)nLinux 2.4 KernelnNo support for other distributionsRed Hat is the market leader for LinuxnIn the USnAnd becoming so in EuropenExcellent support for automated installationScriptable installation (Kickstart)Very good hardware det
20、ectionCopyright 2003 UC RegentsBatch SystemsnPortable Batch SystemMOMnDaemon on every nodenUsed for job launching and health reportingServernOn the frontend onlynQueue definition, and aggregation of node informationSchedulernPolicies for what job to run out of which queue at what timenMaui is the co
21、mmon HPC schedulernSGE - Sun Grid EngineAlternative to PBSIntegrated into Rocks by SCS (Singapore)nScheduler manage scarce resourcesClusters are cheapYou might not want a schedulerCopyright 2003 UC RegentsCommunication LayernNone“Embarrassingly Parallel”nSocketsClient-Server modelPoint-to-point comm
22、unicationnMPI - Message Passing InterfaceMessage PassingStatic model of participantsnPVM - Parallel Virtual MachinesMessage PassingFor Heterogeneous architecturesResource Control and Fault ToleranceCopyright 2003 UC RegentsSockets are low levelnSocketsPoint-to-PointN machines = (n2 - n)/2 connection
23、s1, 3, 6, 10, 15, nMPI/PVMShared virtual channelImplementation could be socketsEasier to programCopyright 2003 UC RegentsRocks Cluster SoftwareCopyright 2003 UC RegentsCluster State ManagementnStatic InformationNode addressesNode typesSite-specific configurationnDynamic InformationCPU utilizationDis
24、k utilizationWhich nodes are onlineCopyright 2003 UC RegentsCluster DatabaseCopyright 2003 UC RegentsGanglianScalable cluster monitoring systemBased on ip multi-castMatt Massie, et al from UCBhttp:/nGmon daemon on every nodeMulticasts system stateListens to other daemonsAll data is represented in XM
25、LnGanglia command linePython code to parse XML to EnglishnGmetricExtends GangliaCommand line to multicast single metricsCopyright 2003 UC RegentsGanglia ScreenshotCopyright 2003 UC RegentsCluster Software ManagementSoftware PackagesnRPMsStandard Red Hat (desktop) packaged softwareOr your own addonsn
26、Rocks-distManages the RPM repositoryThis is the distribution Software ConfigurationnTuning RPMsFor clustersFor your siteOther customizationnXML KickstartProgrammatic System BuildingScalableCopyright 2003 UC RegentsBuilding a Rocks DistributionnStart with Red HatnAdd updates, Rocks (and optional othe
27、r) softwarenAdd Kickstart profilesnModify Red Hat installation boot imagenResulting in a Red Hat compatible Rocks distributionCopyright 2003 UC RegentsKickstartnRed Hats KickstartMonolithic flat ASCII fileNo macro languageRequires forking based on site information and node type.nRocks XML KickstartD
28、ecompose a kickstart file into nodes and a graphnGraph specifies OO frameworknEach node specifies a service and its configurationMacros and SQL for site configurationDriven from web cgi scriptCopyright 2003 UC RegentsSample Node File!DOCTYPE kickstart SYSTEM KICKSTART_DTD Enable SSH &ssh;&ssh;-clien
29、ts&ssh;-server&ssh;-askpasscat > /etc/ssh/ssh_config << EOF Host * ForwardX11 yes ForwardAgent yesEOFchmod o+rx /rootmkdir /root/.sshchmod o+rx /root/.sshCopyright 2003 UC RegentsSample Graph FileDefault Graph for NPACI Rocks. Copyright 2003 UC RegentsKickstart frameworkCopyright 2003 UC Re
30、gentsAppliancesnLaptop / DesktopAppliancesFinal classesNode typesnDesktop IsAstandalonenLaptop IsAstandalonepcmcianCode re-use is goodCopyright 2003 UC RegentsOptional DriversnPVFSParallel Virtual File SystemKernel module built for all nodesInitial support (full support in future version of Rocks)Us
31、er must decide to enablenMyrinetHigh Speed and Low Latency InterconnectGM/MPI for user ApplicationsKernel module built for all nodes with Myrinet cardsCopyright 2003 UC RegentsYour Cluster SoftwareCopyright 2003 UC RegentsLets Build a ClusterHardware SetupnGrab two nodesnUse the cross-over cable to
32、connect eth0 on the frontend to eth0 on the compute nodenHook up a monitor and keyboardCopyright 2003 UC RegentsSoftware InstallationnPut the CD in the frontendnAt the boot: prompt, type:frontendnStart configurationCopyright 2003 UC RegentsCluster InformationCopyright 2003 UC RegentsPartition your D
33、isksCopyright 2003 UC RegentsConfigure Private NetworkCopyright 2003 UC RegentsConfigure Public NetworkCopyright 2003 UC RegentsSet your hostnameCopyright 2003 UC RegentsSet the root passwordCopyright 2003 UC RegentsConfigure NISCopyright 2003 UC RegentsAnd away we goCopyright 2003 UC RegentsAdvance
34、d RocksAdd Compute Node with Insert-ethersnCollect the Ethernet MAC address of cluster nodesnOnly done once, during integrationnPopulates cluster databaseCopyright 2003 UC RegentsDiscovered ApplianceCopyright 2003 UC Regents192.168.254.254Remotely starting reinstallation on two nodes192.168.254.253M
35、onitor the Install with eKVCopyright 2003 UC RegentsGanglia GmetricsnHow to design new metrics for ganglia to track and graph.nCalled user-defined metrics, or gmetrics.1. Write gathering function in Python.2. Push to all compute nodes.3. Allow gschedule to call gatherer repeatedly.4. Watch the graph
36、s grow.Copyright 2003 UC RegentsWrite Metric Gatherer in PythonnWriting a gmetric gatherer is easy in python: Rocks has native support.nGschedule is a daemon running by default on all Rocks nodes. Will call our gatherer function at regular intervals, with some randomization.nWe just need to provide
37、a name and a value for the metric.# SDSC All Hands Meeting 2003 - Advanced Rocks tutorial.# A gmetric.#from gmon.gmetric import Metricfrom string import count,splitclass packets (Metric): Publishes number of out multicast packets on eth0. dev = eth0 def _init_(self): # Publish value every 2 seconds
38、on average. Metric._init_(self, 2) def name(self): return mcast-packets-in-%s % (self.dev) def value(self): dev = open(/proc/net/dev) for line in dev.readlines(): if not count(line, self.dev): continue # Discard dev: fields = split(line,:)1 values = split(fields) dev.close() return int(values7)packe
39、ts.pyCopyright 2003 UC RegentsPush new Gmetric to all NodesnTest on individual nodesLog in (ssh)Copy python source to /opt/ganglia/lib/python/gmon/metrics/nPackagingUse RPM for permanent installationFrontendCompute nodesCopyright 2003 UC RegentsGschedule daemon nThe Rocks gschedule daemon will call
40、our Packets metric repeatedly. ngschedule -debugOr n#service gschedule restartWill publish our metric every 2 seconds.Ganglias gmetad will detect new metric, and graph it. GscheduleTimepublishpublishpublishCopyright 2003 UC RegentsVisualizationnOur new gmetric will be graphed by gmetad.Visible in we
41、bfrontends host view.Only metrics which are numeric and volatile (slope zero) are graphed.nGmetric restrictions:Total metric size 1500 characters.Nonetheless, offers lots of possibilities:nNode temp, etc.Graph of our metricCopyright 2003 UC RegentsMonitoring Grids with GanglianMacro: How to build an
42、d grow a monitoring grid.nA Grid is a collection of Grids and ClustersnA Cluster is a collection of nodesAll ganglia-enabled Copyright 2003 UC RegentsGanglia Meta DaemonnGmeta daemon collects ganglia data from multiple clusters.nKeeps a metrics history over time, used to generate web graphs. nRequir
43、es the trust of each cluster. nTrust is explicitly given MetaGanglia-enabled ClustersABC(TCP)(UDP)(UDP)(UDP)Copyright 2003 UC RegentsAdvanced: Grid HierarchiesnEnables arbitrarily-deep grid nesting in ganglianDistributes metric history collection, a proven bottleneck.Only keep metric history for chi
44、ld clusters.nGanglia Web frontend is now a distributed system.MetaMetaaGrid BGrid CMetaGrid AcbKeeps history for cluster a, but not for b or c.webwebwebCopyright 2003 UC RegentsCreating a New RPMnRPMs are created using spec filesspec files are an RPMs makefilenWell use the Rocks source code infrastr
45、ucture to build a new RPMnFirst, get the Rocks source code:# cd /home/install# mkdir src# cd src# export CVS_RSH=ssh# cvs -d:pserver:anonymouscvs.rocksclusters.org:/home/cvs/CVSROOT/ login# cvs -d:pserver:anonymouscvs.rocksclusters.org:/home/cvs/CVSROOT/ checkout rocksCopyright 2003 UC RegentsCreate
46、 the contrib-new packagenGo to the contrib area of Rocks# cd /home/install/src/rocks/src/contrib/nCreate the new directory:# mkdir newnPopulate new with the skeleton files:# cp skeleton/* newnChange names: skeleton to new# cd new# mv contrib-skeleton.spec.in contrib-new.spec.in# vi MakefilenChange /
47、opt/skeleton to /opt/newCopyright 2003 UC RegentsBuilding the PackagenExecute:# make rpmnThis creates a binary and source RPM:/usr/src/redhat/RPMS/i386/contrib-new-1-1.i386.rpm/usr/src/redhat/SRPMS/contrib-new-1-1.src.rpmCopyright 2003 UC RegentsAdding the New Package to the DistributionnThe distrib
48、ution compiler, rocks-dist, merges RPMs from the following sources:/home/install/ftp.rocksclusters.org/usr/src/redhat/RPMS/home/install/contrib/RPMS/nThe package contrib-new-1-1.i386.rpm is in /usr/src/redhat/RPMS/i386nTo compile a new distro with the contrib-new package:# cd /home/install# rocks-di
49、st distnThis creates a Red Hat compatible distro in /home/install/rocks-dist/nLook under /home/install/rocks-dist for contrib-new-1-1.i386.rpmCopyright 2003 UC RegentsAssigning the New Package to a Cluster AppliancenExtend the compute appliance# cd /home/install/profiles/2.3.2/site-nodes# cp skeleto
50、n.xml extend-compute.xmlnAdd the contrib-new package# vi extend-compute.xmlAdd the line:ncontrib-newTo apply dynamic configuration to components of the contrib-new package, write a program and place it between tagsCopyright 2003 UC RegentsTest the changesnRun kickstart.cgi# cd /home/install# ./kicks
51、tart.cgi -client=compute-0-0 /tmp/ks.cfgnLook for contrib-new in %packages section of /tmp/ks.cfgCopyright 2003 UC RegentsReinstall Compute NodesnUse shoot-node to reinstall compute nodes in order to apply the contrib-new package# shoot-node compute-0-0Copyright 2003 UC RegentsBuilding a Custom Kern
52、el RPMnPick a compute node# ssh compute-0-0Develop on this node, doesnt “trash” the environment on the frontendnCreate a custom .config file.This can be done from scratch or based on a canned Red Hat configuration filen# cd /usr/src/linux-2.4n# cp configs/kernel-2.4.18-i686-smp.config .confign# vi .
53、configCopyright 2003 UC RegentsBuilding a Custom Kernel RPMnBuild a kernel RPM# make rpmnCopy the resulting kernel RPM to the frontend:# scp /usr/src/redhat/RPMS/i686/kernel.rpm frontend-0:/home/install/contrib/RPMS/public/i386nRebuild the distro on the frontend# cd /home/install# rocks-dist distCop
54、yright 2003 UC RegentsCompute Node PartitioningnCreates 4 GB root partition on first driveThis partition is volatile, that is, when the node is reinstalled, this partition is reformattednRemainder of first drive is put into a partition called “/state/partition1”nFor each remaining drives, one partit
55、ion is created per drive and named “/state/partition2”, “/state/partition3”, etc.nAll partitions labeled “/state/partitionn” are not reformatted on reboots.Copyright 2003 UC RegentsExampleRoot Drive18 GB/dev/sda1 /4 GB/dev/sda2 /state/partition114 GBCopyright 2003 UC RegentsExampleSecond Drive36 GB/
56、dev/sdb1 /state/partition236 GBCopyright 2003 UC RegentsExampleThird Drive18 GB/dev/sdc1 /state/partition318 GBCopyright 2003 UC RegentsExampleFourth Drive72 GB/dev/sdd1 /state/partition472 GBCopyright 2003 UC RegentsExampleFifth Drive36 GB/dev/sde1 /state/partition536 GBCopyright 2003 UC RegentsExa
57、mpleSixth Drive181 GB/dev/sdf1 /state/partition6181 GBCopyright 2003 UC RegentsExampleDeviceNameSize/dev/sda1/4 GB/dev/sda2/state/partition114 GB/dev/sdb1/state/partition236 GB/dev/sdc1/state/partition318 GB/dev/sdd1/state/partition472 GB/dev/sde1/state/partition536 GB/dev/sdf1/state/partition6181 G
58、BCopyright 2003 UC RegentsUser-specified PartitioningnOverride the default partitioning configuration file:# cd /home/install/profiles/2.3.2/site-nodes# cp skeleton.xml replace-auto-partition.xmlnAdd your partitions to replace-auto-partition.xml / -size 5000 -ondisk sda swap -size 512 -ondisk sda /m
59、ydata -size 1 -grow -ondisk sda Copyright 2003 UC RegentsBuilding your own CD SetnAfter customizing your cluster distribution, make ISO images to export itnFirst, you must mirror the full Rocks release# cd /home/install# rocks-dist mirrornOr, put in the second (then third) Rocks CD and execute:# roc
60、ks-dist copycdCopyright 2003 UC RegentsBuilding your own CD SetnBuild a CD Set# cd /home/install# rm -rf cdrom# rocks-dist -dist=cdrom cdromnThis creates a CD set under the directory /home/install/cdrom# cd /home/install/cdrom/7.3/en/osCopyright 2003 UC RegentsLabThanksnInspired by a lab conducted a
61、t TACCCopyright 2003 UC RegentsLabnBuilding 2-node x86 clustersnGanglia grid nAdding usersnRunning LinpacknChecking out rocks sourcenCreating a new packagenRecompiling distronReinstall compute nodesnCreating/using a new distributionCopyright 2003 UC RegentsAdd New UsersnUse the standard linux comman
62、d:# useradd This adds user account info to /etc/passwd, /etc/shadow and /etc/groupnAdditionally, adds mountpoint to autofs configuration file (/etc/auto.home) and refreshes the NIS databasenTo delete users, use:# userdel This reverses all the actions taken by useraddCopyright 2003 UC RegentsRun Linp
63、acknLogin as new user:# su - brunoThis prompts for information regarding a new ssh keynSsh is the sole mechanism for logging into compute nodesnCopy over the linpack example files:$ cp /var/www/html/rocks-documentation/2.3.2/examples/* .Copyright 2003 UC RegentsRun LinpacknSubmit a 2-processor Linpa
64、ck job to PBS:$ qsub qsub-test.shnMonitor job progress with:$ showqOr the frontends job monitoring web pageCopyright 2003 UC RegentsScaling Up LinpacknTo scale the job up, edit qsub-test.sh and change:n#PBS -l nodes=2To (assuming you have dual-processor compute nodes):n#PBS -l nodes=2:ppn=2Also chan
65、ge:n/opt/mpich/ethernet/gcc/bin/mpirun -np 2To:n/opt/mpich/ethernet/gcc/bin/mpirun -np 4Copyright 2003 UC RegentsScaling Up LinpacknThen edit HPL.dat and change:n1 PsTo:n2 PsThe number of processors Linpack uses is P * QnTo make Linpack use more memory (and increase performance), edit HPL.dat and ch
66、ange:n1000 NsTo:n4000 NsLinpack operates on an N * N matrixn Submit the (larger) job:$ qsub qsub-test.shCopyright 2003 UC RegentsUsing Linpack Over MyrinetnSubmit the job:$ qsub qsub-test-myri.shnScale up the job in the same manner as described in the previous slides.Copyright 2003 UC RegentsEndGo Home
