Cluster Computing - Revision history

imported>Huston at 15:15, 21 April 2022

2022-04-21T15:15:15Z

← Older revision		Revision as of 15:15, 21 April 2022
Line 11:		Line 11:

	== Hydra / Condor ==		== Hydra / Condor ==
	We used to have a cluster of our own named "Hydra", however it was finally decommissioned in 2012 after having been converted to a general Condor cluster and slowly dismantled. The head node lived on (in spirit, at least) as the controller for our Condor infrastructure, which let you submit jobs to run during idle cycles of desktops and other machines around the department. Alas, by the fall of 2021, desktops in Peyton Hall were a dwindling number and the Condor pool diminished down to just the few compute servers that live in the data center. At that point, there's not much reason to have a complex scheduler system for just two machines, and so Condor (and Hydra) ~~were~~ retired.		We used to have a cluster of our own named "Hydra", however it was finally decommissioned in 2012 after having been converted to a general Condor cluster and slowly dismantled. The head node lived on (in spirit, at least) as the controller for our Condor infrastructure, which let you submit jobs to run during idle cycles of desktops and other machines around the department. Alas, by the fall of 2021, desktops in Peyton Hall were a dwindling number and the Condor pool diminished down to just the few compute servers that live in the data center. At that point, there's not much reason to have a complex scheduler system for just two machines, and so Condor (and Hydra) was retired.


	[[Category:Cluster Computing]]		[[Category:Cluster Computing]]

imported>Huston: No more Condor either

2022-04-21T15:14:41Z

No more Condor either

← Older revision		Revision as of 15:14, 21 April 2022
Line 10:		Line 10:


	== Hydra ==		== Hydra / Condor ==
	We used to have a cluster of our own named "Hydra", however it was finally decommissioned in 2012 after having been converted to a general Condor cluster and slowly dismantled. The head node ~~lives~~ on (in spirit, at least) as the controller for our Condor infrastructure, which ~~lets~~ you submit jobs to run during idle cycles of desktops and other machines around the department.		We used to have a cluster of our own named "Hydra", however it was finally decommissioned in 2012 after having been converted to a general Condor cluster and slowly dismantled. The head node lived on (in spirit, at least) as the controller for our Condor infrastructure, which let you submit jobs to run during idle cycles of desktops and other machines around the department. Alas, by the fall of 2021, desktops in Peyton Hall were a dwindling number and the Condor pool diminished down to just the few compute servers that live in the data center. At that point, there's not much reason to have a complex scheduler system for just two machines, and so Condor (and Hydra) were retired.


	~~=== Submitting jobs~~ to ~~Hydra ===~~
	~~Hydra uses [[Condor]] for job management~~. ~~You~~'~~ll find information about how~~ to ~~use it in the [[~~Condor~~\|Condor article]]~~.


	[[Category:Cluster Computing]]		[[Category:Cluster Computing]]

imported>Huston at 19:51, 24 August 2020

2020-08-24T19:51:52Z

← Older revision		Revision as of 19:51, 24 August 2020
Line 7:		Line 7:

	== Research Computing ==		== Research Computing ==
	Research Computing maintains many HPC clusters. Information on them is available at [~~http~~://~~www~~.princeton.edu~~/researchcomputing~~/ their website]. There's also a page for [~~http~~://~~www~~.princeton.edu/~~researchcomputing~~/~~access~~/ prospective users] describing what is needed to get an account there.		Research Computing maintains many HPC clusters. Information on them is available at [https://researchcomputing.princeton.edu/ their website]. There's also a page for [https://researchcomputing.princeton.edu/education/online-tutorials/getting-started prospective users] describing what is needed to get an account there.

imported>Huston: Hydra is no more, hasn't been for over three years.

2015-10-15T13:11:23Z

Hydra is no more, hasn't been for over three years.

@@ Line 6: / Line 6: @@
-== Hydra ==
+== Research Computing ==
-Hydra is a <s>92</s> ([[Cluster Computing#Status|'''72''']]) node Beowulf cluster housed in the basement server room of Peyton Hall.  Eight of the nodes have only 1GB of memory, another eight have 2GB of memory, and the remainder have 4GB.  The nodes have dual processors ranging in speed from 2.2GHz to 3.06GHz.
+Research Computing maintains many HPC clusters.  Information on them is available at [http://www.princeton.edu/researchcomputing/ their website].  There's also a page for [http://www.princeton.edu/researchcomputing/access/ prospective users] describing what is needed to get an account there.
+== Hydra ==
-=== Status ===
+We used to have a cluster of our own named "Hydra", however it was finally decommissioned in 2012 after having been converted to a general Condor cluster and slowly dismantled.  The head node lives on (in spirit, at least) as the controller for our Condor infrastructure, which lets you submit jobs to run during idle cycles of desktops and other machines around the department.
 === Submitting jobs to Hydra ===
 Hydra uses [[Condor]] for job management.  You'll find information about how to use it in the [[Condor|Condor article]].
 [[Category:Cluster Computing]]

imported>Huston: Removing irrelevant information about PBS, and updated "usage policies"

2009-05-06T21:47:54Z

Removing irrelevant information about PBS, and updated "usage policies"

← Older revision		Revision as of 21:47, 6 May 2009
Line 1:		Line 1:
	No, we're not talking about star clusters - not this time, anyway. We're talking about Beowulf clusters, or to use a more general term "computer clusters". Not the same as what OIT calls their "computer labs".		No, we're not talking about star clusters - not this time, anyway. We're talking about Beowulf clusters, or to use a more general term "computer clusters". Not the same as what OIT calls their "computer labs".

	~~'''NOTE:''' Hydra has undergone some major changes; much of the information below needs to be reworked, but in the mean time please see [[Condor]] for how to use Condor with Hydra.~~
	~~{{update}}~~

	~~----~~


	== Introduction ==		== Introduction ==
	So what is a cluster? It's a set of machines, usually (but not necessarily) on a private network which is attached to a dual-homed "master node". Dual-homed means it sits on two networks at the same time, and may even act as a router between the two. This master node ~~will~~ allow logins, and is where you setup your large parallel jobs. Once the job is submitted, software on the master connects to the drones and runs the job there. This software is designed to fairly execute programs when there is available resources for them, and make sure that someone doesn't start a job on the same nodes that you're using for your processes so that everyone's programs get fair share of the machine.		So what is a cluster? It's a set of machines, usually (but not necessarily) on a private network which is attached to a dual-homed "master node". Dual-homed means it sits on two networks at the same time, and may even act as a router between the two. This master node can allow logins, and is where you setup your large parallel jobs. Once the job is submitted, software on the master connects to the drones and runs the job there. This software is designed to fairly execute programs when there is available resources for them, and make sure that someone doesn't start a job on the same nodes that you're using for your processes so that everyone's programs get fair share of the machine.


Line 19:		Line 14:
	=== Status ===		=== Status ===
	This section will be updated with status information for Hydra as necessary.		This section will be updated with status information for Hydra as necessary.
			* '''2009-05-01:''' Hydra has been reimaged completely with CentOS 5, and is using Condor for scheduling instead of PBS. It no longer accepts user logins directly, as Condor jobs can be submitted from other machines without being logged into Hydra.
	* '''2008-04-24:''' We've done all the testing we can. Access has been restored to all users who previously had access to hydra and who still have a valid Astro account.		* '''2008-04-24:''' We've done all the testing we can. Access has been restored to all users who previously had access to hydra and who still have a valid Astro account.
	* '''2008-04-11:''' The entire cluster has been upgraded to PU_IAS Linux 5.1 (a RedHat Enterprise variant). This distribution has much newer software and libraries than hydra's old distro, which was painfully out of date. Given the latest generation of compilers and MPI libraries, we highly suggest recompiling your code before submitting new jobs. Hydra is currently in a limited test state, we will eventually (soon, hopefully) be reinstating full access to the cluster to all who had accounts, but for now, we are in a shakedown period.		* '''2008-04-11:''' The entire cluster has been upgraded to PU_IAS Linux 5.1 (a RedHat Enterprise variant). This distribution has much newer software and libraries than hydra's old distro, which was painfully out of date. Given the latest generation of compilers and MPI libraries, we highly suggest recompiling your code before submitting new jobs. Hydra is currently in a limited test state, we will eventually (soon, hopefully) be reinstating full access to the cluster to all who had accounts, but for now, we are in a shakedown period.
Line 32:		Line 28:

	=== Getting access to Hydra ===		=== Getting access to Hydra ===
	~~Students or postdocs~~ wishing to use hydra for computations ~~should discuss with~~ their ~~supervisor whether use of hydra is called for (i.e., parallel computations are needed, and whether the code has been debugged and appropriately optimized)~~. ~~The supervisor should [[Requesting assistance\|contact us]] and request~~ that ~~the student or postdoc be given access to hydra~~, ~~and confirm that the research problem calls for parallel computations. We'll add your account to the list of those allowed to login, and create the necessary directories on the~~ scratch disks ~~for you to store data.~~		Anyone wishing to use hydra for computations need only have their data stored somewhere that Hydra nodes can see. Currently that is on /scr/chimera0, but other scratch disks may be added to the pool in time. If you don't have a directory on Chimera and need one, please [[Requesting assistance\|contact us]].


	~~=== Hydra usage policies ===~~
	~~The following policies should~~ be ~~kept in mind while running jobs on the clusters. They are not strict rules, but more like guidelines~~ to ~~make sure that all users get a fair share of~~ the ~~processors.~~
	* Any user who wants to run lengthy jobs on more than about half the available processors at one time should announce this fact to the cluster mailing list in advance. How far in advance should depend on the length of the jobs. For instance, if the jobs will be running for a day or more, then a day or half a day in advance would be appropriate. If ~~the jobs will take a few hours, an hour or so of advance notice is probably enough.~~
	* Several, perhaps 5 or 6, processors should be left available at most or all times for relatively short-running jobs (on the order of an hour or less). Of course, if your job requires all the processors of the cluster, this doesn't mean you ~~can't run it, but should definitely ask around the cluster list first to make sure others~~ don't have ~~jobs they will need to run during that time.~~
	* Jobs should not be run on ~~Hydra itself, nor should any intensive programs be run on it (for example, IDL or Mathematica). The master node is primarily for controlling the cluster~~ and storing input or output data, as well as compiling code that will run on the machines. Running programs such as IDL on the master node takes away from the CPU time and memory available to other users for compiling and the scheduling processes. Processes such as these may be killed, especially if they're eating up resources.
	* If you are submitting for publication a paper that uses computational results obtained on hydra, please ~~acknowledge support from NSF grant AST-0216105~~ [~~e.g., "This research used computational facilities supported by NSF grant AST-0216105"~~]~~. Please send email to draine@astro.princeton.edu with information on the paper (title, author, where submitted) for use in reporting to the NSF on use of the cluster~~.


	=== Submitting jobs to Hydra ===		=== Submitting jobs to Hydra ===
	Hydra uses ~~the Portable Batch System (~~[[~~PBS~~]]) for job management. You'll find information about how to use it in the [[~~PBS~~\|~~PBS~~ article]].		Hydra uses [[Condor]] for job management. You'll find information about how to use it in the [[Condor\|Condor article]].

	== FAQ ==		== FAQ ==
	~~=== I have a bunch of serial jobs to run. ===~~
	Don't use a cluster for them. Clusters are built with high performance networking, high speed disk access, and many tweaks to tune the most performance out of computers for parallel processing jobs - those that use multiple processes spread across many computers. Serial jobs will clog a cluster, since all they do is use a single CPU, and tend to read or write a lot of data to disk (which also clogs the I/O system of the cluster), while parallel jobs tend to talk to each other to exchange data, writing checkpoint information to disk as necessary (and usually only from one process, not all of them). If you have serial jobs to run, you should use [[Condor]] to submit them to the idle machines in the building instead of tying up nodes of the cluster.


	=== Can I speed up my compiling? ===		=== Can I speed up my compiling? ===
	Yes, you can speed things up with the '-j' option to make. From the man page:		Yes, you can speed things up with the '-j' option to make. From the man page:
Line 65:		Line 49:

	== See also ==		== See also ==
	* [[Condor]], ~~a batch~~ scheduling system ~~which is perfect for serial jobs that would otherwise clog~~ the ~~cluster~~.		* [[Condor]], the scheduling system in use on Hydra and other machines on the network.
	* [[MPICH]], an implementation of the Message Passing Interface commonly used in large parallel jobs.		* [[MPICH]], an implementation of the Message Passing Interface commonly used in large parallel jobs.


	[[Category:Cluster Computing]]		[[Category:Cluster Computing]]

imported>Huston at 19:21, 6 May 2009

2009-05-06T19:21:31Z

← Older revision		Revision as of 19:21, 6 May 2009
Line 2:		Line 2:

	'''NOTE:''' Hydra has undergone some major changes; much of the information below needs to be reworked, but in the mean time please see [[Condor]] for how to use Condor with Hydra.		'''NOTE:''' Hydra has undergone some major changes; much of the information below needs to be reworked, but in the mean time please see [[Condor]] for how to use Condor with Hydra.
			{{update}}

	----		----

imported>Huston: No more PBS

2009-05-01T17:51:55Z

No more PBS

← Older revision		Revision as of 17:51, 1 May 2009
Line 1:		Line 1:
	No, we're not talking about star clusters - not this time, anyway. We're talking about Beowulf clusters, or to use a more general term "computer clusters". Not the same as what OIT calls their "computer labs".		No, we're not talking about star clusters - not this time, anyway. We're talking about Beowulf clusters, or to use a more general term "computer clusters". Not the same as what OIT calls their "computer labs".

			'''NOTE:''' Hydra has undergone some major changes; much of the information below needs to be reworked, but in the mean time please see [[Condor]] for how to use Condor with Hydra.

			----

imported>Lkoven: /* Status */

2008-04-24T15:27:08Z

Status

← Older revision		Revision as of 15:27, 24 April 2008
Line 14:		Line 14:
	=== Status ===		=== Status ===
	This section will be updated with status information for Hydra as necessary.		This section will be updated with status information for Hydra as necessary.
			* '''2008-04-24:''' We've done all the testing we can. Access has been restored to all users who previously had access to hydra and who still have a valid Astro account.
	* '''2008-04-11:''' The entire cluster has been upgraded to PU_IAS Linux 5.1 (a RedHat Enterprise variant). This distribution has much newer software and libraries than hydra's old distro, which was painfully out of date. Given the latest generation of compilers and MPI libraries, we highly suggest recompiling your code before submitting new jobs. Hydra is currently in a limited test state, we will eventually (soon, hopefully) be reinstating full access to the cluster to all who had accounts, but for now, we are in a shakedown period.		* '''2008-04-11:''' The entire cluster has been upgraded to PU_IAS Linux 5.1 (a RedHat Enterprise variant). This distribution has much newer software and libraries than hydra's old distro, which was painfully out of date. Given the latest generation of compilers and MPI libraries, we highly suggest recompiling your code before submitting new jobs. Hydra is currently in a limited test state, we will eventually (soon, hopefully) be reinstating full access to the cluster to all who had accounts, but for now, we are in a shakedown period.
	* '''2008-04-11:''' The current state of the hardware is:		* '''2008-04-11:''' The current state of the hardware is:
Line 24:		Line 25:
	* Some usage policies have been posted; please feel free to comment on them by sending mail to the cluster list. They are not strict policies that must be adhered to, but more like general guidelines to be kept in mind while submitting jobs to the systems. They are posted [[Cluster Computing#Hydra usage policies\|below]].		* Some usage policies have been posted; please feel free to comment on them by sending mail to the cluster list. They are not strict policies that must be adhered to, but more like general guidelines to be kept in mind while submitting jobs to the systems. They are posted [[Cluster Computing#Hydra usage policies\|below]].
	* Also, a bug was recently fixed in PBS, which caused every node to report having only 864MB of memory. Each node now properly reports the amount of RAM it has, so everyone should be specifying the amount of memory your jobs require through the PBS resource list ("#PBS -l mem=1380MB", for example). Maui will Do The Right Thing and only assign the job to a node with that much available RAM.		* Also, a bug was recently fixed in PBS, which caused every node to report having only 864MB of memory. Each node now properly reports the amount of RAM it has, so everyone should be specifying the amount of memory your jobs require through the PBS resource list ("#PBS -l mem=1380MB", for example). Maui will Do The Right Thing and only assign the job to a node with that much available RAM.


	=== Getting access to Hydra ===		=== Getting access to Hydra ===

imported>Lkoven: /* Hydra */ Down to 72 nodes now...

2008-04-14T15:42:23Z

Hydra: Down to 72 nodes now...

imported>Lkoven: Updated status with information as of today

2008-04-11T20:34:17Z

Updated status with information as of today

← Older revision		Revision as of 20:34, 11 April 2008
Line 7:		Line 7:

	== Hydra ==		== Hydra ==
	Hydra is a 92 node Beowulf cluster housed in the basement server room of Peyton Hall. Eight of the nodes have only 1GB of memory, another eight have 2GB of memory, and the remainder have 4GB. The nodes have dual processors ranging in speed from 2.2GHz to 3.06GHz.		Hydra is a <s>92</s> ([[Cluster Computing#Status\|'''73''']]) node Beowulf cluster housed in the basement server room of Peyton Hall. Eight of the nodes have only 1GB of memory, another eight have 2GB of memory, and the remainder have 4GB. The nodes have dual processors ranging in speed from 2.2GHz to 3.06GHz.

			'''Not all nodes are online presently. See the status message in the next section.'''


	=== Status ===		=== Status ===
	This section will be updated with status information for Hydra as necessary.		This section will be updated with status information for Hydra as necessary.
			* '''2008-04-11:''' The entire cluster has been upgraded to PU_IAS Linux 5.1 (a RedHat Enterprise variant). This distribution has much newer software and libraries than hydra's old distro, which was painfully out of date. Given the latest generation of compilers and MPI libraries, we highly suggest recompiling your code before submitting new jobs. Hydra is currently in a limited test state, we will eventually (soon, hopefully) be reinstating full access to the cluster to all who had accounts, but for now, we are in a shakedown period.
			* '''2008-04-11:''' The current state of the hardware is:
			** Hydra is currently running with only 73 nodes available. We had several hardware failures during the reinstall. We intend to try and get a few more nodes fixed and added to the pool, stay tuned.
			** Presently there are no 1GB nodes available, and our intent is to make the minimum memory on the whole cluster 2GB per node, if not 4.
			** Hydra's hardware is incapable of running 64-bit, so the upgrade didn't include a move to a 64-bit OS,
	* An additional RAID array has been installed on the Hydra cluster, named Chimera. Its scratch disk is mounted on Hydra as /chimera. To access it from the rest of the department, use the path /peyton/scr/chimera0. It is mounted to all the nodes, also as /chimera, and should be the preferred location for storing inputs and outputs from your programs.		* An additional RAID array has been installed on the Hydra cluster, named Chimera. Its scratch disk is mounted on Hydra as /chimera. To access it from the rest of the department, use the path /peyton/scr/chimera0. It is mounted to all the nodes, also as /chimera, and should be the preferred location for storing inputs and outputs from your programs.
	* The Chimera disk (/chimera on Hydra, and /peyton/scr/chimera0 elsewhere) is truly a scratch disk, in that it is not and will not be backed up. It is a hardware RAID-5 array with one hot spare disk, and should handle most hardware failures without incident.		* The Chimera disk (/chimera on Hydra, and /peyton/scr/chimera0 elsewhere) is truly a scratch disk, in that it is not and will not be backed up. It is a hardware RAID-5 array with one hot spare disk, and should handle most hardware failures without incident.

← Older revision		Revision as of 15:42, 14 April 2008
Line 7:		Line 7:

	== Hydra ==		== Hydra ==
	Hydra is a <s>92</s> ([[Cluster Computing#Status\|'''73''']]) node Beowulf cluster housed in the basement server room of Peyton Hall. Eight of the nodes have only 1GB of memory, another eight have 2GB of memory, and the remainder have 4GB. The nodes have dual processors ranging in speed from 2.2GHz to 3.06GHz.		Hydra is a <s>92</s> ([[Cluster Computing#Status\|'''72''']]) node Beowulf cluster housed in the basement server room of Peyton Hall. Eight of the nodes have only 1GB of memory, another eight have 2GB of memory, and the remainder have 4GB. The nodes have dual processors ranging in speed from 2.2GHz to 3.06GHz.

	'''Not all nodes are online presently. See the status message in the next section.'''		'''Not all nodes are online presently. See the status message in the next section.'''
Line 16:		Line 16:
	* '''2008-04-11:''' The entire cluster has been upgraded to PU_IAS Linux 5.1 (a RedHat Enterprise variant). This distribution has much newer software and libraries than hydra's old distro, which was painfully out of date. Given the latest generation of compilers and MPI libraries, we highly suggest recompiling your code before submitting new jobs. Hydra is currently in a limited test state, we will eventually (soon, hopefully) be reinstating full access to the cluster to all who had accounts, but for now, we are in a shakedown period.		* '''2008-04-11:''' The entire cluster has been upgraded to PU_IAS Linux 5.1 (a RedHat Enterprise variant). This distribution has much newer software and libraries than hydra's old distro, which was painfully out of date. Given the latest generation of compilers and MPI libraries, we highly suggest recompiling your code before submitting new jobs. Hydra is currently in a limited test state, we will eventually (soon, hopefully) be reinstating full access to the cluster to all who had accounts, but for now, we are in a shakedown period.
	* '''2008-04-11:''' The current state of the hardware is:		* '''2008-04-11:''' The current state of the hardware is:
	** Hydra is currently running with only 73 nodes available. We had several hardware failures during the reinstall. We intend to try and get a few more nodes fixed and added to the pool, stay tuned.		** Hydra is currently running with only 72 nodes available. We had several hardware failures during the reinstall. We intend to try and get a few more nodes fixed and added to the pool, stay tuned.
	** Presently there are no 1GB nodes available, and our intent is to make the minimum memory on the whole cluster 2GB per node, if not 4.		** Presently there are no 1GB nodes available, and our intent is to make the minimum memory on the whole cluster 2GB per node, if not 4.
	** Hydra's hardware is incapable of running 64-bit, so the upgrade didn't include a move to a 64-bit OS,		** Hydra's hardware is incapable of running 64-bit, so the upgrade didn't include a move to a 64-bit OS,
Line 40:		Line 40:
	=== Submitting jobs to Hydra ===		=== Submitting jobs to Hydra ===
	Hydra uses the Portable Batch System ([[PBS]]) for job management. You'll find information about how to use it in the [[PBS\|PBS article]].		Hydra uses the Portable Batch System ([[PBS]]) for job management. You'll find information about how to use it in the [[PBS\|PBS article]].


	== FAQ ==		== FAQ ==