Scratch disks - Revision history

imported>Huston: No more depot (leaving it as an example though)

2018-03-20T16:19:23Z

No more depot (leaving it as an example though)

← Older revision		Revision as of 16:19, 20 March 2018
Line 7:		Line 7:

	== What scratch space is available? ==		== What scratch space is available? ==
	=== ~~depot~~ ===		=== General ===
	We have ~~~70TB~~ available at <tt>/~~scr/depot0~~</tt> ~~available to everyone~~ in the department from any departmental Linux computer. To request a directory, [[Requesting assistance\|contact us]].		We have ~75TB available at <tt>/scratch</tt> for anyone in the department from any departmental Linux computer. To request a directory, [[Requesting assistance\|contact us]].


	=== Private servers ===		=== Private servers ===
Line 15:		Line 16:

	=== Desktop scratch space ===		=== Desktop scratch space ===
	Desktops used to have a lot of space left over, however many now use most if not all of that space for the OS. With many desktops being all-in-one devices, there generally isn't any room to install additional disks. Combined with the added reliability of RAID systems and increased network bandwidth, it's generally preferable to use space on a machine such as ~~depot~~ than local scratch space. ~~In some cases, there may be something available, so feel free to ask and we can check the specific machine you're using.~~		Desktops used to have a lot of space left over, however many now use most if not all of that space for the OS. With many desktops being all-in-one devices, there generally isn't any room to install additional disks. Combined with the added reliability of RAID systems and increased network bandwidth, it's generally preferable to use space on a purpose-built machine (such as where /scratch is housed) than local scratch space.


	'''NOTE:'''		'''NOTE:'''
Line 27:		Line 29:
	:''Note that these are all lumped together because they usually have the same answer''		:''Note that these are all lumped together because they usually have the same answer''

	Chances are, you have a symlink in there somewhere to a scratch disk which is currently unreachable. You should never put symlinks to scratch disks in your home directory - if for some reason that disk is not reachable, you run a good risk of not being able to access your account, your home directory, or much of anything else until someone deletes it for you. Make shell aliases such as "export FOO=/~~scr/depot0~~/username" and use 'cd $FOO' if you must have a shortcut - at least then you'll be able to access your files if that disk goes down.		Chances are, you have a symlink in there somewhere to a scratch disk which is currently unreachable. You should never put symlinks to scratch disks in your home directory - if for some reason that disk is not reachable, you run a good risk of not being able to access your account, your home directory, or much of anything else until someone deletes it for you. Make shell aliases such as "export FOO=/scratch/username" and use 'cd $FOO' if you must have a shortcut - at least then you'll be able to access your files if that disk goes down.

imported>Huston at 13:57, 13 October 2015

2015-10-13T13:57:44Z

← Older revision		Revision as of 13:57, 13 October 2015
Line 19:		Line 19:
	'''NOTE:'''		'''NOTE:'''
	: We suggest you think carefully about what files you want to use on local scratch space, however. This space, like all scratch space, is not backed up, and it's on a single hard drive with no redundancy. Do not use it for data that is irreplaceable or would take a long time to regenerate.		: We suggest you think carefully about what files you want to use on local scratch space, however. This space, like all scratch space, is not backed up, and it's on a single hard drive with no redundancy. Do not use it for data that is irreplaceable or would take a long time to regenerate.

	~~If these conditions are OK with you, please [[Requesting assistance\|contact us]] and request access to your local scratch space.~~

imported>Huston: Local scratch is generally small now

2015-10-13T13:55:49Z

Local scratch is generally small now

← Older revision		Revision as of 13:55, 13 October 2015
Line 15:		Line 15:

	=== Desktop scratch space ===		=== Desktop scratch space ===
	~~A number of our desktops~~ have a lot ~~more~~ space ~~on their local hard drives than is used by~~ the OS install and ~~can be allocated~~ as scratch space. ~~If you plan to do a lot of I/O on larger (few GB+) datasets~~, ~~using your local scratch space might~~ be ~~a good way~~ to ~~speed up~~ the ~~process as~~ you ~~are getting rid of network overhead. It~~'~~s also a good way to get temporary data off the main (charged) NFS server~~.		Desktops used to have a lot of space left over, however many now use most if not all of that space for the OS. With many desktops being all-in-one devices, there generally isn't any room to install additional disks. Combined with the added reliability of RAID systems and increased network bandwidth, it's generally preferable to use space on a machine such as depot than local scratch space. In some cases, there may be something available, so feel free to ask and we can check the specific machine you're using.

	'''NOTE:'''		'''NOTE:'''

imported>Lkoven: /* depot */ Changed info about getting access to depot.

2013-11-21T21:43:25Z

depot: Changed info about getting access to depot.

← Older revision		Revision as of 21:43, 21 November 2013
Line 8:		Line 8:
	== What scratch space is available? ==		== What scratch space is available? ==
	=== depot ===		=== depot ===
	We have ~~two 1.1TB arrays~~ available at <tt>/scr/depot0~~</tt> and <tt>/scr/depot1~~</tt> available to everyone in the department from any departmental Linux computer. ~~Please ''read'' the file <tt>/scr/depot0/README_FIRST</tt> for more information on how to use this space~~.		We have ~70TB available at <tt>/scr/depot0</tt> available to everyone in the department from any departmental Linux computer. To request a directory, [[Requesting assistance\|contact us]].


	=== Private servers ===		=== Private servers ===

imported>Huston: /peyton deprecated

2010-05-05T14:38:27Z

/peyton deprecated

← Older revision		Revision as of 14:38, 5 May 2010
Line 3:		Line 3:

	== How do I access scratch disks? ==		== How do I access scratch disks? ==
	Scratch disks are exported via nfs and are available on any department-supported Linux desktop or server via automounts. You can access existing scratch space via the directory structure "<tt>~~/peyton~~/scr/hostname#</tt>" where "<tt>hostname</tt>" is the name of the host the space is exported from (minus the astro.princeton.edu) and "<tt>#</tt>" is the partition number. For example, '<tt>cd ~~/peyton~~/scr/depot0</tt>' would cd into the first (counting from 0) scratch disk on the server depot. If you're using space that's directly attached to your local machine, you can also access it directly via <tt>/scr#</tt> to avoid a bit of NFS overhead.		Scratch disks are exported via nfs and are available on any department-supported Linux desktop or server via automounts. You can access existing scratch space via the directory structure "<tt>/scr/hostname#</tt>" where "<tt>hostname</tt>" is the name of the host the space is exported from (minus the astro.princeton.edu) and "<tt>#</tt>" is the partition number. For example, '<tt>cd /scr/depot0</tt>' would cd into the first (counting from 0) scratch disk on the server depot. If you're using space that's directly attached to your local machine, you can also access it directly via <tt>/scr#</tt> to avoid a bit of NFS overhead.


	== What scratch space is available? ==		== What scratch space is available? ==
	=== depot ===		=== depot ===
	We have two 1.1TB arrays available at <tt>~~/peyton~~/scr/depot0</tt> and <tt>~~/peyton~~/scr/depot1</tt> available to everyone in the department from any departmental Linux computer. Please ''read'' the file <tt>~~/peyton~~/scr/depot0/README_FIRST</tt> for more information on how to use this space.		We have two 1.1TB arrays available at <tt>/scr/depot0</tt> and <tt>/scr/depot1</tt> available to everyone in the department from any departmental Linux computer. Please ''read'' the file <tt>/scr/depot0/README_FIRST</tt> for more information on how to use this space.

imported>Huston at 20:44, 1 March 2010

2010-03-01T20:44:14Z

← Older revision		Revision as of 20:44, 1 March 2010
Line 22:		Line 22:

	If these conditions are OK with you, please [[Requesting assistance\|contact us]] and request access to your local scratch space.		If these conditions are OK with you, please [[Requesting assistance\|contact us]] and request access to your local scratch space.


			== FAQ ==
			=== I can't run <tt>ls</tt> in my home directory ===
			=== I can't login to my account without pressing Ctrl-C to abort something ===
			=== I can't use the window manager's file browser ===
			:''Note that these are all lumped together because they usually have the same answer''

			Chances are, you have a symlink in there somewhere to a scratch disk which is currently unreachable. You should never put symlinks to scratch disks in your home directory - if for some reason that disk is not reachable, you run a good risk of not being able to access your account, your home directory, or much of anything else until someone deletes it for you. Make shell aliases such as "export FOO=/scr/depot0/username" and use 'cd $FOO' if you must have a shortcut - at least then you'll be able to access your files if that disk goes down.

imported>Huston at 13:29, 23 May 2007

2007-05-23T13:29:46Z

← Older revision		Revision as of 13:29, 23 May 2007
Line 1:		Line 1:
	~~==What are scratch disks?==~~
	Scratch disks are usually smaller (but sometimes quite large) drives where you can temporarily store data that does not need to be backed up. They may be used for intermediate stages in a data pipeline, a local copy of a dataset available elsewhere or on a non-online media (CD, DLT, DVD), or temporary storage for outputs of your work. Scratch disks are not [[Backups\|backed up]] at all, though some of them are larger RAID5 disk systems which are more fault-tolerant than a single disk on a workstation. They are usually [[NFS]] mounted across all machines in the building for easy access.		Scratch disks are usually smaller (but sometimes quite large) drives where you can temporarily store data that does not need to be backed up. They may be used for intermediate stages in a data pipeline, a local copy of a dataset available elsewhere or on a non-online media (CD, DLT, DVD), or temporary storage for outputs of your work. Scratch disks are not [[Backups\|backed up]] at all, though some of them are larger RAID5 disk systems which are more fault-tolerant than a single disk on a workstation. They are usually [[NFS]] mounted across all machines in the building for easy access.


	==How do I ~~use~~ access scratch disks?==		== How do I access scratch disks? ==
	Scratch disks are exported via nfs and are available on any department-~~support~~ Linux desktop or server via automounts. You can access existing scratch space via the ~~following~~ directory structure		Scratch disks are exported via nfs and are available on any department-supported Linux desktop or server via automounts. You can access existing scratch space via the directory structure "<tt>/peyton/scr/hostname#</tt>" where "<tt>hostname</tt>" is the name of the host the space is exported from (minus the astro.princeton.edu) and "<tt>#</tt>" is the partition number. For example, '<tt>cd /peyton/scr/depot0</tt>' would cd into the first (counting from 0) scratch disk on the server depot. If you're using space that's directly attached to your local machine, you can also access it directly via <tt>/scr#</tt> to avoid a bit of NFS overhead.
	/peyton/scr/hostname#
	where <tt>hostname</tt> is the name of the host the space is exported from (minus the astro.princeton.edu) and <tt>#</tt> is the partition number ~~so for~~ example
	cd /peyton/scr/depot0
	would cd into the first (counting from 0) scratch disk on the server depot.

	~~If you're using space that's directly attached to your local machine, you can also access it directly via <tt>/scr#</tt> to avoid a bit of NFS overhead.~~

			== What scratch space is available? ==
			=== depot ===
			We have two 1.1TB arrays available at <tt>/peyton/scr/depot0</tt> and <tt>/peyton/scr/depot1</tt> available to everyone in the department from any departmental Linux computer. Please ''read'' the file <tt>/peyton/scr/depot0/README_FIRST</tt> for more information on how to use this space.

	~~==What scratch space is available?==~~
	~~===depot===~~
	We have two 1.1TB arrays available at <tt>/peyton/scr/depot0</tt> and <tt>/peyton/scr/depot1</tt> available to everyone in the department from any departmental Linux computer. Please ''read'' the file <tt>/peyton/scr/depot0/README_FIRST</tt> for more information on how to use this space.

			=== Private servers ===
			There are a number of projects in the department with scratch disks which were purchased for the use of those projects. If you're working on a specific project (such as SDSS or ACT just to name a couple) and need a place to store files related to that project, contact the project's coordinator and ask them to [[Requesting assistance\|contact us]] to request access to their space for your account.

	~~===Private Servers===~~
	There are a number of projects in the department with scratch disks which were purchased for the use of those projects. If you're working on a specific project (such as SDSS or ACT just to name a couple) and need a place to store files related to that project, contact the project's coordinator and ask them to contact us to request access to their space for your account.

			=== Desktop scratch space ===
			A number of our desktops have a lot more space on their local hard drives than is used by the OS install and can be allocated as scratch space. If you plan to do a lot of I/O on larger (few GB+) datasets, using your local scratch space might be a good way to speed up the process as you are getting rid of network overhead. It's also a good way to get temporary data off the main (charged) NFS server.

	~~===Desktop scratch space===~~		'''NOTE:'''
	A number of our desktops have a lot more space on their local hard drives than is used by the OS install and can be allocated as scratch space. If you plan to do a lot of I/O on larger (few GB+) datasets, using your local scratch space might be a good way to speed up the process as you are getting rid of network overhead. It'~~s also a good way to get temporary data off the main (charged) NFS server.~~		: We suggest you think carefully about what files you want to use on local scratch space, however. This space, like all scratch space, is not backed up, and it's on a single hard drive with no redundancy. Do not use it for data that is irreplaceable or would take a long time to regenerate.

	''~~'Note~~'''
	: We suggest you think carefully about what files you want to use on local scratch space, however. This space, like all scratch space, is not backed up, and it's on a single hard drive with no redundancy. Do not use it for data that is irreplaceable or would take a long time to regenerate.

	If these conditions are OK with you, please [[Requesting assistance\|contact us]] and request access to your local scratch space.		If these conditions are OK with you, please [[Requesting assistance\|contact us]] and request access to your local scratch space.

imported>Lkoven at 20:00, 22 May 2007

2007-05-22T20:00:09Z

← Older revision		Revision as of 20:00, 22 May 2007
Line 1:		Line 1:
	~~{{inuse}}~~
	==What are scratch disks?==		==What are scratch disks?==
	Scratch disks are usually smaller (but sometimes quite large) drives where you can temporarily store data that does not need to be backed up. They may be used for intermediate stages in a data pipeline, a local copy of a dataset available elsewhere or on a non-online media (CD, DLT, DVD), or temporary storage for outputs of your work. Scratch disks are not [[Backups\|backed up]] at all, though some of them are larger RAID5 disk systems which are more fault-tolerant than a single disk on a workstation. They are usually [[NFS]] mounted across all machines in the building for easy access.		Scratch disks are usually smaller (but sometimes quite large) drives where you can temporarily store data that does not need to be backed up. They may be used for intermediate stages in a data pipeline, a local copy of a dataset available elsewhere or on a non-online media (CD, DLT, DVD), or temporary storage for outputs of your work. Scratch disks are not [[Backups\|backed up]] at all, though some of them are larger RAID5 disk systems which are more fault-tolerant than a single disk on a workstation. They are usually [[NFS]] mounted across all machines in the building for easy access.

imported>Lkoven at 19:59, 22 May 2007

2007-05-22T19:59:48Z

← Older revision		Revision as of 19:59, 22 May 2007
Line 2:		Line 2:
	==What are scratch disks?==		==What are scratch disks?==
	Scratch disks are usually smaller (but sometimes quite large) drives where you can temporarily store data that does not need to be backed up. They may be used for intermediate stages in a data pipeline, a local copy of a dataset available elsewhere or on a non-online media (CD, DLT, DVD), or temporary storage for outputs of your work. Scratch disks are not [[Backups\|backed up]] at all, though some of them are larger RAID5 disk systems which are more fault-tolerant than a single disk on a workstation. They are usually [[NFS]] mounted across all machines in the building for easy access.		Scratch disks are usually smaller (but sometimes quite large) drives where you can temporarily store data that does not need to be backed up. They may be used for intermediate stages in a data pipeline, a local copy of a dataset available elsewhere or on a non-online media (CD, DLT, DVD), or temporary storage for outputs of your work. Scratch disks are not [[Backups\|backed up]] at all, though some of them are larger RAID5 disk systems which are more fault-tolerant than a single disk on a workstation. They are usually [[NFS]] mounted across all machines in the building for easy access.


	==How do I use access scratch disks?==		==How do I use access scratch disks?==
	Scratch disks are exported via nfs and are available on any department-support Linux desktop or server via automounts. You can access existing scratch space via the following directory structure		Scratch disks are exported via nfs and are available on any department-support Linux desktop or server via automounts. You can access existing scratch space via the following directory structure
Line 10:		Line 12:

	If you're using space that's directly attached to your local machine, you can also access it directly via <tt>/scr#</tt> to avoid a bit of NFS overhead.		If you're using space that's directly attached to your local machine, you can also access it directly via <tt>/scr#</tt> to avoid a bit of NFS overhead.


	==What scratch space is available?==		==What scratch space is available?==
	===depot===		===depot===
Line 22:		Line 26:
	A number of our desktops have a lot more space on their local hard drives than is used by the OS install and can be allocated as scratch space. If you plan to do a lot of I/O on larger (few GB+) datasets, using your local scratch space might be a good way to speed up the process as you are getting rid of network overhead. It's also a good way to get temporary data off the main (charged) NFS server.		A number of our desktops have a lot more space on their local hard drives than is used by the OS install and can be allocated as scratch space. If you plan to do a lot of I/O on larger (few GB+) datasets, using your local scratch space might be a good way to speed up the process as you are getting rid of network overhead. It's also a good way to get temporary data off the main (charged) NFS server.

	We suggest you think carefully about what files you want to use on local scratch space, however. This space, like all scratch space, is not backed up, and it's on a single hard drive with no redundancy. Do not use it for data that is irreplaceable or would take a long time to regenerate.		'''Note'''
			: We suggest you think carefully about what files you want to use on local scratch space, however. This space, like all scratch space, is not backed up, and it's on a single hard drive with no redundancy. Do not use it for data that is irreplaceable or would take a long time to regenerate.
	~~If these conditions are OK with you, please e-mail help@astro and request access to your local scratch space.~~


	~~{{stub}}~~		If these conditions are OK with you, please [[Requesting assistance\|contact us]] and request access to your local scratch space.

imported>Lkoven: Expanded the article

2007-05-22T19:51:42Z

Expanded the article

@@ Line 1: / Line 1: @@
 {{inuse}}
 Scratch disks are usually smaller (but sometimes quite large) drives where you can temporarily store data that does not need to be backed up.  They may be used for intermediate stages in a data pipeline, a local copy of a dataset available elsewhere or on a non-online media (CD, DLT, DVD), or temporary storage for outputs of your work.  Scratch disks are not [[Backups|backed up]] at all, though some of them are larger RAID5 disk systems which are more fault-tolerant than a single disk on a workstation.  They are usually [[NFS]] mounted across all machines in the building for easy access.
 {{stub}}