NCAR Campaign Storage is a resource for medium-term storage of project data, typically for three to five years, by NCAR labs and universities that have project allocations.
Campaign Storage is accessible a number of ways that are described below:
Files stored on this system are not backed up. Users are responsible for replicating any data they feel should be stored at an additional location as backup. Files that are deleted or overwritten cannot be recovered. |
Globus transfersThe Globus mapped collection established for the file system is NCAR Campaign Storage. How to make transfers to and from that collection is documented here: How to make transfers using the command line interface also is covered in detail in this tutorial: Using Globus v5 at NCAR (tutorial)
Data-access nodesThe Campaign Storage file system is mounted on the data-access nodes as /glade/campaign to:
Casper useThe Campaign Storage file system can be accessed from the Casper cluster as /glade/campaign so users are able to:
Data retention policyCampaign Storage is designed to provide medium-term storage for project data, typically for three to five years. While data will not be purged automatically after five years, retaining data longer will reduce the capacity for storing additional, new data. Users are expected to monitor their holdings, remove files that are no longer needed, and move necessary data to other storage options for longer-term preservation. NCAR researchers are expected to collaborate with CISL’s Digital Asset Services Hub (log in to Sundog) to develop data migration plans for storage needs that exceed five years. University researchers are expected to transfer their project data to their home institutions or other alternative storage repositories within one year of their NSF grant expiring. CISL will not award storage space for researchers to carry data forward from one grant to another. AllocationsNCAR labsEach NCAR lab has an allocation of Campaign Storage space and the labs manage how those allocations are used. Users who have questions related to lab allocations should contact the lab's own allocation representative. UniversitiesUniversity users can request Campaign Storage space through the NCAR Resource Allocation System as supplements to their project allocations. Requests must include detailed justification for the amount of space requested. Because NCAR is not currently funded to provide long-term data storage services to the university community, university users' requests for these allocations are prioritized based on the following factors. Higher priority is given to requests if:
Lower priority is given to requests if:
Any data requiring longer storage should be migrated to your home institution or to another appropriate repository. ReportsThe Systems Accounting Manager (SAM) provides overall summary information about the use of Campaign Storage allocations and other allocations. CISL is developing additional tools for use in allocation management. Automated data compressionCampaign Storage has an automated data compression feature for long-duration data sets. Our compression policy targets files that are 180 days old or older and 100MB in size or larger for "z" compression using IBM Spectrum Scale file system mechanisms (details below). The action is transparent to the user – that is, no changes to metadata timestamps or reported size occur, and subsequent reads of the data proceed as usual. During a read, the compressed data are sent to the file system client and then transparently uncompressed for application use. Tool and accounting behaviorThe number of blocks reported consumed by the file will change. Note the following tool-specific behavior:
Individual data sets can be excluded from the compression algorithm, if necessary. To discuss this option, please submit a request through the NCAR Research Computing help desk. Compression detailsWhen a file is considered for compression, the algorithm tests compression of chunks of the file. If the realized compression efficiency of a given chunk is at least 10%, it is then stored compressed on disk. The compression status of a file can be queried via the mmlsattr command. Follow this example:
A file has been compressed if the mmlsattr output:
Several output examples are provided below. User-driven manual compression is also possible before the automated policy is triggered if desired via the mmchattr command:
Examples: commands and outputRun du, ls, stat for an original uncompressed file
Request compression manually
Run du, ls, stat for a compressed file (note that metadata dates are not changed)
List file attributes to verify a compressed file
Request deferred compression of a file
Note that deferred compression is recommended when manually requesting compression for a large number of files. In this case, the mmchattr command will return immediately, and the file compression will occur at the next regularly scheduled system interval. List file attributes (note that "illcompressed" indicates the compression has not yet been applied)
|