table of contents
GFPREP(1) | Gfarm | GFPREP(1) |
NAME¶
gfprep - replicate Gfarm files in parallel
SYNOPSIS¶
gfprep [-nqvdpPBU] [-mx] [-X regexp] [-S source-domainname] [-h source-hostfile] [-D destination-domainname] [-H destination-hostfile] [-j num-of-processes] [-J num-of-processes] [-M total-byte] [-z minimum-byte] [-Z maximum-byte] [-w way-of-scheduling] [-W kilobytes-for-threshold] [-s KB/s-to-simulate] [-F num-for-readahead] [-I sec-to-update] [-N num-of-replicas] path
DESCRIPTION¶
gfprep creates replicas of Gfarm files in parallel.
When the path specifies a directory, files under the directory will be replicated recursively.
Path can be specified by a Gfarm URL or a path on a mount point of gfarm2fs.
A set of source/destination hosts can be specified by a domain name and/or a hostlist file. When both a domain name and a hostlist file are specified, a set of hosts is determined by both conditions. When a set of source hosts is specified, only files stored on the source hosts are replicated.
Less busy file system nodes will be selected for source or destination. The maximum number of parallel replications/copies per host is the number of CPUs. Please see the manual page of gfhost for information about the number of CPUs.
gfprep also retrieves the directory entries in parallel.
GFPREP OPTIONS¶
These are options only for gfprep.
-N num-of-replicas
If this option is not specified, -N 1 is assumed.
-x
-m
This option requires at least one of -S, -h, -D, and -H options.
This option and the -N option are mutually exclusive.
COMMON OPTIONS¶
The following options are common options for gfprep and gfpcopy.
-X regexp
-S source-domainname
-h source-hostfile
If ``-'' is specified, standard input is used to read the host list.
-L
-D destination-domainname
If neither this nor the -H option is specified, replicas may be copied to any available host.
-H destination-hostfile
If ``-'' is specified, standard input is used to read the host list.
-j num-of-processes
The default value is the parameter of client_parallel_copy in gfarm2.conf. (see man gfarm2.conf)
The maximum number of process per file system node for source or destination is the number of CPUs (see man gfhost).
-J num-of-processes
The default value is 8.
-M total-byte
The default value is unlimited.
-z minimum-byte
The default value is unlimited.
-Z maximum-byte
The default value is unlimited.
-w way-of-scheduling
The default behavior is ``noplan''.
``greedy'' scheduling cannot use with the -N option and -m option.
-W kibibytes
This option is effective with -w greedy.
The default value is 50*1024 KiB (50 MiB).
-I sec-to-update
Default is 300 seconds.
-B
High loaded node is defined by having more CPU load than schedule_busy_load_thresh * number of CPUs. For details of schedule_busy_load_thresh, refer to a manual page of gfarm2.conf.
-U
-F num-of-dirents
The default value is 10000.
-s kilobytes-per-second
-n
-p
-P
-q
-v
-d
-?
EXAMPLES¶
To replicate files under the directory recursively.
$ gfprep -N 3 gfarm:///dir
$ cd /mnt/gfarm2fs $ gfprep -N 3 dir
To replicate a file or remove surplus replicas of the file.
$ gfprep -N 3 -x gfarm:///dir/file
To migrate replicas from gfsd1.example.com to other nodes.
$ gfprep -m -S gfsd1.example.com gfarm:///
NOTES¶
When the -L or -m option is not specified, a source node is not always selected within a set of specified source file system nodes.
It is better to execute gfprep command near the metadata server. When you need to execute gfprep command far from the metadata server, increase the parallelism by the -j and -J options.
SEE ALSO¶
14 Jan 2017 | Gfarm |