table of contents
GFPREP(1) | Gfarm | GFPREP(1) |
NAME¶
gfprep - replicate Gfarm files in parallelSYNOPSIS¶
gfprep [-nqvdpPU] [-mx] [-X regexp] [-S source-domainname] [-h source-hostfile] [-D destination-domainname] [-H destination-hostfile] [-j num-of-processes] [-J num-of-processes] [-M total-byte] [-z minimum-byte] [-Z maximum-byte] [-w way-of-scheduling] [-W kilobytes-for-threshold] [-s KB/s-to-simulate] [-F num-for-readahead] [-N num-of-replicas] path
DESCRIPTION¶
gfprep creates replicas of Gfarm files in parallel.If the path parameter is a directory, files under the directory will be replicated recursively.
path is a Gfarm URL or a path on a mountpoint of gfarm2fs.
A set of source/destination hosts can be specified by a domain name and/or a hostlist file. When both a domain name and a hostlist file are specified, a set of hosts is determined by both conditions. When a set of source hosts is specified, only files stored on the source hosts are replicated.
Idler filesystem nodes will be selected for source or destination. The maximum number of parallel to send or receive per host is the number of CPUs. Please see the manual page of gfhost for information about the number of CPUs.
gfprep can also read the directory and get information of the entries in parallel.
GFPREP OPTIONS¶
These are the options only for gfprep.-N num-of-replicas
If this option is not specified, -N 1 is assumed.
-x
-m
This option requires -S or -h or -D or -H options.
This option and -N option are mutually exclusive.
COMMON OPTIONS¶
gfprep has the same options with gfpcopy.-X regexp
-S source-domainname
-h source-hostfile
If ``-'' is specified, standard input is used to read the host list.
-L
-D destination-domainname
If neither this nor the -H option is specified, replicas may be copied to any available host.
-H destination-hostfile
If ``-'' is specified, standard input is used to read the host list.
-j num-of-processes
The default value is the parameter of client_parallel_copy in gfarm2.conf. (see man gfarm2.conf)
The maximum number of process per filesystem node for source or destination is the number of CPUs (see man gfhost).
-J num-of-processes
The default value is 8.
-M total-byte
The default value is unlimited.
-z minimum-byte
The default value is unlimited.
-Z maximum-byte
The default value is unlimited.
-w way-of-scheduling
The default behavior is ``noplan''.
``greedy'' scheduling cannot use the -N option or -m option.
-W kibibytes
This option is effective with -w greedy.
The default value is 50*1024 KiB (50MiB).
-U
-F num-of-dirents
The default value is 10000.
-s kilobytes-per-sec.
-n
-p
-P
-q
-v
-d
-?
EXAMPLES¶
To replicate files under the directory recursively.$ gfprep -N 3 gfarm:///dir
$ cd /mnt/gfarm2fs $ gfprep -N 3 dir
To replicate a file or reduce surplus replicas of the file.
$ gfprep -N 3 -x gfarm:///dir/file
To migrate replicas from gfsd1.example.com to other nodes.
$ gfprep -m -S gfsd1.example.com gfarm:///
NOTES¶
When the -L or -m option is not specified, a source node is not always selected within a set of specified source filesystem nodes.gfprep command should be executed at the same host of gfmd. If the gfmd and gfprep are far apart, the gfprep operation is slow. Specifying a large value by -j or -J options may be effective in such a case.
SEE ALSO¶
gfrep(1), gfpcopy(1), gfhost(1), gfarm2.conf(5)27 Aug 2015 | Gfarm |