.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "PGCOPYDB CLONE" "1" "Mar 15, 2023" "0.11" "pgcopydb" .SH NAME pgcopydb clone \- pgcopydb clone .sp The main pgcopydb operation is the clone operation, and for historical and user friendlyness reasons three aliases are available that implement the same operation: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C pgcopydb clone Clone an entire database from source to target fork Clone an entire database from source to target copy\-db Copy an entire database from source to target .ft P .fi .UNINDENT .UNINDENT .SH PGCOPYDB CLONE .sp The command \fBpgcopydb clone\fP copies a database from the given source Postgres instance to the target Postgres instance. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C pgcopydb clone: Clone an entire database from source to target usage: pgcopydb clone \-\-source ... \-\-target ... [ \-\-table\-jobs ... \-\-index\-jobs ... ] \-\-source Postgres URI to the source database \-\-target Postgres URI to the target database \-\-dir Work directory to use \-\-table\-jobs Number of concurrent COPY jobs to run \-\-index\-jobs Number of concurrent CREATE INDEX jobs to run \-\-split\-tables\-larger\-than Same\-table concurrency size threshold \-\-drop\-if\-exists On the target database, clean\-up from a previous run first \-\-roles Also copy roles found on source to target \-\-no\-role\-passwords Do not dump passwords for roles \-\-no\-owner Do not set ownership of objects to match the original database \-\-no\-acl Prevent restoration of access privileges (grant/revoke commands). \-\-no\-comments Do not output commands to restore comments \-\-skip\-large\-objects Skip copying large objects (blobs) \-\-skip\-extensions Skip restoring extensions \-\-skip\-collations Skip restoring collations \-\-filters Use the filters defined in \-\-restart Allow restarting when temp files exist already \-\-resume Allow resuming operations after a failure \-\-not\-consistent Allow taking a new snapshot on the source database \-\-snapshot Use snapshot obtained with pg_export_snapshot \-\-follow Implement logical decoding to replay changes \-\-plugin Output plugin to use (test_decoding, wal2json) \-\-slot\-name Use this Postgres replication slot name \-\-create\-slot Create the replication slot \-\-origin Use this Postgres replication origin node name \-\-endpos Stop replaying changes when reaching this LSN .ft P .fi .UNINDENT .UNINDENT .SH PGCOPYDB FORK .sp The command \fBpgcopydb fork\fP copies a database from the given source Postgres instance to the target Postgres instance. This command is an alias to the command \fBpgcopydb clone\fP seen above. .SH PGCOPYDB COPY-DB .sp The command \fBpgcopydb copy\-db\fP copies a database from the given source Postgres instance to the target Postgres instance. This command is an alias to the command \fBpgcopydb clone\fP seen above, and available for backward compatibility only. .sp \fBWARNING:\fP .INDENT 0.0 .INDENT 3.5 The \fBpgcopydb copy\-db\fP command is now deprecated and will get removed from pgcopydb when hitting version 1.0, please upgrade your scripts and integrations. .UNINDENT .UNINDENT .SH DESCRIPTION .sp The \fBpgcopydb clone\fP command implements both a base copy of a source database into a target database and also a full \fI\%Logical Decoding\fP client for the \fI\%wal2json\fP logical decoding plugin. .SS Base copy, or the clone operation .sp The \fBpgcopydb clone\fP command implements the following steps: .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .IP 1. 4 \fBpgcopydb\fP calls into \fBpg_dump\fP to produce the \fBpre\-data\fP section and the \fBpost\-data\fP sections of the dump using Postgres custom format. .IP 2. 4 \fBpgcopydb\fP gets the list of ordinary and partitioned tables from a catalog query on the source database, and also the list of indexes, and the list of sequences with their current values. .sp When filtering is used, the list of objects OIDs that are meant to be filtered out is built during this step. .IP 3. 4 The \fBpre\-data\fP section of the dump is restored on the target database using the \fBpg_restore\fP command, creating all the Postgres objects from the source database into the target database. .sp When filtering is used, the \fBpg_restore \-\-use\-list\fP feature is used to filter the list of objects to restore in this step. .IP 4. 4 Then as many as \fB\-\-table\-jobs\fP COPY sub\-processes are started to share the workload and COPY the data from the source to the target database one table at a time, in a loop. .sp A Postgres connection and a SQL query to the Postgres catalog table pg_class is used to get the list of tables with data to copy around, and the \fIreltuples\fP statistic is used to start with the tables with the greatest number of rows first, as an attempt to minimize the copy time. .IP 5. 4 An auxiliary process loops through all the Large Objects found on the source database and copies its data parts over to the target database, much like pg_dump itself would. .sp This step is much like \fBpg_dump | pg_restore\fP for large objects data parts, except that there isn\(aqt a good way to do just that with the tooling. .IP 6. 4 As many as \fB\-\-index\-jobs\fP CREATE INDEX sub\-processes are started to share the workload and build indexes. In order to make sure to start the CREATE INDEX commands only after the COPY operation has completed, a queue mechanism is used. As soon as a table data COPY has completed, all the indexes for the table are queued for processing by the CREATE INDEX sub\-processes. .sp The primary indexes are created as UNIQUE indexes at this stage. .IP 7. 4 Then the PRIMARY KEY constraints are created USING the just built indexes. This two\-steps approach allows the primary key index itself to be created in parallel with other indexes on the same table, avoiding an EXCLUSIVE LOCK while creating the index. .IP 8. 4 As many as \fB\-table\-jobs\fP VACUUM ANALYZE sub\-processes are started to share the workload. As soon as a table data COPY has completed, the table is queued for processing by the VACUUM ANALYZE sub\-processes. .IP 9. 4 An auxilliary process is loops over the sequences on the source database and for each of them runs a separate query on the source to fetch the \fBlast_value\fP and the \fBis_called\fP metadata the same way that pg_dump does. .sp For each sequence, pgcopydb then calls \fBpg_catalog.setval()\fP on the target database with the information obtained on the source database. .IP 10. 4 The final stage consists now of running the \fBpg_restore\fP command for the \fBpost\-data\fP section script for the whole database, and that\(aqs where the foreign key constraints and other elements are created. .sp The \fIpost\-data\fP script is filtered out using the \fBpg_restore \-\-use\-list\fP option so that indexes and primary key constraints already created in steps 6 and 7 are properly skipped now. .UNINDENT .UNINDENT .UNINDENT .SS Postgres privileges, superuser, and dump and restore .sp Postgres has a notion of a superuser status that can be assigned to any role in the system, and the default role \fIpostgres\fP has this status. From the \fI\%Role Attributes\fP documentation page we see that: .INDENT 0.0 .INDENT 3.5 superuser status: .sp \fIA database superuser bypasses all permission checks, except the right to log in. This is a dangerous privilege and should not be used carelessly; it is best to do most of your work as a role that is not a superuser. To create a new database superuser, use CREATE ROLE name SUPERUSER. You must do this as a role that is already a superuser.\fP .UNINDENT .UNINDENT .sp Some Postgres objects can only be created by superusers, and some read and write operations are only allowed to superuser roles, such as the following non\-exclusive list: .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .IP \(bu 2 Reading the \fI\%pg_authid\fP role password (even when encrypted) is restricted to roles with the superuser status. Reading this catalog table is done when calling \fBpg_dumpall \-\-roles\-only\fP so that the dump file can then be used to restore roles including their passwords. .sp It is possible to implement a pgcopydb migration that skips the passwords entirely when using the option \fB\-\-no\-role\-passwords\fP\&. In that case though authentication might fail until passwords have been setup again correctly. .IP \(bu 2 Most of the available Postgres extensions, at least when being written in C, are then only allowed to be created by roles with superuser status. .sp When such an extension contains \fI\%Extension Configuration Tables\fP and has been created with a role having superuser status, then the same superuser status is needed again to pg_dump and pg_restore that extension and its current configuration. .UNINDENT .UNINDENT .UNINDENT .sp When using pgcopydb it is possible to split your migration in privileged and non\-privileged parts, like in the following examples: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ coproc ( pgcopydb snapshot ) # first two commands would use a superuser role to connect $ pgcopydb copy roles \-\-source ... \-\-target ... $ pgcopydb copy extensions \-\-source ... \-\-target ... # now it\(aqs possible to use a non\-superuser role to connect $ pgcopydb clone \-\-skip\-extensions \-\-source ... \-\-target ... $ kill \-TERM ${COPROC_PID} $ wait ${COPROC_PID} .ft P .fi .UNINDENT .UNINDENT .sp In such a script, the calls to \fI\%pgcopydb copy roles\fP and \fI\%pgcopydb copy extensions\fP would be done with connection strings that connects with a role having superuser status; and then the call to \fIpgcopydb clone\fP would be done with a non\-privileged role, typically the role that owns the source and target databases. .sp \fBWARNING:\fP .INDENT 0.0 .INDENT 3.5 That said, there is currently a limitation in \fBpg_dump\fP that impacts pgcopydb. When an extension with configuration table has been installed as superuser, even the main \fBpgcopydb clone\fP operation has to be done with superuser status. .sp That\(aqs because pg_dump filtering (here, there \fB\-\-exclude\-table\fP option) does not apply to extension members, and pg_dump does not provide a mechanism to exclude extensions. .UNINDENT .UNINDENT .SS Change Data Capture using Postgres Logical Decoding .sp When using the \fB\-\-follow\fP option the steps from the \fI\%pgcopydb follow\fP command are also run concurrently to the main copy. The Change Data Capture is then automatically driven from a prefetch\-only phase to the prefetch\-and\-catchup phase, which is enabled as soon as the base copy is done. .sp See the command \fI\%pgcopydb stream sentinel set endpos\fP to remote control the follow parts of the command even while the command is already running. .sp The command \fI\%pgcopydb stream cleanup\fP must be used to free resources created to support the change data capture process. .sp \fBIMPORTANT:\fP .INDENT 0.0 .INDENT 3.5 Make sure to read the documentation for \fI\%pgcopydb follow\fP and the specifics about \fI\%Logical Replication Restrictions\fP as documented by Postgres. .UNINDENT .UNINDENT .SS Change Data Capture Example 1 .sp A simple approach to applying changes after the initial base copy has been done follows: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ pgcopydb clone \-\-follow & # later when the application is ready to make the switch $ pgcopydb stream sentinel set endpos \-\-current # later when the migration is finished, clean\-up both source and target $ pgcopydb stream cleanup .ft P .fi .UNINDENT .UNINDENT .SS Change Data Capture Example 2 .sp In some cases, it might be necessary to have more control over some of the steps taken here. Given pgcopydb flexibility, it\(aqs possible to implement the following steps: .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .IP 1. 3 Grab a snapshot from the source database and hold an open Postgres connection for the duration of the base copy. .sp In case of crash or other problems with the main operations, it\(aqs then possible to resume processing of the base copy and the applying of the changes with the same snapshot again. .sp This step is also implemented when using \fBpgcopydb clone \-\-follow\fP\&. That said, if the command was interrupted (or crashed), then the snapshot would be lost. .IP 2. 3 Setup the logical decoding within the snapshot obtained in the previous step, and the replication tracking on the target database. .sp The following SQL objects are then created: .INDENT 3.0 .INDENT 3.5 .INDENT 0.0 .IP \(bu 2 a replication slot on the source database, .IP \(bu 2 a \fBpgcopydb.sentinel\fP table on the source database, .IP \(bu 2 a replication origin on the target database. .UNINDENT .UNINDENT .UNINDENT .sp This step is also implemented when using \fBpgcopydb clone \-\-follow\fP\&. There is no way to implement Change Data Capture with pgcopydb and skip creating those SQL objects. .IP 3. 3 Start the base copy of the source database, and prefetch logical decoding changes to ensure that we consume from the replication slot and allow the source database server to recycle its WAL files. .IP 4. 3 Remote control the apply process to stop consuming changes and applying them on the target database. .IP 5. 3 Re\-sync the sequences to their now\-current values. .sp Sequences are not handled by Postgres logical decoding, so extra care needs to be implemented manually here. .sp \fBIMPORTANT:\fP .INDENT 3.0 .INDENT 3.5 The next version of pgcopydb will include that step in the \fBpgcopydb clone \-\-snapshot\fP command automatically, after it stops consuming changes and before the process terminates. .UNINDENT .UNINDENT .IP 6. 3 Clean\-up the specific resources created for supporting resumability of the whole process (replication slot on the source database, pgcopydb sentinel table on the source database, replication origin on the target database). .IP 7. 3 Stop holding a snaphot on the source database by stopping the \fBpgcopydb snapshot\fP process left running in the background. .UNINDENT .UNINDENT .UNINDENT .sp If the command \fBpgcopydb clone \-\-follow\fP fails it\(aqs then possible to start it again. It will automatically discover what was done successfully and what needs to be done again because it failed or was interrupted (table copy, index creation, resuming replication slot consuming, resuming applying changes at the right LSN position, etc). .sp Here is an example implement the previous steps: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ pgcopydb snapshot & $ pgcopydb stream setup $ pgcopydb clone \-\-follow & # later when the application is ready to make the switch $ pgcopydb stream sentinel set endpos \-\-current # when the follow process has terminated, re\-sync the sequences $ pgcopydb copy sequences # later when the migration is finished, clean\-up both source and target $ pgcopydb stream cleanup # now stop holding the snapshot transaction (adjust PID to your environment) $ kill %1 .ft P .fi .UNINDENT .UNINDENT .SH OPTIONS .sp The following options are available to \fBpgcopydb clone\fP: .INDENT 0.0 .TP .B \-\-source Connection string to the source Postgres instance. See the Postgres documentation for \fI\%connection strings\fP for the details. In short both the quoted form \fB\(dqhost=... dbname=...\(dq\fP and the URI form \fBpostgres://user@host:5432/dbname\fP are supported. .TP .B \-\-target Connection string to the target Postgres instance. .TP .B \-\-dir During its normal operations pgcopydb creates a lot of temporary files to track sub\-processes progress. Temporary files are created in the directory location given by this option, or defaults to \fB${TMPDIR}/pgcopydb\fP when the environment variable is set, or then to \fB/tmp/pgcopydb\fP\&. .TP .B \-\-table\-jobs How many tables can be processed in parallel. .sp This limit only applies to the COPY operations, more sub\-processes will be running at the same time that this limit while the CREATE INDEX operations are in progress, though then the processes are only waiting for the target Postgres instance to do all the work. .TP .B \-\-index\-jobs How many indexes can be built in parallel, globally. A good option is to set this option to the count of CPU cores that are available on the Postgres target system, minus some cores that are going to be used for handling the COPY operations. .TP .B \-\-split\-tables\-larger\-than Allow \fI\%Same\-table Concurrency\fP when processing the source database. This environment variable value is expected to be a byte size, and bytes units B, kB, MB, GB, TB, PB, and EB are known. .TP .B \-\-drop\-if\-exists When restoring the schema on the target Postgres instance, \fBpgcopydb\fP actually uses \fBpg_restore\fP\&. When this options is specified, then the following pg_restore options are also used: \fB\-\-clean \-\-if\-exists\fP\&. .sp This option is useful when the same command is run several times in a row, either to fix a previous mistake or for instance when used in a continuous integration system. .sp This option causes \fBDROP TABLE\fP and \fBDROP INDEX\fP and other DROP commands to be used. Make sure you understand what you\(aqre doing here! .TP .B \-\-roles The option \fB\-\-roles\fP add a preliminary step that copies the roles found on the source instance to the target instance. As Postgres roles are global object, they do not exist only within the context of a specific database, so all the roles are copied over when using this option. .sp The \fBpg_dumpall \-\-roles\-only\fP is used to fetch the list of roles from the source database, and this command includes support for passwords. As a result, this operation requires the superuser privileges. .sp See also \fI\%pgcopydb copy roles\fP\&. .TP .B \-\-no\-role\-passwords Do not dump passwords for roles. When restored, roles will have a null password, and password authentication will always fail until the password is set. Since password values aren\(aqt needed when this option is specified, the role information is read from the catalog view pg_roles instead of pg_authid. Therefore, this option also helps if access to pg_authid is restricted by some security policy. .TP .B \-\-no\-owner Do not output commands to set ownership of objects to match the original database. By default, \fBpg_restore\fP issues \fBALTER OWNER\fP or \fBSET SESSION AUTHORIZATION\fP statements to set ownership of created schema elements. These statements will fail unless the initial connection to the database is made by a superuser (or the same user that owns all of the objects in the script). With \fB\-\-no\-owner\fP, any user name can be used for the initial connection, and this user will own all the created objects. .TP .B \-\-skip\-large\-objects Skip copying large objects, also known as blobs, when copying the data from the source database to the target database. .TP .B \-\-skip\-extensions Skip copying extensions from the source database to the target database. .sp When used, schema that extensions depend\-on are also skipped: it is expected that creating needed extensions on the target system is then the responsibility of another command (such as \fI\%pgcopydb copy extensions\fP), and schemas that extensions depend\-on are part of that responsibility. .sp Because creating extensions require superuser, this allows a multi\-steps approach where extensions are dealt with superuser privileges, and then the rest of the pgcopydb operations are done without superuser privileges. .TP .B \-\-skip\-collations Skip copying collations from the source database to the target database. .sp In some scenarios the list of collations provided by the Operating System on the source and target system might be different, and a mapping then needs to be manually installed before calling pgcopydb. .sp Then this option allows pgcopydb to skip over collations and assume all the needed collations have been deployed on the target database already. .sp See also \fI\%pgcopydb list collations\fP\&. .TP .BI \-\-filters \ This option allows to exclude table and indexes from the copy operations. See \fI\%Filtering\fP for details about the expected file format and the filtering options available. .TP .B \-\-restart When running the pgcopydb command again, if the work directory already contains information from a previous run, then the command refuses to proceed and delete information that might be used for diagnostics and forensics. .sp In that case, the \fB\-\-restart\fP option can be used to allow pgcopydb to delete traces from a previous run. .TP .B \-\-resume When the pgcopydb command was terminated before completion, either by an interrupt signal (such as C\-c or SIGTERM) or because it crashed, it is possible to resume the database migration. .sp When resuming activity from a previous run, table data that was fully copied over to the target server is not sent again. Table data that was interrupted during the COPY has to be started from scratch even when using \fB\-\-resume\fP: the COPY command in Postgres is transactional and was rolled back. .sp Same reasonning applies to the CREATE INDEX commands and ALTER TABLE commands that pgcopydb issues, those commands are skipped on a \fB\-\-resume\fP run only if known to have run through to completion on the previous one. .sp Finally, using \fB\-\-resume\fP requires the use of \fB\-\-not\-consistent\fP\&. .TP .B \-\-not\-consistent In order to be consistent, pgcopydb exports a Postgres snapshot by calling the \fI\%pg_export_snapshot()\fP function on the source database server. The snapshot is then re\-used in all the connections to the source database server by using the \fBSET TRANSACTION SNAPSHOT\fP command. .sp Per the Postgres documentation about \fBpg_export_snapshot\fP: .INDENT 7.0 .INDENT 3.5 Saves the transaction\(aqs current snapshot and returns a text string identifying the snapshot. This string must be passed (outside the database) to clients that want to import the snapshot. The snapshot is available for import only until the end of the transaction that exported it. .UNINDENT .UNINDENT .sp Now, when the pgcopydb process was interrupted (or crashed) on a previous run, it is possible to resume operations, but the snapshot that was exported does not exists anymore. The pgcopydb command can only resume operations with a new snapshot, and thus can not ensure consistency of the whole data set, because each run is now using their own snapshot. .TP .B \-\-snapshot Instead of exporting its own snapshot by calling the PostgreSQL function \fBpg_export_snapshot()\fP it is possible for pgcopydb to re\-use an already exported snapshot. .TP .B \-\-follow When the \fB\-\-follow\fP option is used then pgcopydb implements Change Data Capture as detailed in the manual page for \fI\%pgcopydb follow\fP in parallel to the main copy database steps. .sp The replication slot is created using the same snapshot as the main database copy operation, and the changes to the source database are prefetched only during the initial copy, then prefetched and applied in a catchup process. .sp It is possible to give \fBpgcopydb clone \-\-follow\fP a termination point (the LSN endpos) while the command is running with the command \fI\%pgcopydb stream sentinel set endpos\fP\&. .TP .B \-\-plugin Logical decoding output plugin to use. The default is \fI\%test_decoding\fP which ships with Postgres core itself, so is probably already available on your source server. .sp It is possible to use \fI\%wal2json\fP instead. The support for wal2json is mostly historical in pgcopydb, it should not make a user visible difference whether you use the default test_decoding or wal2json. .TP .B \-\-slot\-name Logical decoding slot name to use. Defaults to \fBpgcopydb\fP\&. which is unfortunate when your use\-case involves migrating more than one database from the source server. .TP .B \-\-create\-slot Instruct pgcopydb to create the logical replication slot to use. .TP .B \-\-endpos Logical replication target LSN to use. Automatically stop replication and exit with normal exit status 0 when receiving reaches the specified LSN. If there\(aqs a record with LSN exactly equal to lsn, the record will be output. .sp The \fB\-\-endpos\fP option is not aware of transaction boundaries and may truncate output partway through a transaction. Any partially output transaction will not be consumed and will be replayed again when the slot is next read from. Individual messages are never truncated. .sp See also documentation for \fI\%pg_recvlogical\fP\&. .TP .B \-\-origin Logical replication target system needs to track the transactions that have been applied already, so that in case we get disconnected or need to resume operations we can skip already replayed transaction. .sp Postgres uses a notion of an origin node name as documented in \fI\%Replication Progress Tracking\fP\&. This option allows to pick your own node name and defaults to \(dqpgcopydb\(dq. Picking a different name is useful in some advanced scenarios like migrating several sources in the same target, where each source should have their own unique origin node name. .TP .B \-\-verbose Increase current verbosity. The default level of verbosity is INFO. In ascending order pgcopydb knows about the following verbosity levels: FATAL, ERROR, WARN, INFO, NOTICE, DEBUG, TRACE. .TP .B \-\-debug Set current verbosity to DEBUG level. .TP .B \-\-trace Set current verbosity to TRACE level. .TP .B \-\-quiet Set current verbosity to ERROR level. .UNINDENT .SH ENVIRONMENT .sp PGCOPYDB_SOURCE_PGURI .INDENT 0.0 .INDENT 3.5 Connection string to the source Postgres instance. When \fB\-\-source\fP is ommitted from the command line, then this environment variable is used. .UNINDENT .UNINDENT .sp PGCOPYDB_TARGET_PGURI .INDENT 0.0 .INDENT 3.5 Connection string to the target Postgres instance. When \fB\-\-target\fP is ommitted from the command line, then this environment variable is used. .UNINDENT .UNINDENT .sp PGCOPYDB_TABLE_JOBS .INDENT 0.0 .INDENT 3.5 Number of concurrent jobs allowed to run COPY operations in parallel. When \fB\-\-table\-jobs\fP is ommitted from the command line, then this environment variable is used. .UNINDENT .UNINDENT .sp PGCOPYDB_INDEX_JOBS .INDENT 0.0 .INDENT 3.5 Number of concurrent jobs allowed to run CREATE INDEX operations in parallel. When \fB\-\-index\-jobs\fP is ommitted from the command line, then this environment variable is used. .UNINDENT .UNINDENT .sp PGCOPYDB_SPLIT_TABLES_LARGER_THAN .INDENT 0.0 .INDENT 3.5 Allow \fI\%Same\-table Concurrency\fP when processing the source database. This environment variable value is expected to be a byte size, and bytes units B, kB, MB, GB, TB, PB, and EB are known. .sp When \fB\-\-split\-tables\-larger\-than\fP is ommitted from the command line, then this environment variable is used. .UNINDENT .UNINDENT .sp PGCOPYDB_DROP_IF_EXISTS .INDENT 0.0 .INDENT 3.5 When true (or \fIyes\fP, or \fIon\fP, or 1, same input as a Postgres boolean) then pgcopydb uses the pg_restore options \fB\-\-clean \-\-if\-exists\fP when creating the schema on the target Postgres instance. .UNINDENT .UNINDENT .sp PGCOPYDB_SNAPSHOT .INDENT 0.0 .INDENT 3.5 Postgres snapshot identifier to re\-use, see also \fB\-\-snapshot\fP\&. .UNINDENT .UNINDENT .sp TMPDIR .INDENT 0.0 .INDENT 3.5 The pgcopydb command creates all its work files and directories in \fB${TMPDIR}/pgcopydb\fP, and defaults to \fB/tmp/pgcopydb\fP\&. .UNINDENT .UNINDENT .sp XDG_DATA_HOME .INDENT 0.0 .INDENT 3.5 The standard \fI\%XDG Base Directory Specification\fP defines several environment variables that allow controling where programs should store their files. .INDENT 0.0 .INDENT 3.5 \fIXDG_DATA_HOME defines the base directory relative to which user\-specific data files should be stored. If $XDG_DATA_HOME is either not set or empty, a default equal to $HOME/.local/share should be used.\fP .UNINDENT .UNINDENT .sp When using Change Data Capture (through \fB\-\-follow\fP option and Postgres logical decoding with \fI\%wal2json\fP) then pgcopydb pre\-fetches changes in JSON files and transform them into SQL files to apply to the target database. .sp These files are stored at the following location, tried in this order: .INDENT 0.0 .INDENT 3.5 .INDENT 0.0 .IP 1. 3 when \fB\-\-dir\fP is used, then pgcopydb uses the \fBcdc\fP subdirectory of the \fB\-\-dir\fP location, .IP 2. 3 when \fBXDG_DATA_HOME\fP is set in the environment, then pgcopydb uses that location, .IP 3. 3 when neither of the previous settings have been used then pgcopydb defaults to using \fB${HOME}/.local/share\fP\&. .UNINDENT .UNINDENT .UNINDENT .UNINDENT .UNINDENT .SH EXAMPLES .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C $ export PGCOPYDB_SOURCE_PGURI=\(dqport=54311 host=localhost dbname=pgloader\(dq $ export PGCOPYDB_TARGET_PGURI=\(dqport=54311 dbname=plop\(dq $ export PGCOPYDB_DROP_IF_EXISTS=on $ pgcopydb clone \-\-table\-jobs 8 \-\-index\-jobs 12 13:09:08 81987 INFO Running pgcopydb version 0.8.21.gacd2795.dirty from \(dq/Applications/Postgres.app/Contents/Versions/12/bin/pgcopydb\(dq 13:09:08 81987 INFO [SOURCE] Copying database from \(dqpostgres://@:/pagila?\(dq 13:09:08 81987 INFO [TARGET] Copying database into \(dqpostgres://@:/plop?\(dq 13:09:08 81987 INFO Using work dir \(dq/var/folders/d7/zzxmgs9s16gdxxcm0hs0sssw0000gn/T//pgcopydb\(dq 13:09:08 81987 INFO Exported snapshot \(dq00000003\-00076012\-1\(dq from the source database 13:09:08 81991 INFO STEP 1: dump the source database schema (pre/post data) 13:09:08 81991 INFO /Applications/Postgres.app/Contents/Versions/12/bin/pg_dump \-Fc \-\-snapshot 00000003\-00076012\-1 \-\-section pre\-data \-\-file /var/folders/d7/zzxmgs9s16gdxxcm0hs0sssw0000gn/T//pgcopydb/schema/pre.dump \(aqpostgres://@:/pagila?\(aq 13:09:08 81991 INFO /Applications/Postgres.app/Contents/Versions/12/bin/pg_dump \-Fc \-\-snapshot 00000003\-00076012\-1 \-\-section post\-data \-\-file /var/folders/d7/zzxmgs9s16gdxxcm0hs0sssw0000gn/T//pgcopydb/schema/post.dump \(aqpostgres://@:/pagila?\(aq 13:09:08 81991 INFO STEP 2: restore the pre\-data section to the target database 13:09:09 81991 INFO Listing ordinary tables in source database 13:09:09 81991 INFO Fetched information for 21 tables, with an estimated total of 46 248 tuples and 3776 kB 13:09:09 81991 INFO Fetching information for 13 sequences 13:09:09 81991 INFO /Applications/Postgres.app/Contents/Versions/12/bin/pg_restore \-\-dbname \(aqpostgres://@:/plop?\(aq \-\-single\-transaction \-\-clean \-\-if\-exists \-\-use\-list /var/folders/d7/zzxmgs9s16gdxxcm0hs0sssw0000gn/T//pgcopydb/schema/pre.list /var/folders/d7/zzxmgs9s16gdxxcm0hs0sssw0000gn/T//pgcopydb/schema/pre.dump 13:09:09 81991 INFO STEP 3: copy data from source to target in sub\-processes 13:09:09 81991 INFO STEP 4: create indexes and constraints in parallel 13:09:09 81991 INFO STEP 5: vacuum analyze each table 13:09:09 81991 INFO Now starting 8 processes 13:09:09 81991 INFO Reset sequences values on the target database 13:09:09 82003 INFO COPY \(dqpublic\(dq.\(dqrental\(dq 13:09:09 82004 INFO COPY \(dqpublic\(dq.\(dqfilm\(dq 13:09:09 82009 INFO COPY \(dqpublic\(dq.\(dqpayment_p2020_04\(dq 13:09:09 82002 INFO Copying large objects 13:09:09 82007 INFO COPY \(dqpublic\(dq.\(dqpayment_p2020_03\(dq 13:09:09 82010 INFO COPY \(dqpublic\(dq.\(dqfilm_actor\(dq 13:09:09 82005 INFO COPY \(dqpublic\(dq.\(dqinventory\(dq 13:09:09 82014 INFO COPY \(dqpublic\(dq.\(dqpayment_p2020_02\(dq 13:09:09 82012 INFO COPY \(dqpublic\(dq.\(dqcustomer\(dq 13:09:09 82009 INFO Creating 3 indexes for table \(dqpublic\(dq.\(dqpayment_p2020_04\(dq 13:09:09 82010 INFO Creating 2 indexes for table \(dqpublic\(dq.\(dqfilm_actor\(dq 13:09:09 82007 INFO Creating 3 indexes for table \(dqpublic\(dq.\(dqpayment_p2020_03\(dq 13:09:09 82004 INFO Creating 5 indexes for table \(dqpublic\(dq.\(dqfilm\(dq 13:09:09 82005 INFO Creating 2 indexes for table \(dqpublic\(dq.\(dqinventory\(dq 13:09:09 82033 INFO VACUUM ANALYZE \(dqpublic\(dq.\(dqpayment_p2020_04\(dq; 13:09:09 82036 INFO VACUUM ANALYZE \(dqpublic\(dq.\(dqfilm_actor\(dq; 13:09:09 82039 INFO VACUUM ANALYZE \(dqpublic\(dq.\(dqpayment_p2020_03\(dq; 13:09:09 82041 INFO VACUUM ANALYZE \(dqpublic\(dq.\(dqfilm\(dq; 13:09:09 82043 INFO VACUUM ANALYZE \(dqpublic\(dq.\(dqinventory\(dq; \&... \&... \&... 13:09:09 81991 INFO STEP 7: restore the post\-data section to the target database 13:09:09 81991 INFO /Applications/Postgres.app/Contents/Versions/12/bin/pg_restore \-\-dbname \(aqpostgres://@:/plop?\(aq \-\-single\-transaction \-\-clean \-\-if\-exists \-\-use\-list /var/folders/d7/zzxmgs9s16gdxxcm0hs0sssw0000gn/T//pgcopydb/schema/post.list /var/folders/d7/zzxmgs9s16gdxxcm0hs0sssw0000gn/T//pgcopydb/schema/post.dump Step Connection Duration Concurrency \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\- Dump Schema source 355ms 1 Prepare Schema target 135ms 1 COPY, INDEX, CONSTRAINTS, VACUUM (wall clock) both 641ms 8 + 12 COPY (cumulative) both 1s598 8 Large Objects (cumulative) both 29ms 1 CREATE INDEX, CONSTRAINTS (cumulative) target 4s072 12 Finalize Schema target 366ms 1 \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\- Total Wall Clock Duration both 1s499 8 + 12 \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\- .ft P .fi .UNINDENT .UNINDENT .SH AUTHOR Dimitri Fontaine .SH COPYRIGHT 2023, Dimitri Fontaine .\" Generated by docutils manpage writer. .