Backup and Recovery

This chapter describes failover or replacement of the services machine. Refer to the Backup section below for information about possible backup strategies and procedures for implementing a regular backup image or snapshot of the services machine.

Disaster Recovery





If you are running CircleCI in an HA configuration, you must use standard backup mechanisms for the external datastores. Contactsupport@www.drag240sx.com有关详细信息,请访问更多信息文档。

Backing up the Database

如果你有not配置Circleci for外部服务,备份Circleci数据的最佳做法是使用虚拟磁盘的VM快照作为服务机器的根卷。可以在没有停机时间的情况下执行备份,因为底层虚拟磁盘支持这样的操作,就像AWS EB一样。风险较小,这因文件系统和分布而变化,没有重启的快照可能有一些数据损坏,但这在实践中很少见。



Build artifacts, output, and caches are generally stored in object storage services like AWS S3. These services are considered highly redundant and are unlikely to require separate backup. An exception is if your instance is setup to store large objects locally on the Services machine, either directly on-disk or on an NFS volume. In this case, you must separately back these files up and ensure they are mounted back to the same location on restore.

Snapshotting on AWS EBS

There are a few features of AWS EBS snapshots that make the backup process quite easy:

  1. To take a manual backup, choose the instance in the EC2 console and select Actions > Image > Create Image.

  2. Select the No reboot option if you want to avoid downtime. An AMI that can be readily launched as a new EC2 instance for restore purposes is created.

It is also possible to automate this process with the AWS API. Subsequent AMIs/snapshots are only as large as the difference (changed blocks) since the last snapshot, such that storage costs are not necessarily larger for more frequent snapshots, see亚马逊的EBS快照账单有关详细信息的文件。

Restoring From Backup


  1. Launch a fresh EC2 instance using the newly generated AMI from the previous steps

  2. Stop the app in the Management Console (at port 8800) if it is already running

  3. 确保端口8800处于管理控制台中配置的主机名反映了正确的地址。如果此主机名已更改,则需要在相应的Github OAuth应用程序设置中更改它或创建新的OAuth应用程序以测试恢复并登录应用程序。

  4. Update any references to the backed-up instance’s public and private IP addresses in/ etc / default /复制/ etc / default /复制-operator在Debian / Ubuntu或/ etc / sysconfig / *在Rhel / CentOS到新的IP地址。

  5. 从服务框的根目录,运行sudo rm -rf /opt/nomad。State is saved in the/opt/nomad从备份恢复安装时,可能会干扰构建的文件夹。该文件夹及其内容将在开始时由Nomad重新生成。

  6. Restart the app in the Management Console at port 8800.



If you want to clean up any abnormal build records in your database after a recovery, you can delete them by running the following commands on the Services machine replacing the example build URL with an actual URL from your CircleCI application: # Wait for console to load user=> (admin/delete-build "")