Backup and Recovery

This chapter describes failover or replacement of the services machine. Refer to the Backup section below for information about possible backup strategies and procedures for implementing a regular backup image or snapshot of the services machine.

Disaster Recovery

在备用位置指定备用机器,具有相同的服务机器灾难恢复规范。在故障方案中使用备份快照具有热备用定期映像是最佳实践。

至少,提供具有主机名和位置的Circleci安装的系统管理员(即使是共同定位的)在其中安装替代服务器,其中包含使用服务机器配置的最新快照。要完成恢复,请使用安装过程,使用备份图像将图像从该过程中替换。

备份circleci数据

本文档介绍如何备份Circleci应用程序,以便您可以从附加到服务机器的意外或意外丢失的Circleci数据中恢复:

If you are running CircleCI in an HA configuration, you must use standard backup mechanisms for the external datastores. Contactsupport@www.drag240sx.com有关详细信息,请访问更多信息文档。

Backing up the Database

如果你有not配置Circleci for外部服务,备份Circleci数据的最佳做法是使用虚拟磁盘的VM快照作为服务机器的根卷。可以在没有停机时间的情况下执行备份,因为底层虚拟磁盘支持这样的操作,就像AWS EB一样。风险较小,这因文件系统和分布而变化,没有重启的快照可能有一些数据损坏,但这在实践中很少见。

“禁用快照”是指默认情况下已关闭的复制内置快照功能。

备份对象存储

Build artifacts, output, and caches are generally stored in object storage services like AWS S3. These services are considered highly redundant and are unlikely to require separate backup. An exception is if your instance is setup to store large objects locally on the Services machine, either directly on-disk or on an NFS volume. In this case, you must separately back these files up and ensure they are mounted back to the same location on restore.

Snapshotting on AWS EBS

There are a few features of AWS EBS snapshots that make the backup process quite easy:

  1. To take a manual backup, choose the instance in the EC2 console and select Actions > Image > Create Image.

  2. Select the No reboot option if you want to avoid downtime. An AMI that can be readily launched as a new EC2 instance for restore purposes is created.

It is also possible to automate this process with the AWS API. Subsequent AMIs/snapshots are only as large as the difference (changed blocks) since the last snapshot, such that storage costs are not necessarily larger for more frequent snapshots, see亚马逊的EBS快照账单有关详细信息的文件。

Restoring From Backup

在恢复测试备份或在生产中执行还原时,如果其公共或私有IP地址已更改,则可能需要在新启动的实例上进行几个更改:

  1. Launch a fresh EC2 instance using the newly generated AMI from the previous steps

  2. Stop the app in the Management Console (at port 8800) if it is already running

  3. 确保端口8800处于管理控制台中配置的主机名反映了正确的地址。如果此主机名已更改,则需要在相应的Github OAuth应用程序设置中更改它或创建新的OAuth应用程序以测试恢复并登录应用程序。

  4. Update any references to the backed-up instance’s public and private IP addresses in/ etc / default /复制/ etc / default /复制-operator在Debian / Ubuntu或/ etc / sysconfig / *在Rhel / CentOS到新的IP地址。

  5. 从服务框的根目录,运行sudo rm -rf /opt/nomad。State is saved in the/opt/nomad从备份恢复安装时,可能会干扰构建的文件夹。该文件夹及其内容将在开始时由Nomad重新生成。

  6. Restart the app in the Management Console at port 8800.

清理构建记录

虽然文件系统级数据完整性问题罕见和可预防,但在系统上运行时,可能会有一些数据异常,而在系统上运行。例如,仅在备份时间完成的中途的构建可能导致缺少其命令输出的后半部分,并且可能会永久地显示它在应用程序中运行状态。

If you want to clean up any abnormal build records in your database after a recovery, you can delete them by running the following commands on the Services machine replacing the example build URL with an actual URL from your CircleCI application:

Circleci.dev-console # Wait for console to load user=> (admin/delete-build "https://my-circleci-hostname.com/gh/my-org/my-project/1234")