Disaster Recovery for the OpenEdge Database (Part 3)

Recovering from a disaster scenario that affects the OpenEdge Database. It is time to introduce two industry terms around Disaster Recovery being Recovery Point

Objective (RPO) and Recovery Time
Objective (RTO)

Recovery Point Objective (RPO): defines how much data is willing to be lost in a Disaster Recovery Scenario. Let’s assume there are daily backups at might night. If an event occurred that required restoring the database, all the data entered into the database between midnight and the time of the event would be lost. For some environments this may be acceptable, however for most production environments, losing a large amount of data would be unacceptable, especially if the data cannot be reproduced.

Recovery Time Objective (RTO): defines how long it takes to get the environment operational. How long can the system be down during a recovery event? During the time to recover all database operations will be halted. It the database is used as part of a manufacturing process, then the manufacturing itself will be halted until the system is once again operational. Many companies measure downtime in dollars lost by the hour.

Going forward we will use these terms to talk about recovery options.

So far, we have only talked about backups. Backups have many options and can be run online effectively. I have seen environments that run hourly backups, limiting the amount of data loss, however this approach does not scale as the database grows. Many times, it can take hours just to back up the database.

OpenEdge has a feature called After Imaging (AI) that is included in all licenses that include a database. AI can be used the fill the gaps between backups.

OpenEdge After Imaging (AI)

If you recall from Disaster Recovery Part 1, the database has 3 phases of crash recovery. Cluster Fix-up, Redo Phase, and Undo Phase. After Imaging (AI) tracks all the information needed for the Redo Phase of Crash Recovery, In fact the Before Image (BI) file(s) and the AI file(s) for the most part contain the exact same information. The major difference is the BI system reuses space inside the BI file(s), where the AI file(s) are a stream the BI information without any reuse of space.

AI reduces the RPO, you would restore the backup from midnight, then apply all the AI files that were generated between midnight and when the disaster occurred. This will bring the RPO to near zero, however the RTO increases. The RTO increases because instead of just restoring the database and moving on with the data loss, now additional time needs to be taken to apply all the AI files generated between the last backup and the time of the disaster.

Operational Ways to Reduce RTO.

1. Do not compress your backups. Many savy script writers will have a script that looks like this:

#!/bin/sh
DLC=/app/dlc; export DLC
PATH=$DLC/bin:$PATH; export PATH
probkup online /app/db/mfgprod /backups/mfgprod.bak
gzip /backups/mfgprod.bak

This makes sense on the surface, as it saves space with the compressed backup, since it is uncommon to have to use this backup file, however if you do need to use the backup to restore the database, time will be taken to uncompress the backup file, hence increasing the RTO for no real good reason. A better script would be:

#!/bin/sh
DLC=/app/dlc; export DLC
PATH=$DLC/bin:$PATH; export PATH 
gzip /backups/mfgprod.bak   # compress last nights backup
probkup online /app/db/mfgprod /backups/mfgprod.bak

2. Restore the database to a different location on the same machine. This serves 2 purposes. It verifies the backups are good because they are being verified as part of the restore process. It also reduces RTO provided you can point the application environment to the new location of the database. It the database is on the same filesystem, you can likely rename (mv) the database to the new location and use prostrct repair to fix up the database self-location and all the extents. Alternatively, you could use the “builddb” command for prostrct to rebuild the root of the database pointing to the restored extents.

3. Restore the database to a different location on the same machine and apply AI files are they get filled. This will provide you a completely intact database will little to no data-loss (RPO) and the same prosttct repair or builddb commands can bring the environment back up quickly.

All the above actions can be performed without any additional licensing as all the components are licensed and allowed on the same machine.

OOffsite Storage of Backup Files

The next thing to think about is if the production machine itself had a problem and all the files on the machine cannot be accessed. This means you would not be able to restore the database or apply any AI files. A disaster in the making.

Storing the files offsite is a mandatory process. While it significantly increases the RTO in the event the machine goes away, or gets compromised through a ransom attack or other, the alternative is not having any access to the environment and never recovering from a disaster.

You may be tempted to think, what about replacing the machine if the machine fails. Technically all you would need to do is copy the database backup to a different machine, restore it and apply the AI files as they fill up. In the event of a disaster, make a DNS change and presto, the environment is up with small RPO and small RTO. While technically feasible, this requires a Disaster Recovery (DR) license from OpenEdge. The DR license can be purchased, but it lists for 50% of the production list price and maintenance must be paid for this license. It may not feel like this licenses is not being “used”, but as soon as the database has been restored, the license has been used and if you do not have a DR license for that machine, you will be violating your license terms.

What you can do is to install OpenEdge on the machine, copy the backups and copy all the AI files. That keeps you within your license terms without a DR license. In the event of a disaster, the production license transfers to the new machine automatically, then the restore of the database and applying of AI files are allowed.

In our next segment we will talk about ways to reduce the RTO and RPO using a backup machine with a legal license.

OpenEdge System Health Check

Is your Progress OpenEdge database running optimally? Could there be an unknown issue that could potentially damage or bring down your production database?