Here we discuss the management of a server, such as on Amazon Web Services. It is possible to run it anywhere, such as in your workstation or server with the time-limited trial download, but we currently only support AWS for deployment. The time limits are usually several months. We suggest becoming somewhat familiar with the server’s simple directory structure. To launch a server on AWS, go to the AWS Marketplace. You are expected to be familiar with launching and configuring instances on AWS.
Server Launching
A launch script ‘launch-infinitydb-server.sh’ is provided for ‘unix-like’ shells, such as the Linux bash shell on the AWS instances. On windows, ‘cygwin’ may be used to get a ‘unix-like’ shell. The ‘installation directory’ is where the code distribution , such as infinitydb-6.0.5-deployment.zip was unzipped – typically /home/myuser/infdb. Look in infdb/scripts for various tools, such as launch-infinitydb-server.sh and certificate-tool.py. All of the data and per-instance configuration is in /data/infinitydb-home which is on a secondary volume, so the infdb installation directory on the root volume can be changed later, or indeed the entire root volume.
Note that the /data/infinitydb-home/key-store and /data/infinitydb-home/key-store-password directories contain the private SSL keys. Also be sure to check the port accessibility of the configured security groups to keep control until you launch publicly, i.e. until changing the default admin password as described below using temporary password suffixes.
Password Suffixes
When launched, the server will ask you to enter password suffixes like this:
General password suffix (s)hort (m)edium (l)ong (d)isable or any word: <mysuffix> Admin password suffix (s)hort (m)edium (l)ong (d)isable or any word: <anothersuffix>
The server can create and print short, medium, or long random hex suffixes for you, or can disable the admin or general passwords (so you cannot login), or can allow you to enter your own possibly zero-length suffix. In order to log into the server web interface or access via REST, the suffix must be added to the password entered by the user. The admin suffix affects only logging in as the admin user, and the general suffix applies to all other users, although the admin defaults to the general. This scheme makes it safe to launch a server on an open port for maintenance, such as for changing the admin password or the testUser password right after installation. Or, a server can be set up for use by a limited set of users by selectively giving out the suffixes to them, but this is not necessarily strong.
Note that if the suffix is disclosed or weak, a brute-force attack is as effective as without the suffix, and if a user password is disclosed such as the admin default password, a brute-force attack on the suffix becomes easier, so you may want to use the 32-byte long random number suffix you get when you enter ‘l’. Consider the password strength not to be determined by the sum of the lengths but by the longest, or for more safety, the shortest. Don’t assume the suffix will never be disclosed, and use it temporarily and change it often, especially if it is widely distributed. The regular persisted passwords are in the /data/infinitydb-home/data/default/meta-data.infdb file, and are hashed and encrypted (never stored literally), so they are safe once changed, and then you will not need the launch-time suffixes unless further offline work is needed.
In launch-infinitydb-server.sh You may want to uncomment and edit the #GENERAL_PASSWORD_SUFFIX=”–general-password-suffix=<suffix>” and #ADMIN_PASSWORD_SUFFIX=”–admin-password-suffix=<another-suffix> ” lines. If you do not, then on each launch you will be asked again for suffixes.
Don’t Lose the Admin Password
On the first launch, log into the admin account and change the default admin password to something safe. Use a long password suffix for this. Get the default admin password from mail:support@boilerbay.com. You may want to change the default permanently for your own uses.
Note that the admin user password is not possible to circumvent, and if it is lost, the /data/infinitydb-home/data/default/meta-data.infdb (see below about it) will become permanently inaccessible for administration purposes. You should keep backups of the meta-data.infdb file, and it would be reasonable to do that while backing up the database files manually, if you do it that way as described below. However, backup can also be done by transferring data securely between remote servers directly via the remote server feature, in which case server admin or root access is not required.
In the worst case, you can re-create the file by copying the meta-data.infdb in the freely distributed seed database and then manually reconstructing it as the admin user. There is no way to get the user passwords back even if meta-data.infdb could be unencrypted, because the passwords are stored hashed, not literally. If you can get another server running, you can see that the passwords are stored with optional plaintext ‘clues’ visible to the admin which can be used in various ways, such as either being identical to the actual password, or being clues for the users, or being codes that identify passwords in an offline paper list of passwords. (The standard FreeBSD bcrypt hash is currently used, with secure random salt and standard PBE throttling to about 10msec by repeated hashing to slow down brute-force attacks). The admin can set new or pre-existing passwords for each user manually. (see other limitations at the bottom.)
Server Shutdown
The database files have two levels of transactionality: global and optimistic. Both force all data to the file robustly on commit, even if the server fails in any way, losing only uncommitted data. However, the global transactionality allows an unlimited amount of data to accumulate before the commit or rollback, so it is a good idea to do the commit by hand before shutdown. This is a compromise, since users may actually not want a commit to happen. Shutdown rolls back all files instantly, and on restart, the files are recovered instantly back to the last commit. It is best to notify users of the shutdown in advance. To shutdown, simply do a kill -9 pid. It is completely safe other than losing uncommitted data. To manually commit, log in as admin, click on Access Databases, and select database files one by one, clicking on the commit button, and then click back-arrow. Optimistic locking is fully ACID-compliant, and it automatically commits quickly so there is no issue, although it is usually for smaller amounts of data.
Backup
These ways are suggested for backup:
- Manual copying of files or directories via the shell.
- Running multiple servers and transferring data via the web database browser via remote connections.
- AWS snapshotting
- Custom using REST or RemoteItemSpace
Complete Manual Backup
To make a quick, complete local backup, shutdown the server, copy /data/infinitydb-home/data/default to /data/infinitydb-home/data/mybackup or anywhere else, such as by using rsync. Then restart. The server locks the database files and meta-data.infdb during use. Restore by shutting down the server, and copying back to the same place. This captures the data and meta-data.infdb, but not things like the server’s SSL configuration in /data/infinitydb-home/key-store, which you may want to save very securely separately (see SSL configuration).
Per-File Manual Backup
You can also backup or transfer any individual database files, as they are each standalone, and not dependent on any particular server instance. Later you can, for example, shut down the server, copy a database file into /data/infinitydb-home/data/default/files and then use the admin console to create a database by name if necessary and give it access to that database file by using a database name like ‘my/db’ and a URL like file:mydatabasefilename.infdb. The file name can also be absolute, but that may be confusing. The admin console only changes the /data/infinitydb-home/data/default/meta-data.infdb file to remember the set of database names and their associated database files by file name. You can give a selected set of user roles access to this database. The file is compatible with InfinityDB Embedded, and when encryption of database files is later implemented in the server, an encrypted database will be compatible with InfinityDB Encrypted for standalone JVM use.
The meta-data.infdb File
Currently the admin can only change i.e. have an effect on the /data/infinitydb-home/data/default/meta-data.infdb, which is actually a standard InfinityDB Encrypted database file with its password in /data/infinitydb-home/meta-data-password/default/meta-data-password. The admin can only access or make changes from inside the web console, which means a server must be running. Thus the server admin is to some extent isolated from the root user, i.e. the system admin and vice versa.
There is currently no way (in version 6.0.5) to change or ‘rotate’ the metadata password in the file /data/infinitydb-home/meta-data-password/default/meta-data-password, and when that becomes possible in a future version, you will need to keep elsewhere the association between that password and the particular meta-data.infdb files that it corresponds to. If you change the password and securely erase all previous passwords, then of course all backups or disclosed copies of the meta-data.infdb become inaccessible (they are ‘crypto-shredded’).
The contents of the meta-data.infdb are:
- Database names and their associations with local database files by URL like file:mydatabasefile.infdb
- Database names and their associations with remote servers by URL like https://remote.com:37412/my/db. The port is the regular web and REST port plus 1.
- User names and hashed passwords, their role memberships and their owned databases
- Role names and users in those roles
- Permissions of certain roles to be able to read/write/query certain databases
- Further later, but probably backwards compatible
Backup to Another Server
To use online remote backup, set up another server, such as by launching another AWS instance, and connect to it remotely. Then go into the Database Browser by clicking on ‘Access Databases’ in the server’s home page, open the database that is to be backed up and use ‘transfer suffixes’ to copy any desired set of Item suffixes to or from another prefix on any connected server. After the copy is done, be sure to click ‘commit’ while having the target database open.
To connect to a remote server, use the admin console to create a new database name if necessary like ‘mydb/myremote’ and a URL like ‘https://mydomain.com:37412/my/db’. Note the port is the regular server port plus 1. Any port can be used for the server but we traditionally use 37411 (no connection to Chattanooga TN). The database name like ‘my/db’ is any pair of words separated by a slash, where a word is a letter followed by letters, digits, dot, dash, and underscore. All passwords on the remote server must be identical to those on the local server for any users that are to have access (until single-sign on is implemented.) When the remote database is accessed, it is opened, and the remote server actually does not ‘know’ that the remote link has been created. You can use a private database on the remote for backup if you want. (Data transfer is not as fast as, say rsync, however.) A convenient setup is to have a local server on a laptop or desktop system with a remote pointing at the shared server for backup or data sharing.
AWS Snapshot Backup
Amazon Web Services allows you to make a snapshot of any Elastic Block Store volume. You can use this on the /data volume, then restore it to the instance later. Shut down the server during these operations of course. You can also swap the root volume when new versions become available without obsoleting the /data EBS volume.
Note that the SSL private key and related information is copied as well because it is in /data/infinitydb-home/key-store/ and /data/infinitydb-home/key-store-password/. This makes it convenient for single-instance backup but not for wide distribution. Also note that simply deleting the .keystore.p12 file and/or its password or other files does not remove the underlying data from the file system securely, because data is not erased but only forgotten. This is a characteristic of typical file systems. To do a better job, copy zeroes on top of those files and then delete them, making sure that the zeroes are longer than the original files. That will not work on zfs or similar file systems however which use ‘copy on write’. Of course those directories may have already been backed up some other way such as by copying /data/infinitydb-home/, but we keep data separate, in /data/infinitydb-home/data/.
REST Backup
A further means for backup would use REST access to download data and transfer it under program control to another database, server, or other storage. RemoteItemSpace can also be used from Java. This would be a custom solution.
SSL for Https
The ‘infdb/scripts/certificate-tool.py’ python program is used to manage SSL certificates for https support in the /data/key-store directory and /data/key-store-password directory, providing these alternatives:
- self-signing: a default is provided, but you may want to create your own
- let’s encrypt: an easy protocol that instantly creates a 90-day certificate for free
- regular certificate: you have to pay an external certificate authority or be your own CA
By default, the server has a self-signed certificate, so you can access it immediately with encryption but server authentication will not occur, and you will have to tell your web browser to accept the risk. If you want to use http instead of https, you can edit the launch script to set that up, but of course that is risky unless you run behind a firewall or locally, so we strongly recommend SSL. REST access will benefit from SSL also. (We gently suggest that if you turn off server verification or use http in your REST access code, you will possibly have difficulty getting all of the versions of that code to have the verification enabled later.) The key-store-password directory can be given strong permissions, even though the key-store directory is looser. There may be multiple files in both directories that need saving.
The standard OpenSSL is used by certificate-tool.py, so it can be used instead, but we find the OpenSSL commands somewhat obtuse. The .keystore.p12 is in standard PKCS #12 format, with encryption (note the initial dot).
The CA certs trust store is in /home/ec2-user/java/lib/security/cacerts which currently links to /home/ec2-user/jdk-19.0.1/lib/security/cacerts. The currently used trust store path is printed on launch in case you move to a different java. The trust store can be moved to /data/infinitydb-home/trust-store and It can be encrypted as well, with password in /data/infinitydb-home/trust-store/trust-store-password. Trust stores need to be updated from time to time, and as Java versions are produced by ORACLE, the trust stores are updated as well.
Increasing Space
The initial /data Elastic Block Store volume is 100GB using xfs and no partitions. Database file sizes are displayed in the server’s Access Databases or Administrate Databases pages. Remote databases do not show their sizes. From the shell, you can also monitor space with df and du.
To grow a volume, see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/recognize-expanded-volume-linux.html. Shutdown the server. Modify the EBS volume to grow it, then use xfs_growfs on the volume.