Managing an InfinityDB Server

Here we discuss the management of a server on Amazon Web Services. It is also possible to run a server anywhere, such as in your workstation or server with the on-premises license, and you can get a trial for it in the store.

Launching in AWS

In the AWS Marketplace, you can easily launch a server. Then browse https://<serverip>:37411, to see the home page with an image of Boiler Bay. Click on Administrate Users. The browser will complain that the web site is not secure, because while https is enabled, there is no server authentication possible yet, and only encryption will happen. Just accept the supposedly ‘insecure’ login. You can fix this later from the instance shell login, by creating an SSL certificate and giving the server a domain name, or you can edit the launch script to choose http instead of https. Now log in as admin/<instanceid>. You can change the admin password in this page. At this point you may want to change the password for the testUser too, so other non-admin users can get to the demo/readonly and demo/writeable database, which contain interesting example data, such as some documentation and queries. Now browse back home and click on Access Databases. Select demo/writeable to get to the database browser, where you can experiment. At this point, you should familiarize yourself with the basics of the server and database browser access. You need ports 37411 and 37412, plus port 22 for SSH if needed eventually. Or, use the password suffix feature described below.

Backup

The root volume contains all code and gets swapped when new versions are released, while the second much larger volume is /data. Backup involves taking a snapshot of the /data volume, and on restore, creating a volume from it and attaching that to the root. The /data volume contains all databases. AWS is very convenient for managing backup snapshots and creating volumes or new instances from them later for restore. Note that the /data/infinitydb-home/key-store and /data/infinitydb-home/key-store-password directories contain the private SSL keys. Almost all system configuration is in /data/infinitydb-home/data/default/meta-data.infdb and is encrypted, including database names and their corresponding file names, user names, password hashes (not passwords), roles, grants, and permissions. Therefore, snapshotting makes it convenient for single-instance backup.

Directory Copy Backup

Alternatively, the instance may be accessed from the shell, and the /data/infinitydb-home/data/default directory can just be copied, such as to /data/infinitydb-home/data/bkup. Individual database files under /data/infinitydb-home/data/default/files can also be copied, and each database is a single file. For this, however, the server must be shut down, because database files are file locked by the running server. You might want to use ‘rsync’ to copy to a remote computer efficiently. Note that the meta-data.infdb contains all configuration and it lives at /data/infinitydb-home/data/default/ so it will come along for the ride.

Remote Backup With the ‘ItemPacket’ Protocol

Another backup technique can use the ‘ItemPacket’ protocol for user manual transfer of portions or entire databases between servers: this uses the ‘remote database’ feature that can be configured in the admin console as described below. The admin creates a database by providing the URL of the remote server, such as https://mydomain:37412/<database> where <database> is the two-part database name on the remote server, like demo/writeable. Backup is performed manually by users using the ‘Transfer Suffixes’ page of the database browser. as described below The remote server does not need configuration except that the remote must have the same password for the users who use it as on the local server (single-sign-on is in development).

REST Backup

A further means for backup would use REST access to download data and transfer it under program control to another database server, or other storage. This would be a custom solution, such as by using a Python client. One method would be for the Python client to send REST requests with new data to two or more servers. This also allows a simple type of fail-over, especially useful if servers are read-mostly. You could lock out further updates during the degraded periods.

Commit all Databases

Sometimes the entire system must be ‘quiesced’. You may want to do this before a snapshot backup, because globally uncommitted data will not be put into the snapshot. Go to Access Databases, and browse the databases that need to be committed, and click the commit button. Also globally uncommitted data will be rolled back if the server process is manually killed. Fine-grain i.e. ‘Optimistic’ Transactions do not take long to execute so they are normally already committed. Killing the server never leaves the databases in a degraded state, i.e. needing to have logs re-applied because logs are not used, and restart is always instant. The database files have two levels of transactionality: global and optimistic. Both force all data to the database file robustly and durably on commit, even if the server fails in any way, losing only uncommitted data. However, the global transactionality allows an unlimited amount of data to accumulate before the commit or rollback, so it is a good idea to do the commit before shutdown, or stopping on AWS EC2.

Stop and Restart a Server

For instance shell backup by simple copying or other purposes, the server must be stopped. In the instance shell, do sudo ps aux | grep java and sudo kill -9 <serverpid>. The server command line includes ‘java’. To relaunch, go to /home/ec2-user/infdb/scripts and do ./launch-infinitydb-server-on-ec2.sh. Or just reboot with ‘sudo reboot now’. Or, just stop and restart at the AWS EC2 console.

The Launch Script

For some tasks, shell access to the instance is needed. A launch script ‘launch-infinitydb-server.sh’ is provided for ‘unix-like’ shells, such as on standalone Linux, or ‘cygwin’ on windows, in case of an on-premises licensed installation. On AWS, the script is ‘launch-infinitydb-server-on-ec2.sh’, and it calls the generic one.

Password Suffixes

The password suffix feature allows the server to be temporarily launched with controlled secure access. When launched on the command line, the server will ask you to enter password suffixes like this:

General password suffix (s)hort (m)edium (l)ong (d)isable or any word: <mysuffix>
Admin password suffix (s)hort (m)edium (l)ong (d)isable or any word: <anothersuffix>

The server can create and print short, medium, or long random hex suffixes for you or can allow you to enter your own possibly zero-length suffix. In order to log into the server web interface or access via REST, the suffix must be added to the password entered by the user. The admin suffix affects only logging in as the admin user, and the general suffix applies to all other users. This scheme makes it safe to launch a server on an open port for maintenance, such as for changing a password and so on. Or, a server can be set up for use by a limited set of users by selectively giving out the suffixes to them.

Note that if the suffix is disclosed or weak, a brute-force attack is as effective as without the suffix, and if a user password is disclosed such as the admin default password, a brute-force attack on the suffix becomes easier, so you may want to use the 32-byte long random number suffix you get when you enter ‘l’. Consider the password strength not to be determined by the sum of the lengths but by the longest, or for more safety, the shortest. Don’t assume the suffix will never be disclosed, and use it temporarily and change it often, especially if it is widely distributed. The regular persisted passwords are in the /data/infinitydb-home/data/default/meta-data.infdb file, and are hashed and encrypted (never stored literally), so they are safe once changed, and then you will not need the launch-time suffixes unless further offline work is needed. Note that a data backup including /data/infinitydb-home/data/default/metadata.infdb will include the password hashes!

You may want to edit the section with PROMPT_FOR_PASSWORD_SUFFIXES=false or set GENERAL_PASSWORD_SUFFIX=”–general-password-suffix=<general-suffix>” and ADMIN_PASSWORD_SUFFIX=”–admin-password-suffix=<admin-suffix> ” lines to turn off the prompting for manual launches.

Don’t Lose the Admin Password

Note that the admin user password is not possible to circumvent, and if it is lost, the /data/infinitydb-home/data/default/meta-data.infdb will become permanently inaccessible for administration purposes. You should keep backups of the meta-data.infdb file.

In the worst case, you can re-create the file by creating a new /data volume from the original snapshot or a backup, copying the meta-data.infdb, and then manually reconstructing it as the admin user. In the original snapshot case, the admin password starts as the instanceid. There is no way to get the user passwords back because the passwords are stored hashed, not literally. If you can get another server running, you can see that the passwords may be stored with optional plaintext ‘clues’ visible to the admin which can be used in various ways, such as either being identical to the actual password, or being clues for the users, or being codes that identify passwords in an offline paper list of passwords. (The standard FreeBSD bcrypt hash is currently used, with secure random salt and standard PBE throttling to about 10msec by repeated hashing to slow down brute-force attacks). The admin can set new or recovered passwords for each user manually. Only the admin can set passwords (until single-sign-on is implemented). The names of the databases can either come from the backup metadata.infdb or can be guessed from the file names of the database files. You can define a database by giving its file name without deleting the file, and likewise you can ‘forget’ a file.

The meta-data.infdb File

Currently the admin can only change i.e. have an effect on the /data/infinitydb-home/data/default/meta-data.infdb, which is actually a standard InfinityDB Encrypted database file with its password in /data/infinitydb-home/meta-data-password/default/meta-data-password. The admin can only access or make changes from inside the web console, which means a server must be running. Thus the server admin is isolated from the root user, i.e. the system administrator and vice versa.

There is currently no way (in version 6.0.x) to change or ‘rotate’ the metadata password in the file /data/infinitydb-home/meta-data-password/default/meta-data-password, and when that becomes possible in a future version, you will need to keep the association between that password and the particular meta-data.infdb files that it corresponds to, such as in files or subdirectories under /data/infinitydb-home/meta-data-password. This will all become automatic, but until then, just keep both the meta-data.infdb and its password safe. If in the future you change the password and securely erase all copies of the previous password, then of course all backups or disclosed copies of the meta-data.infdb become inaccessible (they are ‘crypto-shredded’). You should keep the permissions on at least the password tight.

The contents of the meta-data.infdb are:

  • Database names and their associations with local database files by URL like file:mydatabasefile.infdb.
  • Database names and their associations with remote servers by URL like https://remote.com:37412/my/db. The port defaults to the regular web and REST port plus 1.
  • User names and hashed passwords, their role memberships and their owned databases
  • Role names and users in those roles
  • Permissions of certain roles to be able to read/write/query certain databases
  • Further later

Remote Servers

To use online remote backup or to build a distributed system, set up another server, such as by launching another AWS instance, and connect to it remotely. Then go into the Database Browser by clicking on ‘Access Databases’ in the server’s home page. You can use ‘transfer suffixes’ to copy, merge or delete any desired set of Item suffixes to or from any pair of prefixes on any local or remote databases. After the copy is done, you may want to click ‘commit’ while having the target database open. Client REST access can use the remote as if it is local.

Connect to a Remote

To connect to a remote server, use the admin console to create a new database name if necessary like ‘mydb/myremote’ and a URL like ‘https://mydomain.com:37412/my/db’. Note the port is the regular server port plus 1. Any port can be used for the server but we traditionally use 37411 . The database name like ‘my/db’ is any pair of words separated by a slash, where a word is a letter followed by letters, digits, dot, dash, and underscore. All passwords on the remote server must be identical to those on the local server for any users that are to have access (until single-sign on is implemented.) (Data transfer is not as fast as, say rsync, however at the file level (but rsync would require shutting down the server, since all InfinityDB Embedded database files have mandatory locks.) A convenient setup is to have a local server on a laptop or desktop system with a remote pointing at the shared server for backup or data sharing. For that, you will need an ‘on-premises’ license, not just an AWS license. The local and remote database names do not need to match. Remote access is fast for writing or reading in bulk with Items having common prefixes, but for reading, each independent operation will involve a TCP turnaround at about 10K/s intra-host or 10/s externally. Queries always execute on the server where they reside but their patterns and results can be redirected when executed in the database browser..

SSL for Https

The ‘/home/ec2-user/infdb/scripts/certificate-tool.py’ python program is used to manage SSL certificates for https support in the /data/infinitydb-home/key-store directory and /data/infinitydb-home/key-store-password directory, providing these alternatives:

  • self-signing: a default is provided, but you may want to create your own
  • let’s encrypt: an easy protocol that instantly creates a 90-day certificate for free
  • regular certificate: you have to pay an external certificate authority or be your own CA

By default, the server has a self-signed certificate, so you can access it immediately with encryption, but server authentication will not occur, and you will have to tell your web browser to accept the risk. If you want to use http instead of https, you can edit the launch script to set that up, but of course that is risky unless you run behind a firewall or locally, so we recommend SSL. REST access will benefit from SSL also. (We gently suggest that if you turn off server verification or use http in your REST access code, you will possibly have difficulty getting all of the versions of that code to have the verification enabled later.) The key-store-password directory can be given strong permissions, even though the key-store directory is looser, and these directories are owned by root. Note that these directories are separate from the data under /data/infinitydb-home/data.

The standard OpenSSL is used by certificate-tool.py, so it can be used instead, but we find the OpenSSL commands somewhat obtuse. The .keystore.p12 is in standard PKCS #12 format, with encryption (note the initial dot).

The CA certs trust store is in /home/ec2-user/java/lib/security/cacerts which currently links to /home/ec2-user/jdk-19.0.1/lib/security/cacerts. The currently used trust store path is printed on launch in case you move to a different java. Java 8 and above is compatible. There is a way move the trust store to /data/infinitydb-home/trust-store by editing /home//ec2-user/web/WEB_INF/web.xml and It can be encrypted as well, with password in /data/infinitydb-home/trust-store/trust-store-password. Trust stores need to be updated from time to time, and as Java versions are produced by ORACLE, the trust stores are updated as well. The cacerts is only used for authenticating remote servers

.Also note that when it is time to change the .keystore.p12 file and/or its password or other files for SSL use, simply deleting does not remove the underlying data from the file system securely, because data is not erased but only forgotten. This is a characteristic of typical file systems. To do a highly secure job, copy zeroes on top of those files and then delete them, making sure that the zeroes are longer than the original files. That will not work on zfs or similar file systems however which use ‘copy on write’, but we use xfs. Of course those directories may have already been backed up some other way such as by copying /data/infinitydb-home/, but we keep data separate, in /data/infinitydb-home/data/default/.

Increasing Space

The initial /data Elastic Block Store volume is 100GB using xfs and no partitions. Database file sizes are displayed in the server’s Access Databases or Administrate Databases pages. Remote databases do not show their sizes. From the instance shell, you can also monitor space with df and du. We have seen that storage on AWS is about $8/month/100GB for a 300 IOPS SSD volume or $17/month/100GB for a 3000 IOPS SSD volume, but these numbers are approximate. Any size can be used and the cost is only the AWS resource fee. Volumes cannot be shrunk. You can save money with snapshotting.

To grow a volume, see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/recognize-expanded-volume-linux.html. Commit necessary databases as described above, and shutdown the server. Modify the EBS volume using the EC2 console to grow it, then use xfs_growfs on the volume.

Permissions

The admin can control who has access to the databases via the Permissions mechanism, at the granularity of role, database, and permission. Each permission granted is a triple of (role, database, permission). With REST access, a failed permission results in an UNAUTHORIZED (401) error. Any combination of read, write, and query are OK. The possible permissions are:

  • read – allows a database to be browsed or retrieved from but not changed
  • write – allows a database to be changed, but not necessarily read. So a ‘write-only’ mode is possible
  • query – depending on further restrictions, queries are allowed. Multiple query permissions may be combined. A query-only setup can be very useful.

Query Permissions

The query permission has optional further syntax for very fine control. For example, below are some query permissions. Each query’s name has an ‘interface’ part and a ‘method’ part, each of which are strings, and occur after the Query class in the Items of the query definition like Query "myinterface" "my method" query pattern....

  • query – any query may be executed
  • query:interface=com.myinterface – this allows queries stored with interface name “com.myinterface” to be executed
  • query:setter – allows any query having the Option ‘setter’ true to be executed. A Query definition must have an Item like Query "myinterface" "my method" query Option setter true
  • query:getter – like setter, but only for queries with Option getter true
  • query:setter,getter – either or both of setter and getter Options must be present on the query
  • query:setter,interface=com.myinterface – the query must have Option setter true, and must be stored under Query "com.myinterface"
  • query:setter,prefix=com.myinterface – like above but the query’s interface name must start with “com.myinterface”. If the query’s interface name is longer than that, there is a dot separator like “com.myinterface.mysubinterface”.

If the colon is replaced by a ? then if a previous version of the server is used that does not ‘understand’ the syntax, then that permission is ignored. More similar syntax is likely to be added in the future.

Interface names are like reversed internet domain names (this idea is in Java), so they are a sequence of lower-case letters, digits, dot, and dash, not starting with a digit. Other than that, any interface names are allowed. A query author will usually use an actual interface name that does correspond to the reverse of an internet domain name they control, so interface names can generally be kept globally unique. The method name is not limited but must be a single string. A typical URL to execute a query might be https://com..mydomain:37411/infinitydb/data/my/db/"com.myinterface"/"my method"?action=execute-query. The quotes and other characters may need to be URL-quoted like %22 in some contexts.

The above are only examples – here is the grammar for completeness:

query_permission := 'query' (':' perm_list | '?' perm_list)?
perm_list := perm (',' perm)*
perm := 'setter' | 'getter' | 'interface=' interface_name | 'prefix=' interface_name
interface_name := interface_char (interface_char | digit)*
interface_char = letter | dot | dash

Adding Web Assets

By putting additional HTML, CSS and JavaScript in the /data/infinltydb-home/web/default directory, you can customize the system to some degree for your own purposes. We recommend leaving the initial home page. The InfinityDB.css controls all of the styles. Changing styles between servers might make it more obvious which server you are looking at. The servlets cannot be customized, and all of them are under <server_url>/infinitydb/data and <server_url>/infinitydb/admin, with more possibly in the future. The format of /data/infinitydb-home/web/default/WEB_INF/web.xml is standard and will be familiar to servlet users but it cannot be edited in v6.x. We do not recommend changing other files there. Future server versions may change or add files by putting them in /home/ec2-user/infdb,/web, such as help files, and they will have to be copied manually to /data/infinitydb-home/web/default.