To see InfinityDB in action, you can experiment with the demo server at infinitydb.com, or you can launch a new server from an AWS subscription as follows:
Basic Launching on AWS
- Launch an InfinityDB Server instance, such as through the AWS marketplace subscriptions page
- Browse to the public DNS or IP. (http access on port 80 is automatically redirected to https on port 443)
- Accept the warning that ‘access is insecure’ and proceed to site. (This means encryption is active, but not server identity verification, which may be fixed later by installing an X509 certificate as described in Administration below.)
- You see the home page. Now click on ‘Access Databases’, ‘Administrate Users’ or ‘Administrate Databases’.
- Log into the server using user name ‘admin’ and the instance id as password.
- Proceed to the Databases Page or the Administration section below
To subscribe, go to https://aws.amazon.com/marketplace and enter InfinityDB in the search box to get to the listing, then click on infinitydb link to get to the actual product page. From there, click on ‘continue to subscribe’ and accept terms. You go into ‘pending’ state for a while (keep refreshing the page in the browser), then you click on ‘configuration’, where you can specify three options, but the best is to leave them default. Use the latest version, but you can change the region if necessary. Click on ‘continue to launch’.
In ‘Launch this Software’, choose one of the two methods: Launch from EC2 Console (more control), or Launch from Website (simplest):
For Launch from Website:
- Action: Launch from Website
- EC2 instance type: t2.medium or your desired more powerful instance, if you want more cores speed. The server can execute any number of queries or REST requests concurrently, while taking advantage of all the cores. Also more cores will make browser access snappier, especially for loading images and blobs.
- VPC settings: default or your preference
- Subnet settings: default or your preference
- Security settings: ensure that ports 22 (ssh), 80 (http) and 443 (https) are open. Our recommended settings are 22, 80, 443, 444, 37411, and 37412. The 3741x are actually just alternatives to 44x, which the root user can select and are by default unused, so you can close them. The first port of a pair – 443 or 37411 – is the web server’s https and the REST access, while the second is a ‘RemoteItemSpaceServer’ which allows databases that appear to be local to be in fact remote. You can do this manually by clicking on those selections, or click ‘create new from seller recommendations’, give it a name and description. You may want to restrict IP access to yourself for high security. Then click save. Port 80 is just a redirector from http to https on port 443, if 443 is used.
- Key pair settings: select a key pair for SSH root access that you have available or create one here. The shell user is ec2-user.
Now in the Web-based method, click Launch. The instance will begin running, but you need to go to the EC2 console anyway to get some information, so for convenience, you can just click the ‘Go to EC2 console’ link (very small near the top).
In the EC2 console, you will be able to start, stop, and terminate the instance, and here you can find the instanceid like i-0f70ac84b7a0c165e, which will be the admin password until you change it. Also you need the public IPv4 or public domain name to access the web interface. Each time you stop and restart the instance, this IP address will change, unless you configure an ‘elastic IP’ which is inexpensive and will provide a fixed public IP address. Once you have an elastic IP you can register a DNS name and point it there, such as by using AWS Route53. If you have multiple instances running, you can give each a name in the control panel, or you can deduce which one it is by the launch time.
The Web UI
Here we introduce web-based access of the InfinityDB Server. Below are some examples of the kinds of views available in the graphical access for the ‘Database Browser’. We will show how to get these views, edit data, and so on, plus how to administer the server. Some good optional background is the ItemSpace Data Model which resembles a document data model like MongoDB but with many extensions.
Browse to the server’s public domain name or IP, such as the demo server at infinitydb.com. At this point you will see the home page, which is an image of boiler bay and links to Access Databases, Administrate Users, and Administrate Databases. (The home page can be replaced with a custom one by the root user.) Click on one of these links and login, then skip down to Databases Page or Administration below.
Full Details for Logging in
There are two parts of the web interface: one for Database Access and one for Administration. Click on Access Databases to go to the Databases page or the other links for administration, which is explained near the bottom. You do not need to be logged in in order to view the home page. If you successfully logged in, skip down to Databases Page.
The InfinityDB server uses ‘Http Basic Authentication’ which means that the browser stores the current user name and password in a special secret local store associated with each particular URL. The InfinityDB server does not keep track of logins in sessions, but instead only checks the user name and password on every page request: the browser includes them securely each time. (This method makes client software simpler.)
- If you are already logged in, no action is required;
- To login: If you are not already logged in, you will be prompted for a user name and password in a sign-in dialog box like the one below that is browser-dependent;
- To logout: In order to logout you need to visit a page other than the home page, click logout, then click it again, and then click cancel in the sign-in dialog. The text ‘logged in as user’ will become ‘logged out’;
- To login as another user: click logout, then click it again, but then don’t cancel; instead enter the new user name and password and click sign in.
Whenever the sign-in dialog below is shown, you may logout by clicking cancel or login by entering a user name and password. If the user name and password entered are not recognized, the sign-in dialog may just be redisplayed. You may need to click the browser’s reload button at some point (at least in Firefox) to get to the sign-in dialog. The currently logged-in user is displayed in the upper left of each page except the home page.
Here is the sign-in dialog for Chrome. The URL being logged into is displayed in it:
Use either the admin user or any other valid one you have available. Sometimes user ‘testUser’ password ‘db’ has been made available, such as at infinitydb.com, and it usually has access to demonstration databases. When a server is initially launched on AWS EC2, you can use the admin user with the AWS instance id as the password. You can continue to be admin as long as you want, and you can change your password and add and configure users, passwords, roles, permissions and databases at any time as described below under Administrate Users and Administrate Databases.
The server can be launched from the shell by the root user in a mode that appends an arbitrary suffix to the passwords, and if so you will need to append that suffix to your regular password. The admin can have its own optional suffix, and the other users can have another optional shared suffix. Usually this suffix mode is temporary, and a normal transition of the EC2-based server from stopped state back to running state will reset the suffixes to empty (unless the root user has edited the launch script.)
Using a Server Without a Valid Certificate
If you are accessing a server that does not already have a valid X509 certificate for proper TLS authentication, then you may need a few more clicks. For example, a newly launched AWS EC2 instance does not have a valid certificate (only a ‘self-signed’ one initially), so if this is your first time using such a server, you may get a message like (in Chrome) ‘Your Connection is Not Private’. Just click on ‘Advanced’ and then click on ‘Proceed to <url>’. Accessing the home page may not immediately show this and it may be different in different browsers. The message will usually not appear again later for that server. The URL line will continue to show ‘Not secure’, but this relates only to server authentication, and encryption is still active.
Once logged in if you clicked on Access Databases in the home page, you see the Databases page. Click on demo/readonly or demo/writeable to go to the Database Browser page. You can create new local or remote databases later.
Database Browser Page
As admin, you can edit either the demo/readonly or the demo/writeable database, but no other user can see or edit them initially. Admin can enable further access later. Traditionally, one gives testUser a password such as ‘db’, puts testUser in the guest role, and gives guest READ WRITE QUERY permission on demo/writeable, and READ QUERY permission on demo/readonly .
The top blue line is the same as long as you are accessing data (not doing administration):
- User ‘admin’ how you are logged in
- Home go to the home page with the picture of boiler bay
- Logout allow you to logout or log back in. Click twice to reach the login dialog, then click cancel or sign in
- Databases go back to the Databases page to choose a database to browse
- Database demo/readonly which database you are using now
- Browser go to the database browser page to see and edit data including queries
- Transfer go to the Transfer Suffixes page to move data around
- Query go to the query page to execute queries
- Recents a selector you can use to return to recently visited prefixes
Database Browser Page Widgets
Here are the lower components of the display, which are specific to browsing and editing data.
- Current Prefix this entire horizontal line is a ‘prefix navigator’. Similar navigators occur elsewhere. It is similar to a current working directory in the shell but follows certain strict formatting rules. The text is in ‘Token’ format. Farther down we explain navigation.
- Load update the screen, leaving text editing mode without saving. See changes from other users or REST clients.
- Insert or Ctrl-Shift-Ins put the Item in the current prefix into the database
- Delete or Ctrl-Shift-Del remove the Item in the current prefix from the database
- Delete Suffixes remove all Items in the database that start with the current prefix. Be careful. Use Rollback if needed.
- Commit make the current changes permanent. This is the ‘global’ transaction, with unlimited transaction size.
- Rollback undo all changes back to the previous commit
- Edit in all but the table view, switch to editing the suffixes in a text area
- Save in all but the table view, finish the editing of the suffixes in the text area. Use load to cancel.
- Files (with an up arrow) select one or multiple files in your local file system to upload. They are stored under the current prefix plus the file name. We suggest setting the current prefix so that it has no existing suffixes. Each file causes a commit.
- Blob (with an up arrow) select one file in your local file system to upload. This is stored directly under the current prefix, with a commit. Blobs are rendered where possible as images, texts, and so on.
- Transactional These ‘optimistic’ transactions are best for relatively small amounts of changes because they use temporary memory and are ‘fine-grained’. The ‘global’ commit feature is always available though, with unlimited data.
- Enter key – go to the current prefix after it has been edited by hand.
The views may be selected in the ‘Show’ dropdown:
The view modes are:
- Show Table a graphical display that looks somewhat ‘tabular’ but actually is very versatile and can represent any possible structure of Items.
- Show i Code Java Style the ‘i-code’ format is for representing a set of Items as text in a very convenient way. This is very useful and is used in other places, such as the Query page. You do not need to know Java.
- Show i Code Python Style like the above, slightly more familiar to Python programmers. You do not need to know Python.
- Show List of Strings the suffixes are Index components followed by String components. A ‘text’. These show up as lists of strings in i -code or JSON, but in table view you see incrementing numbers inside square brackets like  “my text”
- Show List of Comma Separated Values the suffixes are index components followed by components formatted as CSV text. There is more on CSV in PatternQuery Examples. In i code and JSON, these are lists of CSV fields.
- Show Set of Comma Separated Values the suffixes are formatted one Item per line, one component per field as CSV text.
- Show Plain Text Blob a raw sequence of characters of type text/plain. They can be edited in a text area.
- Show Items the underlying most basic form of the data in ‘tokenized’ form, like the current prefix.
The ‘EntityClass’ list at the left is visible when the current prefix is empty. These are the EntityClasses that are the components at the beginning of Items in the database. If you like, you can consider each of these a ‘table’ but that misses the generality of the ItemSpace data model which you should go look at now. Go to Show table view and click on ‘Pictures’ and you see:
Above is the graphical or ‘table’ view of the Pictures EntityClass. It contains some data that looks relatively tabular, but that is just determined by simple rules that control the formatting no matter what data is being viewed. Any set of Items can be viewed in this graphical way, no matter now nested. The images are ‘blobs’ of type ‘image/png’ and were just uploaded as files. Any blob can be stored and represented in the database, like plain texts, pdfs, and so on.
It is possible in the tabular view to edit single cells. Just click on the cell, wait for it to jump up to the top and become surrounded by a dotted line, and then click on it again. The text goes in an editable text area, and you submit the changes by clicking on the check mark, or add a new value (yes there can be multiple values and much more) with the plus, or delete the current value or one of the multiple values with the minus. The text box is not allowed to have EntityClasses (blue) or Attributes (green) so for those, you use the current prefix and the buttons below that. The text must conform to the token format, so strings need double quotes, booleans are just true or false, numbers are 5, 5.0, or 5.0f and so on as described in The ItemSpace Data Model. This is not i-code, where strings by default use single quotes but can also use double quotes. Clicking the load button or clicking in empty space or elsewhere in the table will cancel this mode with no changes.
Click on the ‘X’ button of the current prefix navigator to clear it and go back, so you can click on another EntityClass to explore. Look at Samples or Documentation for example, and try Show Table and Show i-code. To see the actual Items underlying any view, use Show Items.
Current Prefix Navigator
The Current Prefix navigator looks like:
A navigator has these widgets:
- Left or PgUp key shorten the prefix by one component
- Up or Up key move up one Item only changing the rightmost component
- Down or Down key move down one Item changing only the rightmost component
- Right or PgDown key add one component to the prefix by finding the nearest greater Item
- Double-down move to the ‘end’, changing only the last component
- Double right shows a drop-down list of the components one level deeper. You can drill down by selecting one of them to have it appended to the current prefix
- X clears the current prefix
- The Text Area contains the current prefix in ‘Token’ form. Each possible component has one of 12 data types, each having a unique tokenized form.
- Drop-down at right allows choosing from the Items at the current level
Any white, blue, or green box in the table view can be clicked on to put into the navigator the prefix that determines it, and clicking anything in the i code view drills down and extends the navigator.
Here is how the Double-right arrow button works. We are browsing the Query EntityClass, and under that the “examples” string, which is part of the name of a query. The drop down shows the second part of a query name. We can drill down very quickly this way, and it works for any data.
Now with the Pictures EntityClass displayed in table view, click on “pic0” to put Pictures “pic0′ in the current prefix or type it in and press Enter, and switch to ‘Show i Code Java Style’ to see another view. The navigator should show Pictures “pic0”.
Here is the i code view. It is actually a kind of document (like JSON) but more convenient, with fewer nested braces, colons, and quotes. Also i code can contain Blobs, and in fact so can InfinityDB’s extended JSON and extended CSV but there they are just text in a special format. You can edit this as raw text without the syntax coloring and with blobs becoming text by clicking on the Edit button, and you can save it with the Save button or cancel with the Load button. However, the blob part is in a special format that can live in a text file, and you would usually not want to edit that. In order to actually save, you should be browsing the demo/writeable database (just on principle). Click on the image to see it full-screen, where you can save it as a file by using the browser’s context menu. In i-code mode you can drill down by clicking on anything else to move its prefix into the current prefix text area. When editing in a text mode, you should frequently save in order to catch formatting errors you have created and to clean up the formatting.
The Query Page
In order to test and execute PatternQueries, navigate in the database browser page to Query “examples” “Display image” while staying in Show i Code mode, and you see:
Now click on Query in the blue bar at top to see the query page:
Next, type in the i code
name 'pic3' in the request content box, then Save. The quotes are single, not double. It looks like this:
Now click on Execute, to see:
The above ‘output’ is the response content from the query. It shows some deeper structure, with two values for the name inside braces. Attributes are in green, as in the table view.
It is possible for a REST access from client software to execute this query – if given permission by the admin – to retrieve an image by name. The REST query can retrieve all of it or just the image blob. If all of it is retrieved in execute-query mode, it is in JSON form – even the blob, but in execute-get-blob-query mode, the raw bytes of the blob can be retrieved quickly. The request and response content may be very complex, and tens of MB are common, such as for an image. In execute-put-blob-query the raw bytes can be provided in the request content. To understand this query, look at PatternQuery examples or the PatternQuery reference.
The Pattern Prefix and Result Prefix navigators allow the query to be ‘redirected’ so that it matches against a desired different source database and prefix and its results can go into yet another destination database and prefix. This is good for testing and it is more flexible in general, but it cannot be done by REST client accesses, which are allowed to execute queries on only the database containing the query.
Transfer Suffixes Page
In order to move data around in flexible ways, the Transfer Suffixes page can be used. It is very simple: there are two navigators for selecting a source and destination prefix and buttons for performing an operation. Note that the navigators have an additional drop-down to select a database. This way data can be transferred and operated on anywhere, even between pairs of local and remote servers. A database name can be declared by admin to be a local view of a remote database on another accessible server, and from the point of view of users and client software, all the databases appear local.
For more on structuring your own data, see the main server page, which shows some data being edited. See i-code Language and Item Tokens. For how to restructure, input/output and search see PatternQuery Examples, and PatternQuery Reference. PatternQueries are very concise, rich, yet readable and work almost like ‘mini-applications’.
For administration, start at the home page and click on Administrate Users or Administrate Databases, logging in if necessary as admin. If you are not currently admin, you may have to visit a data access page and click on logout twice, then enter admin and the admin password or cancel to logout. Any way you can get to the login dialog box will work. (The HTTP Basic Authentication system is inherently awkward this way, as there is no way in current browsers to logout or re-login without being prompted for a new login by the server.) If this server is an initial launch on AWS EC2, then the password is the instanceid, and it and the server public IP and domain name are on the instance information page. Make sure the port number is as set by the instance’s root user, if it has been changed.
The Administrate Users page looks like this:
To edit something, type in the values you want to add or remove in the appropriate text box and then choose a button below to execute it immediately. (There is no global save or cancel button.) To enable changes, check enable changes and click on Set Enable Changes, so the buttons become active.
The password clue is text added when the password is set in order to make password recovery easier. You can leave it blank for security, or put the actual password there, or put a password number or identifier there that you later look up in a paper book that associates the numbers with the actual passwords. You cannot get the password back once set.
Each user is a member of zero or more roles. Each role may have permissions on zero or more databases. Each permission is a combination of a role, a permission, and a database. For info on the specifics of fine-grained query execution permissions, see Managing an InfinityDB Server near the bottom. The permission format will be improving backwards compatibly over time. If fine-grain query permissions are not necessary, then just use any combination of READ, WRITE, or QUERY. You can enter multiple permissions in the permission box to add or remove them as a group.
Databases are identified by a combination of two words with a slash separator. The words start with a letter, then have letters, numbers, dots, dashes, and underscores.
The Administrate Databases page looks like:
Use Database Files
To create or add an existing database file, enter a database name and a URL plus owner user. The URL can be like file:filename.infdb. The ‘infdb’ will be added if not already there. Use file names that are obvious given the database name. The file will live in /data/infinitydb-home/data/default/files/filename on AWS EC2, where the /data is on the second ‘data’ volume. On on-premises systems, infinitydb-home may be anywhere, but the files are in the same relative place under files. (One convenient place for backups would be infinitydb-home/data/backupname/files, which is up to the root user.) You can ‘forget’ a file or re-establish the reference to it without affecting it.
Refer to Remote Database
To create a remote connection, use a URL like https://remotehost:444/infinitydb/data/blue/green3 or else https://remotehost:34712/infinitydb/data/blue/green3 according to the configuration of the remote. Port 444 is the default on installation. Be sure to include the ‘https://’, not blank or ‘http://’, however if the remote root user has switched the remote to http mode then use ‘http://’ (http mode is discouraged). The port is the regular server port plus 1. To enable remote, this port must be open at the remote server if it is an AWS EC2 instance, which is the default. The database name for this one is blue/green3. Database names are a letter followed by letters, digits, dots, dashes and underscores. There is no save/cancel button, so be careful because changes are immediate. Make sure the remote has a valid SSL/TLS certificate; you can log into the remote to see if the URL bar shows ‘Not secure’ and if so, talk to the remote’s root user.
Remote databases have their own user setup in the metadata configuration of the remote, and if a particular user is not set up in the remote, then that remote is not accessible to that user via the local server. The passwords must be the same also. In the future, we will have single-sign-on using a shared key manager to simplify this and to allow separately configurable users. The structure of the roles, databases, database names and permissions are possible to be different on the remote.
A Remote is Just a Reference
The remote is actually never contacted until a connection is attempted as necessary, at which time a user/password mismatch is detected and an error message may be generated locally. The remote is not actually ‘aware’ of the connection at all so any existing server may be made accessible as a remote by setting up local servers to refer to it. No actions are necessary at the remote, other than perhaps creating the necessary user names and ensuring that the passwords match the local ones. In this way for example, the local servers can be ‘sandboxes’ where data is manipulated before being submitted to a shared remote by using the transfer page. The local servers might be running on laptops, given on-premises licensing.
Getting a Certificate
Later on, you can optionally get a valid certificate so that the browser will stop telling you that access is insecure when you browse the server. You will need to SSH in as root. The server comes with a self-signed X509 certificate that will provide encryption through SSL but cannot be verified as being the authentic by the browser or REST clients. The risk is a ‘Man-in-the-Middle’ attack, which may or may not be important to you. To make this easier, we have provided certificate-tool.py in /home/ec2-user/infdb/scripts. This script calls OpenSSL, but is much more convenient. It can provide a free 90-day certificate via Let’s Encrypt, or install an externally provided certificate or generate another self-signed certificate. There is another very very slight MITM risk if the server is accessed by the initially default http port 80, but that redirects you to https port 443 automatically after which it is secure. Or, you can just browse directly to ‘https://..’ to avoid that risk. The let’s encrypt certificates are not available for the default public DNS names of EC2 instances.
Client REST software can use http and get automatic redirection, or go right to https, but if the self-signed certificate is still being used, client software will have to turn off verification: see the REST documentation and the free open-source client REST helper code in at https://github.com/infinitydb/infinitydb.
The server can be configured by the root user (actually ec2-user on EC2) at any time to use http instead of https. However, with http there is no encryption, even for the credentials in each request, so we do not recommend http. The certificate goes in the data volume. Software clients using REST will have to use verify false until the root user installs the certificate or switches to http mode. See the github code for how to set verify false. Again, http is not recommended because user/password are transmitted in each request, and verify false mode is not for production and may be slower because then SSL connections do not handle keep-alive well, causing transparent SSL link re-negotiation occasionally.
Also see Managing a Server for things the root user must do. The root user’s name is actually ‘ec2-user’ on AWS.