InfinityDB Module Architecture

InfinityDB has are several overall levels of detail which can be broken down by API: The ‘ItemSpace API’ the ‘ConcurrentNavigableMap API’ the PatternQuery API, and the server’s communication APIs which are REST and ItemPacket, The InfinityDB Embedded and InfinityDB Encrypted products are version 4 and version 5, while InfinityDB Server is version 6, including the previous two. Here we describe the various modules and APIs.

ItemSpaces

The ItemSpace (in light blue above) underlies all the other APIs and is provided by all versions. It is an extremely simple but very fast, concurrent, binary interface for use intra-JVM. It is possible to use the Map API instead (in green above) as a higher-level wrapper, but occasionally direct ItemSpace access is still used. An ItemSpace has only a few basic operations – mutations and retrievals take only a few methods, and the data model is almost trivial. An ItemSpace is logically only a set of Items in sorted order, where an Item is a char array from 0 to 1665 in length. (ItemSpaces are not stored as char arrays, however.) ItemSpace is a class (a subclass of ItemOutput) and is frequently subclassed in order to provide additional features, such as the Engine, the Virtual ItemSpaces and the RemoteClientItemSpace and more, shown in blue.

The Components of an Item

Some code deals with Items at the low level, but higher levels use a standard binary encoding to represent a sequence of ‘components’ each of 12 possible fundamental data types which are directly convertible to Java types and more:

  • long: compressed 64 bit numbers
  • double: 64-bit real
  • float: 62 bit real
  • boolean
  • string
  • time and date
  • short char arrays
  • short byte arrays
  • short byte strings (sort like regular strings)
  • index: for lists
  • attribute meta type
  • class meta type

It is easy to store Binary Long Objects or Character Long Objects, optionally associated with mime types. These are accessed like OutputStreams or InputStreams, which are like files but are encoded into Items. There is thus almost no special case access for these, and they integrate with all other ItemSpace uses. Such a ‘file’ uses the index type plus, nested inside it, the short byte array or short char array types.

InfinityDB Engine

In the light blue block, the engine actually stores Items on disk, one ItemSpace per file. The InfinityDB Embedded engine provides transactional, compressing, robust storage. InfinityDB Encrypted also encrypts the files..

Virtual ItemSpaces

A Virtual ItemSpace (also in the light blue block) subclasses ItemSpace, providing additional features by wrapping a ‘base ItemSpace’. These can be used interchangeably anywhere. Mostly, underlying ItemSpace changes reflect immediately in the virtual view ItemSpace. This facility is very powerful and transforms many problems into simple stitching together of these structures.

  • ItemSubspace  virtually hides and restricts by a fixed prefix of an ItemSpace. Multi-threaded;
  • DeltaItemSpace is a mutable view of a fixed underlying ItemSpace with its own commit and rollback, non-concurrently;
  • AndSpace views a logically intersected set of underlying ItemSpaces. Multi-threaded;
  • OrSpace views a logically unioned set of underlying ItemSpaces. Multi-threaded;
  • RangeItemSpace views a limited range of Items. Multi-threaded;
  • VolatileItemSpace stores Items in memory non-persistently. Single-threaded;
  • IncrementalMergingItemSpace views a special kind of index that can be incrementally built and optimized efficiently at any size while +being accessed concurrently. Concurrent deletions are allowed. Text indexing is one use.
  • BuffferedItemSpace increases speed for remote access over the net
  • RangeBufferedItemSpace an alternative to BufferedItemSpace

These light-weight views can be nested, and can underlie the Map interface. An arbitrarily deep nesting of AndSpace and OrSpace can be flattened automatically for best speed, providing a type of instant dynamic query capability without indexes, query compilation, execution, or temporary space usage. There are no locks, inter-thread interference, or transactionality compromises. Some are multi-threaded as noted.

ItemPacket Protocol

In order to communicate with the server, a Java program can use an ItemSpace called RemoteClientItemSpace to transparently view a server’s data as if it were local. Servers can also use this to make remote servers accessible as if they were local to the client server. The protocol is called ‘ItemPacket’, and it is efficient, binary, checksummed, SSL/TLS secure, and transactional. It is interchangeable with any other ItemSpace in the same way as the Virtual ItemSpaces.

Map Interface

Any ItemSpace can be wrapped by an implementation of the java.util.concurrent.ConcurrentNavigableMap interface, in green above. These InfinityDBMaps allow an ItemSpace to be viewed as a standard map of the highest possible capability, so they are compatible with Map, SortedMap, NavigableMap, and ConcurrentNavigableMap. However InfinityDBMap also provides Multi-Map extensions, so any key can be used to access multiple values, and the maps can be recursively nested so that any value becomes the key at a deeper level. The multi-values can be accessed as a NavigableSet as well. Access is transactional, concurrent, multi-core, lockless, and robust. There are no special error cases beyond those required by the Map interfaces themselves. It is possible to do almost anything that is possible with a basic ItemSpace, but much more conveniently. The server and PatternQuery processor are written mostly with InfinityDBMaps.

PatternQueries

PatternQueries (dark blue) are simple non-procedural pattern-based definitions of high-level operations over ItemSpaces. Users provide PatternQuery definitions to the server, which stores them in turn in an ItemSpace and then can execute them on behalf of the user or on behalf of a REST request. Each URL of a REST APIs fires a specific PatternQuery, so application development is rapid and maintenance is easy, with improved security and client isolation.

The REST Protocol

REST is an industry-standard request/response protocol based on HTTP that provides access to the server for Java, JavaScript, and Python clients, and even in the unix shell, the curl command can be used for access. It is SSL/TLS secure. The Python library infinitydb.access provides a set of features that simplify and extend REST, such as for dealing with REST messages as Python dicts. REST uses JSON, but InfinityDB server also can transmit binary blobs of any mimetype efficiently through REST.

Security

The externally-accessible protocols ItemPacket and REST are protected by Secure Sockets Layer SSL/TLS, which is the industry standard protocol for authentication and encryption, and is very strong. Clients are authenticated using HTTP Basic Authentication, which requires username and password access. This method is compatible with all of the client use cases, even unix shell access, and even browsers can ask users for a user name and password for Basic Authentication. Passwords are not stored in the server literally, but are hashed according to industry-standard methods. All such server configuration data is stored encrypted in a single metadata file. The metadata and all InfinityDB Encrypted data is stored encrypted using industry-standard security primitives (Server 6.0.4 does not encrypt databases yet although standalone InfinityDB Encrypted does.) The server admin places users in groups and then gives groups access to particular databases, either local or remote. The admin configures databases and remote connections.