InfinityDB Java NoSQL Database

There are thousands of deployments in current use for years by these large companies and more:

has been shipping InfinityDB Embedded for years to tens of thousands of customers in their successful Crucible and Fisheye repository browser as the foundation for a fast web server, where it gathers and presents repository structure. Atlassian is an Australia-based company that produces collaboration software for software developers.

an international company that uses InfinityDB Embedded in Kuwait in a time-series database for collecting real-time signals from distributed nodes for wellhead health and productivity monitoring. Their system keeps up with a very fast stream of input from hundreds of sensors over radio links, and archives the data for critical later analysis.

is a large Canadian text processing software company that ships InfinityDB Embedded in an enterprise-grade text indexing system. The index provides access to distributed documents based on content.

is an innovative Australian company that uses InfinityDB Encrypted in its  ‘ripple-down-rules’ medical data analysis software products. They use encryption for health security compliance.

These companies and more have relied on InfinityDB Embedded for years for critical data storage of their successful commercial products.  These projects require extreme speed,  zero maintenance, and unique data structuring capabilities. InfinityDB Embedded is licensed for distribution in applications as a jar.

Fast Multi-Core Design

InfinityDB Embedded was already incredibly fast, but then we redesigned it to make use of all cores at the same time, each operating safely on a different thread. Now, InfinityDB Embedded runs at over 1 million ops per second on 8 cores as it scales. You can take advantage of this speed immediately on a server, or you can use multiple threading in your  application. Cores are multiplying at Moore’s-law speed, and applications are adding more and more threads.

Without the multi-core technology in InfinityDB Embedded to avoid inter-thread interference, bottlenecks called ‘convoys’ can occur when threads contend for data. Performance can drop dramatically, even far below single-thread speed. The concurrency algorithm is patented now.

 Zero Administration

InfinityDB Embedded applications can run indefinitely with no DBA attention for installation, management, application upgrade, or schema definitions like create table scripts.

InfinityDB  Embedded uses a single file for all purposes. The combination of this feature and the instant guaranteed recovery on abrupt application termination help make InfinityDB Embedded administrator free. No logs need to be archived or re-applied. There are no configuration files, temporary files, or text logs. No junk files are left behind after any kind of termination, so there is never any cleanup.

 Reliability and Safety

InfinityDB Embedded uses a rugged internal storage update protocol for persistence on demand or cache spilling to disk for large amounts of data that maintains system-wide data integrity, and survives abrupt application termination or other problems. The single data file remains up-to-date, safe, correct, and usable through any event. There is no log-based recovery, hence restart and recovery is immediate in all cases. No unexpected Exceptions are ever thrown: not even due to any kind of deadlock or internal resource limits (optional optimistic locking throws expected Exceptions on conflict however). No dangerous off-heap storage or native libraries are used.

Two kinds of transactions are available:

Data Structures

Applications do not need to invent binary encodings or convert primitives to binary or text. Data is not stored as formatted text or as custom raw binary, but as an intermediate form, with standard pre-defined binary encodings of the individual Java primitives in a consistent way that allows extremely high speed.

InfinityDB Embedded supports all primitive Java data types and more:

• long (stored as compressed bits to handle byte, short, char, with no more space)
• float
• double
• boolean
• String (encoded as UTF-16)
• Date/time
• index (for ‘huge sparse arrays’, lists in JSON, and BLOBs/CLOBs, texts)
• short byte arrays up to 1KB (sort by length first, used for BLOBs)
• short UTF-16 char arrays up to 1K chars (sort by length first, used for CLOBs)
• short byte strings up to 1KB (these sort like strings but with bytes instead of 2-byte chars)
• ‘Classes’ metadata identifiers embedded in Items for delimiting other data, describing the schema
• ‘Attributes’ metadata identifiers

Application-Specific Data Models

InfinityDB provides a rich data representation space for structured, semi-structured, or unstructured data. The  basic data model is simple but flexible enough to be used by the application to define and represent any mixture of images or other BLOBS, texts or other CLOBS, trees, graphs, key/value multi-maps, sets, documents, text indexes or other inversions, huge sparse arrays, tables with an unlimited number of columns of an unlimited number of values or nested structures per column,   Entity-Attribute-Value structures or creative custom structures.

Items and the ItemSpace

All structures in the entire database are represented at the lowest level as a magnitude-ordered set of ‘Items’, where an Item is  logically a char array from 0 to 1665 chars long. The simplicity of the low-level format allows great speed and compression. These ordered Items represent the entire state of the database. ItemSpaces come in a wide variety of implementations, but they all have the same simple structure.

In order to make use of the ItemSpace, an Item is formatted as a packed series of ‘components’ of the 12 primitive data types, each of which is variable-length, compressed, explicitly typed, and self-delimiting. So, an Item can be thought of as a tuple with a variable number of elements of any type. The data in the components is formatted such that the sorting is appropriate: raw floats or other types would not sort properly. The internal binary encoding is done by InfinityDB Embedded in a fixed permanent way  for forwards and backwards compatibility.

All basic access to the database uses a temporary ‘Cu’ cursor containing one Item and no other state. The binary encoding of each component in an Item is unimportant to the application, which uses only Java primitives indirectly to build and examine Items in a Cu cursor.

Prefixes of Items are often used to logically nest Items into arbitrary recursive sub-spaces, i.e. sets of suffixes. Items can be used to represent sets of fixed-length tuples, the equivalent of CSV files or tables, or they can represent paths to JSON terminal values. JSON can be parsed and formatted from the Items.  The JSON is not stored literally: the entire database can be accessed at any level of hierarchical detail, because there is no fixed predefined division between keys and JSON documents. Combinations of tabular, document, or many other structures can be easily intermixed.

Flexible Extensible Data Structures with ‘Class’ and ‘Attribute’ Data Types

If the special Class and Attribute data types are mixed in with the other ‘primitive’ data types in the Items, flexible, ‘incrementally self extending’ structures can be represented. See the InfinityDB Client/Server page for a graphical view of some examples of the flexible structures. An initial Class component is normally used to separate data for unlimited independent uses even without the flexible structuring in a single InfinityDB Embedded file. A Class is encoded as binary but contains a string with an initial capital letter followed by zero or more letters, digits, dot, dash, or underscore (as a regex: [A-Z][A-Za-z0-9._-]*). An Attribute is identical but starts with a lower case letter.

When used to represent a ‘flexible’ tabular structure, keys can be:

• ‘tuples’, where a tuple is any concatenation of zero or more primitives of any type,
• heterogenous – different keys can have different primitive types or tuple types,
• variadic – different keys can  be tuples of a different number of primitive types,
• nestable sparse arrays or lists of unlimited size of any key type, i.e. lists, using the ‘index’ data type

Flexible table column values can be the same as keys plus:

• multi-valued, with no limit on number, and where an absence of any value takes no storage,
• CharacterLongObjects or BinaryLongObjects of unlimited size.

Furthermore, any such flexible structures can be nested by concatenating their Items onto the ends of other Items. A particular set of suffixes can contain any kind of nested structure. The ‘Class’ and ‘Attribute’ data type components can represent four patterns depending on their pairings within each Item:

The GUI display of such flexible structures is very rich – see it in action in InfinityDB Client/Server. The displays look like nestable ‘documents’, tables, lists, trees, and so on. Here is a flexible table with Class “Trees”, a multi-value Attribute, ‘composite’ keys of variable component count, and a nested table “Location”.

Forwards and Backwards Schema Compatibility

The ItemSpace model is inherently extensible, but with the flexible ‘Class’ and ‘Attribute’ metadata data types embedded in the Items, databases become ‘self-describing’ and can be extended in ways that avoid incompatibilities with earlier or later database backups, old or new application versions, or changing or extending data producers and consumers like users, Python scripts, bash ‘curl’ commands,  or IoT’s or distributed databases.

Virtual View ItemSpaces