InfinityDB

The fast instant Map-based noSQL Java Embedded Database

  • Simple Map interface (plus many extensible ‘Engine-level’ models)
  • Strong typing (with no text/binary trap)
  • Runtime schema evolution (provides forwards/backwards compatibility)
  • 1M ops/sec in memory
  • Multi-core concurrency
  • Compression
  • Transactions
  • Self administration (one file)
  • NoSQL

The InfinityDB Java embedded database is as simple to use as the Java Map interface, and is fast, multi-core concurrent, and data-compressing. It requires no schema upgrades or maintenance, and it is ‘no-SQL’ so it is very easy to administer. Yet InfinityDB  Java embedded DBMS provides a fully relational and beyond-relational data representation. Direct control of the data model via the extended java.util.Map interface allows developer creativity in order to reach maximum software capability and future extensibility, while protecting you from the text/binary trap. Data is encoded with efficient fast fine granularity and smooth unlimited growth in size and structure, whether the data fits in memory or not. Data spills to disk transparently from the memory cache as needed. There is only a single file. The noSQL InfinityDB Java embedded DB is safe, even though it avoids a log and has instant restart in all situations.

Simple extended java.util.Map Interface

InfinityDB is accessible via an upper level wrapper that implements and improves on the java.util.concurrent.ConcurrentNavigableMap, thereby providing the capability of a HashMap, TreeMap, ConcurrentHashMap, or ConcurrentSkipListMap.  Data is stored internally as ‘tuples’, which are manipulated by nested InfinityDBMaps, each of which corresponds to an immutable tuple prefix. Nested Maps may be created dynamically if desired to access data in a hierarchy or a wide variety of other structures.

Sophisticated web applications and many other kinds of applications can be supported entirely and simply by InfinityDB – no RDBMS is needed, just the easy extended Map interface and optionally the low-level InfinityDB embedded Java DBMS data model. Initial development is easy, yet the no-SQL InfinityDB supports indefinite application growth and complexity along with increasing data volume and velocity. Backwards compatibility with older databases and even some forwards compatibility is easy because the database is schemaless. The instant InfinityDB Java database avoids the text/binary trap, so applications need not read and slowly parse chunks of limited-size text or devise complex dangerous rigid opaque binary encodings of data.

The InfinityDB no SQL Java embedded DB has been in use for years, with thousands of active installations and more arriving all the time.

Enormous Speed and Concurrency

The processing power of InfinityDB is measured above one million operations per second at the low-level with multiple threads. Its Multi-Core technology takes advantage of the processing power available in modern IoT devices, personal computers, embedded systems, and servers. Cores are multiplying at Moore’s-law speed, and applications are adding more and more threads. Without the multi-core technology supplied by the noSQL InfinityDB to avoid inter-thread interference, bottlenecks called ‘convoys’ can occur when threads contend for data. Performance can drop dramatically, even far below single-thread speed. InfinityDB Java embedded database is completely thread-safe. Modern multi-threaded applications cannot afford failures due to accidental write-sharing or read-write sharing of non-thread-safe Maps or other structures.

Appropriate for small to large installations

You can use InfinityDB as an In-Memory-Only system, or one with more than 100’s of GBs with no code changes. Access to data in the memory cache is fully multi-core, while infrequently used data is paged to disk as necessary. Data is not serialized in slow batches, as it would be with the standard Java Object serialization, XML, or JSON, for example, but is instead operated on efficiently with fine granularity for small or large data sizes regardless of memory capacity. Fine granularity accesses can smoothly transition to coarse block-oriented granularity accesses as needed.

Extensibility and Forwards Compatibility

No data structures have physical or practical size limits but can always expand efficiently from zero values upwards to any size. For example, a multi-value column or attribute, ordered set, or huge sparse array can smoothly extend to any size with fine-grained, efficient and smooth access performance. Data structures that are empty take no space, hence any additional structure requires no reorganization, as each data structure effectively already exists virtually but with no size.

For example, when data is viewed as relations, a new table or column can be added at any time, because all column values are nullable and come into existence on the first use, taking no space until then. There is no limit on tables, columns, values or value size. A single-valued column that is to be converted to multi-value is already in the proper format, and more values can simply be inserted along with it. Because of this generality, such as extending the concept of a column to have multiple values, we call the model ‘Entity-Attribute-Value’, but the extensions are not required, and a basic key/value store is simple as well using just the standard Map view.

Applications can anticipate extensions and often provide some forwards compatibility and can provide vital full backwards compatibility. No upgrade scripts are needed.

Data Structures

The data type for individual values is determined dynamically, yet values are not simple strings to be parsed but are binary, strongly-typed, self-describing, and use minimal space. InfinityDB converts encoded values to Java primitives and back transparently and almost instantly without application effort.

Aggregate data structures for example include potentially large CharacterLongObjects, or BinaryLongObjects which can be added at any time in any context. Typical application-determined structures include relations, key/value maps, EAV or ER triples, taxonomies, trees, text indexes, DAG’s, sets, or general graphs, including extensible dense mixtures of these, as explained in the documentation. These structures all rest on the same fast, trivial Map API or the lower-level ‘engine’, and developers can easily design and optimize creatively.

No Garbage or GC

Space allocation for individual values or aggregates of values is fully dynamic – they take no space until created or after deletion. During growth, values and aggregate structures require minimal space, and during shrinking, they dynamically return all freed space and remain efficient.

Aggregate space allocation is not at the granularity of full data blocks, and aggregates may share one or more data blocks with any other application data transparently. The single data file is 100% efficient with compressed data on initial loading, and stays at least 50% efficient after very large transactions, which may include any amount of data. The file never shrinks, however. By rewriting the data file with trivial application code, 100% efficiency returns.

Deleted values and shrinking or deleted structures generate no garbage, but immediately return freed space. Applications can run forever without gradual unlimited space loss. There is no need for  reorganization or garbage collection.

High Compression on Disk and In Memory

InfinityDB’s continuous, dynamic ZLib and UTF-8 data compression packs data into variable-length blocks, avoiding almost all wasted space that would normally be needed for internal fragmentation. I/O bandwith is reduced accordingly. Variable-length data items, prefix and branch-cell suffix compression are used in the memory cache as well. Data compression means that the branching factor is kept high for fast access, and the OS file cache is better used.

InfinityDB requires no configuration file, no extraneous directories, and no temporary work files. Only one file per database is needed.

Direct No-SQL API

Our trivial API allows you to create the exact data structures you need, whether it is Map-like, relational, Entity-Attribute-Value, custom, or a mixture. All of the data models are compatible and can be combined without loss of generality or restrictions on accessibility from modules that have different views of the data. There is actually only one model that simply encompasses all of the others and can be viewed from any perspective, called the ‘ItemSpace’.

There are only a few data storage and retrieval operations that operate on the no SQL InfinityDB ItemSpace. You gain direct, optimized and low-level access by allocating a cursor, and then using it to insert, delete, update, locate, or scan data in the ItemSpace, in which variable length composite data items contain all of your data. There is no requirement that you refer to any external files such as per-table files or index files whatsoever.

Transactionality

Transactions at the global level can accommodate fast bulk loads, with atomic, consistent, and durable semantics. Fine-grained transactions use optimistic locking and support complete ACID atomic, consistent, isolated and durable semantics. Locks do not follow the usual rules of other DBMS’ but have the equivalent capability as table locks and row locks, index locks, or even single-column value locks and single set element locks. This diversity of lock types is not actually a complex spectrum of details – it follows trivially from the basic data model and is automatic and almost invisible to the programmer. If desired, the programmer can easily control the lock order for maximum concurrency simply by accessing critical data early in the transaction.

No Need for Schema Upgrades

InfinityDB is infinitely, incrementally expandable while maintaining backwards schema compatibility. The schema does not need to be embedded in the database itself, but is implied by the way the application uses the data. This means that no SQL scripts need to be run to accomplish an upgrade. Instead it provides some forwards-compatibility and easy backwards compatibility of databases with applications because future applications do not need to go back and change schema definitions in order to continue operating. Often, future applications require virtually no effort to be backwards-compatible with older databases. Nevertheless, stored data has sensible, self-descriptive regular binary structure that makes it easy to represent and understand. The schema ‘rides along’ with the data as the data is stored. For example, a new table does not need to be created, or columns added to an existing table, before data can be stored that logically belongs in the new table. There is no proliferation of upgrade scripts in a growing directed acyclic graph of schema upgrade paths between versions as applications evolve.

Sensible Data Representation

Data is not stored as raw text or as raw binary, but as an intermediate form, with standard pre-defined binary encodings of the individual data items in a consistent way that allows extremely high speed combined with clarity of representation. The composite binary data items can be converted to regular text for humans by simply printing them while they are in a cursor. Applications do not need to invent binary encodings or use custom Java object serialization formats that will later have upgrade, reliability, flexibility, consistency, documentation, safety, security or coding complexity issues. InfinityDB has more extensibility, flexibility, and self-documenting characteristics than the best text formats without the performance limitations, such as reading and writing entire JSON texts or XML DOMs on each access.

InfinityDb natively supports all common primitive Java data types and more:

  • long (stored as compressed bits so byte, short, and int take no more space)
  • float
  • double
  • boolean
  • String (stored as UTF-8)
  • Date
  • short byte and char arrays (sort by length first)
  • short byte strings (sort like strings but with bytes instead of 2-byte chars)

When used to represent a relational structure, InfinityDB column values can be either:

  • sets of an arbitrary mixture of:
    • composite i.e. a heterogenous concatenation of one or more primitive data types, or
    • any value type identified by a heterogenous composite of primitive data types
  • sparse arrays of unlimited size of any other value type, or
  • CharacterLongObjects or BinaryLongObjects of unlimited size.

In such an extended relational structure, keys can be limited-length Composite and Heterogenous. 

All structures are represented as an ordered set of ‘Items’ which are each a short limited-length composition of one or more arbitrary strongly typed binary-encoded primitives, where Items are accessed independently. Any structure is nullable, but no space is required for such null structures or values, hence all structures are ‘virtual’ and can be inserted anywhere with no other structure changes. The possible structures are not limited to such extended relations however.

On the other hand, object serialization, JSON or XML require slow parsing, do finite chunked access, cannot represent key- or value-ordered sets – especially heterogenous dynamic extensions, need re-organization when schemas change, and cannot grow huge.

Reliability and Safety

InfinityDB uses a rugged internal storage update protocol for persistence on demand or cache spilling to disk for large amounts of data that maintains system-wide ACID properties except Isolation, protects data integrity, and survives abrupt application termination, file system bugs, and kernel panics with no locking. The single data file remains up-to-date, safe, correct, and usable through any event. There is no log-based recovery, hence restart is immediate in all cases. For smaller per-thread transactions, ACID properties including Isolation are also provided via optimistic locking. (There is one exception for power failure that is a fundamental weakness of any software explained in Operating Guidelines.)

See the Manual for detailed information. See Essentials for documents on the internal structure or the principles for constructing any higher-order data model from the trivial underlying ‘ItemSpace‘ data model.

atlassian_logo(1) pacific_knowledge_systems_logo

For licensing, email support@boilerbay.com.