Chapter 4 – Amazon Neptun e
Amazon Neptune Introduction
Amazon Neptune (Neptune) is a graph database engine and -- like RDS and Aurora -- a managed AWS database service.26  Routine database admin tasks, typically performed by DBAs, such as provisioning hardware (i.e., scaling additional compute, memory, and storage resources), performing backups, applying software patches, detecting and reacting to failures, and recovering databases from backups are expedited and often automated by Neptune. Like Amazon Aurora, Neptune is also equipped with solid-state virtual storage facilities which are optimized for database workloads and deliberately engineered to significantly improve database performance, reliability, and availability.  Neptune also promotes high availability by enabling automatic failover to read replicas and data replication across availability zones.
A Neptune graph database is fundamentally different from SQL-based relational database approaches, such as those engines supported by RDS. As previously noted, the data manipulation language (DML) for accessing most relational databases is some version or adaptation of SQL (e.g., an ANSI standard SQL version or a vendor specific SQL version).
In contrast, Neptune is categorized as a NoSQL (i.e., “Non SQL” or "Not Only SQL") database service which essentially means that Neptune provides a non-relational approach to managing, querying and manipulating data.27  Instead of abiding by familiar relational concepts (i.e., tables with columns and foreign keys for implementing relationships), Neptune is based on graph data structures (i.e., vertices, edges, properties …. more on this later).28 Also a Neptune graph database particularly appeals to applications with highly connected datasets and is accessed via well-known graph query languages (GQLs), such as Gremlin and SPARQL (S PARQL P rotocol A nd R DF Q uery L anguage).29
Gremlin, developed by the Apache TinkerPop TM open source project, provides a graph data query language as well as a virtual machine -- akin to a Java Virtual Machine (JVM) -- and is suitable for both OLTP and OLAP applications.
SPARQL is a semantic query language for accessing data stored in RDF (Resource Description Framework) format.30     SPARQL is defined as a formal specification among the family of RDF recommendations from the W3C standards organization (World Wide Web Consortium).  SPARQL supports a query syntax for traversing and navigating data in a graph database.
Regarding highly connected datasets, graph database engines and graph query languages are well-matched for applications that require navigating complex network relationships and also demand rapid results from queries. Complex network relationships typically include numerous, intersecting, many-to-many entity relationships such as those found in applications known as recommender engines .6     Examples of such complex, many-to-many relationships include combinations of the following:
Basic vocabulary of graph databases includes concepts such as vertices (aka nodes ), edges , and properties as the basis for describing, storing and navigating data. These concepts are comparable to fundamental RDBMS concepts (i.e., table rows, foreign keys, columns) and data modeling concepts (i.e. entities, relationships, attributes):
Graph databases supporting these basic graph model concepts (i.e., vertices, edges, labels and properties) are called labelled property graphs (LPGs).33 LPG concepts are comparable to relational vocabulary but the emphasis is somewhat different. The paradigm shift for GQLs is the prominence placed on edges. GQLs elevate edge relationships to a higher stature. Edges are explicitly named (i.e., labelled) components, rather than being subtly inferred (as in the relational world) by foreign keys, creating indexes on foreign keys, and table joins for referencing and comparing values of foreign keys. The GQL focus is storing, searching and navigating edge interconnections between vertices. For many complex OLTP and OLAP applications, edges enable a direct linkage for more succinctly, intuitively, and efficiently traversing the linkages between vertices.
Gremlin Introduction
Unsurprisingly, the Gremlin query language syntax substantially differs from SQL. Specifically, Gremlin queries can concatenate a string of many retrieval operations -- referred to as graph traversal steps -- for successively navigating between vertices via edges.34  For example, a Gremlin query can consist of a sequence of separate steps for traversing from vertex 1 (via a named edge) to vertex 2 -- and then concatenating additional steps for navigating from vertex 2 (via a differently named edge) to vertex 3, etc. The output of a step feeds into the subsequent step as its input. A single Gremlin query statement can traverse many vertices of different labels via different edge types with filters using properties of vertices or edges along the path.
Using Gremlin console commands, a brief summary of Gremlin’s rudimentary graph traversal steps for C reating, R eading, U pdating and D eleting graph data (i.e., fundamental CRUD data operations) -- in contrast to corresponding SQL statements -- is included as follows:
Gremlin Graph Traversal Steps -- Basics
g.addV ('person').property (id, 'PER-0001').
property ('name','Random A. Person').property ('dob', '03/03/1995')
The following statement adds a person who is also an employee. Note that this includes two labels (perhaps connoting that an employee is a sub-type of person ) with an additional property for the employee’s job title.
g.addV (“person::employee”).property (id, 'PER-0002').
property ('name', 'Random B. Person').
property ('dob', '04/04/1990').property ('title','sales person')
g.addE ('friend').from(g.V('PER-0001')).to(g.V('PER-0002')).
property ('weight', 1.0)
g.V().hasLabel ('employee').has ('title',eq ('sales person')).count()
g.V().has ('title','sales person').drop ( )
The previous descriptions and examples of these basic steps represent the tip of the Gremlin iceberg (i.e., a small subset of the overall comprehensive assortment of graph traversal steps supported by Gremlin). These elementary steps are included here simply to provide a glimpse at Gremlin’s fundamental language concepts. A robust collection of many more steps and options are supported by Gremlin for allowing complex data transformations, filtering objects, calculating statistics, etc. Fortunately, abundant documentation describing Gremlin traversal steps (and providing copious examples with subtle nuances) is readily available online.35
SPARQL Introduction
Unlike Gremlin, SPARQL as previously noted is targeted for data stored in RDF format and SPARQL’s syntax and semantics for querying and updating RDF data is more closely aligned with SQL. The RDF graph model (aka a triple store ) stores linked data in the form of data triples derived from statements consisting of a subject + predicate + object . An RDF triple store is considered to be another type of graph database.36  In an RDF graph, the subject and the object become vertices connected by an edge -- the predicate is an edge:
Triple Statement ➔   Subject (Vertex) + Predicate (Edge) + Object (Vertex)
Each triple statement links a subject (i.e., a vertex, typically a discrete entity occurrence such as an individual person, location, product, asset, etc.) via a predicate (edge) to another object (a vertex). In effect, the predicate names the relationship between the subject and object. The object is a discrete entity or can also be a simple literal value (e.g., a character string, date, integer, boolean, etc.) which also is stored as a vertex. Except for literals, objects in a triple statement can be the subject or object in other triple statements, thereby enabling graph traversal from subjects to objects as other subjects. Objects as literal values in essence are property values where the predicate is tantamount to being a property name as well as an edge (more about this later).
Examples of RDF triple statements, corresponding to examples of Gremlin graph traversal steps from in the previous section, are provided as follows:
g.addV ('person').property (id, 'PER-0001').
property ('name','Random A. Person').property ('dob', '03/03/1995')
Corresponding RDF Triple Statements …
... Where “PER-0001” is a Subject … “is a”, “has name”, “was born on” are Predicates … “person” is an Object … ‘Random A. Person’ and ‘03/03/1995’ are literal Objects.
g.addV (“person::employee”).property (id, ‘PER-0002’).
property (‘name’, 'Random B. Person').
property (‘dob’,‘04/04/1990’).property (‘title’,‘sales person’)
Corresponding RDF Triple Statements …
… Where “PER-0002” is a Subject, “employee” is both an Object and Subject (i.e., in 2 separate statements) … “person” is an Object … “is an”, “is a kind of”, “has name”, “was born on”, “has job title” are Predicates … ‘Random B. Person’, ‘04/04/1990’, and ‘sales person’ are literal Objects.
Unlike Gremlin’s labelled property graphs where vertices and edges can each have many properties, RDF subjects and predicates do not have properties directly associated with them.  In RDF, each property of a subject/vertex is implemented as a predicate/edge linking to an object as a literal value. As noted above, the predicate serves not only as an edge but is also analogous to a property name. So, in the previous triple statement ...
PER-0002    has job title    ‘sales person’ …
the predicate “has job title” connects PER-0002 (subject) to the literal value ‘sales person’ (object = a character string) and, in essence, also suggests a property name (i.e., job_title). Each object, therefore, represents either some discrete entity occurrence (i.e., an individual person, place, product, event, thing, concept, etc.) or the value of some property of the subject. So … yes … property values can indeed be linked to a subject but, unlike an LPG, this results in the creation of another vertex and edge in the RDF graph. Clearly for implementing properties of subjects, an RDF graph is likely to contain many more vertices than those that would be required for an LPG.
But what about adding a property to an RDF predicate? Analogous to adding a property to an edge in an LPG, is it possible -- in some way -- to add information about an RDF predicate? Yes … but, sadly, the method for doing such is uncomfortably convoluted!
The RDF specification provides a technique called RDF Reification for adding information about an entire triple statement (e.g., the date the statement was made, the reason for the statement, the individual who made the statement, a rating or weight of the association, etc.).37 In essence, adding information about a triple statement itself is tantamount to adding information about the predicate. In a nutshell -- without the gory details -- reification adds four additional triple statements (aka the “reification quad”) about the original statement itself. These additional statements amount to the inclusion of metadata within the RDF graph for describing the original statement. So, in this context, the RDF graph not only contains application data (i.e., data about the problem domain), but now unfortunately also contains metadata (i.e., data about the data -- data about the statements themselves). This reification quad, therefore, sadly introduces substantially more vertices and edges than would be required for an LPG.
More specifically, the four additional statements, the reification quad, are summarized as follows:
Along with these four additional statements added to the RDF graph, yet another final statement can now be created for achieving the ultimate goal of adding a property value for the original statement, i.e. for a linking another object or literal to the unique identifier for the original statement. Such complexity, in terms of the need for introducing additional vertices and edges -- for simply adding a property to a predicate -- is seemingly breath-taking in comparison to adding a property to an edge in an LPG. But that’s the bad news … Here perhaps is now the good news!
The SPARQL commands for creating, reading, updating, and deleting triple statements (CRUD) as previously indicated are somewhat comparable to SQL (i.e., SQL basic CRUD DML commands: INSERT, SELECT, UPDATE, DELETE). Data professionals (i.e., DBAs, database designers, applications developers, etc.) conversant in SQL are frequently quite comfortable if the need for transitioning to SPARQL arises. Several rudimentary SPARQL commands in comparison to SQL are briefly introduced as follows:
SELECT  ?name
WHERE {
?employee prop:name ?name
?employee prop:jobtitle “sales person”
}
SPARQL variable names begin with a “?”, hence the variables ?employee, ?name, and ?jobtitle in the previous example.
Like the Gremlin introduction of the previous section, this brief introduction to SPARQL commands also represents the tip of the SPARQL iceberg (i.e., a small subset of the overall comprehensive assortment of key words and options supported by SPARQL). The command descriptions included herein simply provide a glimpse at SPARQL’s fundamental language concepts. Fortunately, abundant documentation describing SPARQL commands, examples, key words, options, and complex usage (e.g., joins between triples, outer joins, subqueries, etc.) is also readily available online.38
Neptune Cluster s
A Neptune cluster is very similar to an Aurora cluster. There are, however, some noteworthy differences that justify distinguishing Neptune clusters from Aurora clusters by establishing a new entity, the NEPTUNE CLUSTER entity. Figure 4.1 introduces entities and relationships focusing on Neptune cluster fundamental concepts.
Figure 4.1 Entities and Relationships … Neptune Fundamentals
Figure 4.1 highlights the NEPTUNE CLUSTER entity and reaffirms the explanation of the CLUSTER entity and its sub-type hierarchy in the previous chapter. Similar to Aurora, Neptune also enables the provisioning of a cluster of DB instances, consisting of a primary DB instance and as many as 15 read replica DB instances. The NEPTUNE CLUSTER entity -- at the same level as the AURORA CLUSTER, DOCUMENTDB CLUSTER, and REDSHIFT CLUSTER entities -- is a sub-type of the DB CLUSTER entity. Each Neptune cluster is a kind of DB cluster and, therefore, directly inherits attributes and relationships from the DB CLUSTER entity. Indirectly, it also enjoys inherited attributes and relationships from the CLUSTER entity.
More specifically, as a sub-type of the DB CLUSTER entity, a Neptune cluster (i.e., the graph data maintained by all of its cluster DB instances) is optionally encrypted using a KMS customer master key, and benefits from one or more IAM roles enabling Neptune to access other AWS services on behalf of the Neptune cluster. From Figure 4.1 and as mentioned in the previous chapter, recall that a Neptune cluster -- as a kind of DB cluster -- can be backed up by one or more DB cluster snapshots.
Common attributes inherited by the NEPTUNE CLUSTER entity from the DB CLUSTER entity include the following sample attributes noted as well in the previous chapter:
Indirectly inherited from the CLUSTER entity, a Neptune cluster can serve as a source for event notifications and is associated with many security groups, a single subnet group, parameter group, and an engine version. This means that a cluster of Neptune DB instances is blessed with …
Common attributes indirectly inherited by the NEPTUNE CLUSTER entity from the CLUSTER entity include the sample attributes noted in the previous chapter:
The one-to-many relationship from NEPTUNE CLUSTER entity to the DB INSTANCE entity indicates that each Neptune cluster is composed of one or more DB instances (i.e., all DB instances running the same Neptune engine version). One DB instance is primary and the remaining servers are read replicas. For example, via the Neptune console, when creating a Neptune cluster, a primary DB instance is automatically created for reading, writing and modifying data to the cluster volume. One or more replica DB instances -- possibly in separate availability zones for achieving high availability -- can be added to a Neptune cluster using the Neptune console, CLI, or API.
From 4.1, the optional one-to-many relationships from AURORA CLUSTER and NEPTUNE CLUSTER entities to the DB INSTANCE entity informally convey that each DB instance is exclusively either: 1) A member of an Aurora cluster, or 2) A member of a Neptune cluster, or 3) Neither -- simply a standalone RDS instance.
After a Neptune cluster is created, conventional tools and utilities (i.e., Gremlin, SPARQL, or Neptune utilities) running on client EC2 instances are used for connecting to graph databases. Examples of such utilities include the Gremlin Console, the RDF4J Console for SPARQL access,39 and native Neptune endpoints enabling HTTP/REST queries for Gremlin and SPARQL.
Similar to Aurora, the one-to-many reflexive relationship from the DB INSTANCE entity to itself indicates that possibly many DB instances can serve as read replicas for the primary Neptune DB instance (i.e., a primary DB instance is synchronized to possibly many read replicas). Also similar to Aurora, the one-to-many relationship between CUSTOMER MASTER KEY and DB INSTANCE entities -- for encrypting DB instance data -- applies to Neptune DB instances. Also similar to Aurora, the many-to-many relationship between NEPTUNE CLUSTER and LOG TYPE entities affirms Neptune support for publishing log files to Amazon CloudWatch Logs.
In addition to previously described attributes inherited from its super-type hierarchy (i.e., foreign keys and other attributes inherited from the CLUSTER and DB CLUSTER entities), noteworthy examples of attributes pertinent to the NEPTUNE CLUSTER entity include:
Recall from the previous chapter that these immediately preceding attributes also apply to Aurora clusters. But these attributes are not applicable (as will be explored in subsequent chapters, e.g., Chapter 5 for DocumentDB) to other sub-types of the DB CLUSTER entity. These attributes, therefore, cannot be elevated to the DB CLUSTER entity thereby enabling inheritance by all of its sub-types .
Aurora and Neptune cluster similarities suggest that perhaps Aurora and Neptune clusters might be generalized to yet another higher level of abstraction for hosting their common relationships and attributes (i.e., that the AURORA CLUSTER and NEPTUNE CLUSTER entities become sub-types of a new super-type entity beneath the DB CLUSTER super-type). The author, however, chose not to introduce this additional layer of complexity and defers exploring such (i.e., modifying Figure 4.1 to depict the suggested super-type) as an exercise for curious data modeling students. Recall from the explanation of a conceptual data model (CDM) and a more detailed logical data model (LDM) in the Introduction, that such an exercise might arguably be more appropriate and justifiable when evolving to an LDM from a CDM.
Neptune vs. Aurora Cluster Differences
Clearly from the previous section, a Neptune cluster is similar to an Aurora cluster. Aurora clusters, however, exhibit several noteworthy curiosities in contrast with Neptune clusters which vindicate establishing the NEPTUNE CLUSTER entity as a justifiably distinct sub-type of the DB CLUSTER entity. Figure 4.2 highlights Aurora vs. Neptune major conceptual differences.
Figure 4.2 Entities and Relationships Highlighting Aurora vs. Neptune
From Figure 4.2, the following similarities (i.e., relationships highlighted in green), re-emphasized from the previous section for the reader’s convenience, are highlighted as follows:
Figure 4.2 also highlights AURORA CLUSTER relationships to the BACKTRACK entity, OPTION GROUP entity, and its reflexive relationship (i.e., relationships highlighted in red). The differences? Unlike Aurora …
Lack of Neptune support for backtracking and serverless clusters means that the following attributes -- conveniently repeated here from the previous chapter -- are applicable to Aurora clusters but are not applicable to Neptune clusters: