Chapter 8. Architecture

RT is built like a layer cake. Each layer has no innate knowledge of what sits on top of it.^[11] This chapter takes a tour of RT’s structure and then a more detailed look at RT’s database and object models.

Quick Overview

Before we get into the nitty gritty details of how RT is put together, let’s take a quick look at the RT layer cake and see what we get out of each layer. Figure 8-1 shows the layers and how they fit together.

Figure 8-1. The RT layer cake

SQL Database

Behind everything that RT does, there’s an SQL database. This database stores all of RT’s data and metadata. Some SQL databases provide their own procedural languages that allow developers to implement large applications entirely inside the database. This has the disadvantage of tying an application tightly to a specific database engine, such as MySQL or Oracle. RT takes a different tack, using the database as a datastore. Because of this, RT is able to run on top of a number of different database engines, each of which has different strengths. RT works to make sure that data stored in SQL is relatively easy to query using other tools, though we strongly recommend that you not alter the database by hand.

DBI

Perl’s DBI is the standard SQL Database Interface for Perl programs. It defines an interface that various and sundry database drivers (DBDs) implement in a largely consistent manner. In addition to MySQL, Postgres, and Oracle, you’ll find DBD modules that let you talk to LDAP, flat text files, and even Google as if

they were SQL databases. If you come from the Java world, you can think of DBI as Perl’s equivalent to JDBC.

DBIx::SearchBuilder

DBIx::SearchBuilder is the secret sauce that lets an object-oriented application like RT talk to a table-oriented relational database. SearchBuilder presents a simple object-oriented API to Perl programs and translates those API calls into SQL statements tuned for the specific database. A row from any given database table can be accessed via a subclass of DBIx::SearchBuilder::Record. To search for and work with sets of records all at once, RT provides a Collection class for each database table, which is a subclass of DBIx::SearchBuilder itself.

RT application platform libraries

Until now, we’ve only talked about RT as a ticketing system. Once you start looking at the architecture and APIs, you’ll quickly discover that there’s a whole application platform under the hood in addition to the ticketing system.

RT’s application platform libraries are the guts of RT. They provide database connectivity, logging infrastructure, users, groups, access control, links, and a few other bits. While the ticketing system uses the RT application platform, it’s not the only application that does so. RTFM^[12] is a knowledge-base tool that uses the RT application platform to implement an entirely different application.

RT ticketing system libraries

RT’s ticketing system libraries use the RT application platform as a base to provide tickets, transactions, attachments, custom fields, and queues. The ticketing system defines groups and access control rights and a type of link that teaches the application platform how to deal with links between ticket objects.

Mason handler

The Mason handler is one of many possible applications that runs on top of the RT Core libraries. It provides a wrapper around the Mason templating system.^[13] Mason is conceptually similar to ASP, PHP, or JSP, but for Perl.

RT comes with two main sets of Mason templates out of the box: user interface templates, which are designed for end users to interact with their browsers, and REST templates, which are designed to be easy for other software to interact with. As of version 3.4, the REST templates are designed to present information in RFC822 format, which anything can parse.

Right now, the Mason templates for the RT application platform and RT ticketing system are very tightly integrated, but moving forward, the templates are changing to improve how they are broken out.

HTTP Clients

RT has three HTTP clients: a web browser, the email gateway, and the command-line interface. Most users interact with RT using their browsers. RT’s email gateway is a tiny Perl script that your mail server runs every time an email message destined for RT comes in. The email gateway, in turn, dispatches the message to your RT server over HTTP or HTTPS. It waits around for the RT server to confirm receipt of the message and then exits.

RT’s command-line interface is a web browser. But not like you’re imagining. The CLI communicates with RT over the web, using the same authentication system as the rest of RT, so you can use it from anywhere that you can use RT’s web interface. Like any other web client, it uses GET and POST to download from and upload to your RT server.

Filesystem Layout

In the next sections and chapters, we’ll dive into RT’s internals to get a better sense of how RT is put together. As you’re reading, it may be useful to refer to RT’s source code. All the paths we talk about will be relative to your RT installation path. By default, RT is installed into /opt/rt3, but you or your system administrator may have installed it elsewhere. Some flavors of Unix—such as Debian Linux and FreeBSD—package RT and install its code and configuration files into locations that fit system policy.

lib/

Standard Perl libraries live in lib/. RT.pm contains RT’s basic initialization routines. It reads RT’s configuration file, sets up connections to the database and logging system, and loads system-internal objects.

RT’s other Perl libraries live inside the RT/ subdirectory of lib. To add RT-specific behavior, both RT/Record.pm and RT/SearchBuilder.pm subclass DBIx::SearchBuilder and DBIx::SearchBuilder::Record::Cachable, respectively. RT/Interface/Web.pm handles many of the details of initialization of RT’s Mason handler and provides a number of helper functions for the Mason Interface. RT/Interface/REST.pm provides helper functions for RT’s REST-based web services framework. RT/Interface/Email.pm provides routines for handling incoming email messages and converting them into RT’s preferred formats and encodings.

Standard lexicon files live in lib/RT/I18N/. Lexicon files contain translations of RT into languages other than English. They’re in the extended GNU Gettext format that Locale::Maketext::Lexicon supports.

share/html/

Standard Mason components live in share/html/.

bin/

Executable RT programs live in bin/. rt is the RT command-line tool. It talks to your RT server via HTTP or HTTPS. You can copy it to other servers or your desktop to run. rt-crontool lets you script actions like escalations or reminder email using your system’s cron facility. You can use rt-mailgate to funnel incoming email into RT. Like rt, it can run on any server (such as your mail server) that has Perl installed. webmux.pl is the mod_perl handler for RT. mason_handler.fcgi is the FastCGI handler for RT. standalone_httpd is a full RT webserver in a single tiny Perl script. It’s useful for testing out RT or for your development environment. Because it’s a non-forking, non-threaded Perl script, it might not stand up to full-scale production use.

sbin/

RT system programs live in sbin/. These tools are used during initial installation of RT. They should almost never be needed during regular use of RT. rt-test-dependencies is a tool to help verify that you’ve installed all the Perl modules RT needs to run and to help you install any that you happen to be missing. rt-setup-database is the RT helper script that sets up a database schema, ACLs, and data for RT or any RT extensions that touch the database.

var/

Mason cache data lives in var/mason_data/. Under some database backends, session data and locks may be stored under var/session_data/. If you are using the SQLite backend, its database is under var/ by default, such as the file var/rt3.

etc/

Configuration files live in etc/. RT_Config.pm lists out all of RT’s default configuration parameters. Database configuration parameters are set automatically by RT’s configure script, but just about everything else is the factory default.

When RT starts up, it looks in RT_Config.pm and then overrides the defaults it finds there with local configuration from RT_SiteConfig.pm. This way, you can add any RT configuration you want to RT_SiteConfig.pm, rather than hand-editing the RT_Config.pm file and merging changes every time you upgrade.

The initial database schema also lives in etc/, in multiple different schema.* files customized for each database platform.

local/

Site-local Perl libraries live in local/lib/. Site-local Mason components live in local/html/. Site-local lexicon files live in local/po/.

Unicode

Before RT 3.0, RT was language-agnostic. The user interface was in English,^[14] and RT didn’t do anything special with the text entered by users. As long as you only used the web interface, it worked OK. Users in Japan could enter text in Japanese. Users in Spain could enter text in Spanish. When RT tried to send mail, the recipient wouldn’t be able to read it, since RT didn’t know enough about the content of the message to correctly label it.

RT 3.0 changed all that. The interface has been fully internationalized,^[15] and we’ve standardized on Unicode and the 8-bit UTF-8 encoding. This means that you can use English, Spanish, Russian, Japanese, Korean, and Chinese in the same message, and RT will store and display them just fine.

When a user connects to RT’s web interface, RT tells his browser that it should speak to RT in UTF-8. Because the web came into being after the advent of Unicode, most every web browser knows how to speak UTF-8.

Sadly, the situation isn’t so rosy in the world of email. Every language has its own encoding, sometimes even two or three of them. For email, RT has to do all of the hard work. When it receives an email message, it has a look at the message headers to see if the sender told us what encoding they were using. If the sender didn’t tell RT, it uses the Perl module Encode::Guess to look at the message’s content and guess what the most likely encoding is. Once RT knows what encoding an incoming message is in, it converts the message to UTF-8. When sending messages back out, RT looks at a site’s list of preferred encodings, picks the most apropriate one, and re-encodes the message.

Logical and Object Model

This section takes a tour of RT’s logical model , shown in Figure 8-2. RT maps its logical model closely to its object model , so we’ve condensed the details of the object and logical models throughout the section.

Figure 8-2. The RT Logical Model

Every RT object has a field called ID. This field is an integer that’s automatically set to a unique value when an object is created and never changes. RT uses these values internally as keys to objects. Additionally, many of RT’s objects have fields called Creator, Created, LastUpdated, and LastUpdatedBy. These fields automatically track the users who created and most recently updated these objects and the times of those actions. For clarity, we’ve skipped over those fields in the schema listings below.

Each of the database tables we discuss here corresponds to two RT objects, a record class and a collection class. The methods for each of these classes are split up into several files; see Overlays in Chapter 10 for more details. Method-level documentation for each class is available in Perl’s perldoc format. Each collection class is named the same as RT’s database table for that sort of object. Each record class is named for the singular version of the collection. So if you’re interested in users, for example, you want to look at both RT::Users and RT::User. By default, the primary file for each of these classes lives in /opt/rt3/lib/RT/.

The documentation in both the record and the collection modules gives an overview of the database schema for the class and provides pointers to other files that contain class methods and documentation:

    perldoc /opt/rt3/lib/RT/User.pm

and

    perldoc /opt/rt3/lib/RT/Users.pm

Each of these files will give you an overview of the database schema for the class, as well as provide pointers to other files that contain class methods and documentation.

Users

In RT, a user is any individual who can perform actions within the system. Any time you create, modify, look at, or delete an object, you need to do it as a user.

Out of the box, RT comes with three special users :

RT_System: RT uses the RT_System user to perform actions as the system itself. This includes looking up data for internal use—when you really don’t want access control checks to deny the system access to its own data—as well as automatically reopening a closed ticket when a user sends in email about that ticket. Inside RT, you always can access RT_System as $RT::SystemUser.
Nobody: RT uses the Nobody user primarily to mark tickets that have no owner. For consistency, it’s important that tickets always be owned by somebody, even if that somebody is a dummy user. Inside RT, you can access Nobody as $RT::Nobody.
root: Out of the box, RT comes standard with a single user account called root whose password is password. On Unix systems, root is the superuser. Long, long ago, RT used unix system accounts instead of its own account system. The Unix-y name of this account is a holdover from those days. Once you’ve got an RT system up and running, nothing internal depends on the name root. There isn’t a global object in RT for the root user.

Users have the following fields:

Name: Every user in RT has a Name, which is guaranteed to be unique. While RT doesn’t reference users directly by Name, users authenticate to RT and search for other users by Name.
EmailAddress: A user’s EmailAddress is used by RT to attach incoming email messages to a user account upon receipt. It’s also used by RT to figure out where to send outgoing mail when tickets are updated. No two users may have the same email address, although it’s fine to have many users with no email address.
Password: While RT supports external access control mechanisms, many sites don’t have a universal authentication framework, so RT provides its own password-based system. Internally, RT stores an MD5 hash of a user’s password in the Password field but never exposes it directly to end-users. Unlike most other fields, Password is write-only. RT provides SetPassword and IsPassword methods but not a Password method. RT treats passwords such as *NO-PASSWORD and *LOCK* as locked passwords and never allows password-based access to accounts with such passwords.
Comments: RT is often used in a customer-service scenario, where it’s important to be able to keep notes about users. The Comments field is a scratch-pad about a user, but it isn’t visible to that user unless they’re part of the Privileged group.
Signature: When composing mail from within the web interface, RT automatically will append a user’s Signature to the message.
RealName: In a number of situations, RT will display a user’s RealName, rather than their Name or EmailAddress.
Lang: RT’s web interface provides browser-based language negotiation features, but it’s sometimes useful to override that with a user’s preferred language. The Lang field stores an RFC3066-style language tag.
Gecos: RT provides functionality to allow command-line tools written around the RT framework to map the current Unix username to an RT user’s Gecos field.
NickName, Organization, HomePhone, WorkPhone, MobilePhone, PagerPhone, Address1, Address2, City, State, Zip, Country, FreeformContactInfo: RT provides these fields to allow sites to use it as a contact management system. They’re not used internally.
EmailEncoding, WebEncoding, ExternalContactInfoId, ContactInfoSystem, ExternalAuthId, AuthSystem, Timezone, PGPKey: RT doesn’t currently use these fields.

Groups

A collection of users and other groups can be assigned rights, made watchers of tickets, and so on. Groups can contain users and groups. Groups can’t contain themselves.

Groups have the following fields:

Name

Every group in RT has a name, purely for human convenience. Internally, RT doesn’t care about group names.

Description

Likewise, the Description is provided so that users know what a particular group is for.

Domain

Many parts of RT use the groups system for a wide variety of different purposes. Domain is a high level way to mark what type each group is.

ACLEquivalence

Internally, RT’s access control system only grants rights to groups. When first created, every user has an ACL equivalence group created with only that user as a member. Whenever RT’s API is used to grant rights to an individual user, the right is really granted to that user’s ACL equivalence group.

SystemInternal

RT keeps track of certain user attributes with system-wide meta-groups. Upon installation, RT creates three of these SystemInternal metagroups: Everyone, Privileged, and Unprivileged. Every single user is added as a member of the Everyone group and either the Privileged or Unprivileged group. If a user is Unprivileged, they can’t be granted rights directly, and RT’s web frontend automatically shunts them to a restricted self-service frontend upon login.

UserDefined

UserDefined groups are system-wide groups created by local staff. Generally, they’re used as a system managment convenience. Rights are granted to groups, rather than individual users, and groups are made Ccs or AdminCcs of Tickets or Queues. This makes it easier for administrators to track who can perform actions or who will receive email on a specific topic.

Personal

Personal groups are defined by individual users for their own personal use. Currently, they’re used only by the Delegation system to allow users to delegate rights to other users or groups.

RT::System-Role, RT::Queue-Role and RT::Ticket-Role

Role groups are designed to allow administrators to grant rights to a role and configure mailing rules for a role at either the System or Queue level and have that configuration apply to every user who is operating in that role for a given ticket or queue. As of RT 3.0, the possible roles are Requestor, Cc, AdminCc, and Owner.

Type

For ACLEquivalence groups, the group’s type is 'UserEquiv‘.

For SystemInternal groups, the group’s type is one of 'Everyone', 'Privileged', or 'Unprivileged‘.

For each of the role groups, the type is one of Owner, Requestor, Cc, or AdminCc.

For Personal groups, the type is always “” (an empty string).

Instance

For ACLEquivalence and Personal groups, the group’s instance is the user’s ID.

For SystemInternal and UserDefined groups, the group’s instance is always 0.

For RT::Ticket-Role groups, the group’s instance is the Ticket’s ID.

For RT::Queue-Role groups, the group’s instance is the Queue’s ID.

For RT::System-Role groups, the group’s instance is always 1.

Principals

In RT, each user or group is a type of Principal. Principals are an abstraction that RT uses internally so that rights can be granted to both users and groups and so that both users and groups can be members of a group.

Principals have the following fields:

PrincipalType: A principal’s type is always one of User or Group. It tells RT which sort of object this Principal is. Because a Principal’s ID is always the same as the ID of the associated user or group object, it would be possible (but slow) to deduce this information on the fly.
Disabled: Sometimes a user or group is no longer needed. If you simply deleted it from the database, you’d lose information about what that user or group did in the past. RT provides the Disabled field, which lets you specify that a Principal and its associated user or group is no longer in play.
ObjectId: A principal’s ObjectId is always the same as its ID. At one point, a User’s ID wasn’t the same as the ID of the Principal object. You should never be looking at or using a Principal’s ObjectId.

GroupMembers

The GroupMembers table keeps track of all the users and groups that are members of a group. Generally, developers will interact with the GroupMembers table through the API provided by group objects.

GroupId: GroupId refers to the group where the member belongs. It points to the ID of a group object.
MemberId: MemberId is the ID of a Principal that is a member of the group referenced by the GroupId.

CachedGroupMembers

RT allows both users and groups to be members of groups. Because SQL databases don’t generally provide a means to ask a database for all rows that refer to this row recursively, we have to build up a more complex mapping table.

Generally, everything you ever want to do with CachedGroupMembers is encapsulated in the interface to group objects. RT’s CachedGroupMembers algorithm is quite complex and isn’t designed to be modified by user code. Details of its schema are provided here for completeness.

CachedGroupMembers have the following fields:

MemberId

As in the GroupMembers table, MemberId is a reference to the ID of a Principal that is a member of the group referenced by the GroupId.

Via

In cases where a user or group is a member of a (parent) group, Via will point to the ID of the row in the CachedGroupMembers table for the parent group.

When the group in question is the top-level group, Via points to this row’s ID.^[16]

ImmediateParentId

ImmediateParentId is the ID of the Group that this row’s MemberId is explicitly a member of. It corresponds directly to the GroupId in the GroupMembers table.

Disabled

If this cached group member is a member of this group by way of a disabled group or if this group is disabled, this will be set to 1.

This prevents us from finding members of disabled subgroups when listing off group members recursively. Also, it allows the access control system to elide members of disabled groups.

ACL

A single row in an access control list is an Access Control Entry or ACE. The Access Control List (ACL) table details which rights each Principal has for any ACLed object in RT. ACLed objects include: tickets, queues, and groups.

ACLs have the following fields:

PrincipalType

PrincipalType captures which sort of object a given ACE applies to. For each of the roles—Owner, Requestor, Cc, and AdminCc—the PrincipalType is the name of that role.

For regular groups, the PrincipalType is simply Group.

For an ACE granting rights to a user, the PrincipalType is Group. Behind the scenes, rights are never granted directly to users, but to their ACL equivalence groups.

PrincipalId

PrincipalId is a pointer to the principal to which this ACE applies. In the case of regular groups and role groups, it points to the group in question’s Principal. If the ACE grants rights to a specific user, the PrincipalId points to that user’s ACL equivalence group.

RightName

The RightName is a simple textual identifier for the right being granted. For any object that supports access control, you can get a complete list of what rights it supports with the AvailableRights() method.

ObjectType

Each ACE applies to a specific object. ObjectType is the class of that object. For example, if you granted a right on Queue 1, the class would be RT::Queue.

ObjectId

ObjectId is the ID of the object to which this ACE refers. For example, if you granted a right on Queue 1, the ObjectId would be 1.

DelegatedFrom

RT’s ACL delegation system allows individual users to delegate rights that had been granted to them and automatically removes the delegated rights when the grantor’s rights are revoked. DelegatedFrom is a pointer to the ID of the ACE on which this right is based.

DelegatedBy

DelegatedBy is the Principal ID of the user who delegated this right.

Links

The Links table keeps track of relationships between entities. The object classes wrapping Links have built-in intelligence to do clever things with RT tickets, but you can think of the Links system as a simple store for RDF-style Subject-Predicate-Object triples (Base-Type-Target in RT’s lexicon).^[17]

Links have the following fields:

Base

Base is the URI of the left-hand side of this link relation. It can contain any URI. By default, RT ships with classes that can handle http:, https:, file:, ftp: and fsck.com-rt: URIs.

LocalBase

If this Link’s base is a local ticket, LocalBase is a pointer to the ID of that ticket. This field is a convenience that makes it easier for RT to find links related to a particular ticket.

Type

Type describes what sort of relationship the Base has to the Target. Out of the box, RT supports three simple types of relationships:

RefersTo: The Base refers to the Target. This is a simple weak reference that lets you tie related objects together. For example, “The purchase of new servers refers to the estimates we got from these six vendors.”
DependsOn: The Base in some way depends on the Target. In the ticketing system side of RT, we use DependsOn to track cases when one Ticket must be completed before another one can be completed such as “The purchase of new servers depends on budget approval for the server migration project.”
MemberOf: The Target contains the Base. In the ticketing system side of RT, we use MemberOf to track parent-child relationships such as “The purchase of new servers is a member of the server migration project.”

Target

Target is the URI of the right-hand side of this link relation. It can contain any URI.

LocalTarget

If this Link’s target is a local ticket, LocalTarget is a pointer to the ID of that ticket. This field is a convenience that makes it easier for RT to find links related to a particular ticket.