other versions
- jessie 1:2.2-6+deb8u1
| mapistore-documentation(3) | MAPIProxy | mapistore-documentation(3) |
NAME¶
mapistore-documentation -Contents¶
- •
- Revision History
- •
- 1. Introduction
- •
- 1.1. Purpose and Scope
- •
- 1.2. General Overview
- •
- 2. Technical / MAPI Considerations
- •
- 2.1. MAPI objects
- •
- 2.2. MAPI tables
- •
- 2.3. MAPI properties and mapping
- •
- 2.4. Named properties vs known properties
- •
- 3. MAPIStore architecture
- •
- 3.1. INTERFACE layer
- •
- 3.2. PROCESSING layer
- •
- 3.3. BACKENDS layer
- •
- 3.4. Relationship to OpenChange Dispatcher database
- •
- 3.5. Object mapping
- •
- 4. MAPIStore API
- •
- 4.1. Initialization
- •
- 4.2. Backend contexts
- •
- 5. FSOCPF backend
- •
- 5.1. Definition
- •
- 5.2. Namespace and Attributes
- •
- 5.3. Overview
- •
- 5.4. Documentation and References
Revision History¶
Date Revision Number Author Revision Content 21/05/10 0.3 Julien Kerihuel Add API documentation for initialization, backend connection contexts and add programming samples 21/05/10 0.2 Julien Kerihuel Merge initial Wiki document and add FSOCPF section 20/05/10 0.1 Julien Kerihuel Draft document.1. Introduction¶
1.1. Purpose and Scope¶
MAPIStore is the SAL component of OpenChange server. SAL stands for Storage Abstraction Layer. It is the component used by OpenChange Server to push/get information (messages, folders) to/from storage backends. The following document intends to describe the overall/theoretical SAL behavior and constraints we need to consider when dealing with MAPI/EMSMDB. It also describes the semantics and inner working of its storage backends.1.2. General overview¶
The main objective of mapistore is to provide an interface layer with a common set of atomic functions (operations) used to trigger and dispatch data and commands to the appropriate backend. MAPIStore relies on a backend mechanism specifically designed to transparently handle some of the MAPI semantics required by any Exchange compatible server. The initial idea was to provide to OpenChange a highly customizable storage backend mechanism which would fit in any situation and any environments. One of the greatest limitation we have found with existing groupware is the storage layer which is generally limited to a single solution, service or format and is neither scalable nor modifiable when user requirements evolve upon time. MAPIStore solves this problem and go beyond classical limitations. It is not a revolutionary concept, but the way openchange uses it makes the whole difference and offer administrators an innovative way to customize storage. MAPIStore allows you to:- •
- use a different backend for any top-folder
- •
- transparently move/copy data across backends
- •
- develop new backends quickly
- •
- access all the different backends through an unique API
- •
- Mails stored using an IMAP backend (Cyrus-IMAP or dovecot)
- •
- Calendar items stored in CalDAV or pushed in Google calendar
- •
- Sent emails and archives/backup stored in a compression backend
- •
- Tasks stored in a MySQL database
- •
- Notes stored on the filesystem
2. Technical / MAPI Considerations¶
2.1. MAPI objects¶
An object is a physical (message, folder) or temporary (table) but generic entity which can be represented as a 2 columns fixed array with n rows, where each row contains a property of that entity. One column repesents the property tags (names), and the other represents the property value (or values). From a MAPI perspective (network layer), opening an object means:- •
- opening either a private mailbox or public folder store. These are referenced by EssDN and not IDs
- •
- opening a message given its PR_MID (Message identifier)
- •
- opening a folder/container given its PR_FID (Folder identifier)
2.2. MAPI tables¶
Another category of MAPI objects are tables (also known as views) created with MAPI ROPs such as GetContentsTable, GetHierarchyTable, GetAttachmentTable or GetRulesTable. Views are temporary representation of the elements that a container holds at a specific moment. It can be represented as a list of n rows (elements) with n columns (property values):- •
- GetContentsTables creates a view listing all messages available within a container
- •
- GetHierarchyTable creates a view listing all containers within a container
- •
- GetAttachmentTable creates a view listing all the attachment of a message
- •
- GetRulesTable creates a view listing all permissions associated to a container
2.3. MAPI properties and mapping¶
There is a large set of fixed and known MAPI properties available. If appropriate backends are developed, there can be a 1-1 mapping between MAPI properties and backend properties for some of them. This mapping may even be enough for common purposes. However there will still be a set of MAPI properties which won't fit in this process. There are different way to workaround this issue. For example, Kolab is using a Cyrus/IMAP backend and associate/store MAPI properties to the message using ANNOTATE-MORE within a XML file format. OpenChange provides similar mechanism with its OCPF file format.2.4. Named properties vs known properties¶
OpenChange server needs to support named properties. An initial set of named properties can be defined at provisioning time, but this list must not be static. Users must be able to add, delete, change new entries and create their own set of custom named properties as required.3. MAPIStore Architecture¶
Given that objects representation are similar to SQL database records, an architecture like the sqlite one makes sense for our purpose: Component Brief description INTERFACE convenient top-level functions (Public API) accessed through EMSMDB providers PROCESSING set of atomic operations (open, read, write, close, mkdir, rmdir etc.) BACKENDS The code which does things in the specified storage system (mysql, fsocpf, imap etc.)3.1. INTERFACE layer¶
The interface layer doesn't have any knowledge about mapistore internals, how or where objects are stored. The interface uses MAPI data structure, supplies PR_FID and PR_MID values and assumes the interface layer will return a pack of data it can directly use without further or significant modifications. The interface layer functions should also have (as a parameter) a pointer to a mapistore context with private/opaque set of information (void *) about the object.3.2. PROCESSING layer¶
The processing layer is responsible for:- •
- mapping OpenChange objects identifiers (PR_FID, PR_MID) to unique backends object identifiers (on purpose, depending on the kind of backend).
- •
- format input/output data: glue between INTERFACE and BACKENDS
- •
- relay input requests to the correct backend through atomic operations
- •
- maintain mapistore's integrity
3.3. BACKENDS layer¶
The backends layer has a list of modules identified at mapistore initialization and available across user sessions, which means unique initialization at server start-up. Each module is a backend (fsocpf, sqlite, imap, etc.) and similarly to many other openchange components is loaded as a DSO object (dynamic shared object)3.4. Relationship to OpenChange Dispatcher database¶
MAPIStore and the openchange 'dispatcher' database (openchange.ldb) are completely unrelated. MAPIStore is a standalone API and developers can use it independently from OpenChange server. However, the mapistore API has initially been designed to be used by OpenChange server, and OpenChange server is using a tiny indexing database which describes user mailboxes top level containers. In openchange.ldb the mapistore_uri attribute is attached to top level containers and its value points to a valid mapistore URI (namespace + path). Note that a single user can have several different types of mapistore databases in use (one for each of the top level containers). The is the only relationship between the database and the store: The database points to store locations.3.5. Object mapping¶
MAPIStore needs to maintain a hash database linking unique OpenChange identifiers to unique backend identifiers. This hash table can be stored within a TDB database. MAPIStore is responsible for managing IDs mapping between OpenChange objects and backend specific objects. It maintains a list of free identifiers which it reallocates on demand whenever a backend needs (mapistore_register_id()) one or where it wants to release one (mapistore_unregister_id()).4. MAPIStore API¶
MAPIStore relies on the talloc library for memory allocation.4.1. Initialization¶
If there was a 'hello mapistore' program, it would only require to make 2 calls:- •
- mapistore_init:
The initialization routine initializes the mapistore general context used along all mapistore calls, the mapping context databases (described below) and finally load all the backends available (DSO). When this operation is successful, developers are ready to make mapistore calls.
- •
- mapistore_release:
The release operation uninitializes the mapistore general context, closes connections to database and frees the remaining allocated memory.
#include <mapistore/mapistore.h>
int main(int ac, const char *av[])
{
TALLOC_CTX *mem_ctx;
struct mapistore_context *mstore_ctx;
int retval;
/* Step 1. Create the talloc memory context */
mem_ctx = talloc_named(NULL, 0, "mapistore_sample1");
/* Step 2. Initialize mapistore system */
mstore_ctx = mapistore_init(mem_ctx, NULL);
if (!mstore_ctx) {
exit (1);
}
/* Step 3. Uninitialize mapistore system */
retval = mapistore_release(mstore_ctx);
if (retval != MAPISTORE_SUCCESS) {
exit (1);
}
return 0;
}
$ export PKG_CONFIG_PATH=/usr/local/samba/lib/pkgconfig $ gcc mapistore_sample1.c -o mapistore_sample1 `pkg-config --cflags --libs libmapistore` $ ./mapistore_sample1 $
4.2. Backend contexts¶
MAPIStore registers and loads its backends upon initialization. It means they are only instantiated/initialized one time during the whole server lifetime and the same code is used for all users and all mapistore folders. These backend contexts (or connection contexts) are identified by a context id, which is an unsigned 32 bit integer which references the context during its lifetime. If OpenChange is used in a very large environment with many top folders (which implies the same number of mapistore contexts), or if OpenChange server has an incredibly long uptime, it would be possible to run out of available context identifiers. In order to prevent this situation from happening, mapistore implements context databases where it stores available/free/used context identifiers:- •
- mapistore_id_mapping_used.tdb: TDB database with used IDs
- •
- mapistore_id_mapping_free.tdb: TDB database with available pool of IDs
- •
- mapistore_set_mapping_path: Defines the path where context databases are stored. Call to this function is optional and default path would be used instead. However if a call to this function has to be made, it must be done before any call to mapistore (even mapistore_init).
- •
- mapistore_add_context: Add a new connection context to mapistore
- •
- mapistore_del_context: Delete a connection context from mapistore
#include <mapistore/mapistore.h>
int main(int ac, const char *av[])
{
TALLOC_CTX *mem_ctx;
struct mapistore_context *mstore_ctx;
int retval;
uint32_t context_id = 0;
uint32_t context_id2 = 0;
/* Step 1. Create the talloc memory context */
mem_ctx = talloc_named(NULL, 0, "mapistore_sample1");
/* Step 2. Set the mapping path to /tmp */
retval = mapistore_set_mapping_path("/tmp");
if (retval != MAPISTORE_SUCCESS) {
exit (1);
}
/* Step 3. Initialize mapistore system */
mstore_ctx = mapistore_init(mem_ctx, NULL);
if (!mstore_ctx) {
exit (1);
}
/* Step 4. Add connection contexts */
retval = mapistore_add_context(mstore_ctx, "fsocpf:///tmp/Inbox", &context_id);
if (retval != MAPISTORE_SUCCESS) {
exit (1);
}
retval = mapistore_add_context(mstore_ctx, "fsocpf:///tmp/Sent Items", &context_id2);
if (retval != MAPISTORE_SUCCESS) {
exit (1);
}
/* Step 5. Release connection contexts */
retval = mapistore_del_context(mstore_ctx, context_id);
retval = mapistore_del_context(mstore_ctx, context_id2);
/* Step 6. Uninitialize mapistore system */
retval = mapistore_release(mstore_ctx);
if (retval != MAPISTORE_SUCCESS) {
exit (1);
}
return 0;
}
$ ./mapistore_sample2 sqlite3 backend initialized fsocpf backend initialized namespace is fsocpf:// and backend_uri is '/tmp/Inbox' [fsocpf_create_context:49] namespace is fsocpf:// and backend_uri is '/tmp/Sent Items' [fsocpf_create_context:49] $
5. FSOCPF Backend¶
5.1. Definition¶
FSOCPF stands for FileSystem and OpenChange Property Files. It is a backend designed to help developers testing OpenChange server code easily. The main idea is to have a backend we can manipulate, analyze and modify from the Linux console without having to develop a specific tool. This backend uses the UNIX filesystem for folder semantics and the OCPF file format as a way to store MAPI objects easily.5.2. Namespace and Attributes¶
The namespace for this backend is:fsocpf://The mapistore_uri attribute for the folder definition in openchange.ldb must be a valid path where the last part of the URI is the FSOCPF container folder to create on the filesystem.
5.3. Overview¶
[+] Private user storage space
|
+-[+] Top-MAPIStore folder (Inbox)
|
+-[+] 0xf1000001 (mapistore folder1)
| |
| +-[+] .properties (OCPF)
| |
| +-[+] 0xe10000001.ocpf (message - OCPF)
| |
| +-[+] 0xe10000001 (attachment folder)
| |
| +-[+] 1.ocpf (PR_ATTACH_NUM)
| |
| +-[+] 1.data (attachment / stream data)
|
+-[+] 0xf2000001 (mapistore folder2)
|
+-[+] .properties (OCPF)
The figure above exposes the storage architecture of the FSOCPF backend using a
real-world example. In this use case, we have decided to associate the FSOCPF
backend to the Inbox folder. It means that any folder created under Inbox or
any message stored within Inbox at any level of depth is stored and retrieved
from the path defined in openchange.ldb.
In openchange.ldb, the mapistore_uri attribute of the Inbox record points to:
fsocpf://Private user storage space/Inbox
where Private user storage space can for example be
/usr/local/samba/private/mapistore/$usernameUnder Inbox, we have created 2 folders:
- •
- 0xf1000001 folder1
- •
- 0xf2000001 folder2
5.4. Documentation and References¶
- •
- OpenChange Property File format (OCPF) documentation
| Tue Apr 25 2017 | Version 2.2 |