Serialization

Serialization is the process of transforming an EMF model into its textual representation. Thereby, serialization complements parsing and lexing.

In Xtext, the process of serialization is split into three steps:

  1. Matching the model elements with the grammar rules and creating a stream of tokens. This is done by the parse tree constructor.

  2. Mixing existing hidden tokens (whitespace, comments, etc.) into the token stream. This is done by the hidden token merger.

  3. Adding needed whitespace or replacing all whitespace using a formatter.

Serialization is invoked when calling XtextResource .save(...). Furthermore, SerializerUtil provides resource-independent support for serialization.

The Contract

The contract of serialization says that a model that is serialized to its textual representation and then loaded (parsed) again should yield a loaded model that equals the original model. Please be aware that this does not imply, that loading a textual representation and serializing it back produces identical textual representations. For example, this is the case when a default value is used in a textual representation and the assignment is optional. Another example is:

MyRule:
  (xval+=ID | yval+=INT)*;
  

MyRule in this example reads ID- and INT-elements which may occur in an arbitrary order in the textual representation. However, when serializing the model all ID-elements will be written first and then all INT-elements. If the order is important it can be preserved by storing all elements in the same list – which may require wrapping the ID- and INT-elements into objects.

Parse Tree Constructor

The parse tree constructor usually does not need to be customized since it is automatically derived from the Xtext Grammar. However, it can be a good idea to look into it to understand its error messages and its runtime performance.

For serialization to succeed, the parse tree constructor must be able to consume every element of the to-be-serialized EMF model. To consume means, in this context, to write the element to the textual representation of the model. This can turn out to be a not-so-easy to fulfill requirement, since a grammar usually introduces implicit constraints to the EMF model. Example:

MyRule:
  (sval+=ID ival+=INT)*;

This example introduces the constraint sval.size() == ival.size(). Models which violate this constraint are sort of valid EMF models, but they can not be serialized. To check whether a model complies with all constraints introduced by the grammar, the only way is currently to invoke the parse tree constructor. If this changes at some day, there will be news in bugzilla 239565.

For the parse tree constructor, this can lead to the scenarios, where

To understand error messages and performance issues of the parse tree constructor it is important to know that it implements a backtracking algorithm. This basically means that the grammar is used to specify the structure of a tree in which one path (from the root node to a leaf node) is a valid serialization of a specific model. The parse tree constructor’s task is to find this path – with the condition, that all model elements are consumed while walking this path. The parse tree constructor’s strategy is to take the most promising branch first (the one that would consume the most model elements). If the branch leads to a dead end (for example, if a model element needs to be consumed that is not present in the model), the parse tree constructor goes back the path until a different branch can be taken. This behavior has two consequences:

Transient Values

Transient values are values or model elements which are not persisted (written to the textual representation in the serialization phase). If a model contains model elements which can not be serialized with the current grammar, it is critical to mark them transient using the ITransientValueService, or serialization will fail. The default implementation marks all model elements transient, that are unset or equal to their default value.

Unassigned Text

Unassigned text can be necessary due to data type rule calls or terminal rule calls which do not reside within an assignment. Example:

PluralRule:
  'contents:' count=INT Plural;
  
terminal Plural: 
  'item' | 'items';
  

Valid models for this example are contents 1 item or contents 5 items. However, it is not stored in the semantic model whether the keyword item or items has been parsed. This is due to the fact that the rule call Plural is unassigned. However, the parse tree constructor needs to decide which value to write during serialization. This decision can be be made by implementing the IUnassignedTextSerializer.

Cross Reference Serializer

The cross reference serializer specifies which values are to be written to the textual representation for cross references. This behavior can be customized by implementing ICrossReferenceSerializer. The default implementation delegates to ILinkingService, which may be the better place for customization.

Hidden Token Merger

After the parse tree constructor has done its job to create a stream of tokens which are to be written to the textual representation, the hidden token merger ( IHiddenTokenMerger) mixes existing hidden tokens into this token stream. The default implementation uses the hidden tokens (whitespace, line breaks, comments) from the node model. The IHiddenTokenMerger is the factory for a token stream which is fed by the parse tree constructor and which writes to another token stream.

Token Stream

The parse tree constructor, the hidden token merger and the formatter use an ITokenStream for their output, and the latter two for their input as well. This makes them chainable. Token streams can be converted to a String using the TokenStringBuffer and to an OutputStream using the TokenOutputStream. Maybe there will be an implementation to reconstruct a node model as well at some point in the future. While providing fast output due to the stream pattern, token streams allow easy manipulation of the stream, such as mixing in whitespace or manipulating them.

public interface ITokenStream {
  public void close() throws IOException;
  public void writeHidden(
    EObject grammarElement, String value) throws IOException;
  public void writeSemantic(
    EObject grammarElement, String value) throws IOException;
}