Differences

In this section differences between oAW Xtext and TMF Xtext are outlined and explained. We’ll start from the APIs such as the grammar language and the validation and finish with the different hooks for customizing linking and several UI aspects, such as outline view and content assist. We’ll also try to map some of the oAW Xtext concepts to their counterparts in TMF Xtext.

Differences in the grammar language

When looking at a TMF Xtext grammar the first time it looks like one has to provide additional information which was not necessary in oAW Xtext. In oAW Xtext *.xtxt files started with the first production rule where in TMF Xtext one has to declare the name of the language followed by declaration of one or more used/generated Ecore models:

TMF Xtext heading information


  grammar my.namespace.Language with org.eclipse.xtext.common.Terminals
  generate myDsl "http://www.namespace.my/2009/MyDSL"
 
  FirstRule : ...

In oAW Xtext this information was provided through the generator (actually it is contained in the *.properties file) but we found that these things are very important for a complete description of a grammar. Therefore we made that information becoming a part of the grammar language in order to have self-describing grammars and allow for sophisticated static analysis, etc..

Apart from the first two lines the grammar languages of both versions are more or less compatible. The syntax for all the different EBNF concepts (alternatives, groups, cardinalities) is similar. Also assignments are syntactically and semantically identical in both versions. However in TMF Xtext some concepts have been generalized and improved:

String rules become Datatype rules

The very handy String rules are still present in TMF Xtext but we generalized them so that you don’t need to write the ‘String’ keyword in front of them and at the same time these rules can not only produce EStrings but (as the name suggests) any kind of EDataType. Every parser rule that neither includes assignments nor calls any other that does, returns an EDataType containing the consumed data. Per default this is an EString but you can now simply create a parser rule returning other EDataTypes as well (see Datatype rules).


  Float returns ecore::EDouble : INT ('.' INT)?;

Enum rules

Enum rules have not changed significantly. The keyword has changed to be all lower case (‘enum’ instead of ‘Enum’). Also the right-hand side of the assignment is now optional. That is in oAW Xtext:


  Enum MyEnum : foo='foo' | bar='bar';
  

becomes


  enum MyEnum : foo='foo' | bar='bar';
   

and because the name of the literal equals the literal value one can omit the right-hand side in this case and write:


  enum MyEnum : foo | bar;

Native rules

Another improvement is that we could replace the blackbox native rules with full-blown EBNF syntax. That is native rules become terminal rules and are no longer written as a string literal containing ANTLR syntax.

Example :


  Native FOO : "'f' 'o' 'o'";

becomes


  terminal FOO : 'f' 'o' 'o';

See the reference documentation for all the different expressions possible in terminal rules.

No URI terminal rule anymore

We decided to remove the URI terminal. The only reason for the existence was to mark the model somehow so that the framework knows what information to use in order to load referenced models. Instead we decided to solve this similar to how we imply other defaults: by convention.

So instead of using a special token which is syntactically a STRING token, the default import mechanism now looks for EAttributes of type EString with the name ‘importURI’. That is if you’ve used the URI token like this:


  Import : 'import' myReference=URI;

you’ll have to rewrite it that way


  Import : 'import' importURI=STRING;

Although this changes your Ecore model, one usually never used this reference explicitly as it was only there to be used by the default import mechanism. So we assume and hope that changing the reference is not a big deal for you.

Return types

The syntax to explicitly declare the return type of a rule has changed. In oAW Xtext (where this was marked as ‘experimental’) the syntax was:


  MyRule [MyType] : foo=ID;  

in TMF Xtext we have a keyword for this :


  MyRule returns MyType : foo=ID;

This is a bit more verbose, but at the same time more readable. And as you don’t have to write the return type in most situations, it’s good to have a more explicit, readable syntax.

Differences in Linking

The linking has been completely redesigned. In oAW Xtext linking was done in a very naive way: To find an element one queries a list of all ‘visible’ EObjects, then filters out what is not needed and tries to find a match by comparing the text written for the crosslink with the value returned by the id() extension. As a side-effect of link_feature() the reference is set.

The code about selecting and filtering allElements() usually has been duplicated in the corresponding content assist function, so that linking and content assist are semantically in sync. If you’re good (we usually were not) you externalized that piece of code and reused the same extension in content assist and linking.

To put it bluntly this approach could be summarized in two steps:

  1. Give me the whole universe including every unregarded object in the uncharted backwaters of the unfashionable end of the western spiral arm of the galaxy and squeeze it into an Arraylist

  2. From this, select the one I need

This was not only very expensive but also lacks an important abstraction: the notion of scopes.

The idea of scopes

In TMF Xtext we’ve introduced scopes and scope providers that are responsible for creating scopes. A scope is basically a set of name->value pairs. Scopes are implemented upon Iterables and are nested to build a hierarchy. With scopes we declare “visible” objects in a lazy and cost-saving way where the linker only navigates as far as necessary to find matching objects. The content assist reuses this set of visible objects to offer only reachable objects.

When the linking has to be customized scoping is where most of the semantics typically goes into. By implementing an IScopeProvider for your language linking and content assist will automatically be kept in sync since both use the scope provider.

The provided default implementation is semantically mostly identical to how the default linking worked in oAW Xtext:

  1. Elements which have an attribute ‘name’ will be made visible by their name

  2. Referenced resources will be put on the (outer) scope by using the ‘importURI’- naming convention and will only be loaded if necessary

  3. The available elements are filtered by the expected type (i.e. the type of the reference to be linked)

Migration

We expect the migration of linking to be very simple if you’ve not changed the default semantics that much. We’ve already migrated a couple of projects and it wasn’t too hard to do so. If you have changed linking (and also content assist) a lot, you’ll have to translate the semantics to the IScopeProvider concept. This might be a bit of work, but it is worth the effort as this will clean up your code base and better separate concerns.

Differences in UI customizing

In oAW Xtext several UI services such as content assist, outline view or the label provider have been customized using Xtend. In TMF Xtext there is no Xtend API for these aspects. Extensive model computations for the content assist is most probably not necessary anymore- it reuses scopes. And since we provide a declarative Java API that mimics the polymorphic dispatch and relies on static Ecore classes you will gain nearly the same expressiveness as before while increasing maintainability and performance.

Beside the API change in favor of Java we have to mention that in TMF Xtext the outline view does not support multiple view points so far. This is just because we didn’t manage to get this included. We don’t think that view points are a bad idea in general, but we decided that other things were more important.