N4JS Design Specification

Keywords and identifiers have to be distinguished by the lexer. Therefore, there is no means to decide upfront whether a certain keyword is actually used as a keyword or whether it is used as an identifier in a given context. This limitation is idiomatically overcome by a data type rule for valid identifiers. This data type rule enumerates all keywords which may be used as identifiers and the pure IDENTIFIER terminal rule as seen in Keywords as Identifier listing.

The ambiguity between shift operators and nested generics arises also from the fact, that Antlr lexer upfront without any contextual information. When implemented naively, the grammar will be broken, since a token sequence a>>b can either be part of List<List<a>> b or it can be part of a binary operation int c = a >> b. Therefore the shift operator may not be defined with a single token but has to be composed from individual characters (see Shift Operator listing).

The following figure, Types and Type References, shows the classes of the type model and their inheritance relations, both the actual type definitions as defined in Types.xcore and the type references defined in TypeRefs.xcore. The most important type reference is the ParameterizedTypeRef; it is used for most user-defined references, for both parameterized and non-parameterized references. In the latter case, the list of type arguments is empty.

Most types are self-explanatory. TypeDefs is the container element used in N4TS. Note that not all types and properties of types are available in N4JS – some can only be used in the N4TS language or be inferred by the type system for internal purposes. Some types need some explanation:

We distinguish four kinds of types as summarized in Kind Of Types. Role is an internal construct for different kind of users who can define the special kind of type. The language column refers to the language used to specify the type; which is either N4JS or N4TS.

The built-in and primitive types are not defined by the user, i.e. N4JS programmer. Instead, they are defined in special internal files using the internal N4TS language: builtin_js.n4ts, builtin_n4.n4ts, primitives_js.n4ts, primitives_n4.n4ts.

Originally, the N4JS type system could be called with any EObject at any given time, without any knowledge of the context. While this looked flexible in the beginning, it caused severe problems solving some inference cases, e.g., the rule environment had to be prepared from the outside and recursion problems could occur, and it was also inefficient because some things had to be recalculated over and over again (although caching helped).

It is better to do type inferencing (that is, computing the type of expressions in general) in a controlled manner. That is, instead of randomly computing the type of an expression in the AST, it is better to traverse the AST in a well-defined traversal order. That way, it is guaranteed that certain other nodes have been visited and, if not, either some special handling can kick in or an error can be reported. This could even work with XSemantics and the declarative style of the rules. The difference is that by traversing the AST in a controlled manner, the rules can make certain assumptions about the content of the rule-environment, such as that it always contains information about type variable bindings and that it always contains information about expected types etc.

In that scenario, all AST nodes are visited and all types (and expected types) are calculated up-front. Validation and other parts then do not need to actually compute types (by calling the actual, Xsemantics-generated type system); instead, at that time all types have already been calculated and can simply be retrieved from the cache (this is taken care of by the type system facade N4JSTypeSystem).

This also affects scoping, since all cross-references have to be resolved in this type computation step. However, even for scoping this has positive effects: E.g., the receiver type in property access expressions is always visited before visiting the selector. Thus it is not necessary to re-calculate the receiver type in order to perform scoping for the selector.

The above refactoring was done in summer 2015. After this refactoring, we are still using Xsemantics to compute the types, i.e. the type judgement in Xsemantics was largely kept as before. However, the type judgment is invoked in a controlled traversal order for each typable AST node in largely one step (controlled by ASTProcessor and TypeProcessor).

The upshot of this one-step type inference is that once it is completed, the type for every typable AST node is known. Instead of storing this information in a separate model, this information will be stored and persisted in the type model directly, as well as in transient fields of the AST ^[1]. Currently, this applies only to types, not expected types; the inference of expected types could / should be integrated into the one-step inference as part of a future, further refactoring.

The up-front type inference of the entire AST is part of the post-processing of every N4JSResource and is thus triggered when post-processing is triggered. This happens when

Usually this happens after the types builder was run with preLinking==false and before validation takes place. For details, see classes PostProcessingAwareResource and N4JSPostProcessor.

The traversal order during post-processing is a bit tricky, as some things need to be done in a top-down order (only few cases, for now ^[2]), others in a bottom-up order (e.g. the main typing of AST nodes), and there is a third case in which several AST nodes are processed together (constraint-based type inference).

Figure Order in which AST nodes are being processed during post-processing. provides an example of an AST and shows in which order the nodes are processed. Green numbers represent top-down processing, red numbers represent bottom-up processing and blue numbers represent the processing of the surrounding yellow nodes in a single step.

In the code, this is controlled by class ASTProcessor. The two main processing methods are

The common processing of groups of adjacent yellow nodes (represented in the figure by the two yellow/brown triangles) is achieved by PolyProcessor telling the TypeProcessor to

While typing the entire AST, cross-references need special care. Three cases of cross-references need to be distinguished:

Note that for references to an ancestor (upward references) or successor (downward references) within an AST, the classification as a forward or backward reference depends on whether we are in top-down or bottom-up processing. Figure Upward Downward illustrates this: the left and right side show the same AST but on the left side we assume a top down processing whereas on the right we assume a bottom up processing. On both sides, backward references are shown in green ink (because they are unproblematic and always legal) and forward references are shown in red ink. Now, looking at the two arrows pointing from a node to its parent, we see that it is classified as a backward reference on the left side (i.e. top down case) but as a forward reference on the right side (i.e. bottom down case). Conversely, an arrow from a node to its child is classified as a forward reference on the left side and as a backward reference on the right side. Arrows across subtrees, however, are classified in the same way on the left and right side (see the horizontal arrows at the bottom).

An important exception to the basic traversal order shown in Figure Order in which AST nodes are being processed during post-processing. is that the body of all functions (including methods) and field accessors is postponed until the end of processing. This is used to avoid unnecessary cycles during type inference due to a function’s body making use of the function itself or some other declarations on the same level as the containing function. For example, the following code relies on this:

Similar situation using fields and methods:

For details of this special handling of function bodies, see method ASTProcessor#isPostponedNode(EObject) and field ASTMetaInfoCache#postponedSubTrees and the code using it. For further investigation, change isPostponedNode() to always return false and debug with the two examples above (which will then show the incorrect errors mentioned in the XPECT comments) or run tests to find more cases that require this handling.

Polymorphic expressions, or poly expressions for short, are expressions for which the actual type depends on the expected type and/or the expected type depends on the actual type. They require constraint-based type inference because the dependency between the actual and expected type can introduce dependency cycles between the types of several AST nodes which are best broken up by using a constraint-based approach. This is particularly true when several poly expressions are nested. Therefore, poly expressions are inferred neither in top-down nor in bottom-up order, but all together by solving a single constraint system.

Only a few types of expressions can be polymorphic; they are called poly candidates: array literals, object literals, call expressions, and function expressions. The following rules tell whether a poly candidate is actually poly:

This is a simplified overview of these rules, for details see method #isPoly(Expression) in AbstractPolyProcessor.

The main logic for inferring the type of poly expressions is found in method #inferType() in class PolyProcessor. It is important to note that this method will only be called for root poly expressions (see above). In short, the basic approach is to create a new, empty InferenceContext, i.e. constraint system, add inference variables and constraints for the root poly expression and all its nested poly expressions, solve the constraint system and use the types in the solution as the types of the root and nested poly expressions. For more details see method #inferType() in class PolyProcessor.

So, this means that nested poly expressions do not introduce a new constraint system but instead simply extend their parent poly’s constraint system by adding additional inference variables and constraints. But not every nested expression that is poly is a nested poly expression in that sense! Sometimes, a new constraint system has to be introduced. For example:

For details see method #isRootPoly() in AbstractPolyProcessor and its clients.

The simple constraint solver used by the N4JS type system, mainly for the inference of poly expressions, is implemented by class InferenceContext and the other classes in package org.eclipse.n4js.typesystem.constraints.

The constraint solving algorithm used here is largely modeled after the one defined in The Java Language Specification 8, Chapter 18, but was adjusted in a number of ways, esp. by removing functionality not required for N4JS (e.g. primitive types, method overloading) and adding support for specific N4JS language features (e.g. union types, structural typing).

For details see the API documentation of class InferenceContext.

An instance of the IResourceDescriptions can be acquired from the resource description provider. Just like all services in Xtext, this can be injected into the client code as well. The resource descriptions accepts a non-null resource set. The resource set argument is mandatory to provide the index with the proper state. We are differentiating between three different state. The first one is the persisted one, basically the builder state is a resource description as well, and it provides a content that is based on the persisted state of the files (in our case the modules and package.json file) on the file system. The second one is the live scoped index, this is modification and dirty state aware. Namely when using this resource descriptions then an object description will be searched in the resource set itself first, then in the dirty editor’s index, finally among the persisted ones. The third index is the named builder scoped one. This index should not be used by common client code, since it is designed and implemented for the Xtex builder itself.

A resource set and hence the index can be acquired from the N4JS core, in such cases an optional N4JS project can be specified. The N4JS project argument is used to retrieve the underlying Eclipse resource project (if present) and get the resource set from the resource set provider. This is completely ignored when the application is running in headless mode and Eclipse resource projects are not available. It is also important to note that the resource set is always configured to load only the persisted states.

When the Eclipse platform is running, the workspace is available and the all N4JS projects are backed by an Eclipse resource project. With the Eclipse resource project the resource sets can be initialized properly via the resource set initializer implementations. This mechanism is used to get the global objects (such as console) and the built-in types (such as string, number) into the resource set via the corresponding resource set adapters. In the headless case a special resource set implementation is used; ResourceSetWithBuiltInScheme. This implementation is responsible to initialize the globals and the built-in types into itself.

A crucial point in the workflow described above is the combination of types model building and type inference. In some cases, the type of a given element is not directly stated in the AST but has to be inferred from an expression and other types. For example, when a variable declaration does not declare the variable’s type explicitly but provides an initializer expression, the actual type of the variable is inferred to be the type of the expression.

However, the types builder cannot be allowed to use type inference, mainly for two reasons:

Therefore, whenever the type of a particular element has to be inferred, the types builder will use a special type reference called DeferredTypeRef ^[3], in order to defer the actual type inference to a later stage, i.e. the post-processing stage.

Whenever type inference would be required to obtain the actual type of an element, the types builder will insert a stub to defer actual type inference (see previous section). A dedicated subclass of TypeRef, called DeferredTypeRef, is used that contains neither the actual type information nor any information necessary to perform the type inference at a later point in time. Later, this DeferredTypeRef will be replaced during post-processing, see TypeDeferredProcessor.

All DeferredTypeRefs will be replaced by the actual types during post-processing. One important reason for resolving all DeferredTypeRefs as early as possible is that they are not suited for serialization and therefore have to be removed from the types model before populating the Xtext index, which includes serializing the TModule into the user data of the root element. This is always assured by the logic that manages the triggering of the post-processing phase.

To manually trigger resolution of all DeferredTypeRefs in a given types model, simply call method performPostProcessing(CancelIndicator) of the containing N4JSResource (should never be required by client code such as validations).

Currently, DeferredTypeRefs are created by the types builder only in these cases:

Note that this overview might easily get out-dated; see references to class DeferredTypeRef in the code.

Whenever a change in the workspace happens …​

Invoked: once for each project containing a changed resource and dependant projects.
Input: one instance of BuildData, as created by XtextBuilder, containing:

To conclude this section, we briefly describe the state of the above five phases through a simple example. Assume a workspace with four N4JS projects: P1, P2, P3 and PX. Each project has one single module with one single publicly visible class. Also let’s assume project P2 depends on P1 and P3 depends on P2. Project PX have no dependencies to other projects. Project P1 has a module A.n4js with a class A, project P2 has one single module B.n4js. This module has a public exported class B which extends class A. Furthermore, project P3 has one single module: C.n4js. This module contains one exported public class C which extends B. Finally, project PX has a module X.n4js containing a class X that has no dependencies to any other classes. The figure below picture depicts the dependencies between the projects, the modules and the classes as described above.

For the sake of simplification, the table below describes a symbol table for all resources:

Let assume auto-build is enabled and the workspace contains no errors and/or warnings. We make one simple modification and expect one single validation error in class C after the incremental builder finished its processing; we delete the method foo() from class A.

After deleting the method in the editor and saving the editor content, a workspace modification operation will run and that will trigger an auto-build job. The auto-build job will try to build the container project P1 of module A. Since the project is configured with the Xtext builder command, a builder state update will be performed through the Xtext builder. Initially, due to an Eclipse resource change event (we literally modify the resource from the editor and save it), the ToBeBuilt instance wrapped into the BuildData will contain the URI of the module A marked for an update. When updating the copied index content, module A will be queued for a build. While processing the queued elements for project P1, module A will be processed and will be added to the allDeltas set. Besides that, it will be added to the changedDeltas set as well. That is correct, because its TModule information has been changed after deleting the public foo() method. When queuing affected resources, iterating through the set of allRemainingURIs, we recognize that module B is affected. That is indeed true; module B imports the qualified name of class A from module A and project P2 has a direct dependency to P1. In this builder state phase, when building project P1, module C is not considered as affected. Although class C from module C also imports the qualified name of class A from module A, project P3 does not have a direct dependency to project P1. When module B becomes enqueued for a forthcoming build phase, we assume its TModule information is obsolete. We invalidate this TModule related user data information on the resource description by wrapping the resource description into a custom implementation (ResourceDescriptionWithoutModuleUserData). Due to this wrapping the resource description for module B will be marked as changed (IResourceDescription.Delta#haveEObjectDescriptionsChanged()) whenever the old and current states are being compared.

The Eclipse builder will recognize (via IProjectDescription#getDynamicReferences()) that project P2 depends on project P1 so the Xtext builder will run for project P2 as well. At the previous phase we have enqueued module B for the build. We will therefore run into a builder state update again. We do not have any resource changes this time, so ToBeBuilt will be empty. Since ToBeBuilt is empty, we do not have to update the copied Xtext index state before the builder state setup phase. As the result of the previous builder state, phase module B is already enqueued for a build. When processing B we register it into the allDeltas set. That happens for each resource being processed by the builder state. But it will be registered into the changedDeltas because we have previously wrapped module B into a customized resource description delta to hide its obsolete TModule related user data information. Based on the builder state rules and logic described above, module C will be marked as an affected resource, will be queued for build and will be wrapped into a customized resource description delta to hide its TModule related user data information.

In the next builder state phase, when building project P3, we apply the same logic as we applied for project P2. The builder state will process module C and will update the Xtext index state. No additional resources will be found as affected ones, nothing will be queued for build. The build will terminate, since there were no changed IResource instances and the build queue is empty.

The outcome of the incremental build will be a workspace that contains exactly one validation error. The error will be associated with module C which was exactly our expectation, however, we have to clarify that transitive C dependency was built due to wrong reasons. Module C was build because we wrapped module B to hide its user data information and not because it imports and uses class A from module A which should be the logical and correct reason.

When a N4JS file is changed in way that requires reparsing the file, the underlying resource is completely unloaded and loaded again. By this the also the elements at the Xtext index are recreated again, belonging to this resource (i.e. new entries for new elements in the resource, update index elements of changed elements, delete index entries for deleted elements).

When Eclipse is closed the Xtext index is serialized in a file.

When starting Eclipse again, the Xtext index is restored from this file:

Every internal graph refers to a single control flow container. The graph consists of nodes and edges, where the edges are instances of ControlFlowEdge, and nodes are instances of Node. Additionally, a so called complex node is used to group nodes that belong to the same CF element.

Consider that source code elements can be nested like expressions that have sub-expressions as in 1 + 2 * 3. Also statements can contain other statements like in if (true) return;. The design of the control flow graph deals with this nesting by mapping CF elements to several nodes. All nodes of one CF element are aggregated into the complex node.

The example in the figure above shows the internal graph produced by the source code 1+2. Additionally, a simpler version of the internal graph is shown (called User Graph View), with which client analyses deal. The user graph view is only a view on the internal graph, but does not exist as an own instance. In the figure, the nesting of the integer literals becomes obvious: The control flow edges of delegating nodes targets entry nodes of different CF elements. Also, there are CF edges from the exit nodes of these nested CF elements to return the control flow.

In the above figure, the complex node of the for statement is shown. The details of the complex nodes of the nested CF elements (such as the initializer or the body statement) are omitted. The figure displays the control flow fork after the condition and also shows the Repeat edge that targets the for body. The node called Catch Node is used in situations when there are jumping control flows introduced for instance by a continue statement. The catch will will then be the target of an control flow edge that starts at the continue statement.

The graph of nodes and edges is constructed in the following order.

The internal graph contains many nodes to simplify the graph construction. However, these nodes carry no relevant information when traversing the graph. Consequently, in an optimization step, they are removed from the graph for performance reasons.

A node removal for a node $n_2$ is done by replacing the path $n_1->n_2->n_3$ by the new path $n_1->n_3$. These removals are done for delegating nodes that only have one incoming and one outgoing edge.

A second kind but similar optimization reduces the number of helper nodes that are used as entry nodes. In case a complex nodes consists only of exactly one entry and one exit node, both of these nodes are collapsed into one node. This remaining node then is the representing node of the AST element.

To implement client analyses based on the control flow graph, the three classes GraphVisitor, GraphExplorer and BranchWalker are provided. They provide the means to visit CF elements in a control flow graph and also to traverse single control flow paths. The method N4JSFlowAnalyses#analyze can execute several client analyses in one run to maintain scalability.

The data flow graph provides means to reason about symbols, effects, data flow, aliases and guards in the control flow graph. The main classes of the data flow API are DataflowVisitor and Assumption.

Note that the term value use means either write or method call of a variable. The term value definition means that a variable is written.

This approach provides a very pragmatic and simple solution to support external libraries in both in the IN4JSCore and in the IBuilderState. While IN4JSCore supports a completely transparent way of external libraries via the IN4JSProject interface all over in the application, the IBuilderState is responsible for keeping the Xtext index content up to date with the external libraries. Below picture depicts the hierarchy between the ordinary IResource and the IExternalResource instances. As described above each external resource is backed by a java.io.File resource and each access and operation being invoked on the IResource interface will be delegated to this backing resource.

External library workspace is an extension of the InternalN4JSWorkspace. This workspace is used for storing and managing external libraries all over the application. External libraries can be registered into the workspace by providing one to many external library root folder locations. The provided root folder locations will be visited in an ordered fashion and the contained external libraries (N4JS projects) will be registered into the application. If an external library from a root folder has been registered, then a forthcoming occurrence of an external library with the same artefact identifier (and same folder name) will be ignored at all. For instance let assume two external library root locations are available ER1 and ER2, also ER1 contains P1 and P2 external libraries, while ER2 contains P2 and P3. After registering the two roots into the workspace ER1 will be processed first, and P1 and P2 will be registered to the workspace, when processing the forthcoming ER2 root, P2 will be ignored at all as an external with the same name exists. Finally P3 will be registered to the workspace. External libraries cannot be registered directly into the workspace it is done automatically by the External Library Preference Store and by the npm Manager.

This persistent cache is used for storing an ordered enumeration of registered external library root folder locations. Whenever its internal state is being persisted after a modification, all registered modification listeners will be synchronously notified about this change. All listeners will receive the store itself with the updated state. There are a couple of registered listeners all over the application listening to store update events but the most important one is the External Library Workspace itself. After receiving an external library preference store update event, the external library workspace will calculate the changes from its own state: creates a sort of difference by identifying added, removed and modified external libraries. Also tracks external library root location order changes. Once the workspace has calculated the changes^[12] it will interact with the External Library Builder Helper which will eventually update the persisted Xtext index directly through the IBuilderState. After the Xtext index content update all ordinary workspace projects that directly depend either on a built or a cleaned external library will be automatically rebuilt by the external library workspace.

This service is responsible for downloading, installing third party npm dependencies into the local file system. This is done directly by npm from Node.js. Once an npm package has been downloaded and installed it will be registered into the external library workspace. As part of the registration, the Xtext index content will be updated and all dependent ordinary workspace projects will be rebuilt automatically. An npm package cannot be installed via the Library Manager if it already installed previously.

This builder is responsible for updating the persisted Xtext index state with external library content directly through the IBuilderState. When providing a subset of external libraries to either build or clean, internally it orders the provided external libraries based on the project dependencies. Also, it might skip building all those external libraries that have are being shadowed by a workspace counterpart. An external library is being shadowed by an ordinary workspace project, if the workspace project is accessible and has exactly the same project name as the external library.

By default Xtext provides a way to fix corrupted index or to recreate it from scratch in case of its absence. Such inconsistent index states could occur due to application crashes or due to non-graceful application shutdowns. Although this default recovery mechanism provided by Xtext works properly, it is provided only for projects that are available in the Eclipse based workspace (org.eclipse.core.resources.IWorkspace) but non of the external libraries are not available from the Eclipse based workspace, so inconsistent external library index content cannot be recovered by this default mechanism. N4JS IDE contributes its own logic to recover index state of external N4JS libraries. When the default Xtext index recovery runs, then it will trigger a external reload as well. This external reload is guaranteed to run always after the default recovery mechanism.

This preference page provides a way to configure the external libraries by adding and removing external library root folders, also allows the user to reorder the configured external library root locations. Besides that, npm packages can be installed into the application as external libraries. Neither removing nor reordering built-in external libraries are supported, hence these operations are disabled for built-ins on the preference page. No modifications will take effect unless the changes are persisted with the Apply button. One can reset the configurations to the default state by clicking on the Restore Defaults button then on the Apply button. The Reload button will check whether new type definition files are available for npm dependencies, then reloads the persistent Xtext index content based on the available external libraries. Once the external library reloading has been successfully finished, all dependent workspace projects will be rebuilt as well. From the preference page one can export the installed and used third party npm packages as a target platform. This exported target platform file can be used with the headless compiler. After setting up the headless compiler with this exported target platform file, the headless tool will collect and download all required third party npm dependencies.

In some cases there is a need for custom npm settings, e.g. custom npm registry. Those kind of configurations are supported via .npmrc file (see https://docs.npmjs.com/files/npmrc).

In N4JSIDE user can specify path to his custom configuration file in the preference page.

For the commandline N4JSC.jar provides special option -npmrcRootLocation that allows headless compiler to use custom settings.