N4JS Design Specification

Keywords and identifiers have to be distinguished by the lexer. Therefore, there is no means to decide upfront whether a certain keyword is actually used as a keyword or whether it is used as an identifier in a given context. This limitation is idiomatically overcome by a data type rule for valid identifiers. This data type rule enumerates all keywords which may be used as identifiers and the pure IDENTIFIER terminal rule as seen in Keywords as Identifier listing.

The ambiguity between shift operators and nested generics arises also from the fact, that Antlr lexer upfront without any contextual information. When implemented naively, the grammar will be broken, since a token sequence a>>b can either be part of List<List<a>> b or it can be part of a binary operation int c = a >> b. Therefore the shift operator may not be defined with a single token but has to be composed from individual characters (see Shift Operator listing).

Every internal graph refers to a single control flow container. The graph consists of nodes and edges, where the edges are instances of ControlFlowEdge, and nodes are instances of Node. Additionally, a so called complex node is used to group nodes that belong to the same CF element.

Consider that source code elements can be nested like expressions that have sub-expressions as in 1 + 2 * 3. Also statements can contain other statements like in if (true) return;. The design of the control flow graph deals with this nesting by mapping CF elements to several nodes. All nodes of one CF element are aggregated into the complex node.

The example in the figure above shows the internal graph produced by the source code 1+2. Additionally, a simpler version of the internal graph is shown (called User Graph View), with which client analyses deal. The user graph view is only a view on the internal graph, but does not exist as an own instance. In the figure, the nesting of the integer literals becomes obvious: The control flow edges of delegating nodes targets entry nodes of different CF elements. Also, there are CF edges from the exit nodes of these nested CF elements to return the control flow.

In the above figure, the complex node of the for statement is shown. The details of the complex nodes of the nested CF elements (such as the initializer or the body statement) are omitted. The figure displays the control flow fork after the condition and also shows the Repeat edge that targets the for body. The node called Catch Node is used in situations when there are jumping control flows introduced for instance by a continue statement. The catch will will then be the target of an control flow edge that starts at the continue statement.

The graph of nodes and edges is constructed in the following order.

The internal graph contains many nodes to simplify the graph construction. However, these nodes carry no relevant information when traversing the graph. Consequently, in an optimization step, they are removed from the graph for performance reasons.

A node removal for a node $n_2$ is done by replacing the path $n_1->n_2->n_3$ by the new path $n_1->n_3$. These removals are done for delegating nodes that only have one incoming and one outgoing edge.

A second kind but similar optimization reduces the number of helper nodes that are used as entry nodes. In case a complex nodes consists only of exactly one entry and one exit node, both of these nodes are collapsed into one node. This remaining node then is the representing node of the AST element.

To implement client analyses based on the control flow graph, the three classes GraphVisitor, GraphExplorer and BranchWalker are provided. They provide the means to visit CF elements in a control flow graph and also to traverse single control flow paths. The method N4JSFlowAnalyses#analyze can execute several client analyses in one run to maintain scalability.

The data flow graph provides means to reason about symbols, effects, data flow, aliases and guards in the control flow graph. The main classes of the data flow API are DataflowVisitor and Assumption.

Note that the term value use means either write or method call of a variable. The term value definition means that a variable is written.

This approach provides a very pragmatic and simple solution to support external libraries in both in the IN4JSCore and in the IBuilderState. While IN4JSCore supports a completely transparent way of external libraries via the IN4JSProject interface all over in the application, the IBuilderState is responsible for keeping the Xtext index content up to date with the external libraries. Below picture depicts the hierarchy between the ordinary IResource and the IExternalResource instances. As described above each external resource is backed by a java.io.File resource and each access and operation being invoked on the IResource interface will be delegated to this backing resource.

External library workspace is an extension of the InternalN4JSWorkspace. This workspace is used for storing and managing external libraries all over the application. External libraries can be registered into the workspace by providing one to many external library root folder locations. The provided root folder locations will be visited in an ordered fashion and the contained external libraries (N4JS projects) will be registered into the application. If an external library from a root folder has been registered, then a forthcoming occurrence of an external library with the same artefact identifier (and same folder name) will be ignored at all. For instance let assume two external library root locations are available ER1 and ER2, also ER1 contains P1 and P2 external libraries, while ER2 contains P2 and P3. After registering the two roots into the workspace ER1 will be processed first, and P1 and P2 will be registered to the workspace, when processing the forthcoming ER2 root, P2 will be ignored at all as an external with the same name exists. Finally P3 will be registered to the workspace. External libraries cannot be registered directly into the workspace it is done automatically by the External Library Preference Store and by the npm Manager.

This persistent cache is used for storing an ordered enumeration of registered external library root folder locations. Whenever its internal state is being persisted after a modification, all registered modification listeners will be synchronously notified about this change. All listeners will receive the store itself with the updated state. There are a couple of registered listeners all over the application listening to store update events but the most important one is the External Library Workspace itself. After receiving an external library preference store update event, the external library workspace will calculate the changes from its own state: creates a sort of difference by identifying added, removed and modified external libraries. Also tracks external library root location order changes. Once the workspace has calculated the changes^[1] it will interact with the External Library Builder Helper which will eventually update the persisted Xtext index directly through the IBuilderState. After the Xtext index content update all ordinary workspace projects that directly depend either on a built or a cleaned external library will be automatically rebuilt by the external library workspace.

This service is responsible for downloading, installing third party npm dependencies into the local file system. This is done directly by npm from Node.js. Once an npm package has been downloaded and installed it will be registered into the external library workspace. As part of the registration, the Xtext index content will be updated and all dependent ordinary workspace projects will be rebuilt automatically. An npm package cannot be installed via the Library Manager if it already installed previously.

This builder is responsible for updating the persisted Xtext index state with external library content directly through the IBuilderState. When providing a subset of external libraries to either build or clean, internally it orders the provided external libraries based on the project dependencies. Also, it might skip building all those external libraries that have are being shadowed by a workspace counterpart. An external library is being shadowed by an ordinary workspace project, if the workspace project is accessible and has exactly the same project name as the external library.

By default Xtext provides a way to fix corrupted index or to recreate it from scratch in case of its absence. Such inconsistent index states could occur due to application crashes or due to non-graceful application shutdowns. Although this default recovery mechanism provided by Xtext works properly, it is provided only for projects that are available in the Eclipse based workspace (org.eclipse.core.resources.IWorkspace) but non of the external libraries are not available from the Eclipse based workspace, so inconsistent external library index content cannot be recovered by this default mechanism. N4JS IDE contributes its own logic to recover index state of external N4JS libraries. When the default Xtext index recovery runs, then it will trigger a external reload as well. This external reload is guaranteed to run always after the default recovery mechanism.

This preference page provides a way to configure the external libraries by adding and removing external library root folders, also allows the user to reorder the configured external library root locations. Besides that, npm packages can be installed into the application as external libraries. Neither removing nor reordering built-in external libraries are supported, hence these operations are disabled for built-ins on the preference page. No modifications will take effect unless the changes are persisted with the Apply button. One can reset the configurations to the default state by clicking on the Restore Defaults button then on the Apply button. The Reload button will check whether new type definition files are available for npm dependencies, then reloads the persistent Xtext index content based on the available external libraries. Once the external library reloading has been successfully finished, all dependent workspace projects will be rebuilt as well. From the preference page one can export the installed and used third party npm packages as a target platform. This exported target platform file can be used with the headless compiler. After setting up the headless compiler with this exported target platform file, the headless tool will collect and download all required third party npm dependencies.

In some cases there is a need for custom npm settings, e.g. custom npm registry. Those kind of configurations are supported via .npmrc file (see https://docs.npmjs.com/files/npmrc).

In N4JSIDE user can specify path to his custom configuration file in the preference page.

For the commandline N4JSC.jar provides special option -npmrcRootLocation that allows headless compiler to use custom settings.

By default, built-in external libraries (such as node_modules, N4JS Language, N4JS Runtime and Mangelhaft), also known as "shipped code", are only available when the N4JS IDE is running in production mode, i.e. when Platform#inDebugMode() and Platform#inDevelopmentMode() both return false). This means built-in external libraries are not available in the N4JS IDE when it is started from Eclipse development environment or when running any plug-in, plug-in UI and/or SWTBot tests.

There are two ways of enabling built-in external libraries in NON-production mode:

The first option is only intended for the case of starting an N4JS IDE from an Eclipse development environment; in all kinds of tests, the helper methods mentioned above should be used instead (i.e. option 2.). In other words, the configuration property org.eclipse.n4js.includesBuiltInLibraries should only be used in our two launch configurations "N4JS__IDE.launch" and "N4JS_N4__IDE.launch" that are checked-in to our repository.