Effective and eventual immutability

For now, the analyser’s binaries have not been uploaded to a central jar repository (like MavenCentral) yet. Please clone the following projects from GitHub:

The support jar has been compiled with the Java 10+ API; this can easily be stripped down to Java 8 if required. The analyser makes extensive use of the Java 16 API and language features.

The IntelliJ plugin is still in its infancy. It is not yet available in IntelliJ’s plugin repository. To experiment with it, clone

The annotations and support classes can be compiled with any JDK providing the Java 10+ API. Execute:

to make the jar, with reference org.e2immu:e2immu-support:0.2.0, locally available.

Please ensure you have at least a Java 16 JDK. Gradle 7.0 (lower versions do not play nice with Java 16) is provided via the Gradle wrapper.

To make the jars available, publish them to your local Maven repository:

This pushes the following jars to your local Maven repository:

Of course, version numbers may have changed when you read this.

The annotation store is a separate project, consisting of two Java classes. Build it with:

and start the annotation store with

The build.gradle file provides support to change the port, in case 8281 is already occupied. For example,

will start the annotation store listening to port 9999. Please make sure this change of port is reflected in the configuration of the IDE plugin, which also needs to connect to the annotation store.

The plugin is compiled with Java 8 language features only.

Starting your first project, basic use of the e2immu analyser looks like

Having installed e2immu 's IntelliJ IDEA plugin, and having started a local annotation server, you can edit your project, occasionally run the analyser, and make use of pre-annotated libraries.

The analyser produces computed annotations, errors and warnings, which you can of course read from the command line. It also pushes these errors, warnings, and computed annotations to the annotation server, which is continuously consulted by the highlighter plugin. So while you’re working on your project, each time you run the analyser, your editor is updated. We’re suggesting that you confirm critical annotations in the source code. They will get the annotation type VERIFY which is the default for source code read by the analyser, they allow you to 'stabilize' your code with respect to class types like @Container or {e2immutable}.

This set-up will get you pretty far, as long as

The e2immu analyser is set up to read, in order of decreasing priority,

If your project is or becomes a library for other projects to use, the computed annotations have to be made available to the users of the library:

If you are annotating external libraries with e2immu annotations, there are two options

Annotated API sources can be generated by the analyser from jars and XML annotations, presenting only those types, methods and fields that your project is using.

Updated annotation files can be generated by the analyser from the combination of annotated API sources and existing annotation files.

The input to the analyser is largely controlled by the following primary locations

Then there are typical options like

The following options are available to control the output to the annotation server:

The following options are available to control the output of Annotation XML files written:

The easiest way to use the analyser is via the Gradle plugin.

The list of properties configurable differs slightly from the one of the command line. Gradle takes care of source and class path.

The modification status of a class often depends on foreign method calls: calls on objects defined outside your own code base. In some situations, their source code is available, but that need not be the case. So to build up a fair picture, one could say that the e2immu analyser would have to parse all types and methods "from the ground up", and decide on their modification status in an incremental way. This procedure would be slow and hugely impractical.

On top of the incremental problem, APIs often come in the form of an interface without implementation. Expressing the modification status by hand (the terminology we use is contracting, or writing the contract) is the only way forward.

Thirdly, manually annotating APIs can help you override the implied modification status, or any other feature supported by the analyser. Here are two simple examples why that can come in handy:

Here, we acknowledge that add modifies the list, but we prohibit passing null as an element. Secondly, we may decide to ignore modifications to the output stream System.out, by writing

so that we can still claim that the following method is not modifying:

It should be clear from these examples, and from the terminology used throughout the rest of the documentation, that we express the modification status, and other aspects of the code, by means of Java annotations.

In this section, we describe a practical way of annotating foreign APIs; we call it simply Annotated API files. The next section deals with a more compact form, annotation XML files, which are less readable but (maybe) simpler and faster to load.

Should we then start to systematically annotate all the libraries that our project uses? That is one way of approaching the problem; however, because we have a source code analyser at hand, we can easily detect exactly which types, methods, and fields are used by our code. The analyser’s command line interpreter (CLI) provides options for generating a template Annotated API file for a whole library, for selected packages, or for exactly those types, methods and fields required.

The following Gradle task, taken from the e2immu/annotation-store project, contains the code to produce a single IoVertxCore.java file:

The task can be run with the command ./gradlew -b build-api.gradle runIoVertxAAPI.

Annotated API files are standard Java files, they will be inspected by a standard Java parser, so all normal syntax rules need to be followed. They deviate in the following way:

This excerpt from the annotated API file for java.util used by the tests in the analyser, shows what this looks like:

Once all relevant types, methods and fields can be written, they can be annotated, as in:

Also on display here are Companion methods, static methods describing either how the state of a Collection instance changes after a modifying method call, or certain edge cases can be resolved using this state information.

All e2immu annotations have a parameter of the enum type AnnotationType, which takes 4 different values:

The list of available annotations can be found here. In Annotated API files, CONTRACT is the default type, and needs not be specified.

Modification of parameters is seen as any modification to the entire object graph of the parameter.

Modification of a field is seen as a modification to the accessible content of the field, i.e., as a modification which happens explicitly inside the type of the field.

A method is modifying when it makes a modification to any of the fields of the type of the method. Modifications to a field of a different type within the same primary type TODO.

A modification to a (necessarily non-private) foreign static field is called a static side effect. Type which has methods which make static side effects, is marked with @StaticSideEffects . Static side effect marking can be disabled by annotating a static, non-private field with @IgnoreModifications .

We divide the object graph of the fields into an accessible part, and a hidden part. The depth of the object graph plays the primary role in separating both parts: most implementations do not access all types that are theoretically reachable from the fields. Secondly, we note that every class C that can be sub-classed, i.e., which has not been marked final, and every interface I, can have a subclass or implementation S which has fields that are outside its reach. So when faced with the formal type C or I, a concrete object of type S may be present. This object carries a part hidden to the methods and field accessors of C or I.

We speak of the hidden content of the types when we refer to the hidden part of its fields' object graph. Hidden contents are omni-present in Java.

In e2immu , the hidden content of a type T is computed on a type-by-type basis: it consists of a set of parameterized types H1, …​, Hn. A type H belongs to the hidden content of T when it is part of the object graph of the fields, and it is not explicit in T.

Formally, a type E is explicit in T when either

If the type S extends the type C, then types which are transparent in C may not be transparent in S anymore. The type S can also introduce new transparent types. However, a type cannot be transparent in S and explicit in C: all explicit types of C are also explicit in S.

If a type N is a nested class of an enclosing type E, then all explicit types of N are also explicit in E. A type E and its inner class I (nested, not static) share the same set of transparent and explicit types.

For practical reasons, java.lang.Object, java.lang.String and java.lang.Class are always explicit.

A type is a container when only private methods or private constructors modify their parameters. To the outside world, all parameters must be @NotModified.

When a modifying method is called on a parameter, but this modification is semantically deemed to be irrelevant, then @IgnoreModifications can be added to the parameter. This is implicitly the case when the parameter is of one of the abstract types in java.util.function, such as Function, Consumer, etc. The modifications caused by their concrete implementations are typically outside the scope of the type. The typical example of this situation is the forEach method of the java.util.Collection classes: iterating over the collection should be non-modifying, and modifications to the hidden content are semantically irrelevant to the collection.

Two variables are linked when a modification in one may imply a modification in the other. We differentiate between modifications to the accessible content (normal linking), and modifications to the hidden content (hidden content linking).

Linking typically occurs when the object graphs of the variables (partially) overlaps.

A method is dependent when one of the fields of the type links to the return value of the method. A constructor or method parameter is dependent when it links to one of the fields of the type.

A method or parameter is independent when it is not dependent. However, given the possibility of hidden content linking, we differentiate between different levels of independence. When the hidden content of the field being linked to the return value or parameter is mutable or level 1 immutable, we speak of level 1 independence, and mark with {independent1}. When the hidden content is level 2 immutable, we speak of level 2 independence, and write @Independent1(level=2). When no modifications are possible at all, the hidden content must be recursively immutable, and we speak of independence, marked as @Independent .

The computation of linking follows the rules:

The computation of hidden content linking is similar but not quite identical:

TODO

TODO

For each of the annotations, we answer a couple of standard questions:

This classification hopefully helps to see the wood for the trees in the long list.

In this section we summarize the relations between annotations.

The errors described in this section relate to bad practices that are not tolerated by the analyser:

Errors of this type are typically trivial to clean up, so why not do it immediately?

The following errors relate to the annotations you added to your source code, to verify that a type, method or field has a given property.

The main orchestrating class of e2immu is the Parser class, which initiates the different phases of e2immu :

The key classes for configuration and input preparation are, easily enough, Configuration, Input, Resources, ClassPath.

The e2immu project currently depends on two libraries to help transform the Java source code and byte code into objects: JavaParser for the source, and ASM for the byte code. Currently, the byte code inspector does not inspect statements.

Byte code inspection is done on-demand, because the system libraries provided with the Java Runtime are too extensive to convert a priori. The main class responsible is the ByteCodeInspector.

Inspection using JavaParser is primarily carried out by the TypeInspector and the MethodInspector.

Both inspection types fill up a TypeMap (which exists as a builder and a final, immutable implemention constructed after the resolution phase), which keeps track of all types known to e2immu . A TypeContext object provides support for translating locally known simple type names to fully qualified ones.

The end result of inspection are the types

The resolution phase has two primary goals. Firstly, it parses method bodies into statements (implementations of the interface Statement), starting with Block, and expressions (implementations of Expression). Central to the inspection of expressions is the ExpressionContext. Secondly, the resolution phase resolves all remaining type and method determination issues:

Even though JavaParser has its own symbol-solving module, e2immu uses its own implementation. The most complicated of the resolution classes is ParseMethodCallExpr, which determines the exact method call. A lot of logic dealing with the type hierarchy is in ParameterizedType, the class responsible for representing types decorated with concrete values for type parameters.

The Resolver is the main class here, it adds TypeResolution to TypeInfo, and MethodResolution to MethodInfo.

The output of the resolution phase is a list of types, sorted in order of dependency. This sorting step is necessary for analysis, to the extent that circular dependencies between types make analysis a lot harder. The class SortedType transfers information from the Resolver into the PrimaryTypeAnalyser.

In e2immu , a number of different analysers cooperate to determine the semantic properties of the code:

There are obvious and strong dependencies between these analysers, e.g., the modification aspect of a field depends on determining whether there are statements that modify it. At the same time, there are dependencies between different aspects of the code, e.g., between methods calling other methods, methods initialising fields, etc. There is no easy way to determine an exact execution order for this multitude of analyser instances.

To this end, the analyser implements an iterative approach, that gives each analyser one opportunity per iteration to determine its values and properties based on previously established facts.

When a value or property cannot yet be determined, typically because a dependency has not yet been met, this value or property is delayed. The delay system keeps track of the causes of the delay, to allow for detecting circular dependencies. The latter can then be broken, to allow the iterative system to continue. As soon as a value has been determined, it is written out for others to refer to, and cannot be changed anymore.

The analysers themselves consist of a number of components (or tasks), which are executed as part of the iteration. This execution takes place in a fixed order; it starts at the first, not-yet-executed (in the first iteration) or non-delayed (in the subsequent ones) component, and visits all others in the chain. The main loop stops iterating when all components have been resolved (are in the done state, rather than the delayed state).

The order of the main analyser groups is

Inside the primary type (the .java file) the methods are processed alphabetically, as are the fields and nested types. This orchestration is the responsibility of the PrimaryTypeAnalyser, which is also used, recursively, to analyse types defined inside statements. It employs the AnalyserComponents class to execute the components.

Annotated API files (or classes) are analysed by the AnnotatedAPIAnalyser, which is a shallow version of the primary type analyser, the type analyser (ComputingTypeAnalyser) and the field analyser (FieldAnalyser). Methods without code are analysed by the ShallowMethodAnalyser, rather than the ComputingMethodAnalyser and ComputingParameterAnalyser.

In the special situation of sealed classes, we need a variant on the shallow analysers which aggregate data from the computed analysers in the child types. These are the AggregatingTypeAnalyser, AggregatingMethodAnalyser, and AggregatingParameterAnalyser.

Due to the hierarchical nature of statements, statement analysers and statement analysis objects are also structured hierarchically. The statements of a method are internally numbered from 0 onward. Because of the hierarchical nature, a dotted system is used: each sub-block introduces a dot, a sub-block number, and a dot again. So 0.1.2 indicates the 3rd statement in the 2nd block of the first statement of the method. The second block can be a "catch"-block, or the "else" block in an "if-else" statement.

The general Statement interface has an implementation per type of statement specific to the Java language. Many of them contain expressions, represented by the Expression interface. Again, the typical range of expression implementations exist, mostly corresponding to expressions existing in the language. Specific to this implementation are DelayedExpression, DelayedVariableExpression, and PropertyWrapper.

Expressions can be evaluated in the context in which they appear, the EvaluationContext. The result of this evaluation is an EvaluationResult object, which is subsequently processed by the statement analyser. When dependencies inside the expression have not been resolved yet (e.g., the return value of a method is still unknown, or a variable doesn’t have a value yet), the end result is an expression which contains delayed components. Each expression answers the isDelayed method, and can return a CausesOfDelay object to identify exactly what the reasons for the delay are.

Some evaluation leads to simplification of the expression, which is sometimes a reason to emit a warning to the developer. When a complex expression evaluates to a constant, for example, it is likely that they should know about this.

Each analyser has a corresponding analysis object, which contains the results of the analysis. After analysis, the analyser is dropped, the analysis object remains. Each analysis component consists of two implementations: the builder, which holds the values while the analyser is alive, and the effectively immutable implementation which survives the analyser. There is currently one exception: the StatementAnalyser only has a StatementAnalysis data companion, which holds eventually immutable objects. The MethodAnalysis interface has a MethodAnalysisImpl immutable implementation, and a MethodAnalysisImpl.Builder mutable builder.

An important aspect of the builders is that writing information is constrained: while causes of delay can be overwritten in each iteration, once a value has been determined for a property or some piece of information the analyser has to store, it cannot be changed anymore. To this end, we employ a variation of eventually final helper classes, such as EventuallyFinal and VariableFirstThen, with setVariable and setFinal write methods.

The StatementAnalysis data object holds a number of sub-objects, where data is stored per topic:

The analysers compute semantic information, some of which can be expressed as numeric values for properties, many of which apply to most of the analysers. These properties can then be visualised, either by coloring, highlighting in an IDE, or by adding annotations.

A property (implemented by the enumeration Property) has a numeric value when it is not delayed. Otherwise, it takes a CausesOfDelay value, which enumerates the reasons why no value was computed for this property. The property-value map Properties maps Property to DV. The latter stands for delayable value.

There are four types of properties:

A statement analyser, and associated statement analysis builder object, is present for each statement. The builder holds information about the state of all the variables known to the statement.

Parameters are known from the first statement onwards; fields are only introduced in the statement that refers to them. All subsequent statements will also know the statement.

Each variable in the statement analyser has values for three "levels"

Obviously, not all statements allow for nested statements; they will not have an M level. Information about a variable is stored in a VariableInfoContainer, which holds VariableInfo instances for each of the three levels. A VariableInfo object holds:

When a field is detected during the evaluation phase, it cannot yet have a value at the initial level. A delayed value expression is returned for this field. In the next iteration, the field analyser may provide an initial value for the field.

Variable fields and loop variables have "local copies" which exist starting from the 2nd iteration.

A variable which points to a generic value (of type Instance) is evaluated to a VariableExpression, rather than the value itself. When variable b is assigned to variable a, a can have this VariableExpression as a value. No further redirections are possible.

Variables are identified by their fully qualified name. Local copies can be identified by the $ sign and the suffix, either specifying the latest assignment and read statement ids, or the statement time.

Variables link to other variables at different levels:

The context properties are assigned to clusters of variables: after executing a=b; b.doSomething(), for example, we must set the context not null value of a to not-null as well.

The static assignment level is different from the three other levels in that it is available from the first iteration onward. Local copies of variable fields and variables assigned to in a loop are only introduced in the 2nd iteration; however, they simply expand the cluster and cannot cause different context property values.

The only context property which is computed across the first three levels (i.e., static and dynamic assignment, and linking at the accessible content level) is context modification: after executing s = t.subList(0, 3); s.add(x); both s and t are likely to be modified.

The result of linking is stored in the linkedVariables field of VariableInfo; it is of type LinkedVariables.

Parameters start with values, nullable, mutable in case of self-references; then they rely on ENN, ExtImm.

this also starts with a value, relies on ENN, ExtImm

statical assignment linking → CNN, CIMM - delays only on dependent methods

CondMgr works on delayed values, to assist in correct CNN values

field value ← last statement + initialiser values + CNN, CIMM

in statements, delayed variable value is replaced by field value + IMM + NNE

Rest of linking follows as soon as there are values → CM → type immutable

Breaking a circular computation CNN → field → ENN → NNP → CNN deactivates this local CNN.

ENN, ExtImm can always augment on fields when value was chosen after breaking circular computation

The output system of e2immu is very lightweight, yet sufficiently flexible to be parameterizable between extremely compact and nicely readable. Output elements (implementations of OutputElement, such as Text, Symbol, Space, and Guide) are collected in an OutputBuilder, and finally emitted by the Formatter which is parameterized by FormatterOptions. The OutputBuilder implements the Collector interface from the JDK streaming package, which allows for simple conversion and collection of lists and streams of statements, expressions, types, etc. into OutputElement and then OutputBuilder instances.

At the moment the implementation is not tied to the inspection system, so there is no way to link back to line numbers and symbol positions of the original source. It is also not possible to maintain the existing formatting.

Errors and warnings are stored in an enumeration called Message.Label, and are decorated with a Location and some textual information. They are collected in a Messages object. Resource files allow for translation of the messages in other languages.

The Location object holds an Identifier, which can be of the PositionalIdentifier variety that refers to the line number and position in the original source code, provided by the JavaParser.

The Gradle plugin greatly facilitates the use of the analyser by integrating it your project’s build process. We based its initial implementation on the one from SonarQube.

The list of properties configurable differs slightly from the one of the command line. Gradle takes care of source and class path.

The key-value store is a simple, two-class implementation of an independent key-value store acting as a bridge between the e2immu analyser and the IntelliJ IDEA plugin. It is mostly agnostic to the purpose in the project: it stores key-value pairs in sub-stores called projects.

Unless directed differently by the e2immu-port parameter, the store starts listening on port 8281 for HTTP communication. The protocol summary is:

Projects that do not exist will be created on-demand. The bulk get operation may receive more elements than that it asked for: depending on the effects of recent set operations, the store may include recently updated keys that were asked for recently as well.

Start the store by running:

In another terminal, experiment with the curl statements:

Removing a key-value pair only works via the PUT method:

Without a plugin for an IDE, the analyser would be hard to use, and the main purpose of the project, namely, unobtrusively assisting in better coding, would remain very far off. The current plugin for IntelliJ IDEA is absolutely minimal. We based our initial implementation on the Return statement highlighter plugin by Edoardo Luppi.