tl;drFor the upcoming version of Xtext we are revising the approach to building. It appears to be promising to rethink the overall lifecycle of the Xtext builder to aim at:
- Better user experience by introducing a non-blocking build infrastructure
- Improved performance due to improved parallelization
- Incremental builds also on the command line
The ProblemThe Xtext framework implements an Eclipse builder and is thereby immediately affected by the builder's user experience metaphor (even bad experience is still experience). Whenever a file is saved or a project is explicitly built, the user is essentially locked out from doing work in the editor.
Go Home, Eclipse! You're drunk!That's not because the editor isn't generally usable during the build. But it turns out, that it becomes quite a habit to Eclipse users to save early and save often. As soon as you wrote some code and you save the file that you're working on, the builder kicks in and tries to validate the new file state. Since you are continuing to edit, it's quite likely that you hit save again and are confronted with that modal dialog with greetings from the 90s. Of course you don't see this message all the time when you save a file since the incremental build is usually quite fast, but when you see it, it is definitely not what you expected.
Generally speaking, the Eclipse builder is responsible to produce artifacts from the source files in a project. There may be different builders configured for the very same project and the term artifact does not only describe compilation results in the form of new files, but also validation markers. While a builder is running for a project, it holds an acquired lock not only for that project including its contained files and folders but for the entire workspace. This ensures that there are no intermitted events that remove or modify any state on disk (details have been discussed here). And this is where the trouble starts from the users perspective.
On the one hand, the locking prevents from unexpected modifications within Eclipse, on the other hand it gets in the way of users since they can no longer work without interruption. The thing was apparently designed to ensure consistency within the workspace between sources and compilation result. But if you look into the dirty corners, the paid price is way too high. The blocking mechanism introduces only the impression of safety but can never guarantee it. Literally every external process may still perform I/O operations on the very same files and the build would go bananas since the state known to Eclipse is no longer in sync with the actual state on disk. But that's probably another can of worms that is not subject of this post. Instead let's focus on ways to improve the situation which may lead to a more responsive UI.
For Xtext, we are currently analyzing how we can change the way we build files and projects. Rather than getting in the way of the user, we are thinking about performing the build in the background without unnecessary blocking. The main goal with that regards is to move the complete build out of the coarse grained project lock and break it into manageable, smaller pieces. E.g. as soon as the files are loaded, they don't need to be locked anymore. In the validation phase only the markers are written but not the entire files. For incremental builds, only a small subset of files needs to be considered in the first place.
This breakdown of locking is desirable on various level. First and foremost, the user experience would be improved a lot since Xtext would present fewer blocking dialogs to the user. Another positive effect is that the build and its lifecycle would be essentially decoupled from the Eclipse builder and its related UI components. By factoring out the build cycle, Xtext can support incremental compilation on the command line, too.
In times of many-cores, it also becomes more and more interesting to parallelize the build to go full throttle with todays CPUs. The leverage the potential there, the build process itself has to be analyzed carefully. The Xtext build inherently runs in multiple passes that are currently strictly sequential, especially in the context of Eclipse projects. These steps are performed for each individual project during a build.
- First of all, the resources in a project have to be indexed to collect all the reachable names and derive an index structure that can be seen as a global symbol table.
- After all symbols and names are known, the cross references are resolved and their resolution state is cached as reference descriptions. Currently also the validation is performed on that stage but that can be seen as step 2.5
- The last step is the code generation. All resources are processed to create derived artifacts from them.
But there's even more that we are discussing right now about the way Xtext projects are build within Eclipse. We are looking into means to preserve the index state if a project is closed by the user, for example. Instead of rebuilding the entire project, the builder state would be available immediately after the project is reopened, similar as with plain Java projects. Also the general handling of archives and resources in these archives is under review. For bigger projects, it may pay off to have precomputed information available that is packaged together with the resources in the archive.
In the end, the overall goal is to improve the perceived performance and the responsiveness of the IDE. Never ever should a user action be blocked by some task the IDE is performing in the background. The build should also be decoupled from the Eclipse infrastructure. With that regards, the contracts for each build step have to be sharpened and of course correctness should not be traded for concurrency. Exciting times!