Friday, November 6, 2015

Improved Grammar Inheritance

Since the very first day of Xtext, it was possible to extend another grammar to mixin its rule declarations to reuse or specialize them. For most use cases that was straightforward and a perfect match. For others it was rather cumbersome so far because the original declaration was no longer reachable from the sub-language. Copy and paste was the only solution to that problem. The good news? The situation changes with Xtext 2.9 significantly.
The newly introduced super call allows to override a rule and still use its super implementation without the need to duplicate it. Along with super, Xtext 2.9 also provides a way to call inherited or locally declared rules explicitly. Explicit rule calls will overrule the polymorphism that is usually applied in the context of grammar inheritance. As a language library designer you get fine grained control over the syntax, even if your language is supposed to be extended by sub-languages.
But let's look at an example:
grammar SuperDsl
  with org.eclipse.xtext.common.Terminals
  'element' name=ID
  'thing' name=SuperDsl::ID
terminal ID: ('a'..'z')+;

grammar SubDsl with SuperDsl
    super // 1; or super::Element
  | 'element' name=super::ID // 2
  | 'element' name=Terminals::ID // 3
terminal ID: 'id';
Here we see different use cases for the super call and also for qualified rule calls.  The  first occurrence of super (1) illustrates the shortest possible notation to reach out to the super implementation. If you override a rule and want to use the original declaration in the rule's body, you can simply call super from there.
It is also possible to use a qualified::RuleCall. Qualified invocations point directly to the referenced rule. The qualifier can either be a generic super qualifier (2) or an explicit language name (3). The latter provides a way to skip the immediate super language and reach out to its parent. This offers great flexibility. You can ensure that you call the rule from your own grammar, even if a sub-language will override the declaration. The benefit is illustrated by the rule Thing. It calls the ID declaration from SuperDsl explicitly thus it will also do so from the SubDsl. As long as you do not explicitly override the declaration of Thing, its syntax will not change in any inheritor from SuperDsl.
Long story short: super calls add a lot more flexibility for language mixins and greatly reduce the need to copy and paste entire rule bodies in the sub-language. Go ahead and download the latest milestone to give it a try!


Thomas Baar said...

Hi Sebastian,

thanks for this nice post and for (probably, just my guess) implementing the improvements wrt. grammar mixing.

I downloaded the milestone and tried your example. However, I got an error
"Cannot add supertype 'Element' to sealed type 'Element'." in the grammar editor of Sub.
The error disappears when I comment out the access to 'super:

//super // 1; or super::Element
'element' name=super::ID // 2
| 'element' name=Terminals::ID // 3

Unfortunately, I still cannot make the Sub-example run due to the error
"Cannot find GenModel for Sup" while buiding the infrastructure for Sub.
In Xtext 2.8, I knew how to register the super grammar in the mwe2-file.
In Xtext 2.9, however, I simply do not know how to do it, since the format of .mwe2-file seems to have changed drastically (while documentation have not changed

Thus, could you please provide a reference where one can obtain the whole source-code
of your project? Thanks in advanced!

Best regards,

P.S. In Xtext 2.8 I had enormous problems with grammar inheritance when the
inheritance depth was greater than two. So, would your example still work if we, for example,
add another grammar SubSub inheriting from Sub and we would like to refine definitions from Sup and Sub within SubSub?

Sebastian Zarnekow said...

Hi Thomas,

thanks for nice words.
Apparently I oversimplified the example. The overriding rule 'Element' needs a return clause. It will automatically be added if you use content assist to create the rule stub. Eventually it would read similat to this snippet:

Element returns mydsl::Element:

The poor feedback will be better with this feature

Multiple inheritance levels shouldn't be a problem at all. What issues did you face?

Regarding the mwe2 file: Yes, we are still working on the documentation. Please try to register 'referencedResource' in the language configuration.

david said...

Hi Sebastian,

I'm experiencing the same problem with the mwe2 workflow when trying to reuse grammars. With xtext 2.8 I had something like this inside bean = StandaloneSetup{} to reuse an existing grammar:

registerGeneratedEPackage = "org.reuseddsl.DSLPackage"
registerGenModeFile = "platform:/resource/org.reuseddsl/model/generated/MyDSL.genmodel"

What do I do with these lines in XText 2.9?

Best regards,

Aykut K said...

Thanks for the article and the improvement.

How I used to handle that was really ugly. I put an extension point to the parent grammar (some rule that avoids matching anything).

Literal returns Expr:
StringLiteral |
VariableRefLiteral |

LiteralExtension hidden(): '##LiteralExtensionDoNotMatch##';

Then override LiteralExtension in the sub-grammar.

Regarding this feature, is it possible to have more than 2 levels?

G0 -> Element: ...
G1 extends G0 -> Element: super ...
G2 extends G1 -> Element: super ...

Nesredin Mahmud said...

xtext 2.8.4

I want to extend a super grammar rule. That is add sub rules (delegate) on top of existing rules on the super grammar.

super // 1;
'element' name=super::ID // 2
| 'element' name=Terminals::ID // 3
error "super cannot be resolved to a rule"!?
I thought 'super' was a predefined concept in xtext...

Tnx :)

Nesredin Mahmud said...

I used the wrong version. 'super' does work on the 2.9 :) perfect!

Knut Wannheden said...

Hi Sebastian,

Looks interesting. I wonder what implications use case (3) will have. Specifically when referencing terminal rules, will this mean that the parser for the sub grammar will have both terminal rules and that the sub grammar is responsible to make sure they don't conflict?

Sebastian Zarnekow said...

Hey Knut,

yes, that's exactly how its supposed to work. If you use super::TerminalRule somewhere, the overriding terminal must not conflict with the overridden terminal. One notable exception: if super::TerminalRule is only used from other terminal rules, it becomes a terminal fragment in the generated Antlr stuff. It won't conflict with any other terminal in that case.

aliya seen said...

I want someone to fix my sentence fragment also that must be live with correct english.

rohit nagar said...

Hi Sebastian,

I want to write grammar in multiple xtext file and later I want to use all these file into single xtext file. mean I want to use multiple inheritance but I am not able to do that. Shall I have to use only one grammar file with another grammar file?

obat gatal kulit said...

Yes I’m seeking for someone, to help me. So that some day I will be the someone to help some other one.
Obat kencing nanah di apotik