A scheme for lr(k) parsing with error recovery part iii: error correction

The paper is the third in a series of three papers devoted to a detailed study of LR(k) parsing with error recovery and correction. A new class of syntax errors is introduced, called (k)-local parser defined errors, which suit better than the conventional minimum distance errors for characterization of error detection and recovery in LR(k) parsing. The question whether a given string has n k-local parser defined errors for some integer n is shown to be decidable. Using the formalization of LR(k) parsing and error recovery presented in the first and the second paper in the series it is shown that the canonical LR(k) parser of an LR(k) grammar always has an error recovering extension which is able to produce a correction for any terminal string containing only (k)-local parser defined errors.     

A method for transforming grammars into LL(k) form

Summary A new method for transforming grammars into equivalent LL(k) grammars is studied. The applicability of the transformation is characterized by defining a subclass of LR(k) grammars, called predictive LR(k) grammars, with the property that a grammar is predictive LR(k) if and only if the corresponding transformed grammar is LL(k). Furthermore, it is shown that deterministic bottom-up parsing of a predictive LR(k) grammar can be done by the LL(k) parser of the transformed grammar. This parsing method is possible since the transformed grammar always left-to-right covers the original grammar. The class of predictive LR(k) grammars strictly includes the class of LC(k) grammars (the grammars that can be parsed deterministically in the left-corner manner). Thus our transformation is more powerful than the one previously available, which transforms LC(k) grammars into LL(k) form.     

Ll-regular grammars

Culik II and Cohen introduced the class of LR-regular grammars, an extension of the LR(k) grammars.

In this paper we consider an analogous extension of the LL(k) grammars called the LL- regular grammars. The relation of this class of grammars to other classes of grammars will be shown. Any LL-regular grammars is an LR-regular grammar. Properties of LL(k) grammars can be generalized to properties of LL-regular grammars.     

Lar(m,c, l) grammars

We present the class of LAR(M, C, L) context-free grammars: a generalization of several classes of fixed and arbitrary lookahead LR grammars that appear in the literature. The parser construction is based on three parameters M, C and L; M and C determine the type of parser, and L is the amount of lookahead. Specific settings of these parameters yield fixed-lookahead grammar classes such as LALR(k), SLR(k) and NQLALR, along with several arbitrary lookahead classes, all of which are subsets of LR-Regular. Thus both fixed and arbitrary lookahead LR techniques are described (and implemented) by one powerful model.     

Dynamic LL(k) parsing

A new class of context-free grammars, called dynamic context-free grammars, is introduced. These grammars have the ability to change the set of production rules dynamically during the derivation of some terminal string. The notion of LL() parsing is adapted to this grammar model. We show that dynamic LL() parsers are as powerful as LR() parsers, i.e. that they are capable to analyze every deterministic context-free language while using only one symbol of lookahead. Received: 24 August 1994 / 5 January 1996     

A new error repair and recovery scheme for lexical and syntactic analysis

This paper describes a two-level error repair and recovery scheme applicable to table- driven parsers and scanners. For parsers, the first level of the scheme tries to locally correct erroneous text by performing insertions, deletions and replacements of tokens around the error detection point, and matching these source modifications against an ordered list of correction models. If this local repair of text fails, a global recovery is initiated, which skips the text up to a “key terminal” and pops the parse stack. Lexical error processing is based on similar principles. The main advantages of the scheme are its power, efficiency and language independence. It can be parameterized by the grammar writer, uses the normal analysis tables and does not slow down the analysis of correct portions of text. Furthermore it can be easily implemented in automatically generated analysers such as the ones constructed by our system SYNTAX.     

Agile Parsing in TXL

Syntactic analysis forms a foundation of many source analysis and reverse engineering tools. However, a single standard grammar is not always appropriate for all source analysis and manipulation tasks. Small custom modifications to the grammar can make the programs used to implement these tasks simpler, clearer and more efficient. This leads to a new paradigm for programming these tools: agile parsing. In agile parsing the effective grammar used by a particular tool is a combination of two parts: the standard base grammar for the input language, and a set of explicit grammar overrides that modify the parse to support the task at hand. This paper introduces the basic techniques of agile parsing in TXL and discusses several industry proven techniques for exploiting agile parsing in software source analysis and transformation.     

Object-oriented Tree Traversal with JJForester

We want to use the advanced language processing technology available in the asf+sdf Meta-Environment in combination with general purpose programming languages. In particular, we want to combine the syntax definition formalism sdf and the associated components that support generalized LR parsing, with the object-oriented language Java. To this end, we implemented JJForester, a tool that generates class structures from sdf grammar definitions. The generated class structures implement a number of design patterns to facilitate construction and traversal of parse trees represented by object structures. In a detailed case study, we demonstrate how program analyses and transformations can be constructed with JJForester.     

Sequential classification of strings generated by SCFG's

A sequential classification algorithm (SCA) for stochastic context-free languages is given. The algorithm is an extension of Earley's parsing algorithm for context-free languages. The SCA uses an optimum decision rule and a suboptimal stopping rule. The time bound is proportional ton ³ (n is the length of the string). A parametere* in the SCA allows an easy control over the probability of error. It is shown theoretically and experimentally that a considerable amount of processing time can be saved when using the SCA compared to the time required when using Earley's algorithm.This work was supported by the AFOSR Grant 74–2661.     

Efficient LL(1) error correction and recovery using only insertions

An LL(1)-based error-corrector which operates by insertion-only is studied. The corrector is able to correct and parse any input string. It is efficient (linear in space and time requirements) and chooses least-cost insertions (as defined by the user) in correcting syntax errors. Moreover, the error-corrector can be generated automatically from the grammar and a table of terminal symbol insertion costs. This method is also very well suited for use as an automatic error-recovery technique in LL(1) parsers. The class of LL(1) grammars correctable by this method contains (with minor modifications) grammars used to specify most common programming languages. Preliminary results suggest that this method can be used to advantage in LL(1)-driven compilers.A preliminary version of this paper was presented at the Fourth ACM Symposium on Principles of Programming LanguagesResearch supported in part by National Science Foundation Grant MCS78-02570     

Software visualization of LR parsing and synthesized attribute evaluation

Visual YACC is a tool that automatically creates visualizations of the YACC LR parsing process and synthesized attribute computation. The Visual YACC tool works by instrumenting a standard YACC grammar with graphics calls that draw the appropriate data structures given the current actions by the parser. The new grammar is processed by the YACC tools and the resulting parser displays the parse stack and parse tree for every step of the parsing process of a given input string. Visual YACC was initially designed to be used in compiler construction courses to supplement the teaching of parsing and syntax directed evaluation. We have also found it to be useful in the difficult task of debugging YACC grammars. In this paper, we describe this tool and how it is used in both contexts. We also detail two different implementations of this tool: one that produces a parser written in C with calls to Motif; and a second implementation that generates Java source code. Copyright © 1999 John Wiley & Sons, Ltd.     

A new parsing method for non-LR(1) grammars

One of the difficult problems that faces a compiler writer is to devise a grammar that is suitable for both efficient parsing and semantic attribution. This paper describes a system that resolves conflicts in LR(1) parsing by taking advantage of information in the parse tree. The system, which functions as part of a compiler generator, rewrites the user's grammar to remove parsing conflicts. It then places code into the generated compiler that rewrites the parse tree during parsing so as to produce the tree of the original grammar. The compiler writer can then write the semantic attribution to fit his or her original grammar without any knowledge of the changes made. The method is expected to be efficient in most cases, even in parsing systems that do not explicitly build the entire parse tree. The method complements previous work in its capabilities and advantages. The system has been implemented and integrated into a compiler generator system.     

Methods for the automatic construction of error correcting parsers

Summary Methods for the automatic construction of error handling parsers are presented. The resulting parsers are capable of correcting all syntax errors by insertion and/or deletion of terminal symbols to the right of the error location. Thus, the output of the parser always corresponds to a syntactically valid program. This contributes significantly to the reliability and robustness of a compiler. The speed of parsing correct parts of a program is not affected by the presence of the error handling capability. The correction algorithm is easy to implement. Apart from the parsing tables only one character per parser state is required to control the correction process. The method is applicable to a wide class of stack automata including LL(k), LR(k), SLR(k), and LALR(k) parsers. It is shown that for LL(k) grammars error correction can be obtained as a byproduct of the canonical LL(k) parser generation. A similar result can be obtained for LR(k) grammars if the parser generator is slightly modified. The method has been successfully added to an LALR(1) parser generator.     

Two iteration theorems for the LL(k) languages

The structure of derivation trees over an LL(k) grammar is explored and a property of these trees obtained which is shown to characterize the LL(k) grammars. This characterization, called the LL(k) Left Part Theorem, makes it possible to establish a pair of iteration theorems for the LL(k) languages. These theorems provide a general and powerful method of showing that a language is not LL(k) when that is the case. They thus provide for the first time a flexible tool with which to explore the structure of the LL(k) languages and with which to discriminate between the LL(k) and LR(k) language classes.Examples are given of LR(k) languages which, for various reasons, fail to be LL(k). Easy and rigorous proofs to this effect are given using our LL(k) iteration theorems. In particular, it is proven that the dangling-ELSE construct allowed in PL/I and Pascal cannot be generated by any LL(k) grammar. We also give a new and straightforward proof based on the LL(k) Left Part Theorem that every LL(k) grammar is LR(k).     

A music notation construction engine for optical music recognition

Optical music recognition (OMR) systems are used to convert music scanned from paper into a format suitable for playing or editing on a computer. These systems generally have two phases: recognizing the graphical symbols (such as note‐heads and lines) and determining the musical meaning and relationships of the symbols (such as the pitch and rhythm of the notes). In this paper we explore the second phase and give a two‐step approach that admits an economical representation of the parsing rules for the system. The approach is flexible and allows the system to be extended to new notations with little effort—the current system can parse common music notation, Sacred Harp notation and plainsong. It is based on a string grammar and a customizable graph that specifies relationships between musical objects. We observe that this graph can be related to printing as well as recognizing music notation, bringing the opportunity for cross‐fertilization between the two areas of research. Copyright © 2003 John Wiley & Sons, Ltd.     

Parsing regular grammars with finite lookahead

Summary Not every unambiguous regular grammar can be parsed by a finite state machine, even if a lookahead facility is added to the machine's capabilities. Those which can be parsed with a fixed lookahead of k are said to be FL(k). If such a grammar has n non-terminals, it never needs more than (n(n–1)/2) + 1 lookahead, and there exist grammars which do require this much. An algorithm is presented for determining whether a grammar is fixed lookahead parsable, and if so, for finding the minimum lookahead needed. The algorithm sets up and solves a set of O(n²) equations using O(n⁴) steps. Two parsing methods for FL(k) grammars are discussed. One uses a large precomputed parsing table, and operates in real time. The second parses an input string in time proportional to its length, while using approximately 3n storage locations.     

采用预测策略的Earley算法

在自然语言处理中,句法分析主要有基于统计的方法和基于规则的方法。Earley算法是一种基于规则的方法,可以分析任意上下文无关文法(CFG),而不需要对文法进行修改。详细分析了Earley算法的特点。在通常的Earley算法中增加了多种预测机制,这些预测机制借鉴了LL,LR以及SLR等确定性分析算法的一些思想,并对这几种不同的预测机制及其组合在相同条件下进行了中文句法分析实验。结果显示,引入这些预测机制通常可以减少产生项目的数量,从而节省存储空间,减少运行时间。     

A language independent error recovery method for LL(1) parsers

An efficient and systematic LL(1) error recovery method is presented that has been implemented for an LL(1) parser generator. Error messages which provide good diagnostic information are generated automatically. Error correction is done by discarding some input symbols and popping up some symbols from the parsing-stack in order to restore the parser to a valid configuration. Thus, symbol deletions and insertions are simulated. The choice between different possible corrections is made by comparing the cost of the inserted (popped) symbols with the reliability value of the recovery symbol (the first input symbol that is not discarded). Our concept of reliability is based on the observation that input symbols differ from each other in their ability to serve as recovery points. A high reliability value of a symbol asserts that it was probably not placed in the input by accident. So it is reasonable not to discard that symbol but to resume parsing. This is done even if a string with high insert-cost has to be inserted before that symbol in order to fit it to the part of the program that has already been analysed. The error recovery routine is invoked only when an error is detected. Thus, there is no additional time required for parsing correct programs. Error-correcting parsers for different languages, including Pascal, have been generated. Some experimental results are summarized.