From: W. Trevor King Date: Tue, 5 Oct 2010 13:10:49 +0000 (-0400) Subject: Added PLY, yacc2dot, and inline pages. X-Git-Url: http://git.tremily.us/?a=commitdiff_plain;h=ba9185cb0e83d226c46a223a668b7eefd52811f2;p=blog.git Added PLY, yacc2dot, and inline pages. Also added a Makefile to build generated files (e.g. posts/yacc2dot/*.png). --- diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..aca3484 --- /dev/null +++ b/Makefile @@ -0,0 +1,11 @@ +# Build everything necessary to compile the wiki. + +SUBDIRS = posts/yacc2dot +TARGETS = all clean + +all : + +$(TARGETS) : + @for i in $(SUBDIRS); do \ + echo "make $@ in $$i..."; \ + (cd $$i; $(MAKE) $(MFLAGS) $@); done diff --git a/ikiwiki.setup b/ikiwiki.setup index 4d3203a..64d2e02 100644 --- a/ikiwiki.setup +++ b/ikiwiki.setup @@ -29,7 +29,7 @@ use IkiWiki::Setup::Standard { # rcs backend to use rcs => 'git', # plugins to add to the default configuration - add_plugins => [qw{goodstuff lockedit mdwn_itex sidebar}], + add_plugins => [qw{goodstuff linktoimgonly lockedit mdwn_itex sidebar}], # plugins to disable disable_plugins => [qw{editpage htmlscrubber passwordauth smiley}], # location of template files @@ -108,6 +108,8 @@ use IkiWiki::Setup::Standard { #allowatom => 0, # urls to ping (using XML-RPC) on feed update #pingurl => 'http://rpc.technorati.com/rpc/ping', + # templates for which you want raw content + raw_templates => [qw{raw}], # lockedit plugin # PageSpec controlling which pages are locked diff --git a/posts/PLY.mdwn b/posts/PLY.mdwn new file mode 100644 index 0000000..629bd73 --- /dev/null +++ b/posts/PLY.mdwn @@ -0,0 +1,10 @@ +David Beazley has written up a pure-python Lex/YACC module +called... [Python Lex/YACC][PLY] (or PLY). If you haven't worked with +YACC-type parsers before, you might want to look at my [[implicit +multiplication notes|yacc2dot]]. + +[PLY]: http://www.dabeaz.com/ply/ + +[[!tag tags/programming]] +[[!tag tags/python]] +[[!tag tags/tools]] diff --git a/posts/inline.mdwn b/posts/inline.mdwn new file mode 100644 index 0000000..4600cb9 --- /dev/null +++ b/posts/inline.mdwn @@ -0,0 +1,12 @@ +While writing the [[yacc2dot]] page I ran across some difficulties +inlining raw files (i.e. those whose type/extension ikiwiki doesn't +know about). Fixed with the following changes: + +* Altered `inline.pm` which reads the content of raw files if the + selected template is in the `raw_templates` config array. +* New `raw.tmpl` to minimally format the raw contents (formatting + still needs work). + +Changes made in my [branch][] of the IkiWiki Git repository. + +[branch]: http://www.physics.drexel.edu./~wking/code/git/git.php?p=ikiwiki.git diff --git a/posts/yacc2dot.mdwn b/posts/yacc2dot.mdwn new file mode 100644 index 0000000..cea1306 --- /dev/null +++ b/posts/yacc2dot.mdwn @@ -0,0 +1,155 @@ +YACC precedence and shift/reduce errors +======================================= + +While I was working on a BibTeX parser, I ran into some trouble with +shift/reduce errors in `yacc`. The problem essentially boiled down +to: How do you define implicit multiplication in `yacc`? + +Keep reading for my intro to `yacc`, or, if you just want the simple +solution, [skip to the end](#implicit_multiplication). + +Related work +------------ + +After much googling about, I would like to recommend the following sources: + +* David Beazley's [PLY documentation][DB] which has an good + introduction to shift/reduce errors with a nice walkthrough + shift/reduce example; +* Stephen Johnson's [yacc introduction][SJ] (see Section 4: How the + Parser Works), which helped me understand the state-stack concept + which hadn't sunk in until then; +* Leon Kaplan's [yaccviso documenation][LK], which reminded me that + visualizations are good, and inspired my resurrection of the + `yacc2dot` tool in Python; and +* Xavier Leroy et al.'s [ocamlyacc documentation][XL], + where rule-vs-operator precedence finally clicked. + +For further details and entertaining browsing, you can take a slow +meander through the [Bison manual][bison]. + +Tools +----- + +The following graphics were generated from the `y.output` +files output by `yacc` using [my yacc2dot script](#yacc2dot) and +[Graphviz dot][graphviz]. + +Example 1: Single, explicit addition +------------------------------------ + +As a brief introduction to how parsers work, consider the *really* +simple grammar + +[[!inline pagenames="p.y" template="raw" feeds="no"]] + +which generates the following `yacc` output and `dot` image: + +[[!inline pagenames="p.output" template="raw" feeds="no"]] +[[!img p.png alt="Visualization of plus-only grammar" + title="Visualization of plus-only grammar" ]] + +Parsing input like `X PLUS X` would look like + + + + + + + + + +
StepState stackSymbol stackSymbol queueAction
00$acceptX PLUS X $endShift X
101$accept XPLUS X $endShift PLUS
2013$accept X PLUSX $endShift X
30135$accept X PLUS X$endReduce using rule 1, pop back to state 0, and goto state 2
402$accept expr$endShift $end
5024$accept expr $endAccept using rule 0
+ +The key step here is 3-to-4. We reduce using Rule 1 + + (1) expr: X PLUS X + +Since there are three symbols on the rule's right hand side, we pop +three symbols from the stack. The symbols we pop match the right hand +side, which is why we can apply the rule. We also pop three *states* +(1, 3, and 5), which returns us to state 0. We look up state 0's +action for rules reducing to expr, and see that we're supposed to go +to state 2, so we do. Whew! + +It's a lot of words, and not very intuitive for me the first time +around, but the underlying idea is pretty simple. + +The tokens `$accept` and `$end` are inserted by the parser so it knows +if the symbols left after parsing are acceptable or not. + +The input token string `X PLUS X` is the only valid token string for +this grammar. For extra credit, you can figure out where other input +strings go wrong. For example, consider `X` and `X PLUS X PLUS X`. + +More examples +------------- + +In my gradual work up to implicit multiplication, I went through the +following progressively more complicated examples. + + + + + + + + + + + + + + + + + + + +
grammaryacc outputyacc2dot outputnotes
[[pr.y]][[pr.output]][[!ltio pr.png]]Recursive addition (allow X PLUS X PLUS X PLUS ...). Already + a shift/reduce error!
[[pr_prec.y]][[pr_prec.output]][[!ltio pr_prec.png]]Add precedence (PLUS > X) to solve in favor of reduction.
[[pt.y]][[pt.output]][[!ltio pt.png]]Explicit addition and multiplication with precedence.
[[t_implicit.y]][[t_implicit.output]][[!ltio t_implicit.png]]Implicit multiplication. Shift/reduce error, but no operator handle + for assigning higher precedence to reduction...
+ + +Implicit multiplication +----------------------- + +The lack of an operator handle to assign precedence to an +operator-less rule had me stumped for a few days, until I found the +[OCaml site][XL] with a nice, explicit listing of the shift/reduce +decision process. + +The shift/reduce decision is not a problem of which operator has a +higher precedence, but about whether a given *token* has a higher +precedence than a given *rule*. All we have to do is assign +precedence to *both* the tokens *and* the rules. + +[[!inline pagenames="t_implicit_prec.y" template="raw" feeds="no"]] + +which generates the following `yacc` output and `dot` image: + +[[!inline pagenames="t_implicit_prec.output" template="raw" feeds="no"]] + +[[!img t_implicit_prec.png + alt="Visualization of implicit_multiplication grammar" + title="Visualization of implicit multiplication grammar" ]] + + +yacc2dot +-------- + +I'll post my `yacc2dot` code once I polish it up a little bit. +Take a look at [yaccviso][] if you're impatient. + + +[DB]: http://www.dabeaz.com/ply/ply.html#ply_nn23 +[SJ]: http://dinosaur.compilertools.net/yacc/index.html +[LK]: http://www.lo-res.org/~aaron/yaccviso/docu.pdf +[XL]: http://caml.inria.fr/pub/docs/manual-ocaml/manual026.html#htoc140 +[bison]: http://www.gnu.org/software/bison/manual/ +[graphviz]: http://www.graphviz.org/ +[yaccviso]: http://directory.fsf.org/project/yaccviso/ + +[[!tag tags/programming]] +[[!tag tags/python]] +[[!tag tags/tools]] diff --git a/posts/yacc2dot/Makefile b/posts/yacc2dot/Makefile new file mode 100644 index 0000000..29e3af3 --- /dev/null +++ b/posts/yacc2dot/Makefile @@ -0,0 +1,12 @@ +SOURCES = $(shell ls *.y) + +all : $(SOURCES:%.y=%.output) $(SOURCES:%.y=%.png) + +%.output : %.y + yacc -v "$<" && mv y.output "$@" + +%.png : %.output yacc2dot + cat "$<" | ./yacc2dot | dot -Tpng > "$@" + +clean : + rm -f *.png *.output y.tab.c y.tab.h diff --git a/posts/yacc2dot/README b/posts/yacc2dot/README new file mode 100644 index 0000000..71f1e58 --- /dev/null +++ b/posts/yacc2dot/README @@ -0,0 +1,52 @@ +The best explaination I've seen so far. + +Reference: + +The Objective Caml system +release 3.10 +Documentation and user's manual +Xavier Leroy +(with Damien Doligez, Jacques Garrigue, Didier RÃy and JÃÃe Vouillon) +May 16, 2007 +Part III: The Objective Caml tools +Chapter 12 Lexer and parser generators (ocamllex, ocamlyacc) +12.4.2 Declarations +http://caml.inria.fr/pub/docs/manual-ocaml/manual026.html#htoc140 + +License: + +The Objective Caml system is copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007 Institut National de Recherche en Informatique et en Automatique (INRIA). INRIA holds all ownership rights to the Objective Caml system. + +The Objective Caml system is open source and can be freely redistributed. See the file LICENSE in the distribution for licensing information. + +The present documentation is copyright (C) 2007 Institut National de Recherche en Informatique et en Automatique (INRIA). The Objective Caml documentation and user's manual may be reproduced and distributed in whole or in part, subject to the following conditions: + + * The copyright notice above and this permission notice must be preserved complete on all complete or partial copies. + * Any translation or derivative work of the Objective Caml documentation and user's manual must be approved by the authors in writing before distribution. + * If you distribute the Objective Caml documentation and user's manual in part, instructions for obtaining the complete version of this manual must be included, and a means for obtaining a complete version provided. + * Small portions may be reproduced as illustrations for reviews or quotes in other works without this permission notice if proper citation is given. + +The explaination: + +Declarations are given one per line. They all start with a % sign. + +%token constr ... constr + Declare the given symbols constr ... constr as tokens (terminal symbols). These symbols are added as constant constructors for the token concrete type. +%token < typexpr > constr ... constr + Declare the given symbols constr ... constr as tokens with an attached attribute of the given type. These symbols are added as constructors with arguments of the given type for the token concrete type. The typexpr part is an arbitrary Caml type expression, except that all type constructor names must be fully qualified (e.g. Modname.typename) for all types except standard built-in types, even if the proper open directives (e.g. open Modname) were given in the header section. That's because the header is copied only to the .ml output file, but not to the .mli output file, while the typexpr part of a %token declaration is copied to both. +%start symbol ... symbol + Declare the given symbols as entry points for the grammar. For each entry point, a parsing function with the same name is defined in the output module. Non-terminals that are not declared as entry points have no such parsing function. Start symbols must be given a type with the %type directive below. +%type < typexpr > symbol ... symbol + Specify the type of the semantic attributes for the given symbols. This is mandatory for start symbols only. Other nonterminal symbols need not be given types by hand: these types will be inferred when running the output files through the Objective Caml compiler (unless the -s option is in effect). The typexpr part is an arbitrary Caml type expression, except that all type constructor names must be fully qualified, as explained above for %token. +%left symbol ... symbol +%right symbol ... symbol +%nonassoc symbol ... symbol + Associate precedences and associativities to the given symbols. All symbols on the same line are given the same precedence. They have higher precedence than symbols declared before in a %left, %right or %nonassoc line. They have lower precedence than symbols declared after in a %left, %right or %nonassoc line. The symbols are declared to associate to the left (%left), to the right (%right), or to be non-associative (%nonassoc). The symbols are usually tokens. They can also be dummy nonterminals, for use with the %prec directive inside the rules. + + The precedence declarations are used in the following way to resolve reduce/reduce and shift/reduce conflicts: + + * Tokens and rules have precedences. By default, the precedence of a rule is the precedence of its rightmost terminal. You can override this default by using the %prec directive in the rule. + * A reduce/reduce conflict is resolved in favor of the first rule (in the order given by the source file), and ocamlyacc outputs a warning. + * A shift/reduce conflict is resolved by comparing the precedence of the rule to be reduced with the precedence of the token to be shifted. If the precedence of the rule is higher, then the rule will be reduced; if the precedence of the token is higher, then the token will be shifted. + * A shift/reduce conflict between a rule and a token with the same precedence will be resolved using the associativity: if the token is left-associative, then the parser will reduce; if the token is right-associative, then the parser will shift. If the token is non-associative, then the parser will declare a syntax error. + * When a shift/reduce conflict cannot be resolved using the above method, then ocamlyacc will output a warning and the parser will always shift. diff --git a/posts/yacc2dot/p.y b/posts/yacc2dot/p.y new file mode 100644 index 0000000..94a56db --- /dev/null +++ b/posts/yacc2dot/p.y @@ -0,0 +1,5 @@ +%token X +%token PLUS +%% +expr : X PLUS X + ; diff --git a/posts/yacc2dot/pr.y b/posts/yacc2dot/pr.y new file mode 100644 index 0000000..4f8bb3f --- /dev/null +++ b/posts/yacc2dot/pr.y @@ -0,0 +1,6 @@ +%token X +%token PLUS +%% +expr : X + | expr PLUS expr + ; diff --git a/posts/yacc2dot/pr_prec.y b/posts/yacc2dot/pr_prec.y new file mode 100644 index 0000000..3e798d3 --- /dev/null +++ b/posts/yacc2dot/pr_prec.y @@ -0,0 +1,6 @@ +%token X +%left PLUS +%% +expr : X + | expr PLUS expr + ; diff --git a/posts/yacc2dot/pt.y b/posts/yacc2dot/pt.y new file mode 100644 index 0000000..edb8834 --- /dev/null +++ b/posts/yacc2dot/pt.y @@ -0,0 +1,8 @@ +%token X +%left PLUS /* precedence from lowest to highest */ +%left TIMES +%% +expr : X + | expr PLUS expr + | expr TIMES expr + ; diff --git a/posts/yacc2dot/pt_implicit.y b/posts/yacc2dot/pt_implicit.y new file mode 100644 index 0000000..16929e4 --- /dev/null +++ b/posts/yacc2dot/pt_implicit.y @@ -0,0 +1,7 @@ +%token X +%left PLUS /* precedence from lowest to highest */ +%% +expr : X + | expr PLUS expr + | expr expr /* implicit times */ + ; diff --git a/posts/yacc2dot/t_implicit.y b/posts/yacc2dot/t_implicit.y new file mode 100644 index 0000000..c9436e4 --- /dev/null +++ b/posts/yacc2dot/t_implicit.y @@ -0,0 +1,5 @@ +%token X +%% +expr : X + | expr expr /* implicit times */ + ; diff --git a/posts/yacc2dot/t_implicit_prec.y b/posts/yacc2dot/t_implicit_prec.y new file mode 100644 index 0000000..c4a7407 --- /dev/null +++ b/posts/yacc2dot/t_implicit_prec.y @@ -0,0 +1,6 @@ +%nonassoc X /* low precedence for operator shifting */ +%left IMP_MULT /* high precedence for implicit multiplication */ +%% +expr : X + | expr expr %prec IMP_MULT /* implicit multiplication */ + ; diff --git a/posts/yacc2dot/yacc2dot b/posts/yacc2dot/yacc2dot new file mode 100755 index 0000000..35c40ee --- /dev/null +++ b/posts/yacc2dot/yacc2dot @@ -0,0 +1,584 @@ +#!/usr/bin/python +# +# convert yacc's verbose output to a dot-language diagram + +from sys import stdin, stdout, stderr + +DOT_HEAD = """ +digraph yacc { + + epsilon = 0.01; + nodesep = 0.2; /* inches */ + + node [shape = record]; + +""" + +DOT_GOTO_EDGE = """ + edge [color = blue]; +""" + +DOT_REDUCE_EDGE = """ + edge [color = red]; +""" + +DOT_TAIL = "}\n" + + + + +class action (object) : + """ + Parse the text defining a given rule: + >>> a = action(' TEXT shift, and go to state 1') + >>> print a.key + TEXT + >>> print a.action + shift + >>> a = action(' $default accept') + >>> print a.action + accept + >>> a = action(' $default reduce using rule 1 (sum)') + >>> print a.action + reduce + >>> print a.reduce_rule + 1 + >>> a = action(' PLUS shift, and go to state 3') + >>> print a.goto + 3 + >>> a = action(' sum go to state 2') + >>> print a.action + go + >>> print a.goto + 2 + """ + def __init__(self, string) : + self.rule = string + split = string.split() + self.key = split[0] + if split[1][-1] == ',' : + self.action = split[1][0:-1] + elif split[1][0] == '[': + self.action = 'IGNORED' + else : + self.action = split[1] + + if self.action == 'reduce' : + self.reduce_rule = int(split[-2]) + elif self.action == 'shift' : + self.goto = int(split[-1]) + elif self.action == 'go' : + self.goto = int(split[-1]) + def __str__(self) : + if self.action == 'go' or self.action == 'shift': + details = ', goto %d' % self.goto + elif self.action == 'reduce' : + details = ', rule %d' % self.reduce_rule + elif self.action == 'accept' : + details = '' + elif self.action == 'IGNORED' : + details = '' + else : + raise Exception, 'wierd action %s' % self.action + + return '' % (self.key, self.action, details) + def __repr__(self) : + return self.__str__() + +class state (object) : + """ + Parse the text defining a given state: + >>> state_text = '''state 0 + ... + ... 0 $accept: . authors $end + ... + ... TEXT shift, and go to state 1 + ... LBRACE shift, and go to state 2 + ... + ... authors go to state 3 + ... author go to state 4 + ... text go to state 5 + ... + ... ''' + >>> s = state(state_text) + >>> print s.id + 0 + >>> print s.rules + [(0, '$accept: . authors $end')] + >>> print s.tok_act + [, ] + >>> print s.rule_act + [, , ] + >>> print s.goes_to() + [1, 2, 3, 4, 5] + """ + def __init__(self, state_text, debug=False) : + # because the output of `yacc -v grammar.y' is so regular, + # we'll sneak by with a HACK parsing job ;). + self.id = None + self.rules = [] # list of "rule string"s ###(, , [], pos) + self.tok_act = [] + self.rule_act = [] + lookfor = 'state' + hadtext = False # so consecutive blank lines don't keep switching lookfor state + for line in state_text.splitlines() : + if debug : print >> stderr, '%s : %s' % (lookfor, line) + split = line.split() + if len(split) < 1 : # blank line + if hadtext == False : + continue # ignore + elif lookfor == 'rules' : + lookfor = 'tok-act' + continue + elif lookfor == 'tok-act' : + lookfor = 'rule-act' + continue + elif split[0] == 'state' : # 'state ' + assert lookfor == 'state', "Multiple states in '%s'" % state_text + assert len(split) == 2, "%d items in state line '%s'" % line + self.id = int(split[1]) # store as self.id + lookfor = 'rules' + hadtext = False + continue + elif lookfor == 'rules' : # e.g ' 2 expr: expr . PLUS expr' + rule_id = int(split[0]) + rule_text = split[1:] + if rule_text[0] == '|' : # e.g. second line of + # expr: X + # | expr PLUS expr + rule_text.pop(0) + prev_rule = self.rules[len(self.rules)-1] + p_text_split = prev_rule[1].split() + rule_text.insert(0, p_text_split[0]) + self.rules.append((rule_id, " ".join(rule_text))) + elif lookfor == 'tok-act' : + self.tok_act.append(action(line)) + elif lookfor == 'rule-act' : + self.rule_act.append(action(line)) + else : + raise Exception, "Unrecognized lookfor '%s' for line '%s'" % (lookfor, line) + hadtext = True + def __cmp__(self, other) : + return cmp(self.id, other.id) + def __str__(self) : + return '' % self.id + def __repr__(self) : + return self.__str__() + def goes_to(self) : + "Return a list of integer states that this node goes to (not reduces to)." + ret = [] + for a in self.tok_act : + if hasattr(a, 'goto') : + ret.append(a.goto) + for a in self.rule_act : + if hasattr(a, 'goto') : + ret.append(a.goto) + return ret + +class tnode (object) : + """ + Tree structure, with backlinks. + >>> root = tnode("root") + >>> root.add_child("child0") + >>> root.add_child("child1") + >>> root.child[0].add_child("child00") + >>> root.child[0].add_child("child01") + >>> print root.child[0].child + [, ] + >>> print root.child[0].parent + + >>> print root.find("child01") + + """ + def __init__(self, value=None, parent=None) : + self.value = value + self.parent = parent + self.child = [] + def add_child(self, child_value) : + self.child.append(tnode(child_value, self)) + def build(self, fn_get_children) : + "use gn_get_children(value) = [values' children] to recursively build the tree" + children = fn_get_children(self.value) + for child in children : + self.add_child(child) + for cnode in self.child : + cnode.build(fn_get_children) + def find(self, value) : + "depth first search" + if self.value == value : + return self + else : + for cnode in self.child : + match = cnode.find(value) + if match != None : + return match + return None + def __str__(self) : + string = '' % str(self.value) + return string + def __repr__(self) : + return self.__str__() + +class yaccout (object) : + """ + Parse the output of `yacc -v file.y' + For example, for sum.y = + %token X + %token PLUS + %% + sum : X PLUS X + We get: + >>> yacc_text = ''' + ... Grammar + ... + ... 0 $accept: sum $end + ... + ... 1 sum: X PLUS X + ... + ... + ... Terminals, with rules where they appear + ... + ... $end (0) 0 + ... error (256) + ... X (258) 1 + ... PLUS (259) 1 + ... + ... + ... Nonterminals, with rules where they appear + ... + ... $accept (5) + ... on left: 0 + ... sum (6) + ... on left: 1, on right: 0 + ... + ... + ... state 0 + ... + ... 0 $accept: . sum $end + ... + ... X shift, and go to state 1 + ... + ... sum go to state 2 + ... + ... + ... state 1 + ... + ... 1 sum: X . PLUS X + ... + ... PLUS shift, and go to state 3 + ... + ... + ... state 2 + ... + ... 0 $accept: sum . $end + ... + ... $end shift, and go to state 4 + ... + ... + ... state 3 + ... + ... 1 sum: X PLUS . X + ... + ... X shift, and go to state 5 + ... + ... + ... state 4 + ... + ... 0 $accept: sum $end . + ... + ... $default accept + ... + ... + ... state 5 + ... + ... 1 sum: X PLUS X . + ... + ... $default reduce using rule 1 (sum) + ... ''' + >>> y = yaccout(yacc_text) + >>> print y.states + [, , , , , ] + >>> print y.states[0].tok_act + [] + >>> print y.root.child[0].child + [>] + >>> print y.root.child[0].parent + > + """ + def __init__(self, yacc_text, debug=False): + # because the output of `yacc -v grammar.y' is so regular, + # we'll sneak by with a HACK parsing job ;). + self.read_states(yacc_text, debug=debug) + #self.gen_tree() + #self.reduce_link(debug=debug) + def read_states(self, yacc_text, debug=False): + "read in all the states" + self.states = [] + lookfor = 'first state' + for line in yacc_text.splitlines() : + split = line.split(' ') + if lookfor == 'first state' : + if split[0] != 'state' : + continue + else : + lookfor = 'states' + state_text = [line] + continue + else : + if split[0] == 'state' : # we've hit the next state + self.states.append(state('\n'.join(state_text), debug=debug)) + state_text = [line] + continue + else : + state_text.append(line) + # end of file, so process the last state + self.states.append(state('\n'.join(state_text), debug=debug)) + def state_goes_to(self, state) : + ret = [] + for id in state.goes_to() : + for s in self.states : + if s.id == id : + ret.append(s) + return ret + def gen_tree(self): + "generate the state dependency tree" + self.root = tnode(self.states[0]) + self.root.build(self.state_goes_to) + def reduce_action(self, state, action, debug): + if action.action == 'reduce' : + if debug : print >> stderr, 'reduce from %s with rule %d::' % (state, action.reduce_rule), + # find the rule text + rule = None + for r in state.rules : + if r[0] == action.reduce_rule : + rule = r[1] # e.g. 'sum: X PLUS X .' + if debug : print >> stderr, rule + # find the number of args in the rule + split = rule.split() + args = len(split) - 2 # 2 for 'sum:' and '.' + # find the state in the tnode tree + tnode = self.root.find(state) + for i in range(args) : # take arg steps up the tree + tnode = tnode.parent + tstate = tnode.value # reduction target state + action.reduce_to = tstate.id # state we reduce to + action.reduce_targ = split[0][0:-1] # rule we reduce to 'sum' + i = 0 + for a in tstate.tok_act : + if a.key == action.reduce_targ : + action.reduce_targ_i = i + i += 1 + for a in tstate.rule_act : + if a.key == action.reduce_targ : + action.reduce_targ_i = i + i += 1 + if debug : print >> stderr, 'to state %d' % action.reduce_to + def reduce_link(self, debug=False): + "generate the reduce_to backlinks" + for state in self.states : + for a in state.tok_act : + self.reduce_action(state, a, debug) + for a in state.rule_act : + self.reduce_action(state, a, debug) + + +def dot_state (state) : + """ + Print a dot node for a given state. + Node type must be 'record'. + layout: + state %d + rules + ... + a0 | a1 | ... + >>> state_text = '''state 0 + ... + ... 0 $accept: . authors $end + ... + ... TEXT shift, and go to state 1 + ... LBRACE shift, and go to state 2 + ... + ... authors go to state 3 + ... author go to state 4 + ... text go to state 5 + ... + ... ''' + >>> s = state(state_text) + >>> print dot_state(s), + state0 [label = "{ state 0 | { TEXT | LBRACE | authors | author | text } }"]; + """ + label = '{ state %d | ' % state.id + for rule in state.rules : + label += '(%d) %s\\n' % (rule[0], rule[1]) + label += ' | {' + a = 0 + for action in state.tok_act[:-1] : + label += ' %s |' % (a, action.key,) + a += 1 + if len(state.tok_act) > 0 : + label += ' %s' % (a, state.tok_act[-1].key) + a += 1 + for action in state.rule_act : + label += ' | %s' % (a, action.key) + a += 1 + label += ' } }' + + string = ' state%d [label = "%s"];\n' % (state.id, label) + return string + +def dot_goto(id, a, action) : + """ + Print dot links for a given action (action a for state id). + >>> a = action(' TEXT shift, and go to state 1') + >>> print dot_goto(5,8,a), + state5:a8 -> state1:s; + >>> a = action(' $default reduce using rule 1 (sum)') + >>> print dot_goto(0,1,a), + + >>> a = action(' sum go to state 2') + >>> print dot_goto(0,1,a), + state0:a1 -> state2:s; + """ + if hasattr(action, 'goto') : + string = ' state%d:a%d -> state%d:s;\n' % (id, a, action.goto) + else : + string = '' + return string + + +def dot_gotos(state) : + """ + Print dot links for a given state. + >>> state_text = '''state 0 + ... + ... 0 $accept: . authors $end + ... + ... TEXT shift, and go to state 1 + ... LBRACE reduce using rule 1 (braces) + ... + ... authors go to state 3 + ... author go to state 4 + ... text go to state 5 + ... + ... ''' + >>> s = state(state_text) + >>> print dot_gotos(s), + state0:a0 -> state1:s; + state0:a2 -> state3:s; + state0:a3 -> state4:s; + state0:a4 -> state5:s; + """ + string = "" + a = 0 + for action in state.tok_act : + string += dot_goto(state.id, a, action) + a += 1 + for action in state.rule_act : + string += dot_goto(state.id, a, action) + a += 1 + return string + +def dot_reduce(id, a, action) : + """ + Print dot reduce links for a reduction action (action a for state id). + """ + if action.action == 'reduce' : + string = ' state%d:a%d -> state%d:a%d;\n' % (id, a, action.reduce_to, action.reduce_targ_i) + else : + string = '' + return string + + +def dot_reduces(state) : + """ + Print dot reduce links for a given state. + """ + string = "" + a = 0 + for action in state.tok_act : + string += dot_reduce(state.id, a, action) + a += 1 + for action in state.rule_act : + string += dot_reduce(state.id, a, action) + a += 1 + return string + +def yacc2dot(yaccout) : + string = DOT_HEAD + string += "\n" + for state in yaccout.states : + string += dot_state(state) + string += "\n" + string += DOT_GOTO_EDGE + for state in yaccout.states : + string += dot_gotos(state) + string += "\n" + string += DOT_REDUCE_EDGE + #for state in yaccout.states : + # string += dot_reduces(state) + string += "\n" + string += DOT_TAIL + return string + +def open_IOfiles(ifilename=None, ofilename=None, debug=False): + if ifilename : + if debug : print >> stderr, "open input file '%s'" % ifilename + ifile = file(ifilename, 'r') + else : + ifile = stdin + if ofilename : + if debug : print >> stderr, "open output file '%s'" % ofilename + ofile = file(ofilename, 'w') + else : + ofile = stdout + return (ifile, ofile) + +def close_IOfiles(ifilename=None, ifile=stdin, + ofilename=None, ofile=stdout, + debug=False): + if ifilename : + if debug : print >> stderr, "close input file '%s'" % ifilename + ifile.close() + if ofilename : + if debug : print >> stderr, "close output file '%s'" % ofilename + ofile.close() + +def _test(): + import doctest + doctest.testmod() + +if __name__ == "__main__": + from optparse import OptionParser + + parser = OptionParser(usage="usage: %prog [options]", version="%prog 0.1") + + parser.add_option('-f', '--input-file', dest="ifilename", + help="Read input from FILE (default stdin)", + type='string', metavar="FILE") + parser.add_option('-o', '--output-file', dest="ofilename", + help="Write output to FILE (default stdout)", + type='string', metavar="FILE") + parser.add_option('-t', '--test', dest="test", + help="Run the yacc2dot test suite", + action="store_true", default=False) + parser.add_option('-v', '--verbose', dest="verbose", + help="Print lots of debugging information", + action="store_true", default=False) + + (options, args) = parser.parse_args() + parser.destroy() + + ifile,ofile = open_IOfiles(options.ifilename, options.ofilename, + options.verbose) + + if options.test : + _test() + else : + text = ifile.read() + y = yaccout(text, options.verbose) + dot = yacc2dot(y) + print >> ofile, dot + + close_IOfiles(options.ifilename, ifile, + options.ofilename, ofile, options.verbose) +