From: Armin Ronacher Date: Fri, 27 Apr 2007 16:24:19 +0000 (+0200) Subject: [svn] many jinja changes: X-Git-Tag: 2.0rc1~339 X-Git-Url: http://git.tremily.us/?a=commitdiff_plain;h=fb5bebc338ef37d38aa19195394663fdc90782dd;p=jinja2.git [svn] many jinja changes: - improved generated bytecode - improved streaming system - buffer variable substitution syntax --HG-- branch : trunk --- diff --git a/CHANGES b/CHANGES index 68a5383..be02762 100644 --- a/CHANGES +++ b/CHANGES @@ -35,7 +35,8 @@ Version 1.1 - developer friendly traceback is now toggleable -- silent variable name failure is now toggleable +- the variable failure is now pluggable by replacing the undefined + singleton for an environment instance - fixed issue with old-style classes not implementing `__getitem__` (thanks to Axel Böhm for discovering that bug) @@ -68,10 +69,18 @@ Version 1.1 - You can now use the environment to just tokenize a template. This can be useful for syntax highlighting or other purposes. -- added optional c-implementation of the context baseclass. +- added optional C-implementation of the context baseclass. + +- you can now use optional parentheses around macro defintions. Thus it's + possible to write ``{% macro foo(a, b, c) %}`` instead of ``{% macro + foo a, b, c %}``. + +- additional macro arguments now end up in `varargs`. - implemented `{% call %}` - unsure if this makes it into the final release. +- it's not possible to stream templates. + Version 1.0 ----------- diff --git a/TODO b/TODO index 1659c39..ae57435 100644 --- a/TODO +++ b/TODO @@ -2,8 +2,25 @@ TODO List for Jinja =================== -- Improve the context lookup (maybe with an optional C extension) +1.1: -- `include` and `extends` should work with dynamic data too. In order to - support this the blocks should be stored as importable functions in the - generated source. + - Improve the context lookup (maybe with an optional C extension) [DONE] + - make Undefined exchangeable [DONE] + - implement block.super [DONE] + - Implement a `IntrospectionPrinter` that works like pprint but it outputs + either plain text or html. It would also have to cover changing names of + the special builtins True, False etc to lowercase in order to not + confuse people. + - decide on `{% call %}` + - speed up jinja import + - add optional zlib compression of template bytecode + - write more unittests!!!! + +1.2: + + - `include` and `extends` should work with dynamic data too. In order to + support this the blocks should be stored as importable functions in the + generated source. + - add support for `{% include myfile = 'myfile.html' %}` and give the + template designer to access variables and macros defined in the template + root or requirements namespace. diff --git a/docs/src/api.txt b/docs/src/api.txt new file mode 100644 index 0000000..5086953 --- /dev/null +++ b/docs/src/api.txt @@ -0,0 +1,315 @@ +======================= +Context and Environment +======================= + +The two central objects in Jinja are the `Environment` and `Context`. Both +are designed to be subclassed by applications if they need to extend Jinja. + +Environment +=========== + +The initialization parameters are already covered in the `Quickstart`_ thus +not repeated here. + +But beside those configurable instance variables there are some functions used +in the template evaluation code you may want to override: + +**def** `parse` *(source, filename)*: + + Parse the sourcecode and return the abstract syntax tree. This tree of + nodes is used by the `translators`_ to convert the template into + executable source- or bytecode. + +**def** `lex` *(source, filename)*: + + Tokenize the given sourcecode and return a generator of tuples in the + form ``(lineno, token, value)``. The filename is just used in the + exceptions raised. + + **New in Jinja 1.1** + +**def** `from_string` *(source)*: + + Load and parse a template source and translate it into eval-able Python + code. This code is wrapped within a `Template` class that allows you to + render it. + +**def** `get_template` *(name)*: + + Load a template from a loader. If the template does not exist, you will + get a `jinja.exceptions.TemplateNotFound` exception. + +**def** `to_unicode` *(self, value)*: + + Called to convert variables to unicode. Per default this checks if the + value is already unicode. If not it's converted to unicode using the + charset defined on the environment. + + Also `None` is converted into an empty string per default. + +**def** `get_translator` *(self, context)*: + + Return the translator used for i18n. A translator is an object that + provides the two functions ``gettext(string)`` and + ``ngettext(singular, plural, n)``. Both of those functions have to + behave like the `ugettext` and `nugettext` functions described in the + python `gettext documentation`_. + + If you don't provide a translator a default one is used to switch + between singular and plural forms. + + Have a look at the `i18n`_ section for more information. + +**def** `get_translations` *(self, name)*: + + Get the translations for the template `name`. Only works if a loader + is present. See the `i18n`_ section for more details. + +**def** `get_translations_for_string` *(self, string)*: + + Get the translations for the string `string`. This works also if no + loader is present and can be used to lookup translation strings from + templates that are loaded from dynamic resources like databases. + +**def** `apply_filters` *(self, value, context, filters)*: + + Now this function is a bit tricky and you usually don't have to override + it. It's used to apply filters on a value. The Jinja expression + ``{{ foo|escape|replace('a', 'b') }}`` calls the function with the + value of `foo` as first parameter, the current context as second and + a list of filters as third. The list looks like this: + + .. sourcecode:: python + + [('escape', ()), ('replace', (u'a', u'b'))] + + As you can see the filter `escape` is called without arguments whereas + `replace` is called with the two literal strings ``a`` and ``b``, both + unicode. The filters for the names are stored on ``self.filters`` in a + dict. Missing filters should raise a `FilterNotFound` exception. + + **Warning** this is a Jinja internal method. The actual implementation + and function signature might change. + +**def** `perform_test` *(self, context, testname, args, value, invert)*: + + Like `apply_filters` you usually don't override this one. It's the + callback function for tests (``foo is bar`` / ``foo is not bar``). + + The first parameter is the current contex, the second the name of + the test to perform. the third a tuple of arguments, the fourth is + the value to test. The last one is `True` if the test was performed + with the `is not` operator, `False` if with the `is` operator. + + Missing tests should raise a `TestNotFound` exception. + + **Warning** this is a Jinja internal method. The actual implementation + and function signature might change. + +**def** `get_attribute` *(self, obj, attribute)*: + + Get `attribute` from the object provided. The default implementation + performs security tests. + + **Warning** this is a Jinja internal method. The actual implementation + and function signature might change. + +**def** `get_attributes` *(self, obj, attributes)*: + + Get some attributes from the object. If `attributes` is an empty + sequence the object itself is returned unchanged. + +**def** `call_function` *(self, f, context, args, kwargs, dyn_args, dyn_kwargs)*: + + Call a function `f` with the arguments `args`, `kwargs`, `dyn_args` and + `dyn_kwargs` where `args` is a tuple and `kwargs` a dict. If `dyn_args` + is not `None` you have to add it to the arguments, if `dyn_kwargs` is + not `None` you have to update the `kwargs` with it. + + The default implementation performs some security checks. + + **Warning** this is a Jinja internal method. The actual implementation + and function signature might change. + +**def** `call_function_simple` *(self, f, context)*: + + Like `call_function` but without arguments. + + **Warning** this is a Jinja internal method. The actual implementation + and function signature might change. + +**def** `finish_var` *(self, value, ctx)*: + + Postprocess a variable before it's sent to the template. + + **Warning** this is a Jinja internal method. The actual implementation + and function signature might change. + +.. admonition:: Note + + The Enviornment class is defined in `jinja.environment.Environment` + but imported into the `jinja` package because it's often used. + +Context +======= + +Jinja wraps the variables passed to the template in a special class called a +context. This context supports variables on multiple layers and lazy (deferred) +objects. Often your application has a request object, database connection +object or something similar you want to access in filters, functions etc. + +The default context object is defined in `jinja.datastructure`. If you want +to provide your own context object always subclass the default one. This +ensures that the class continues working after Jinja upgrades. + +Beacause of that you can easily subclass a context to add additional variables +or to change the way it behaves. + +**def** `pop` *(self)*: + + Pop the outermost layer and return it. + +**def** `push` *(self, data=None)*: + + Push a dict to the stack or an empty layer. + + Has to return the pushed object. + +**def** `to_dict` *(self)*: + + Flatten the context and convert it into a dict. + +**def** `__getitem__` *(self, name)*: + + Resolve an item. Per default this also resolves `Deferred` objects. + +**def** `__setitem__` *(self, name, value)*: + + Set an item in the outermost layer. + +**def** `__delitem__` *(self, name)*: + + Delete an item in the outermost layer. Do not raise exceptions if + the value does not exist. + +**def** `__contains__` *(self, name)*: + + Return `True` if `name` exists in the context. + +**attribute** `cache`: + + The cache is a dict which can be used by filters, test functions + and global objects to cache data. It's also used by the environment + to cache often used tests and filters. + +**attribute** `translate_func`: + + This property is created on first access and returns a translation + function used by the rendering process to translate strings with the + translator defined on the environment. + +.. admonition:: Note + + The context uses a stack of dicts internally to represent the + layers of variables. It contains at least 3 levels available on + the context with some attributes. Those are: + + `globals`: + + The reference to the global namespace of the environment. + It's the lowest namespace on the stack and thus immutable + + `initial`: + + The initial namespace. Contains the values passed to the + context in the render function. It also contains the resolved + deferred values for bot the `initial` and the `globals` + namespace. + + `current`: + + The reference to the current active namespace. When the + context is initialized this automatically points to an + empty namespace. + + The number of layers on the stack are theoretically unlimited. + Some elements in the template language like loops, blocks, + macros and others push and pop the layer on entering and leaving + the section. + + This is done in order to keep the namespace clean. + + Note that since Jinja 1.1 the context object is a subclass of the + `BaseContext`, a much simpler class that just implements a stack + like namespace for python. If the `_speedups` extension was + compiled for jinja the base class will be + `jinja._speedups.BaseContext` otherwise `jinja._native.BaseContext`. + + Since you cannot reproduce completely the same semantics in python + and the C API there are some things you should keep in mind: + + - The `stack` attribute of the context maps to the real layers + on the stack, thus you can modify the items but the list as + such is meant to be read only. + + - `globals`, `current` and `initial` are read only attributes that + map to layers on the stack which you can of course modify. + + +Exceptions +========== + +During parsing and evaluation Jinja raises a couple of Jinja specific +exceptions. All of those exceptions are defined in the `jinja.exceptions` +module and are subclasses of the `TemplateError` class defined there. + +Here a list of exceptions that could occur: + +`SecurityException`: + + An exception that is raised if the template tried to access something + it should not access. In the default configuration this exception + will get caught in the Jinja rendering process and silenced. + + If however the environment is configured to not silently fail it + could happen that this exception reaches the application. + +`FilterNotFound`: + + Raised if the template tried to apply a filter that does not exist. + Since this exception is a subclass of `KeyError` too you can catch + it this way too. + +`FilterArgumentError`: + + Raised if the filter received an argument that it couldn't handle. + It's a subclass of `TypeError` too so you can catch it this way too. + +`TestNotFound`: + + Raised if the template tried to perform a test that does not exist. + Since this exception is a subclass of `KeyError` too you can catch + it this way too. + +`TestArgumentError`: + + Raised if a test function received an argument that it couldn't handle. + It's a subclass of `TypeError` too so you can catch it this way too. + +`TemplateNotFound`: + + Raised if a template does not exist. Subclass of `IOError` too. + +`TemplateSyntaxError`: + + Subclass of `SyntaxError` and used to indicate an syntax error. + +`TemplateRuntimeError`: + + Generic runtime error exception which can occour at various places. + + +.. _i18n: i18n.txt +.. _translators: translators.txt +.. _Quickstart: devintro.txt +.. _gettext documentation: http://docs.python.org/lib/module-gettext.html diff --git a/docs/src/designerdoc.txt b/docs/src/designerdoc.txt index 1173763..e826d8f 100644 --- a/docs/src/designerdoc.txt +++ b/docs/src/designerdoc.txt @@ -235,25 +235,6 @@ available per default: *new in Jinja 1.1* -`flush` - - Jinja 1.1 includes a new stream interface which can be used to speed - up the generation of big templates by streaming them to the client. - Per default Jinja flushes after 40 actions automatically. If you - want to force a flush at a special point you can use `flush()` to - do so. (eg, end of a loop iteration etc) - - .. sourcecode:: jinja - - {% for items in seq %} - {% for item in items %} -
  • {{ item|e }}
    • - {% endfor %} - {{ flush() }} - {% endfor %} - - *new in Jinja 1.1* - Loops ===== @@ -504,6 +485,38 @@ You can also specify more than one value: {{ show_dialog('Warning', 'something went wrong i guess') }} +*Improvements in Jinja 1.1*: + + Starting with Jinja 1.1 it's possible to use optional parentheses + around the macro arguments: + + .. sourcecode:: html+jinja + + {% macro foo(a, b) %} + ... + {% endmacro %} + + Additionally extra arguments passed to the macro end up in the + special variable `varargs`. So you can have a macro like this: + + .. sourcecode:: html+jinja + + {% macro make_list() %} + {% if varargs %} + + {% endif %} + {% endmacro %} + + {{ make_list("John", "Jane", "Marcus", "Heinrich") }} + + If a macro parameter is called `varargs` the additional extra + arguments are not accessible. + + Inheritance =========== @@ -691,11 +704,84 @@ The variable ``angryname`` just exists inside the macro, not outside it. Defined macros appear on the context as variables. Because of this, they are affected by the scoping too. A macro defined inside of a macro is just available -in those two macros (the macro itself and the macro it's defined in). For `set` -and `macro` two additional rules exist: If a macro is defined in an extended -template but outside of a visible block (thus outside of any block) will be -available in all blocks below. This allows you to use `include` statements to -load often used macros at once. +in those two macros (the macro itself and the macro it's defined in). + +Template Globals +================ + +A special threatment exists for template code outside of visible blocks in +child templates. This code will be executed **before** the layout template +code. Thus it can be used to propagate values back to the layout template or +import macros from templates for rendering. + +Such code can output data but it won't appear in the final rendering. So no +additional whitespace will pollute the template. + +Because this code is executed before the actual layout template code it's +possible that the layout code overrides some of those variables. Usually +this is not a problem because of different variable names but it can be +a problem if you plan to specify default values. + +In that case you have to test if the variable is not defined before setting +it: + +.. sourcecode:: jinja + + {% if not page_title %} + {% set page_title = 'Default Page Title' %} + {% endif %} + +You can of course also use the `|default` filter. + +.. admonition:: Explanation + + This template stored as `a.html`: + + .. sourcecode:: html+jinja + + {{ title|default('Untitled') }} + {% block body %}{% endblock %} + + ...and this child template stored as `b.html`: + + .. sourcecode:: html+jinja + + {% extends 'a.html' %} + {% include 'macros.tmpl' %} + {% set title = 'My Page' %} + {% block body %}{{ wrap(42) }}{% endblock %} + + ...and this code in `macros.tmpl`: + + .. sourcecode:: html+jinja + + {% macro wrap(text) %} + [{{ text }}] + {% endmacro %} + + ..will translate to something with the same semantics as this (just + that the value is not stored in a variable): + + .. sourcecode:: html+jinja + + {% filter capture('captured', true) %} + {% macro wrap(text) %} + [{{ text }}] + {% endmacro %} + {% set title='My Page' %} + {% endfilter %} + {{ title|default('Untitled') }} + + {{ wrap(42) }} + + +.. admonition:: Note + + This implementation was improved in Jinja 1.1. In Jinja 1.0 blocks that + were not top-level were not propagated to the layout template. This made + it impossible to use conditional expressions for inclusion in non root + templates. + Undefined Variables =================== @@ -704,48 +790,53 @@ If you have already worked with python you probably know about the fact that undefined variables raise an exception. This is different in Jinja. There is a special value called `undefined` that represents values that do not exist. -This special variable works complete different from any variables you maybe -know. If you print it using ``{{ variable }}`` it will not appear because it's -literally empty. If you try to iterate over it, it will work. But no items -are returned. Comparing this value to any other value results in `false`. -Even if you compare it to itself: +Depending on the configuration it will behave different. + +In order to check if a value is defined you can use the `defined` test: .. sourcecode:: jinja - {{ undefined == undefined }} - will return false. Not even undefined is undefined :) - Use `is defined` / `is not defined`: + {{ myvariable is not defined }} + will return true if the variable does not exist. - {{ undefined is not defined }} - will return true. +`SilentUndefined`: -There are also some additional rules regarding this special value. Any -mathematical operators (``+``, ``-``, ``*``, ``/``) return the operand -as result: + The silent `undefined` is the default behavior. The `undefined` object works + complete different from any variables you maybe know. If you print it + using ``{{ variable }}`` it will not appear because it's literally empty. + If you try to iterate over it, it will work. But no items are returned. -.. sourcecode:: jinja + In order to check if a value is defined you can use the `defined` test: + + There are also some additional rules regarding this special value. Any + mathematical operators (``+``, ``-``, ``*``, ``/``) return the operand + as result: - {{ undefined + "foo" }} - returns "foo" + .. sourcecode:: jinja - {{ undefined - 42 }} - returns 42. Note: not -42! + {{ undefined + "foo" }} + returns "foo" -In any expression `undefined` evaluates to `false`. It has no length, all -attribute calls return undefined, calling too: + {{ undefined - 42 }} + returns 42. Note: not -42! -.. sourcecode:: jinja + In any expression `undefined` evaluates to `false`. It has no length, all + attribute calls return undefined, calling too: - {{ undefined.attribute().attribute_too[42] }} - still returns `undefined`. + .. sourcecode:: jinja + + {{ undefined.attribute().attribute_too[42] }} + still returns `undefined`. -Starting with Jinja 1.1 it's possible to control the `undefined` behavior -in the Jinja environment, thus on the application side. If you receive -a `TemplateRuntimeError` when operating on an undefined variable the -environment was created with `silent` disabled. +`ComplainingUndefined`: -In a non silent environment the error you get tells you the line number -and the name of the variable / attribute which could not be found. + Starting with Jinja 1.1 it's possible to replace the default undefined + object with different values. The other common undefined object which + comes with Jinja is the `ComplainingUndefined` object. + + It raises exceptions as soon as you either render it or want to iterate + over it or try to access attributes etc. + Escaping ======== @@ -957,9 +1048,8 @@ change. {% endmacro %} - Called the normal way `caller` will be undefined and thus return nothing - in silent mode or raise an exception in non silent. But if you call it - with the new `{% call %}` tag you can pass it some data: + Called the normal way `caller` will be undefined, but if you call it + using the new `{% call %}` tag you can pass it some data: .. sourcecode:: html+jinja diff --git a/docs/src/devintro.txt b/docs/src/devintro.txt index bcfb046..596ab67 100644 --- a/docs/src/devintro.txt +++ b/docs/src/devintro.txt @@ -74,6 +74,22 @@ addition to the initialization values: There are also some internal functions on the environment used by the template evaluation code to keep it sandboxed. +Undefined Values +================ + +If a template designer tries to access a not defined value the return value +will be the `undefined_singleton` specified in the environment. The default +one is the `SilentUndefined` which fails in no case. Additionally there is +a special undefined type called the `ComplainingUndefined` which is located +in the `jinja.datastructure` module. It will raise exceptions when compared +with other types, or when rendered. + +Theoretically you can provide your own singleton by subclassing +`AbstractUndefindedType` and creating an instance of it using `make_undefined` +(both located in `jinja.datastructure`) but those two types should cover +the basic use cases. The `Undefined` object in that module exists for +backwards compatibility and is an alias for `SilentUndefined`. + Automatic Escaping ================== diff --git a/docs/src/streaming.txt b/docs/src/streaming.txt index ef91b0e..d872dc1 100644 --- a/docs/src/streaming.txt +++ b/docs/src/streaming.txt @@ -10,91 +10,65 @@ If you for example generate a file with a couple of megabytes you may want to pass the stream to the WSGI interface in order to keep the amount of memory used low and deliver the output to the browser as fast as possible. -The streaming works quite simple. Whenever an item is returned by the -internal generator it's passed to a `TemplateStream` which buffers 40 events -before yielding them. Because WSGI usually slows down on too many flushings -this is the integrated solution for this problem. The template designer -can further control that behavior by placing a ``{{ flush() }}`` call in -the template. Whenever the `TemplateStream` encounters an object marked as -flushable it stops buffering and yields the item. - -Note that some constructs are not part of the stream. For example blocks -that are rendered using ``{{ super() }}`` are yielded as one flushable item. - -This restriction exists because template designers may want to operate on -the return value of a macro, block etc. - -The following constructs yield their subcontents as one event: ``blocks`` -that are not the current one (eg, blocks called using ``{{ super() }}``), -macros, filtered sections. - -The TemplateStream -================== - -You can get a new `TemplateStream` by calling the `stream()` function on -a template like you would do for rendering. The `TemplateStream` behaves -like a normal generator/iterator. However, as long as the stream is not -started you can modify the buffer threshold. Once the stream started -streaming it looses the `threshold` attribute. - -Explicit flushing: +The streaming interface is straightforward. Instead of using `render()` you +can call `stream()` which does pretty much the same but doesn't return a +string but a `TemplateStream`: .. sourcecode:: pycon - >>> tmpl = evironment.from_string("") - >>> s = tmpl.stream(seq=range(3)) - >>> s.next() - u'' - >>> - -Implicit flushing after 6 events: + >>> tmpl = env.from_string("") + >>> stream = tmpl.stream(seq=range(4)) + >>> stream.next() + '' + +As you can see each iteration is threated as template event here. But also +other tags trigger events. Basically every tag yields one event, the +`print` tag too. The only exception is the variable substitution syntax which +is inserted into the template text data. + +Because some protocols like `WSGI` flush after each iteration if passed as +response iterable it's better to buffer some events internally. You can do +this by enable buffering using `enable_buffering` and passing it the buffer +size which must be greater than one: .. sourcecode:: pycon - >>> tmpl = environment.from_string("") - >>> s = tmpl.stream(seq=range(6)) - >>> s.threshold = 6 - >>> s.next() - u'' + >>> stream.enable_buffering(size=3) + >>> stream.next() + u'' -General `TemplateStream` behavior: +This will buffer 3 events before yielding. Disabling buffering works using the +`disable_buffering` method. You can enable and disable buffering on the fly also +if you have already iterated over that stream. To check if you are in buffered +or unbuffered mode you can use the `.buffered` property: .. sourcecode:: pycon - >>> s = tmpl.stream(seq=range(6)) - >>> s.started + >>> stream.buffered False - >>> s.threshold - 40 - >>> s.next() - u'' - >>> s.started + >>> stream.enable_buffering(20) + >>> stream.buffered True - >>> s.threshold - Traceback (most recent call last): - File "", line 1, in - AttributeError: 'TemplateStream' object has no attribute 'threshold' + >>> stream.disable_buffering() + >>> stream.buffered + False +.. admonition:: Note -Stream Control -============== + Jinja uses buffering internally for some constructs like macros. A macro + call is yielded as one event, independently of the internal structure. -The stream control is designed so that it's completely transparent. When used -in non stream mode the invisible flush tokens disappear. In order to flush -the stream after calling a specific function all you have to do is to wrap -the return value in a `jinja.datastructure.Flush` object. This however -bypasses the automatic filtering system and converts the value to unicode. + The same applies to recursive for loops and `{% filter %}` tags. diff --git a/jdebug.py b/jdebug.py index 257ddea..487ec66 100644 --- a/jdebug.py +++ b/jdebug.py @@ -33,9 +33,20 @@ if os.environ.get('JDEBUG_SOURCEPRINT'): PythonTranslator.translate = debug_translate -def p(x, f=None): +def p(x=None, f=None): + if x is None and f is not None: + x = e.loader.get_source(f) print PythonTranslator(e, Parser(e, x, f).parse()).translate() def l(x): for item in e.lexer.tokenize(x): print '%5s %-20s %r' % item + +if __name__ == '__main__': + if len(sys.argv) > 1: + from jinja import FileSystemLoader + e.loader = FileSystemLoader(sys.argv[1]) + if len(sys.argv) > 2: + p(f=sys.argv[2]) + else: + p(sys.stdin.read()) diff --git a/jinja/_native.py b/jinja/_native.py index aec56ab..8d42c7a 100644 --- a/jinja/_native.py +++ b/jinja/_native.py @@ -14,13 +14,12 @@ :license: BSD, see LICENSE for more details. """ from jinja.datastructure import Deferred, Undefined -from jinja.exceptions import TemplateRuntimeError class BaseContext(object): - def __init__(self, silent, globals, initial): - self._silent = silent + def __init__(self, undefined_singleton, globals, initial): + self._undefined_singleton = undefined_singleton self.current = current = {} self.stack = [globals, initial, current] self._push = self.stack.append @@ -65,9 +64,7 @@ class BaseContext(object): else: d[name] = rv return rv - if self._silent: - return Undefined - raise TemplateRuntimeError('%r is not defined' % name) + return self._undefined_singleton def __setitem__(self, name, value): """ diff --git a/jinja/_speedups.c b/jinja/_speedups.c index 86eca95..dc71a1f 100644 --- a/jinja/_speedups.c +++ b/jinja/_speedups.c @@ -18,8 +18,7 @@ #include /* Set by init_constants to real values */ -static PyObject *Undefined, *Deferred, *TemplateRuntimeError; -static Py_UNICODE *amp, *lt, *gt, *qt; +static PyObject *Deferred; /** * Internal struct used by BaseContext to store the @@ -39,7 +38,7 @@ typedef struct { struct StackLayer *initial; /* initial values */ struct StackLayer *current; /* current values */ long stacksize; /* current size of the stack */ - int silent; /* boolean value for silent failure */ + PyObject *undefined_singleton; /* the singleton returned on missing values */ } BaseContext; /** @@ -52,108 +51,11 @@ init_constants(void) PyObject *datastructure = PyImport_ImportModule("jinja.datastructure"); if (!datastructure) return 0; - PyObject *exceptions = PyImport_ImportModule("jinja.exceptions"); - if (!exceptions) { - Py_DECREF(datastructure); - return 0; - } - Undefined = PyObject_GetAttrString(datastructure, "Undefined"); Deferred = PyObject_GetAttrString(datastructure, "Deferred"); - TemplateRuntimeError = PyObject_GetAttrString(exceptions, "TemplateRuntimeError"); - - amp = ((PyUnicodeObject*)PyUnicode_DecodeASCII("&", 5, NULL))->str; - lt = ((PyUnicodeObject*)PyUnicode_DecodeASCII("<", 4, NULL))->str; - gt = ((PyUnicodeObject*)PyUnicode_DecodeASCII(">", 4, NULL))->str; - qt = ((PyUnicodeObject*)PyUnicode_DecodeASCII(""", 6, NULL))->str; - Py_DECREF(datastructure); - Py_DECREF(exceptions); return 1; } -/** - * SGML/XML escape something. - * - * XXX: this is awefully slow for non unicode objects because they - * get converted to unicode first. - */ -static PyObject* -escape(PyObject *self, PyObject *args) -{ - PyUnicodeObject *in, *out; - Py_UNICODE *outp; - int i, len; - - PyObject *text = NULL, *use_quotes = NULL; - - if (!PyArg_ParseTuple(args, "O|O", &text, &use_quotes)) - return NULL; - int quotes = use_quotes && PyObject_IsTrue(use_quotes); - in = (PyUnicodeObject*)PyObject_Unicode(text); - if (!in) - return NULL; - - /* First we need to figure out how long the escaped string will be */ - len = 0; - for (i = 0; i < in->length; i++) { - switch (in->str[i]) { - case '&': - len += 5; - break; - case '"': - len += quotes ? 6 : 1; - break; - case '<': - case '>': - len += 4; - break; - default: - len++; - } - } - - /* Do we need to escape anything at all? */ - if (len == in->length) - return (PyObject*)in; - - out = (PyUnicodeObject*)PyUnicode_FromUnicode(NULL, len); - if (!out) { - Py_DECREF(in); - return NULL; - } - - outp = out->str; - for (i = 0; i < in->length; i++) { - switch (in->str[i]) { - case '&': - Py_UNICODE_COPY(outp, amp, 5); - outp += 5; - break; - case '"': - if (quotes) { - Py_UNICODE_COPY(outp, qt, 6); - outp += 6; - } - else - *outp++ = in->str[i]; - break; - case '<': - Py_UNICODE_COPY(outp, lt, 4); - outp += 4; - break; - case '>': - Py_UNICODE_COPY(outp, gt, 4); - outp += 4; - break; - default: - *outp++ = in->str[i]; - }; - } - - Py_DECREF(in); - return (PyObject*)out; -} - /** * Deallocator for BaseContext. * @@ -176,8 +78,8 @@ BaseContext_dealloc(BaseContext *self) /** * Initializes the BaseContext. * - * Like the native python class it takes a flag telling the context - * to either fail silently with Undefined or raising a TemplateRuntimeError. + * Like the native python class it takes a reference to the undefined + * singleton which will be used for undefined values. * The other two arguments are the global namespace and the initial * namespace which usually contains the values passed to the render * function of the template. Both must be dicts. @@ -185,21 +87,15 @@ BaseContext_dealloc(BaseContext *self) static int BaseContext_init(BaseContext *self, PyObject *args, PyObject *kwds) { - PyObject *silent = NULL, *globals = NULL, *initial = NULL; + PyObject *undefined = NULL, *globals = NULL, *initial = NULL; - static char *kwlist[] = {"silent", "globals", "initial", NULL}; - if (!PyArg_ParseTupleAndKeywords(args, kwds, "OOO", kwlist, - &silent, &globals, &initial)) + if (!PyArg_ParseTuple(args, "OOO", &undefined, &globals, &initial)) return -1; if (!PyDict_Check(globals) || !PyDict_Check(initial)) { PyErr_SetString(PyExc_TypeError, "stack layers must be dicts."); return -1; } - self->silent = PyObject_IsTrue(silent); - if (self->silent == -1) - return -1; - self->current = PyMem_Malloc(sizeof(struct StackLayer)); self->current->prev = NULL; self->current->dict = PyDict_New(); @@ -218,6 +114,9 @@ BaseContext_init(BaseContext *self, PyObject *args, PyObject *kwds) Py_INCREF(globals); self->initial->prev = self->globals; + self->undefined_singleton = undefined; + Py_INCREF(undefined); + self->stacksize = 3; return 0; } @@ -347,7 +246,13 @@ BaseContext_getitem(BaseContext *self, PyObject *item) int isdeferred; struct StackLayer *current = self->current; - if (!PyString_Check(item)) + /* allow unicode keys as long as they are ascii keys */ + if (PyUnicode_CheckExact(item)) { + item = PyUnicode_AsASCIIString(item); + if (!item) + goto missing; + } + else if (!PyString_Check(item)) goto missing; /* disallow access to internal jinja values */ @@ -387,15 +292,8 @@ BaseContext_getitem(BaseContext *self, PyObject *item) } missing: - if (self->silent) { - Py_INCREF(Undefined); - return Undefined; - } - if (name) - PyErr_Format(TemplateRuntimeError, "'%s' is not defined", name); - else - PyErr_SetString(TemplateRuntimeError, "value is not a string"); - return NULL; + Py_INCREF(self->undefined_singleton); + return self->undefined_singleton; } /** @@ -407,7 +305,13 @@ BaseContext_contains(BaseContext *self, PyObject *item) char *name; struct StackLayer *current = self->current; - if (!PyString_Check(item)) + /* allow unicode objects as keys as long as they are ASCII */ + if (PyUnicode_CheckExact(item)) { + item = PyUnicode_AsASCIIString(item); + if (!item) + return 0; + } + else if (!PyString_Check(item)) return 0; name = PyString_AS_STRING(item); @@ -432,13 +336,23 @@ BaseContext_contains(BaseContext *self, PyObject *item) static int BaseContext_setitem(BaseContext *self, PyObject *item, PyObject *value) { - if (!PyString_Check(item)) { - PyErr_SetString(PyExc_TypeError, "expected string argument"); - return -1; + /* allow unicode objects as keys as long as they are ASCII */ + if (PyUnicode_CheckExact(item)) { + item = PyUnicode_AsASCIIString(item); + if (!item) { + PyErr_Clear(); + goto error; + } } + else if (!PyString_Check(item)) + goto error; if (!value) return PyDict_DelItem(self->current->dict, item); return PyDict_SetItem(self->current->dict, item, value); + +error: + PyErr_SetString(PyExc_TypeError, "expected string argument"); + return -1; } /** @@ -536,9 +450,6 @@ static PyTypeObject BaseContextType = { }; static PyMethodDef module_methods[] = { - {"escape", (PyCFunction)escape, METH_VARARGS, - "escape(s, quotes=False) -> string\n\n" - "SGML/XML a string."}, {NULL, NULL, 0, NULL} /* Sentinel */ }; diff --git a/jinja/datastructure.py b/jinja/datastructure.py index e6dedf4..01952df 100644 --- a/jinja/datastructure.py +++ b/jinja/datastructure.py @@ -34,9 +34,20 @@ def unsafe(f): return f -class UndefinedType(object): +def make_undefined(implementation): """ - An object that does not exist. + Creates an undefined singleton based on a given implementation. + It performs some tests that make sure the undefined type implements + everything it should. + """ + self = object.__new__(implementation) + self.__reduce__() + return self + + +class AbstractUndefinedType(object): + """ + Base class for any undefined type. """ __slots__ = () @@ -44,6 +55,32 @@ class UndefinedType(object): raise TypeError('cannot create %r instances' % self.__class__.__name__) + def __setattr__(self, name, value): + raise AttributeError('%r object has no attribute %r' % ( + self.__class__.__name__, + name + )) + + def __eq__(self, other): + return self is other + + def __ne__(self, other): + return self is not other + + def __copy__(self): + return self + __deepcopy__ = __copy__ + + def __reduce__(self): + raise TypeError('undefined objects have to provide a __reduce__') + + +class SilentUndefinedType(AbstractUndefinedType): + """ + An object that does not exist. + """ + __slots__ = () + def __add__(self, other): """Any operator returns the operand.""" return other @@ -79,10 +116,6 @@ class UndefinedType(object): """The unicode representation is empty.""" return u'' - def __repr__(self): - """The representation is ``'Undefined'``""" - return 'Undefined' - def __int__(self): """Converting `Undefined` to an integer ends up in ``0``""" return 0 @@ -91,33 +124,54 @@ class UndefinedType(object): """Converting `Undefined` to an float ends up in ``0.0``""" return 0.0 - def __eq__(self, other): - """`Undefined` is not equal to anything else.""" - return False - - def __ne__(self, other): - """`Undefined` is not equal to anything else.""" - return True - def __call__(self, *args, **kwargs): """Calling `Undefined` returns `Undefined`""" return self - def __copy__(self): - """Return a copy.""" - return self + def __reduce__(self): + """Helper for pickle.""" + return 'SilentUndefined' - def __deepcopy__(self, memo): - """Return a deepcopy.""" - return self + +class ComplainingUndefinedType(AbstractUndefinedType): + """ + An object that does not exist. + """ + __slots__ = () + + def __iter__(self): + """Iterating over `Undefined` returns an empty iterator.""" + if False: + yield None + + def __nonzero__(self): + """`Undefined` is considered boolean `False`""" + return False + + def __str__(self): + """The string representation raises an error.""" + raise TemplateRuntimeError('Undefined object rendered') + + def __unicode__(self): + """The unicode representation raises an error.""" + self.__str__() + + def __call__(self, *args, **kwargs): + """Calling `Undefined` returns `Undefined`""" + raise TemplateRuntimeError('Undefined object called') def __reduce__(self): """Helper for pickle.""" - return 'Undefined' + return 'ComplainingUndefined' -#: the singleton instance of UndefinedType -Undefined = object.__new__(UndefinedType) +#: the singleton instances for the undefined objects +SilentUndefined = make_undefined(SilentUndefinedType) +ComplainingUndefined = make_undefined(ComplainingUndefinedType) + +#: jinja 1.0 compatibility +Undefined = SilentUndefined +UndefinedType = SilentUndefinedType class FakeTranslator(object): @@ -172,14 +226,6 @@ class TemplateData(Markup): """ -class Flush(TemplateData): - """ - After a string marked as Flush the stream will stop buffering. - """ - - jinja_no_finalization = True - - # import these here because those modules import Deferred and Undefined # from this module. try: @@ -197,7 +243,7 @@ class Context(BaseContext): def __init__(self, *args, **kwargs): environment = args[0] - super(Context, self).__init__(environment.silent, + super(Context, self).__init__(environment.undefined_singleton, environment.globals, dict(*args[1:], **kwargs)) self._translate_func = None @@ -480,48 +526,53 @@ class TokenStream(object): class TemplateStream(object): """ - Pass it a template generator and it will buffer a few items - before yielding them as one item. Useful when working with WSGI - because a Jinja template yields far too many items per default. - - The `TemplateStream` class looks for the invisble `Flush` - markers sent by the template to find out when it should stop - buffering. + Wraps a genererator for outputing template streams. """ def __init__(self, gen): - self._next = gen.next - self._threshold = None - self.threshold = 40 + self._gen = gen + self._next = gen._next + self.buffered = False + + def disable_buffering(self): + """ + Disable the output buffering. + """ + self._next = self._gen.next + self.buffered = False + + def enable_buffering(self, size=5): + """ + Enable buffering. Buffer `size` items before + yielding them. + """ + if size <= 1: + raise ValueError('buffer size too small') + self.buffered = True + + def buffering_next(): + buf = [] + c_size = 0 + push = buf.append + next = self._gen.next + + try: + while True: + item = next() + if item: + push(item) + c_size += 1 + if c_size >= size: + raise StopIteration() + except StopIteration: + if not size: + raise + return u''.join(buf) + + self._next = buffering_next def __iter__(self): return self - def started(self): - return self._threshold is not None - started = property(started) - def next(self): - if self._threshold is None: - self._threshold = t = self.threshold - del self.threshold - else: - t = self._threshold - buf = [] - size = 0 - push = buf.append - next = self._next - - try: - while True: - item = next() - if item: - push(item) - size += 1 - if (size and item.__class__ is Flush) or size >= t: - raise StopIteration() - except StopIteration: - pass - if not size: - raise StopIteration() - return u''.join(buf) + return self._next() diff --git a/jinja/defaults.py b/jinja/defaults.py index 43b67a1..1a0e10c 100644 --- a/jinja/defaults.py +++ b/jinja/defaults.py @@ -11,7 +11,7 @@ from jinja.filters import FILTERS as DEFAULT_FILTERS from jinja.tests import TESTS as DEFAULT_TESTS from jinja.utils import debug_helper, safe_range, generate_lorem_ipsum, \ - watch_changes, flush + watch_changes __all__ = ['DEFAULT_FILTERS', 'DEFAULT_TESTS', 'DEFAULT_NAMESPACE'] @@ -21,6 +21,5 @@ DEFAULT_NAMESPACE = { 'range': safe_range, 'debug': debug_helper, 'lipsum': generate_lorem_ipsum, - 'watchchanges': watch_changes, - 'flush': flush + 'watchchanges': watch_changes } diff --git a/jinja/environment.py b/jinja/environment.py index 026506d..c4a01ef 100644 --- a/jinja/environment.py +++ b/jinja/environment.py @@ -12,7 +12,7 @@ import re from jinja.lexer import Lexer from jinja.parser import Parser from jinja.loaders import LoaderWrapper -from jinja.datastructure import Undefined, Markup, Context, FakeTranslator +from jinja.datastructure import SilentUndefined, Markup, Context, FakeTranslator from jinja.utils import collect_translations, get_attribute from jinja.exceptions import FilterNotFound, TestNotFound, \ SecurityException, TemplateSyntaxError, TemplateRuntimeError @@ -48,7 +48,7 @@ class Environment(object): filters=None, tests=None, context_class=Context, - silent=True, + undefined_singleton=SilentUndefined, friendly_traceback=True): """ Here the possible initialization parameters: @@ -97,10 +97,8 @@ class Environment(object): `context_class` the context class this template should use. See the `Context` documentation for more details. - `silent` set this to `False` if you want to receive - errors for missing template variables or - attributes. Defaults to `False`. *new in - Jinja 1.1* + `undefined_singleton` The singleton value that is used for missing + variables. *new in Jinja 1.1* `friendly_traceback` Set this to `False` to disable the developer friendly traceback rewriting. Whenever an runtime or syntax error occours jinja will @@ -132,7 +130,7 @@ class Environment(object): self.tests = tests is None and DEFAULT_TESTS.copy() or tests self.default_filters = default_filters or [] self.context_class = context_class - self.silent = silent + self.undefined_singleton = undefined_singleton self.friendly_traceback = friendly_traceback # global namespace @@ -213,15 +211,18 @@ class Environment(object): """ Convert a value to unicode with the rules defined on the environment. """ - if value in (None, Undefined): + # undefined and None expand to "" + if value in (None, self.undefined_singleton): return u'' + # things that are already unicode can pass. As long as nobody + # does ugly things with the class it works for jinja too elif isinstance(value, unicode): return value - else: - try: - return unicode(value) - except UnicodeError: - return str(value).decode(self.charset, 'ignore') + # otherwise try to use __unicode__ or decode __str__ + try: + return unicode(value) + except UnicodeError: + return str(value).decode(self.charset, 'ignore') def get_translator(self, context): """ @@ -291,10 +292,9 @@ class Environment(object): return get_attribute(obj, name) except (AttributeError, SecurityException): pass - if self.silent: - return Undefined - raise TemplateRuntimeError('attribute %r or object %r not defined' % ( - name, obj)) + if obj is self.undefined_singleton: + return getattr(self.undefined_singleton, name) + return self.undefined_singleton def get_attributes(self, obj, attributes): """ @@ -317,7 +317,7 @@ class Environment(object): kwargs.update(dyn_kwargs) if getattr(f, 'jinja_unsafe_call', False) or \ getattr(f, 'alters_data', False): - return Undefined + return self.undefined_singleton if getattr(f, 'jinja_context_callable', False): args = (self, context) + args return f(*args, **kwargs) @@ -329,7 +329,7 @@ class Environment(object): """ if getattr(f, 'jinja_unsafe_call', False) or \ getattr(f, 'alters_data', False): - return Undefined + return self.undefined_singleton if getattr(f, 'jinja_context_callable', False): return f(self, context) return f() @@ -340,8 +340,10 @@ class Environment(object): evaluator the source generated by the python translator will call this function for all variables. """ - if value is Undefined or value is None: + if value is None: return u'' + elif value is self.undefined_singleton: + return unicode(value) elif getattr(value, 'jinja_no_finalization', False): return value val = self.to_unicode(value) diff --git a/jinja/filters.py b/jinja/filters.py index 49e07b4..08208ca 100644 --- a/jinja/filters.py +++ b/jinja/filters.py @@ -12,7 +12,7 @@ import re from random import choice from urllib import urlencode, quote from jinja.utils import urlize, escape -from jinja.datastructure import Undefined, Markup, TemplateData +from jinja.datastructure import Markup, TemplateData from jinja.exceptions import FilterArgumentError @@ -114,7 +114,8 @@ def do_escape(attribute=False): return s elif hasattr(s, '__html__'): return s.__html__() - #: small speedup + #: small speedup, do not convert to unicode if we already + #: have an unicode object. if s.__class__ is not unicode: s = env.to_unicode(s) return e(s, attribute) @@ -150,7 +151,7 @@ def do_xmlattr(): raise TypeError('a dict is required') result = [] for key, value in d.iteritems(): - if value not in (None, Undefined): + if value not in (None, env.undefined_singleton): result.append(u'%s="%s"' % ( e(env.to_unicode(key)), e(env.to_unicode(value), True) @@ -236,7 +237,7 @@ def do_default(default_value=u'', boolean=False): {{ ''|default('the string was empty', true) }} """ def wrapped(env, context, value): - if (boolean and not value) or value in (Undefined, None): + if (boolean and not value) or value in (env.undefined_singleton, None): return default_value return value return wrapped @@ -314,9 +315,7 @@ def do_first(): try: return iter(seq).next() except StopIteration: - if env.silent: - return Undefined - raise TemplateRuntimeError('%r is empty' % seq) + return env.undefined_singleton[0] return wrapped @@ -328,9 +327,7 @@ def do_last(): try: return iter(_reversed(seq)).next() except StopIteration: - if env.silent: - return Undefined - raise TemplateRuntimeError('%r is empty' % seq) + return env.undefined_singleton[-1] return wrapped @@ -342,9 +339,7 @@ def do_random(): try: return choice(seq) except IndexError: - if env.silent: - return Undefined - raise TemplateRuntimeError('%r is empty' % seq) + return env.undefined_singleton[0] return wrapped @@ -655,7 +650,7 @@ def do_capture(name='captured', clean=False): def wrapped(env, context, value): context[name] = value if clean: - return Undefined + return TemplateData() return value return wrapped diff --git a/jinja/nodes.py b/jinja/nodes.py index cba063d..f3af166 100644 --- a/jinja/nodes.py +++ b/jinja/nodes.py @@ -73,6 +73,28 @@ class Text(Node): return 'Text(%r)' % (self.text,) +class DynamicText(Node): + """ + Note that represents normal text with string formattings. + Those are used for texts that contain variables. The attribute + `variables` contains a list of Print tags and nothing else. + """ + + def __init__(self, lineno, format_string, variables): + self.lineno = lineno + self.format_string = format_string + self.variables = variables + + def get_items(self): + return [self.format_string] + list(self.variables) + + def __repr__(self): + return 'DynamicText(%r, %r)' % ( + self.text, + self.variables + ) + + class NodeList(list, Node): """ A node that stores multiple childnodes. diff --git a/jinja/parser.py b/jinja/parser.py index f93a1d5..c76f990 100644 --- a/jinja/parser.py +++ b/jinja/parser.py @@ -87,9 +87,9 @@ class Parser(object): #: set of directives that are only available in a certain #: context. - self.context_directives = set(['elif', 'else', 'endblock', - 'endfilter', 'endfor', 'endif', 'endmacro', 'endraw', - 'endtrans', 'pluralize' + self.context_directives = set([ + 'elif', 'else', 'endblock', 'endfilter', 'endfor', 'endif', + 'endmacro', 'endraw', 'endtrans', 'pluralize' ]) self.tokenstream = environment.lexer.tokenize(source, filename) @@ -223,7 +223,8 @@ class Parser(object): def handle_macro_directive(self, lineno, gen): """ - Handle {% macro foo bar, baz %}. + Handle {% macro foo bar, baz %} as well as + {% macro foo(bar, baz) %}. """ try: macro_name = gen.next() @@ -240,7 +241,13 @@ class Parser(object): 'as macro name.' % macro_name[2], lineno, self.filename) - ast = self.parse_python(lineno, gen, 'def %s(%%s):pass' % + # make the punctuation around arguments optional + arg_list = list(gen) + if arg_list and arg_list[0][1:] == ('operator', '(') and \ + arg_list[-1][1:] == ('operator', ')'): + arg_list = arg_list[1:-1] + + ast = self.parse_python(lineno, arg_list, 'def %s(%%s):pass' % str(macro_name[2][:-1])) body = self.subparse(end_of_macro, True) self.close_remaining_block() @@ -620,12 +627,42 @@ class Parser(object): """ def finish(): """Helper function to remove unused nodelists.""" + if data_buffer: + flush_data_buffer() if len(result) == 1: return result[0] return result + def flush_data_buffer(): + """Helper function to write the contents of the buffer + to the result nodelist.""" + format_string = [] + insertions = [] + for item in data_buffer: + if item[0] == 'variable': + p = self.handle_print_directive(*item[1:]) + format_string.append('%s') + insertions.append(p) + else: + format_string.append(item[2].replace('%', '%%')) + # if we do not have insertions yield it as text node + if not insertions: + result.append(nodes.Text(data_buffer[0][1], + (u''.join(format_string)).replace('%%', '%'))) + # if we do not have any text data we yield some variable nodes + elif len(insertions) == len(format_string): + result.extend(insertions) + # otherwise we go with a dynamic text node + else: + result.append(nodes.DynamicText(data_buffer[0][1], + u''.join(format_string), + insertions)) + # clear the buffer + del data_buffer[:] + lineno = self.tokenstream.last[0] result = nodes.NodeList(lineno) + data_buffer = [] for lineno, token, data in self.tokenstream: # comments if token == 'comment_begin': @@ -635,11 +672,16 @@ class Parser(object): # parse everything till the end of it. elif token == 'variable_begin': gen = self.tokenstream.fetch_until(end_of_variable, True) - result.append(self.directives['print'](lineno, gen)) + data_buffer.append(('variable', lineno, tuple(gen))) # this token marks the start of a block. like for variables # just parse everything until the end of the block elif token == 'block_begin': + # if we have something in the buffer we write the + # data back + if data_buffer: + flush_data_buffer() + gen = self.tokenstream.fetch_until(end_of_block, True) try: lineno, token, data = gen.next() @@ -686,7 +728,7 @@ class Parser(object): # tokens just exist in block or variable sections. (if the # tokenizer is not brocken) elif token in 'data': - result.append(nodes.Text(lineno, data)) + data_buffer.append(('text', lineno, data)) # so this should be unreachable code else: diff --git a/jinja/tests.py b/jinja/tests.py index 6960a04..8f0e395 100644 --- a/jinja/tests.py +++ b/jinja/tests.py @@ -9,7 +9,6 @@ :license: BSD, see LICENSE for more details. """ import re -from jinja.datastructure import Undefined number_re = re.compile(r'^-?\d+(\.\d+)?$') @@ -45,7 +44,7 @@ def test_defined(): See also the ``default`` filter. """ - return lambda e, c, v: v is not Undefined + return lambda e, c, v: v is not e.undefined_singleton def test_lower(): diff --git a/jinja/translators/python.py b/jinja/translators/python.py index 62bcd39..f2195fd 100644 --- a/jinja/translators/python.py +++ b/jinja/translators/python.py @@ -16,6 +16,21 @@ It also adds debug symbols used by the traceback toolkit implemented in `jinja.utils`. + Implementation Details + ====================== + + Some of the semantics are handled directly in the translator in order + to speed up the parsing process. For example the semantics for the + `is` operator are handled here, changing this would require and + additional traversing of the node tree in the parser. Thus the actual + translation process can raise a `TemplateSyntaxError` too. + + It might sound strange but the translator tries to keep the generated + code readable as much as possible. This simplifies debugging the Jinja + core a lot. The additional processing overhead is just relevant for + the translation process, the additional comments and whitespace won't + appear in the saved bytecode. + :copyright: 2007 by Armin Ronacher. :license: BSD, see LICENSE for more details. """ @@ -36,28 +51,25 @@ from jinja.utils import translate_exception, capture_generator, \ _debug_re = re.compile(r'^\s*\# DEBUG\(filename=(?P.*?), ' r'lineno=(?P\d+)\)$') +# For Python2.3 compatibiilty +try: + set +except NameError: + from sets import Set as set +# For Python 2.3/2.4 compatibility try: GeneratorExit except NameError: class GeneratorExit(Exception): - """For python2.3/python2.4 compatibility""" - + pass +# For Pythons without conditional expressions try: - set -except NameError: - from sets import Set as set - - -def _to_tuple(args): - """ - Return a tuple repr without nested repr. - """ - return '(%s%s)' % ( - ', '.join(args), - len(args) == 1 and ',' or '' - ) + exec '0 if 0 else 0' + have_conditional_expr = True +except SyntaxError: + have_conditional_expr = False class Template(object): @@ -161,7 +173,7 @@ class PythonTranslator(Translator): 'true': 'True', 'false': 'False', 'none': 'None', - 'undefined': 'Undefined' + 'undefined': 'undefined_singleton' } #: bind the nodes to the callback functions. There are @@ -173,6 +185,7 @@ class PythonTranslator(Translator): # jinja nodes nodes.Template: self.handle_template, nodes.Text: self.handle_template_text, + nodes.DynamicText: self.handle_dynamic_text, nodes.NodeList: self.handle_node_list, nodes.ForLoop: self.handle_for_loop, nodes.IfCondition: self.handle_if_condition, @@ -254,6 +267,15 @@ class PythonTranslator(Translator): """ return (' ' * (self.indention * 4)) + text + def to_tuple(self, args): + """ + Return a tuple repr without nested repr. + """ + return '(%s%s)' % ( + ', '.join(args), + len(args) == 1 and ',' or '' + ) + def nodeinfo(self, node, force=False): """ Return a comment that helds the node informations or None @@ -292,7 +314,7 @@ class PythonTranslator(Translator): n.filename) filters.append('(%r, %s)' % ( n.node.name, - _to_tuple(args) + self.to_tuple(args) )) elif n.__class__ is ast.Name: filters.append('(%r, ())' % n.name) @@ -301,7 +323,8 @@ class PythonTranslator(Translator): 'hardcoded function name from the ' 'filter namespace', n.lineno, n.filename) - return 'apply_filters(%s, context, %s)' % (s, _to_tuple(filters)) + self.used_shortcuts.add('apply_filters') + return 'apply_filters(%s, context, %s)' % (s, self.to_tuple(filters)) def handle_node(self, node): """ @@ -323,8 +346,19 @@ class PythonTranslator(Translator): """ Reset translation variables such as indention or cycle id """ + #: current level of indention self.indention = 0 + #: each {% cycle %} tag has a unique ID which increments + #: automatically for each tag. self.last_cycle_id = 0 + #: set of used shortcuts jinja has to make local automatically + self.used_shortcuts = set(['undefined_singleton']) + #: set of used datastructures jinja has to import + self.used_data_structures = set() + #: set of used utils jinja has to import + self.used_utils = set() + #: flags for runtime error + self.require_runtime_error = False def translate(self): """ @@ -349,17 +383,26 @@ class PythonTranslator(Translator): # update the blocks there. Once this is done we drop the current # template in favor of the new one. Do that until we found the # root template. - requirements_todo = [] parent = None overwrites = {} blocks = {} + requirements = [] + outer_filename = node.filename or '