50/44c5d089e8676a7077854d7fcb1e03bb60d1f0

   1 Return-Path: <wking@tremily.us>\r
   2 X-Original-To: notmuch@notmuchmail.org\r
   3 Delivered-To: notmuch@notmuchmail.org\r
   4 Received: from localhost (localhost [127.0.0.1])\r
   5         by olra.theworths.org (Postfix) with ESMTP id 59509429E35\r
   6         for <notmuch@notmuchmail.org>; Mon,  3 Feb 2014 03:10:17 -0800 (PST)\r
   7 X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
   8 X-Amavis-Alert: BAD HEADER SECTION, Duplicate header field: "References"\r
   9 X-Spam-Flag: NO\r
  10 X-Spam-Score: 0\r
  11 X-Spam-Level: \r
  12 X-Spam-Status: No, score=0 tagged_above=-999 required=5\r
  13         tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001]\r
  14         autolearn=disabled\r
  15 Received: from olra.theworths.org ([127.0.0.1])\r
  16         by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
  17         with ESMTP id YYMknRkc3ykA for <notmuch@notmuchmail.org>;\r
  18         Mon,  3 Feb 2014 03:10:09 -0800 (PST)\r
  19 Received: from qmta07.westchester.pa.mail.comcast.net\r
  20         (qmta07.westchester.pa.mail.comcast.net [76.96.62.64])\r
  21         by olra.theworths.org (Postfix) with ESMTP id 4C3DA431E64\r
  22         for <notmuch@notmuchmail.org>; Mon,  3 Feb 2014 03:10:09 -0800 (PST)\r
  23 Received: from omta14.westchester.pa.mail.comcast.net ([76.96.62.60])\r
  24         by qmta07.westchester.pa.mail.comcast.net with comcast\r
  25         id Mn3z1n0011HzFnQ57nA9a7; Mon, 03 Feb 2014 11:10:09 +0000\r
  26 Received: from odin.tremily.us ([24.18.63.50])\r
  27         by omta14.westchester.pa.mail.comcast.net with comcast\r
  28         id Mn881n00d152l3L3an89be; Mon, 03 Feb 2014 11:08:09 +0000\r
  29 Received: from mjolnir.tremily.us (unknown [192.168.0.140])\r
  30         by odin.tremily.us (Postfix) with ESMTPS id 1BEBFFB4D54;\r
  31         Mon,  3 Feb 2014 03:00:42 -0800 (PST)\r
  32 Received: (nullmailer pid 700 invoked by uid 1000);\r
  33         Mon, 03 Feb 2014 10:59:41 -0000\r
  34 From: "W. Trevor King" <wking@tremily.us>\r
  35 To: notmuch@notmuchmail.org\r
  36 Subject: [PATCH 10/17] nmbug-status: Add Page and HtmlPage for modular\r
  37         rendering\r
  38 Date: Mon,  3 Feb 2014 02:59:28 -0800\r
  39 Message-Id:\r
  40  <fd0d9baf256cf815007c2f9f32fc5f484e680766.1391424512.git.wking@tremily.us>\r
  41 X-Mailer: git-send-email 1.8.5.2.8.g0f6c0d1\r
  42 In-Reply-To: <cover.1391424512.git.wking@tremily.us>\r
  43 References: <cover.1391424512.git.wking@tremily.us>\r
  44 In-Reply-To: <cover.1391424512.git.wking@tremily.us>\r
  45 References: <cover.1391424512.git.wking@tremily.us>\r
  46 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net;\r
  47         s=q20121106; t=1391425809;\r
  48         bh=lkyEHMaK0pWitOsDQufbYK9aEYIMwylLXL0ZMnpMSsI=;\r
  49         h=Received:Received:Received:Received:From:To:Subject:Date:\r
  50         Message-Id;\r
  51         b=CBHmANbcwj9AXJZ+aTm56FEbaHgI4uy1b/t0CW0woHPrTjwY5hHcVdNgXal1eYZhp\r
  52         dwcMKNFFJdONf9nWz3bJ2S0jyllcpuoUCAjXpwJwHom1RDGIRP/8iNA/3qmkG7hcIQ\r
  53         c4Dk/9ajCSe/lcUk7FBlD6FWirkEAlgx9TWt7STMWMw+fFlFafseRwgWcVXhD6pnim\r
  54         pU2uJMV6XLcxHM8yLL5x1EeHEuBx5OH2NFETvPBs6PjeI6HvuJmHSV6zWG7f4uwCw+\r
  55         QVTV7mkW8mDWHlp9wf1MWuRps4CgGk3S904j0Iz0L+22Hj4mBLEP3FSVP4Oe9I6D/T\r
  56         PVYCM2kgIrp3w==\r
  57 X-BeenThere: notmuch@notmuchmail.org\r
  58 X-Mailman-Version: 2.1.13\r
  59 Precedence: list\r
  60 List-Id: "Use and development of the notmuch mail system."\r
  61         <notmuch.notmuchmail.org>\r
  62 List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
  63         <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
  64 List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
  65 List-Post: <mailto:notmuch@notmuchmail.org>\r
  66 List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
  67 List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
  68         <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
  69 X-List-Received-Date: Mon, 03 Feb 2014 11:10:17 -0000\r
  70 \r
  71 I was having trouble understanding the logic of the longish print_view\r
  72 function, so I refactored the output generation into modular bits.\r
  73 The basic text rendering is handled by Page, which has enough hooks\r
  74 that HtmlPage can borrow the logic and slot-in HTML generators.\r
  75 \r
  76 By modularizing the logic it should also be easier to build other\r
  77 renderers if folks want to customize the layout for other projects.\r
  78 \r
  79 Timezones\r
  80 =========\r
  81 \r
  82 This commit has not effect on the output, except that some dates have\r
  83 been converted from the sender's timezone to UTC due to:\r
  84 \r
  85   -            val = m.get_header(header)\r
  86   -            ...\r
  87   -            if header == 'date':\r
  88   -                val = str.join(' ', val.split(None)[1:4])\r
  89   -                val = str(datetime.datetime.strptime(val, '%d %b %Y').date())\r
  90   ...\r
  91   +                value = str(datetime.datetime.utcfromtimestamp(\r
  92   +                    message.get_date()).date())\r
  93 \r
  94 I also tweaked the HTML header date to be utcnow instead of the local\r
  95 now() to make all times independent of the generator's local time.\r
  96 This matches Gmane, which converts all Date headers to UTC (although\r
  97 they use a 'GMT' suffix).  Notmuch uses\r
  98 g_mime_utils_header_decode_date to calculate the UTC timestamps, but\r
  99 uses a NULL tz_offset which drops the information we'd need to get\r
 100 back to the sender's local time [1].  With the generator's local time\r
 101 arbitrarily different from the sender's and viewer's local time,\r
 102 sticking with UTC seems the best bet.\r
 103 \r
 104 [1]: https://developer.gnome.org/gmime/stable/gmime-gmime-utils.html#g-mime-utils-header-decode-date\r
 105 ---\r
 106  devel/nmbug/nmbug-status | 292 +++++++++++++++++++++++++++--------------------\r
 107  1 file changed, 171 insertions(+), 121 deletions(-)\r
 108 \r
 109 diff --git a/devel/nmbug/nmbug-status b/devel/nmbug/nmbug-status\r
 110 index 22b6b10..7778029 100755\r
 111 --- a/devel/nmbug/nmbug-status\r
 112 +++ b/devel/nmbug/nmbug-status\r
 113 @@ -5,10 +5,13 @@\r
 114  # dependencies\r
 115  #       - python 2.6 for json\r
 116  #       - argparse; either python 2.7, or install separately\r
 117 +#       - collections; python 2.7\r
 118  \r
 119  from __future__ import print_function\r
 120 +from __future__ import unicode_literals\r
 121  \r
 122  import codecs\r
 123 +import collections\r
 124  import datetime\r
 125  import email.utils\r
 126  import locale\r
 127 @@ -24,6 +27,7 @@ import subprocess\r
 128  \r
 129  \r
 130  _ENCODING = locale.getpreferredencoding() or sys.getdefaultencoding()\r
 131 +_PAGES = {}\r
 132  \r
 133  \r
 134  def read_config(path=None, encoding=None):\r
 135 @@ -50,104 +54,175 @@ def read_config(path=None, encoding=None):\r
 136      return json.load(fp)\r
 137  \r
 138  \r
 139 -class Thread:\r
 140 -    def __init__(self, last, lines):\r
 141 -        self.last = last\r
 142 -        self.lines = lines\r
 143 -\r
 144 -    def join_utf8_with_newlines(self):\r
 145 -        return '\n'.join( (line.encode('utf-8') for line in self.lines) )\r
 146 -\r
 147 -\r
 148 -def output_with_separator(threadlist, sep):\r
 149 -    outputs = (thread.join_utf8_with_newlines() for thread in threadlist)\r
 150 -    print(sep.join(outputs))\r
 151 -\r
 152 -\r
 153 -def print_view(database, title, query, comment,\r
 154 -               headers=('date', 'from', 'subject')):\r
 155 -\r
 156 -    query_string = ' and '.join(query)\r
 157 -    q_new = notmuch.Query(database, query_string)\r
 158 -    q_new.set_sort(notmuch.Query.SORT.OLDEST_FIRST)\r
 159 -\r
 160 -    last_thread_id = ''\r
 161 -    threads = {}\r
 162 -    threadlist = []\r
 163 -    out = {}\r
 164 -    last = None\r
 165 -    lines = None\r
 166 -\r
 167 -    if output_format == 'html':\r
 168 -        print('<h3><a name="%s" />%s</h3>' % (title, title))\r
 169 -        print(comment)\r
 170 -        print('The view is generated from the following query:')\r
 171 -        print('<blockquote>')\r
 172 -        print(query_string)\r
 173 -        print('</blockquote>')\r
 174 -        print('<table>\n')\r
 175 -\r
 176 -    for m in q_new.search_messages():\r
 177 -\r
 178 -        thread_id = m.get_thread_id()\r
 179 -\r
 180 -        if thread_id != last_thread_id:\r
 181 -            if threads.has_key(thread_id):\r
 182 -                last = threads[thread_id].last\r
 183 -                lines = threads[thread_id].lines\r
 184 +class Thread (list):\r
 185 +    def __init__(self):\r
 186 +        self.running_data = {}\r
 187 +\r
 188 +\r
 189 +class Page (object):\r
 190 +    def __init__(self, header=None, footer=None):\r
 191 +        self.header = header\r
 192 +        self.footer = footer\r
 193 +\r
 194 +    def write(self, database, views, stream=None):\r
 195 +        if not stream:\r
 196 +            try:  # Python 3\r
 197 +                byte_stream = sys.stdout.buffer\r
 198 +            except AttributeError:  # Python 2\r
 199 +                byte_stream = sys.stdout\r
 200 +            stream = codecs.getwriter(encoding='UTF-8')(stream=byte_stream)\r
 201 +        self._write_header(views=views, stream=stream)\r
 202 +        for view in views:\r
 203 +            self._write_view(database=database, view=view, stream=stream)\r
 204 +        self._write_footer(views=views, stream=stream)\r
 205 +\r
 206 +    def _write_header(self, views, stream):\r
 207 +        if self.header:\r
 208 +            stream.write(self.header)\r
 209 +\r
 210 +    def _write_footer(self, views, stream):\r
 211 +        if self.footer:\r
 212 +            stream.write(self.footer)\r
 213 +\r
 214 +    def _write_view(self, database, view, stream):\r
 215 +        if 'query-string' not in view:\r
 216 +            query = view['query']\r
 217 +            view['query-string'] = ' and '.join(query)\r
 218 +        q = notmuch.Query(database, view['query-string'])\r
 219 +        q.set_sort(notmuch.Query.SORT.OLDEST_FIRST)\r
 220 +        threads = self._get_threads(messages=q.search_messages())\r
 221 +        self._write_view_header(view=view, stream=stream)\r
 222 +        self._write_threads(threads=threads, stream=stream)\r
 223 +\r
 224 +    def _get_threads(self, messages):\r
 225 +        threads = collections.OrderedDict()\r
 226 +        for message in messages:\r
 227 +            thread_id = message.get_thread_id()\r
 228 +            if thread_id in threads:\r
 229 +                thread = threads[thread_id]\r
 230              else:\r
 231 -                last = {}\r
 232 -                lines = []\r
 233 -                thread = Thread(last, lines)\r
 234 +                thread = Thread()\r
 235                  threads[thread_id] = thread\r
 236 -                for h in headers:\r
 237 -                    last[h] = ''\r
 238 -                threadlist.append(thread)\r
 239 -            last_thread_id = thread_id\r
 240 -\r
 241 +            thread.running_data, display_data = self._message_display_data(\r
 242 +                running_data=thread.running_data, message=message)\r
 243 +            thread.append(display_data)\r
 244 +        return list(threads.values())\r
 245 +\r
 246 +    def _write_view_header(self, view, stream):\r
 247 +        pass\r
 248 +\r
 249 +    def _write_threads(self, threads, stream):\r
 250 +        for thread in threads:\r
 251 +            for message_display_data in thread:\r
 252 +                stream.write(\r
 253 +                    ('{date:10.10s} {from:20.20s} {subject:40.40s}\n'\r
 254 +                     '{message-id-term:>72}\n'\r
 255 +                     ).format(**message_display_data))\r
 256 +            if thread != threads[-1]:\r
 257 +                stream.write('\n')\r
 258 +\r
 259 +    def _message_display_data(self, running_data, message):\r
 260 +        headers = ('thread-id', 'message-id', 'date', 'from', 'subject')\r
 261 +        data = {}\r
 262          for header in headers:\r
 263 -            val = m.get_header(header)\r
 264 -\r
 265 -            if header == 'date':\r
 266 -                val = str.join(' ', val.split(None)[1:4])\r
 267 -                val = str(datetime.datetime.strptime(val, '%d %b %Y').date())\r
 268 -            elif header == 'from':\r
 269 -                (val, addr) = email.utils.parseaddr(val)\r
 270 -                if val == '':\r
 271 -                    val = addr.split('@')[0]\r
 272 -\r
 273 -            if header != 'subject' and last[header] == val:\r
 274 -                out[header] = ''\r
 275 +            if header == 'thread-id':\r
 276 +                value = message.get_thread_id()\r
 277 +            elif header == 'message-id':\r
 278 +                value = message.get_message_id()\r
 279 +                data['message-id-term'] = 'id:"{}"'.format(value)\r
 280 +            elif header == 'date':\r
 281 +                value = str(datetime.datetime.utcfromtimestamp(\r
 282 +                    message.get_date()).date())\r
 283              else:\r
 284 -                out[header] = val\r
 285 -                last[header] = val\r
 286 -\r
 287 -        mid = m.get_message_id()\r
 288 -        out['id'] = 'id:"%s"' % mid\r
 289 -\r
 290 -        if output_format == 'html':\r
 291 -\r
 292 -            out['subject'] = '<a href="http://mid.gmane.org/%s">%s</a>' % (\r
 293 -                quote(mid), out['subject'])\r
 294 -\r
 295 -            lines.append(' <tr><td>%s' % out['date'])\r
 296 -            lines.append('</td><td>%s' % out['id'])\r
 297 -            lines.append('</td></tr>')\r
 298 -            lines.append(' <tr><td>%s' % out['from'])\r
 299 -            lines.append('</td><td>%s' % out['subject'])\r
 300 -            lines.append('</td></tr>')\r
 301 -        else:\r
 302 -            lines.append('%(date)-10.10s %(from)-20.20s %(subject)-40.40s\n%(id)72s' % out)\r
 303 -\r
 304 -    if output_format == 'html':\r
 305 -        output_with_separator(threadlist,\r
 306 -                              '\n<tr><td colspan="2"><br /></td></tr>\n')\r
 307 -        print('</table>')\r
 308 -    else:\r
 309 -        output_with_separator(threadlist, '\n\n')\r
 310 -\r
 311 +                value = message.get_header(header)\r
 312 +            if header == 'from':\r
 313 +                (value, addr) = email.utils.parseaddr(value)\r
 314 +                if not value:\r
 315 +                    value = addr.split('@')[0]\r
 316 +            data[header] = value\r
 317 +        next_running_data = data.copy()\r
 318 +        for header, value in data.items():\r
 319 +            if header in ['message-id', 'subject']:\r
 320 +                continue\r
 321 +            if value == running_data.get(header, None):\r
 322 +                data[header] = ''\r
 323 +        return (next_running_data, data)\r
 324 +\r
 325 +\r
 326 +class HtmlPage (Page):\r
 327 +    def _write_header(self, views, stream):\r
 328 +        super(HtmlPage, self)._write_header(views=views, stream=stream)\r
 329 +        stream.write('<ul>\n')\r
 330 +        for view in views:\r
 331 +            stream.write(\r
 332 +                '<li><a href="#{title}">{title}</a></li>\n'.format(**view))\r
 333 +        stream.write('</ul>\n')\r
 334 +\r
 335 +    def _write_view_header(self, view, stream):\r
 336 +        stream.write('<h3><a name="{title}" />{title}</h3>\n'.format(**view))\r
 337 +        if 'comment' in view:\r
 338 +            stream.write(view['comment'])\r
 339 +            stream.write('\n')\r
 340 +        for line in [\r
 341 +                'The view is generated from the following query:',\r
 342 +                '<blockquote>',\r
 343 +                view['query-string'],\r
 344 +                '</blockquote>',\r
 345 +                ]:\r
 346 +            stream.write(line)\r
 347 +            stream.write('\n')\r
 348 +\r
 349 +    def _write_threads(self, threads, stream):\r
 350 +        if not threads:\r
 351 +            return\r
 352 +        stream.write('<table>\n')\r
 353 +        for thread in threads:\r
 354 +            for message_display_data in thread:\r
 355 +                stream.write((\r
 356 +                    '<tr><td>{date}\n'\r
 357 +                    '</td><td>{message-id-term}\n'\r
 358 +                    '</td></tr>\n'\r
 359 +                    '<tr><td>{from}\n'\r
 360 +                    '</td><td>{subject}\n'\r
 361 +                    '</td></tr>\n'\r
 362 +                    ).format(**message_display_data))\r
 363 +            if thread != threads[-1]:\r
 364 +                stream.write('<tr><td colspan="2"><br /></td></tr>\n')\r
 365 +        stream.write('</table>\n')\r
 366 +\r
 367 +    def _message_display_data(self, *args, **kwargs):\r
 368 +        running_data, display_data = super(\r
 369 +            HtmlPage, self)._message_display_data(\r
 370 +                *args, **kwargs)\r
 371 +        if 'subject' in display_data and 'message-id' in display_data:\r
 372 +            d = {\r
 373 +                'message-id': quote(display_data['message-id']),\r
 374 +                'subject': display_data['subject'],\r
 375 +                }\r
 376 +            display_data['subject'] = (\r
 377 +                '<a href="http://mid.gmane.org/{message-id}">{subject}</a>'\r
 378 +                ).format(**d)\r
 379 +        return (running_data, display_data)\r
 380 +\r
 381 +\r
 382 +_PAGES['text'] = Page()\r
 383 +_PAGES['html'] = HtmlPage(\r
 384 +    header='''<?xml version="1.0" encoding="utf-8" ?>\r
 385 +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">\r
 386 +<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">\r
 387 +<head>\r
 388 +<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />\r
 389 +<title>Notmuch Patches</title>\r
 390 +</head>\r
 391 +<body>\r
 392 +<h2>Notmuch Patches</h2>\r
 393 +Generated: {date}<br />\r
 394 +For more infomation see <a href="http://notmuchmail.org/nmbug">nmbug</a>\r
 395 +<h3>Views</h3>\r
 396 +'''.format(date=datetime.datetime.utcnow().date()),\r
 397 +    footer='</body>\n</html>\n',\r
 398 +    )\r
 399  \r
 400 -# parse command line arguments\r
 401  \r
 402  parser = argparse.ArgumentParser()\r
 403  parser.add_argument('--text', help='output plain text format',\r
 404 @@ -177,34 +252,9 @@ else:\r
 405      import notmuch\r
 406  \r
 407  if args.text:\r
 408 -    output_format = 'text'\r
 409 +    page = _PAGES['text']\r
 410  else:\r
 411 -    output_format = 'html'\r
 412 -\r
 413 -# main program\r
 414 +    page = _PAGES['html']\r
 415  \r
 416  db = notmuch.Database(mode=notmuch.Database.MODE.READ_ONLY)\r
 417 -\r
 418 -if output_format == 'html':\r
 419 -    print('''<?xml version="1.0" encoding="utf-8" ?>\r
 420 -<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">\r
 421 -<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">\r
 422 -<head>\r
 423 -<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />\r
 424 -<title>Notmuch Patches</title>\r
 425 -</head>\r
 426 -<body>\r
 427 -<h2>Notmuch Patches</h2>\r
 428 -Generated: {date}<br />\r
 429 -For more infomation see <a href="http://notmuchmail.org/nmbug">nmbug</a>\r
 430 -<h3>Views</h3>\r
 431 -<ul>'''.format(date=datetime.datetime.utcnow().date()))\r
 432 -    for view in config['views']:\r
 433 -        print('<li><a href="#%(title)s">%(title)s</a></li>' % view)\r
 434 -    print('</ul>')\r
 435 -\r
 436 -for view in config['views']:\r
 437 -    print_view(database=db, **view)\r
 438 -\r
 439 -if output_format == 'html':\r
 440 -    print('</body>\n</html>')\r
 441 +page.write(database=db, views=config['views'])\r
 442 -- \r
 443 1.8.5.2.8.g0f6c0d1\r
 444 \r