2c/0c55fad0fb9c07b8093772eb5204c524853a42

   1 Return-Path: <dkg@fifthhorseman.net>\r
   2 X-Original-To: notmuch@notmuchmail.org\r
   3 Delivered-To: notmuch@notmuchmail.org\r
   4 Received: from localhost (localhost [127.0.0.1])\r
   5  by arlo.cworth.org (Postfix) with ESMTP id ADF3E6DE0B38\r
   6  for <notmuch@notmuchmail.org>; Fri, 22 Jan 2016 09:02:43 -0800 (PST)\r
   7 X-Virus-Scanned: Debian amavisd-new at cworth.org\r
   8 X-Spam-Flag: NO\r
   9 X-Spam-Score: -0.013\r
  10 X-Spam-Level: \r
  11 X-Spam-Status: No, score=-0.013 tagged_above=-999 required=5\r
  12  tests=[AWL=-0.013] autolearn=disabled\r
  13 Received: from arlo.cworth.org ([127.0.0.1])\r
  14  by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024)\r
  15  with ESMTP id W4AgVm5pY6tx for <notmuch@notmuchmail.org>;\r
  16  Fri, 22 Jan 2016 09:02:38 -0800 (PST)\r
  17 Received: from che.mayfirst.org (che.mayfirst.org [209.234.253.108])\r
  18  by arlo.cworth.org (Postfix) with ESMTP id 4C4F16DE0B2F\r
  19  for <notmuch@notmuchmail.org>; Fri, 22 Jan 2016 09:02:38 -0800 (PST)\r
  20 Received: from fifthhorseman.net (unknown [38.109.115.130])\r
  21  by che.mayfirst.org (Postfix) with ESMTPSA id 000A3F984;\r
  22  Fri, 22 Jan 2016 12:02:35 -0500 (EST)\r
  23 Received: by fifthhorseman.net (Postfix, from userid 1000)\r
  24  id 7376C20112; Fri, 22 Jan 2016 12:02:35 -0500 (EST)\r
  25 From: Daniel Kahn Gillmor <dkg@fifthhorseman.net>\r
  26 To: Antoine Amarilli <a3nm@a3nm.net>, notmuch@notmuchmail.org\r
  27 Subject: Re: Searching messages by size with notmuch\r
  28 In-Reply-To: <20160122151318.GA17099@mu.a3nm.net>\r
  29 References: <20160122151318.GA17099@mu.a3nm.net>\r
  30 User-Agent: Notmuch/0.21+67~g41ad7ff (http://notmuchmail.org) Emacs/24.5.1\r
  31  (x86_64-pc-linux-gnu)\r
  32 Date: Fri, 22 Jan 2016 12:02:35 -0500\r
  33 Message-ID: <87twm5r1hw.fsf@alice.fifthhorseman.net>\r
  34 MIME-Version: 1.0\r
  35 Content-Type: multipart/mixed; boundary="=-=-="\r
  36 X-BeenThere: notmuch@notmuchmail.org\r
  37 X-Mailman-Version: 2.1.20\r
  38 Precedence: list\r
  39 List-Id: "Use and development of the notmuch mail system."\r
  40  <notmuch.notmuchmail.org>\r
  41 List-Unsubscribe: <https://notmuchmail.org/mailman/options/notmuch>,\r
  42  <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
  43 List-Archive: <http://notmuchmail.org/pipermail/notmuch/>\r
  44 List-Post: <mailto:notmuch@notmuchmail.org>\r
  45 List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
  46 List-Subscribe: <https://notmuchmail.org/mailman/listinfo/notmuch>,\r
  47  <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
  48 X-List-Received-Date: Fri, 22 Jan 2016 17:02:43 -0000\r
  49 \r
  50 --=-=-=\r
  51 Content-Type: text/plain\r
  52 \r
  53 On Fri 2016-01-22 10:13:18 -0500, Antoine Amarilli wrote:\r
  54 \r
  55 > After chatting on #notmuch, I wanted to suggest a feature which would be\r
  56 > useful, at least to me: searching for messages by size.\r
  57 >\r
  58 > My use case would be to look for long messages, but dkg on IRC mentioned\r
  59 > that it could also be useful to clean up messages to save disk space.\r
  60 >\r
  61 > It is unclear whether the size of a message should be defined as that of\r
  62 > a single copy of the message, or that of all copies; and it is unclear\r
  63 > whether it should be the total size (for my purposes I would have been\r
  64 > interested in the size of the plaintext part of the message only).\r
  65 \r
  66 Note that "the plaintext part of the message" might mean the sum of\r
  67 multiple plaintext parts too -- consider this message as an example:\r
  68 \r
  69 \r
  70 --=-=-=\r
  71 Content-Type: image/png\r
  72 Content-Disposition: inline; filename=stock_smiley-7.png\r
  73 Content-Transfer-Encoding: base64\r
  74 \r
  75 iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\r
  76 AAAN1wAADdcBQiibeAAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAAAAXdEVY\r
  77 dEF1dGhvcgBMYXBvIENhbGFtYW5kcmVp35EaKgAAACl0RVh0RGVzY3JpcHRpb24AQmFzZWQgb2Yg\r
  78 SmFrdWIgU3RlaW5lciBkZXNpZ26ghAVzAAAP7UlEQVRo3r2Ze4xcV3nAf+fce+exs++112snjhM7\r
  79 NiFxRIKSUPIghBSImiht1BpVqYpAIJAQIKqqoJKgFkSrQhCggugDaEJbpZVKVBISEoLCQylpAgmJ\r
  80 A9gkcezYjp+7OzM7szNz7z3nfF//uHftjbWxTQId6dPO7J255/t973OuUVVexcsclycsjFlYiKBr\r
  81 Ya2BtoFxhaAwEOgKjIQf0tU382YBtPj5K1fCvAqAJeUt7IqgH4MkoAlIDEkE1oBonyBDRB6Mg3kH\r
  82 eBgJsFtgm7waiFcKsGT1CEigUoW4lhHXDWnDUK0ZXAUSG4OCz3MkrWD74AZg+6A5uHwJwmDQVwAR\r
  83 v3Lld8SQJDA01DpycOa5x//n90KeviWJorNsUp2K4uqwjepV8QMnPu+J5PPO+YM2qf/wrC2X3jez\r
  84 ZdN+yHpwQVaY///FA8bAXxvYFtOs1L531+dvjjS8qzo6ffH68y5m1brNlaQxRVQZwURVUAEToSFF\r
  85 8h55f57moV1u/zPbGSwcfNaJvfOia9/6tTWbLlyAcz1Q5sVvL4QM7Eju/fvPXE+efeGcrZfNbNh6\r
  86 ZbU+uR5CHzTD2Bi1McbEYGNQQTVgxKHBg61ioiHS9ovs3/FItuPJR3tE0S033fSpOzj77LyE+O0A\r
  87 fOu2mzb5/uDOmY1bL3zdFdfV68PjSHoE4gq2PolJRjBxHWwFY6Miv1HQgIYcQoq6RSRtoT7HVleR\r
  88 9RbZ/uPvpC88s/0QSX3bzZ+46En4K/mNA/zHx992dVJN7r78rTeOrDrzXCvdPWAV25jBDk1iKiOY\r
  89 pAFRBWMTMLZMF1AVEA8hQ/0AzbvIoIn0joKAHd5A69Bu/d69dw36afau9332oW+ebijZ0/nSv3/s\r
  90 qvdVG0P3v+X6bWPjNW+z3fcgeQvianmHAMajODAOJV8mGZgcTI4aBxRV1FgwcYK6NvneBxhNcnPD\r
  91 TTcPjQwPf+MfPvzGT2OM+Y0A/NtfXPmescnpL1573Y31uPUU7uDDmCjBIKApaB/VHkgP6KHaR00h\r
  92 mD5qBig90D5oD9XFQqSH6gDUgbW4I48RtZ7khhv+oL52es1HvvKBy/7uVQN846NvuqJSa3zpyive\r
  93 VJdd9+DnnsZYi2oGoQuuhbgm6puItEA6GLpAD0MJQSFqFlHtoNJCfQtcE81bqO9A6KNGcHO/wD9/\r
  94 D9dc+5ahWr3xoS+8/5KbXzHA7X9+9Xqreu9Vb7yszoGH8c1fAQENfTRvI9kckh+BfBbjmyBdhB5C\r
  95 VobOUrjkQIZqitIH6UJoIu4omh+GbBbyFup6GPX49vPIvu9z/bVX1KMo/upn33vpZa8IwGj2X5ds\r
  96 fe1ovb+XfP/DhfK+j+YLaD6PZLNoPo+GFvc8+DhGUozmGHWgvpgWNAAeJWBw5bWMu7/zGBramLyJ\r
  97 ZLNINlt40/cw4sgPPEplcQ/X/c4FQxZ/1yc/eU38awF89SOXv314eHzr2tXjNn3mvrKK5KhfRPM2\r
  98 ZG3EtVC/yG3//Agf+PgD3HnXwyBZMSGow6gv4ls9RhxIDpLypX+8nw/e+l0+90+PIKGLug6aLyBZ\r
  99 G1wXlRRVIX3uftasmmTdqrGJ6gsL7z19AGOMlfDl129Z38j3/KhQAkHFQUiR0Eekh5Wirt/3gz3c\r
 100 eustfPv+JyH0IPQxMkBlUDQ2GRTJKgMIfb517xPceust3PPQLjRkRW+QPsigCE9xGBVQR7b7+1x5\r
 101 4aaGivztJ99/ydBpAXzlQ2/4w1WT4zPjiccd/QUYMGgxFojHiGCCR0qr7tq3wNe//i8889xh8F0k\r
 102 dIsElx5GuqA9TOiioYOGDr/adaT4/p6Fwqvqyh7hQAQklC3A4I7uZDj2nHfGaNX0wp+d1jBnxb93\r
 103 8/TocL7/p8cnXGPAaHnjgCIY9Swu9mku5DS3b2eoHqO+jY0EY3LUVMDGqATwGbgB3dYss3M9Zue2\r
 104 AzA7v8hUxS0bgaRYpxwXjYF8/0+54Mzzhn6xv/0e4G9O6oE73n1NTUxy9dT4KKG1d9kVfel7EUwI\r
 105 NJJArVLcYnqqhmZtNG+ibg5cE+PnwM8X4po0kj5DtcJmSWyYGBIIHhUpjHVC71UgtPYxNT6KGrP2\r
 106 E+983TknBViop2+bGBlydA8hLj12U10yvioqxQctE3TjGTUAtmxogGsVlSmbh3wWSWchmy0+Z/NY\r
 107 12Lz2cMAnLOuTowrCoTK8UWWQJbW9jmhc5iNqxqEnN8/aQhpcNdND8fDrrnvuAlecsNSRNAgqAS2\r
 108 bhpmePVWXveaJpp1MeRFAocYjMEoxRTqMkLuuOi8EaqjW9gwuhcJHoKUuwFZZrClNQslfGsfZ07O\r
 109 1HYeaP0R8MWX9YBIOLcaR8b128eU1WXKqwKihctFMAqf+8trOHdtxsc/fDXq+mjWQdMFZLCApm0k\r
 110 bUPaRrIuNgi33Xot56zJ+Pwtb4IQUA3Fllm0VPylBlMF129TTyJEzVkn94BwRhwZXN4nluLHZulG\r
 111 ophyEVNCqCirpldxx5ffgQyOElpZkY/BYSKLlB5ABHUeNTA6uYZ//co2wtwu8nYAX3iy8GwZoroM\r
 112 RhSX96kmljwwdVKAgJmOLORpn1qkGBV6qVKPlCQulDZBjv2VPCMc2o72D6GuDZIiscWEANaCKQ0p\r
 113 Wlg7pGj2FCYaIrRehPJeiKKlF5Y8HIKSOaWqikv7RFFEEKrvfvc5tdtv35OuDCCMqAq5d0U5E6Ve\r
 114 UfY3A1NjlvGk9ERQNAjGe0J3DslamCTGxBFIBJGgtiy/qiAUOeMD6o+gzqEuFP9bEjkuiwNhoSPM\r
 115 1BRxSu4dIoo1mtXmKlPAgZcBkO4gzap1WyWXAVVRrAjrxyxPHvaM9uC89RYTCQSDegEbwC/FYJHg\r
 116 RLY4USk3NIVVS4X9cgmFF4JAEJxTduz3iBMumALNlSCKs1V8npJ5raaL+fzJkvjoYpqTmwqZAKGw\r
 117 uBXh4nWG+U7gP/+3z479OeoEDaFQxElh0VIk92ju0dyhuUOcQ/NQiAulJ4rfiitgXjjq+O/HB6gI\r
 118 W9cUeUMQMgFnqywOBqhKdvsPjofPSkl8YDHNzx+yFTIPIoINggaLjZTLN0Ts7hge2jHgx8/lXPHa\r
 119 OhecXSSqZSnZLcYa1JjyvKos7yKFQY5BB8QHnn0x48ndKb1+4G2viZmuFZBL+ZEJuKjCQi8D0fmT\r
 120 JrGHXZ2B/93JpGYGwZAGZah0sQkBiQwbJ2KmL2vw/Wc99/+sx4+fSdl8ZpVNaytsWFfFeMVG5Rxg\r
 121 TmiApVKdrmf78wOe2uNI08D5Z1S54QKoBAdOES9oULIAg2BwtRrNvkNU9p28Col94GhX/2T92tHR\r
 122 QQcGAeplfGowGC+oFUaSnBsvHmfPYoMnn59n594BT+/JsNEiZ61OWDsVE8eWyEIUGWIDcx3PkaZn\r
 123 dsGTpkolUjauW8WFG0aYsbNI2kN8wHg51igHodAhVEbZf8ANnNhvnhSgWus8uDhoVPqSUDUV+iGn\r
 124 FpSGFzQyZcIW9d1mXTY2crZcfxUdX+NXO3/O3n37SNPArgOCyLLBRg2RUSzKqkbMea/fysZN51Br\r
 125 /xI3t4uQF2FVKB5QJ6Re6QdDqjGehIPtzDiN7z7lscrH3nHBAxsm9e3rojka2RyTCUxUlKQSYZII\r
 126 k8TYJII4QmMLNsI2JqiccSmMbiFXw2CxRdqdZ7F9GBEYnTqT4ckZao1JbNZGWzvJX3wUyVPwRXhS\r
 127 FgByT3CeZg5NZ+glkxwMkzy6x++548HdG085Todgvnaww+XTMxMjvf48FaskHsZNODZaF6dOpmhS\r
 128 EfjOPHn3QdQ+SDK1mZEz3sj4yNmYDVvBRhAc6aEn6O98jJD1MSqYsmlZ0aJQ+AC+SOxugF4w9IPB\r
 129 j07x3O68F8R87bQOtowx5iM3nffMxqmweSZuMpQ1mUhgLFFGKgYii63EaBxBbNHIIljEWkRBDagY\r
 130 pJyNjw1mxmBUMUaxZf2ORDA+YELA+IBmgb4X2s7QdtBLJjgSRnlkt2l1GtmZ99xzoH/KHZmqahDz\r
 131 gReaUS+Px+hJRNdD1xt6uR4rgSwTE47Xd5cHcufJs0CWheKvCzjn8U6KCuMFnAfnUe/BFz1i4IWu\r
 132 N3Q99ILFVcf5+QFdDIGPraT8y27qv3T3zodyL0/ta2nw9dUlAHS8oecUzYvFyZcgCkviBRuO13Dx\r
 133 xUyDF/CKCYLxgnEB6wMm95gy7nte6HjDgivWyoemeX42o92nedjsuf3lNvXxCnt6A5iFNPvTIJWf\r
 134 VeJkfKY6SSdrlps+Q1CloYEkUggWYsEaC6ZoYLEW4SJl9FiKySICIopB0FKChUDXw2Jp+Y6HNJng\r
 135 6MDw9IGo7zx//MNHEGOM0RUOco/lgDl+Flk+NiLedvU5V00Ox3efv7pXn0pSavkCjUgYjqERKUMx\r
 136 1CMwkUWtRY1BjEEwxWB5bJHiYCAqu3UkxcyUBaUfoOcNvVBYPo3HWNA6Dz8fD7opH7zvsf13Fj32\r
 137 +LOD5SArAUSlZxIg2Xb1hndNDEWfPn96UJ9uWJKsTU36NCIYiqAWKTULVQvGFkmuAmrN8T2uKVxh\r
 138 pDjdyAOkAqkY+h76AfpicY01NAfCT16I+62+3PHtxw5+ojzeK5+rEU4FYJcBVJfkxjesf+fUaHLL\r
 139 5qledf1khboNRP05ajhqFmoRVKwSG0gsRKZMLnN8gyUKXsEJRa8IlKMC5JVRZGg1L853+eneJO0O\r
 140 wh33P37oM0AKZKUsHV/I6QAkywDqQO3SLVNv3nzGyOc2jKfVTdPWToyME/k+ZvEIcUipWKiUU3RS\r
 141 jkIvOV0QcGXY5wq5WEJ9AhrT9LOMZw50+OWRanZgNvvUT56d/RYwWAEg/DoAlRKgVkKMr5mqX3L5\r
 142 lsmPNmp25typQbxhqsLE5FqqsUUHLUK/iRksYBDsslluKR80HsLUx6E6itbG6HZavHCoxS8Px66b\r
 143 0d2+e+G2fbO9HwHt8lh7Sfn8dAFOzIFqCVIDGsAMsHHzutGrtm4YvWG4po2zxgbx2jHL5PgkI+Or\r
 144 qQ9PkiQVVAPqHcZaMBFBlbTfoddp0uk22XukzwvtxHcGkXvuUO+7O/a2vwvsAo4AvdL6+TLrn1YO\r
 145 LPUGW0LEJ4TTMLAKmAZmXnvW2BUbVtffUK/FZzaiENaMZJWhJFCLhOFazFC1Qpp7epmj7yD1EYcX\r
 146 K66bJzbN/ZED89kTO/cuPOREDgFHgTlg8YSw8cuqkLwswAl9YOkBV3yCR5Z7YwwYAUaSxE5snBm5\r
 147 eO1E9aKhip2OIjsWWzukxlRRdaLa8z50Uq9zR1vZ07sP9x7vZ24O6AKdUpZbfbnF/bFIPFkfOAFg\r
 148 qR8sz4toGUx8Qp7US7DKMuglI5SHnseUyUtFByfEt1+m9JIIy4+4TgfgJCBmWXjZFcDMMsWPP6I8\r
 149 XohkGYiuoOhy0VMpfkqAFUBOhGGZomaFayu9dAWRFa5xKsVPG+BlQJYDcYLSpwI48b2uNBFzmq//\r
 150 AxyL7Nqf76KTAAAAAElFTkSuQmCC\r
 151 --=-=-=\r
 152 Content-Type: text/plain\r
 153 \r
 154 \r
 155 Would you count the total of all related text/plain parts?  or all\r
 156 text/* parts?  or, if you have a multipart/alternative node in the MIME\r
 157 tree, would you report it as the maximum of any of the text/*\r
 158 alternatives? \r
 159 \r
 160 if you're really interested in textual analysis, then the "size of\r
 161 plaintext" might instead be better measured in words or paragraphs,\r
 162 rather than octets.  Also, you might want to ignore quoted text and only\r
 163 measure non-quoted text (this is particularly relevant for\r
 164 conversations where people top-post and don't trim, or else you're\r
 165 actually measuring just how deep in the thread a given message is).\r
 166 \r
 167 I'm not trying to say that these metrics are impossible, just pointing\r
 168 out that the underlying data formats can be much more complicated than\r
 169 most people think about with mail.  The decision about what to count and\r
 170 how to count it greatly effects the possible use cases.\r
 171 \r
 172 > Ideally I'd say that all of these could make sense.\r
 173 \r
 174 They could indeed, but I think we could motivate this much better as\r
 175 initial work by picking one particular use case, and implementing it.\r
 176 The work would be something like:\r
 177 \r
 178  a) choose the metric we care about, and describe concretely how to\r
 179     calculate it for a given rfc822 file.\r
 180 \r
 181  b) assign and name a new notmuch_value_t to to identify the metric\r
 182 \r
 183  c) update notmuch_database_add_message to insert that new value when a\r
 184     file is added\r
 185 \r
 186  d) consider what workflows are available to update the database for\r
 187     already-indexed documents that do not have this value.\r
 188  \r
 189  e) resolve what to do about documents associated with multiple filenames\r
 190 \r
 191  f) define how to include it in searches (this is probably a\r
 192     NumberValueRangeProcessor, see\r
 193     file:///usr/share/doc/xapian-doc/valueranges.html)\r
 194 \r
 195  g) update documentation for notmuch cli tools.\r
 196 \r
 197 \r
 198 If you work through this process for one particular "message size" use\r
 199 case and document the steps, then we could presumably handle the other\r
 200 "message size" metrics in exactly the same way, modifying only steps (a)\r
 201 and (e) depending on the metric.\r
 202 \r
 203 > Would anyone else on the list be interested by such a feature?\r
 204 \r
 205 I'm definitely interested, but don't have a lot of time to work on it.\r
 206 \r
 207     --dkg\r
 208 \r
 209 --=-=-=--\r