18/f18b0824aefc24c1a5e733fa43f485a3b48fa4

   1 Return-Path: <amdragon@mit.edu>\r
   2 X-Original-To: notmuch@notmuchmail.org\r
   3 Delivered-To: notmuch@notmuchmail.org\r
   4 Received: from localhost (localhost [127.0.0.1])\r
   5         by olra.theworths.org (Postfix) with ESMTP id 0625A429E2E\r
   6         for <notmuch@notmuchmail.org>; Tue, 22 Nov 2011 19:17:45 -0800 (PST)\r
   7 X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
   8 X-Spam-Flag: NO\r
   9 X-Spam-Score: -0.7\r
  10 X-Spam-Level: \r
  11 X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5\r
  12         tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
  13 Received: from olra.theworths.org ([127.0.0.1])\r
  14         by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
  15         with ESMTP id ewbX6LcRWrfv for <notmuch@notmuchmail.org>;\r
  16         Tue, 22 Nov 2011 19:17:44 -0800 (PST)\r
  17 Received: from dmz-mailsec-scanner-4.mit.edu (DMZ-MAILSEC-SCANNER-4.MIT.EDU\r
  18         [18.9.25.15])\r
  19         by olra.theworths.org (Postfix) with ESMTP id 88563431FB6\r
  20         for <notmuch@notmuchmail.org>; Tue, 22 Nov 2011 19:17:44 -0800 (PST)\r
  21 X-AuditID: 1209190f-b7f6e6d0000008df-0c-4ecc65d81f11\r
  22 Received: from mailhub-auth-3.mit.edu ( [18.9.21.43])\r
  23         by dmz-mailsec-scanner-4.mit.edu (Symantec Messaging Gateway) with SMTP\r
  24         id E1.15.02271.8D56CCE4; Tue, 22 Nov 2011 22:17:44 -0500 (EST)\r
  25 Received: from outgoing.mit.edu (OUTGOING-AUTH.MIT.EDU [18.7.22.103])\r
  26         by mailhub-auth-3.mit.edu (8.13.8/8.9.2) with ESMTP id pAN3HhvQ002702; \r
  27         Tue, 22 Nov 2011 22:17:44 -0500\r
  28 Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91])\r
  29         (authenticated bits=0)\r
  30         (User authenticated as amdragon@ATHENA.MIT.EDU)\r
  31         by outgoing.mit.edu (8.13.6/8.12.4) with ESMTP id pAN3HgZJ006884\r
  32         (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NOT);\r
  33         Tue, 22 Nov 2011 22:17:43 -0500 (EST)\r
  34 Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.77)\r
  35         (envelope-from <amdragon@mit.edu>)\r
  36         id 1RT3NT-0006uq-44; Tue, 22 Nov 2011 22:20:03 -0500\r
  37 Date: Tue, 22 Nov 2011 22:20:03 -0500\r
  38 From: Austin Clements <amdragon@MIT.EDU>\r
  39 To: Tom Bulli <mrbulli@yahoo.com>\r
  40 Subject: Re: Notmuch indexing 21 million emails\r
  41 Message-ID: <20111123032002.GK9351@mit.edu>\r
  42 References: <1321930927.73603.YahooMailNeo@web36506.mail.mud.yahoo.com>\r
  43 MIME-Version: 1.0\r
  44 Content-Type: text/plain; charset=iso-8859-1\r
  45 Content-Disposition: inline\r
  46 Content-Transfer-Encoding: 8bit\r
  47 In-Reply-To: <1321930927.73603.YahooMailNeo@web36506.mail.mud.yahoo.com>\r
  48 User-Agent: Mutt/1.5.21 (2010-09-15)\r
  49 X-Brightmail-Tracker:\r
  50  H4sIAAAAAAAAA+NgFprFKsWRmVeSWpSXmKPExsUixCmqrXsj9YyfwYo73BZdd0+xW1y/OZPZ\r
  51         gcnj2apbzB6zZh1mCmCK4rJJSc3JLEst0rdL4Mp4uU6tYAZ3Revlf6wNjF85uhg5OSQETCT2\r
  52         vm9jhrDFJC7cW8/WxcjFISSwj1Hi0qsWRghnA6PElcnLoJyTTBK39jZBlS1hlDh5qZkRpJ9F\r
  53         QFVi6ayDYLPYBDQktu1fDhYXEVCU+PFmPlicWcBcom1bE1Ccg0NYQF9i+kMNkDCvgLbEz4Nf\r
  54         WUBsIQF3iesHtzJCxAUlTs58wgLRqiOxc+sdNpBWZgFpieX/OCDC8hLNW2eDTecU8JCY2bWU\r
  55         CcQWFVCRmHJyG9sERuFZSCbNQjJpFsKkWUgmLWBkWcUom5JbpZubmJlTnJqsW5ycmJeXWqRr\r
  56         opebWaKXmlK6iREcBZL8Oxi/HVQ6xCjAwajEwxt18rSfEGtiWXFl7iFGSQ4mJVHerJQzfkJ8\r
  57         SfkplRmJxRnxRaU5qcWHGCU4mJVEeK+5A+V4UxIrq1KL8mFS0hwsSuK8jTsc/IQE0hNLUrNT\r
  58         UwtSi2CyMhwcShK8aUlAjYJFqempFWmZOSUIaSYOTpDhPEDDn4Is5i0uSMwtzkyHyJ9iVJQS\r
  59         540ASQiAJDJK8+B6YUnqFaM40CvCvM3xQFU8wAQH1/0KaDAT0OBpa0+ADC5JREhJNTB2CE+V\r
  60         aZixwermnKcs8Ztm8TB5hPl63Vt0Lat2UkONCdccpSS7A9ovSrM/xkWVl1o8VFEq2hlqVLZk\r
  61         SStDxk/l6UapVU5B38Nvnu295n5RrrEvKrs/PDLvatV8e+npeu5FLAqKgj9Lzj2Umd+5Xavk\r
  62         xCT396q56y6Jsfs7vl8cveW8s9ayL0osxRmJhlrMRcWJAGkHfagtAwAA\r
  63 Cc: "notmuch@notmuchmail.org" <notmuch@notmuchmail.org>\r
  64 X-BeenThere: notmuch@notmuchmail.org\r
  65 X-Mailman-Version: 2.1.13\r
  66 Precedence: list\r
  67 List-Id: "Use and development of the notmuch mail system."\r
  68         <notmuch.notmuchmail.org>\r
  69 List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
  70         <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
  71 List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
  72 List-Post: <mailto:notmuch@notmuchmail.org>\r
  73 List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
  74 List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
  75         <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
  76 X-List-Received-Date: Wed, 23 Nov 2011 03:17:45 -0000\r
  77 \r
  78 Quoth Tom Bulli on Nov 21 at  7:02 pm:\r
  79 > I have a project where I need to search about 21 emails - and\r
  80 > decided to use "notmuch" for it.  The system is a Debian Squeeze,\r
  81 > the notmuch version is "0.8-1~bpo60+1" from "kyria's" private\r
  82 > repository.\r
  83 > \r
  84 > I am running the "notmuch new" for approx. 4 days now - and\r
  85 > according to "not,uch count" it has indexed about 4.5 million\r
  86 > emails.\r
  87 > \r
  88 > Is this expected performance?  Is there any way to speed that up?\r
  89 \r
  90 Currently, notmuch is much more optimized for search than it is for\r
  91 indexing.  This is unfortunate for the initial indexing process and\r
  92 seems to be becoming increasingly unfortunate.\r
  93 \r
  94 There are some things you can try.  One is to use an SSD if you aren't\r
  95 already, since constructing the index requires a lot of random IO.\r
  96 You can also try libeatmydata to disable fsync's, which may improve\r
  97 your IO performance, with the obvious crash-safety caveats.  However,\r
  98 unless you have a lot of RAM, I suspect your index has long outgrown\r
  99 your buffer cache, so this may have limited impact.\r
 100 \r
 101 Since you're going to the trouble of indexing 21 million emails, you\r
 102 might want to try 0.10 (under freeze right now, to be released very,\r
 103 very soon).  It won't improve your indexing time, but if you're doing\r
 104 searches with non-trivial numbers of results, emails indexed with 0.10\r
 105 will search much faster.\r
 106 \r
 107 Sorry I don't have better news, but I hope this helps.\r