portage.git
16 years agoIn SequentialTaskQueue.clear(), remove the exit listener from each task before
Zac Medico [Thu, 17 Jul 2008 06:31:00 +0000 (06:31 -0000)]
In SequentialTaskQueue.clear(), remove the exit listener from each task before
cancelling it.

svn path=/main/trunk/; revision=11095

16 years agoAdd the new --anon option to the getopt args.
Zac Medico [Thu, 17 Jul 2008 04:50:49 +0000 (04:50 -0000)]
Add the new --anon option to the getopt args.

svn path=/main/trunk/; revision=11094

16 years agoWhen SpawnProcess._start() spawns a process in the background, use /dev/null
Zac Medico [Thu, 17 Jul 2008 04:20:12 +0000 (04:20 -0000)]
When SpawnProcess._start() spawns a process in the background, use /dev/null
so that any attempts to read from stdin will immediately return EOF instead
of blocking indefinitely. TODO: Use job control functions like tcsetpgrp()
to control access to stdin.

svn path=/main/trunk/; revision=11093

16 years agoEliminate duplicate code by making EbuildProcess inherit from SpawnProcess
Zac Medico [Thu, 17 Jul 2008 03:52:24 +0000 (03:52 -0000)]
Eliminate duplicate code by making EbuildProcess inherit from SpawnProcess
instead of SubProcess.

svn path=/main/trunk/; revision=11092

16 years agoAdd --anon to the usage message.
Zac Medico [Wed, 16 Jul 2008 21:57:04 +0000 (21:57 -0000)]
Add --anon to the usage message.

svn path=/main/trunk/; revision=11091

16 years agoAdd --anon option for anonsvn.gentoo.org users.
Zac Medico [Wed, 16 Jul 2008 21:52:01 +0000 (21:52 -0000)]
Add --anon option for anonsvn.gentoo.org users.

svn path=/main/trunk/; revision=11090

16 years agoRemove TODO since it no longer exists.
Zac Medico [Wed, 16 Jul 2008 21:11:32 +0000 (21:11 -0000)]
Remove TODO since it no longer exists.

svn path=/main/trunk/; revision=11089

16 years agoAvoid raising an IndexError from deprecated_profile_check() if the deprecated
Zac Medico [Wed, 16 Jul 2008 20:43:12 +0000 (20:43 -0000)]
Avoid raising an IndexError from deprecated_profile_check() if the deprecated
file happens to be empty. Instead just refer to the "Gentoo Upgrading Guide".
Thanks to armin76 for reporting.

svn path=/main/trunk/; revision=11088

16 years agoPass background=True into prefetcher constructors.
Zac Medico [Wed, 16 Jul 2008 08:47:54 +0000 (08:47 -0000)]
Pass background=True into prefetcher constructors.

svn path=/main/trunk/; revision=11076

16 years agoAdd a progress counter showing "x of y complete" to
Zac Medico [Wed, 16 Jul 2008 08:37:30 +0000 (08:37 -0000)]
Add a progress counter showing "x of y complete" to
Scheduler._display_status().

svn path=/main/trunk/; revision=11075

16 years ago* Add xterm titles support to Scheduler._display_status().
Zac Medico [Wed, 16 Jul 2008 07:38:07 +0000 (07:38 -0000)]
* Add xterm titles support to Scheduler._display_status().
* Tweak emergelog() so that it won't interfere with the scheduler summary.

svn path=/main/trunk/; revision=11073

16 years agoAdd async support for calling functions in misc-functions.sh:
Zac Medico [Wed, 16 Jul 2008 07:09:51 +0000 (07:09 -0000)]
Add async support for calling functions in misc-functions.sh:

* Create a new MiscFunctionsProcess class derived from SpawnProcess.

* Add "background" support to the SpawnProcess class and split out a
  _spawn() template function so MiscFunctionsProcess can override it.

* Rename the existing EbuildPhase class to EbuildProcess.

* Create a new EbuildPhase class derived from CompositeTask. This class
  spawns the EbuildProcess and MiscFunctionsProcess classes as sub-tasks.

svn path=/main/trunk/; revision=11072

16 years agoRemove the config._filter_calling_env attribute since it's not really needed.
Zac Medico [Wed, 16 Jul 2008 03:32:53 +0000 (03:32 -0000)]
Remove the config._filter_calling_env attribute since it's not really needed.
Replace it with an simple os.path.exists() call inside config.environ().

svn path=/main/trunk/; revision=11071

16 years agoShow LDFLAGS in emerge --info <pkg> output. Thanks to dirtyepic for this
Zac Medico [Wed, 16 Jul 2008 02:38:53 +0000 (02:38 -0000)]
Show LDFLAGS in emerge --info <pkg> output. Thanks to dirtyepic for this
patch.

svn path=/main/trunk/; revision=11070

16 years agoPass the whole dblink instance into Scheduler._dblink_display_merge() in case
Zac Medico [Wed, 16 Jul 2008 01:42:42 +0000 (01:42 -0000)]
Pass the whole dblink instance into Scheduler._dblink_display_merge() in case
it's ever needed.

svn path=/main/trunk/; revision=11069

16 years agoRedirect dblink.merge() output to the log file when --jobs > 1.
Zac Medico [Wed, 16 Jul 2008 01:37:27 +0000 (01:37 -0000)]
Redirect dblink.merge() output to the log file when --jobs > 1.

svn path=/main/trunk/; revision=11068

16 years agoMake dblink._display_merge() detect scheduler presence internally.
Zac Medico [Wed, 16 Jul 2008 01:15:18 +0000 (01:15 -0000)]
Make dblink._display_merge() detect scheduler presence internally.

svn path=/main/trunk/; revision=11067

16 years agoAdapt the dblink merge display callback to use log levels from the logging
Zac Medico [Wed, 16 Jul 2008 01:09:46 +0000 (01:09 -0000)]
Adapt the dblink merge display callback to use log levels from the logging
module so that it can be used as an interface for both writemsg() and
writemsg_stdout().

svn path=/main/trunk/; revision=11066

16 years agoRedirect dblink.unmerge() output to the log file when --jobs > 1.
Zac Medico [Wed, 16 Jul 2008 00:44:58 +0000 (00:44 -0000)]
Redirect dblink.unmerge() output to the log file when --jobs > 1.

svn path=/main/trunk/; revision=11065

16 years agoWhen --jobs > 1, display a status message each time that
Zac Medico [Wed, 16 Jul 2008 00:11:44 +0000 (00:11 -0000)]
When --jobs > 1, display a status message each time that
Scheduler._schedule_tasks() is called. For example:

>>> Jobs:  2 running,  0 merges, load average: 0.34, 0.46, 0.27

svn path=/main/trunk/; revision=11064

16 years agoAdd some cross references between emerge's new parallel build options and
Zac Medico [Tue, 15 Jul 2008 23:19:16 +0000 (23:19 -0000)]
Add some cross references between emerge's new parallel build options and
MAKEOPTS.

svn path=/main/trunk/; revision=11063

16 years agoMake EbuildFetcher copy FETCHCOMMAND and RESUMECOMMAND to the fetch
Zac Medico [Tue, 15 Jul 2008 22:31:13 +0000 (22:31 -0000)]
Make EbuildFetcher copy FETCHCOMMAND and RESUMECOMMAND to the fetch
environment, in case the default values have been overridden.

svn path=/main/trunk/; revision=11062

16 years agoIn Scheduler._restart_if_necessary(), use list.remove() instead of assuming
Zac Medico [Tue, 15 Jul 2008 21:17:37 +0000 (21:17 -0000)]
In Scheduler._restart_if_necessary(), use list.remove() instead of assuming
that portage is first in the resume list.

svn path=/main/trunk/; revision=11061

16 years agoSince lots of packages will have to be fixed before stable portage can abort
Zac Medico [Tue, 15 Jul 2008 10:45:42 +0000 (10:45 -0000)]
Since lots of packages will have to be fixed before stable portage can abort
with "this code is not 64bit clean", convert the die call to an eerror message
instructing the user to file a bug at bugs.gentoo.org. This should allow us
to stabilize portage-2.2 without having to ensure that the entire stable tree
is 64 bit clean first.

svn path=/main/trunk/; revision=11057

16 years agoWhen packages fail with --jobs > 1 and the "echo" elog module is not enabled,
Zac Medico [Tue, 15 Jul 2008 09:50:41 +0000 (09:50 -0000)]
When packages fail with --jobs > 1 and the "echo" elog module is not enabled,
emulate elog's mod_echo module to show the 'error' level messages, which should
include the important die message. This is implemented by adding a private
hook inside elog_process() which the scheduler uses to collect error messages.

svn path=/main/trunk/; revision=11056

16 years agoWhen there is more than one failure, show a summary packages that failed to
Zac Medico [Tue, 15 Jul 2008 08:58:05 +0000 (08:58 -0000)]
When there is more than one failure, show a summary packages that failed to
build or install. This is especially useful when using --jobs or --keep-going
since there can be multiple failures.

svn path=/main/trunk/; revision=11055

16 years ago* Fix --keep-going breakage that occured when Scheduler._choose_pkg() was
Zac Medico [Tue, 15 Jul 2008 08:28:16 +0000 (08:28 -0000)]
* Fix --keep-going breakage that occured when Scheduler._choose_pkg() was
  introduced.

* Handle multiple failed builds when buildinding in parallel.

* Fix Scheduler._do_merge_exit() to remove the correct package since it's
  not always at position 0 when using --jobs.

in parallel, so

svn path=/main/trunk/; revision=11054

16 years agoMake AsynchronousTask "registered" attributes private since poll() and wait()
Zac Medico [Tue, 15 Jul 2008 03:03:42 +0000 (03:03 -0000)]
Make AsynchronousTask "registered" attributes private since poll() and wait()
do the job now. Also, hide SubProcess.files since there's no reason to expose
it.

svn path=/main/trunk/; revision=11053

16 years agoUse self.registered to return early from SubProcess.poll() when possible,
Zac Medico [Tue, 15 Jul 2008 02:49:31 +0000 (02:49 -0000)]
Use self.registered to return early from SubProcess.poll() when possible,
and remove the related code from SequentialTaskQueue.schedule() since
this makes it obsolete.

svn path=/main/trunk/; revision=11052

16 years agoAdd a PollScheduler._next_poll_event() method to maintain a central event
Zac Medico [Tue, 15 Jul 2008 02:11:39 +0000 (02:11 -0000)]
Add a PollScheduler._next_poll_event() method to maintain a central event
queue between _poll_loop() and _schedule_wait(). This solves an intermittent
KeyError from _poll_loop() which occured when an event handler called
_schedule_wait() which unregistered an event handler for an event that
_poll_loop() had already recieved but hadn't processed yet.

svn path=/main/trunk/; revision=11051

16 years agoSplit out a _main_loop_cleanup() method from Scheduler._merge() and call it
Zac Medico [Tue, 15 Jul 2008 00:50:13 +0000 (00:50 -0000)]
Split out a _main_loop_cleanup() method from Scheduler._merge() and call it
before restarting in _restart_if_necessary() to kill any parallel-fetch tasks
that may still be running.

svn path=/main/trunk/; revision=11050

16 years agoSplit out a generic SubProcess._unregister() method that's called from
Zac Medico [Mon, 14 Jul 2008 23:31:51 +0000 (23:31 -0000)]
Split out a generic SubProcess._unregister() method that's called from
_wait() to make sure everything is cleaned up before wait() returns.
This might solve an intermittent KeyError that is sometimes raised from
the main poll() loop when a handler is no longer registered for a poll()
event that has occurred.

svn path=/main/trunk/; revision=11049

16 years agoUnregister from poll() objects before closing file in order to hopefully
Zac Medico [Mon, 14 Jul 2008 09:00:03 +0000 (09:00 -0000)]
Unregister from poll() objects before closing file in order to hopefully
avoid stale events that can trigger a KeyError in Scheduler._poll_loop()
if the event's file descriptor no longer has a registered handler.

svn path=/main/trunk/; revision=11047

16 years agoCall prepare_build_dirs() prior to pkg_preinst so that logging works
Zac Medico [Mon, 14 Jul 2008 06:48:10 +0000 (06:48 -0000)]
Call prepare_build_dirs() prior to pkg_preinst so that logging works
properly.

svn path=/main/trunk/; revision=11046

16 years ago* Fix doebuild so "returnpid" works for preinst and postinst.
Zac Medico [Mon, 14 Jul 2008 06:38:30 +0000 (06:38 -0000)]
* Fix doebuild so "returnpid" works for preinst and postinst.
* Add async support for pkg_prerm and pkg_postrm.
* Add missing "treetype" parameter to dblink constructor calls.
* Fix SubProcess._wait() to check self.returncode _after_ calling
  the scheduler.

svn path=/main/trunk/; revision=11045

16 years agoAdd async execution support for pkg_preinst and pkg_postinst. This allows the
Zac Medico [Mon, 14 Jul 2008 03:47:47 +0000 (03:47 -0000)]
Add async execution support for pkg_preinst and pkg_postinst. This allows the
scheduler's poll loop to run so that other parallel tasks aren't starved for
output handling while pkg_preinst and pkg_postinst are executing.

svn path=/main/trunk/; revision=11044

16 years agoRemove redundant failed_packages check from Scheduler._main_loop() since
Zac Medico [Sun, 13 Jul 2008 23:25:23 +0000 (23:25 -0000)]
Remove redundant failed_packages check from Scheduler._main_loop() since
it's now handled inside the _schedule() call.

svn path=/main/trunk/; revision=11043

16 years agoEnsure that that Scheduler._main_loop() doesn't return before the merge
Zac Medico [Sun, 13 Jul 2008 22:13:52 +0000 (22:13 -0000)]
Ensure that that Scheduler._main_loop() doesn't return before the merge
queue is empty.

svn path=/main/trunk/; revision=11042

16 years agoRemove all traces of the "poll" parameter for the PollScheduler constructor
Zac Medico [Sun, 13 Jul 2008 13:38:24 +0000 (13:38 -0000)]
Remove all traces of the "poll" parameter for the PollScheduler constructor
since it was only used by can_poll_pty().

svn path=/main/trunk/; revision=11040

16 years ago* Since can_poll_pty() is unsafe due to he possibility of the poll() call
Zac Medico [Sun, 13 Jul 2008 13:24:39 +0000 (13:24 -0000)]
* Since can_poll_pty() is unsafe due to he possibility of the poll() call
  blocking forever, replace it with a simpler test on /dev/null that's
  derived from a test in a glib2 configure script.

* Remove PtyReaderTestCase since it has the same problem as can_poll_pty().

svn path=/main/trunk/; revision=11039

16 years agoIn can_poll_tty(), direct bash's stderr into the pty since we don't want
Zac Medico [Sun, 13 Jul 2008 11:54:13 +0000 (11:54 -0000)]
In can_poll_tty(), direct bash's stderr into the pty since we don't want
so see any error message it generates (due to PipeReader closing it's
end of the pty). The error message is irrelevant since all we want to
know is whether or not the data is successfully transmitted through the
pty.

svn path=/main/trunk/; revision=11037

16 years agoChanges to pym/_emerge/__init__.py from the previous commit weren't meant
Zac Medico [Sun, 13 Jul 2008 07:34:06 +0000 (07:34 -0000)]
Changes to pym/_emerge/__init__.py from the previous commit weren't meant
to be committed yet.

svn path=/main/trunk/; revision=11035

16 years agoFix broken binpkg path comparison in in binarytree.inject() so that duplicate
Zac Medico [Sun, 13 Jul 2008 07:30:43 +0000 (07:30 -0000)]
Fix broken binpkg path comparison in in binarytree.inject() so that duplicate
entries are correctly from the Packages index.

svn path=/main/trunk/; revision=11034

16 years agoIn order to avoid having BlockerDB instantiate a new FakeVartree for each
Zac Medico [Sun, 13 Jul 2008 05:29:35 +0000 (05:29 -0000)]
In order to avoid having BlockerDB instantiate a new FakeVartree for each
merge, add a FakeVartree.sync() method and so that the same FakeVartree
can be reused. Package counters and timestamps are used to validate package
instances during each sync() call.

svn path=/main/trunk/; revision=11033

16 years agoFix Scheduler._schedule_tasks() so that it never schedules a task when a
Zac Medico [Sun, 13 Jul 2008 04:30:06 +0000 (04:30 -0000)]
Fix Scheduler._schedule_tasks() so that it never schedules a task when a
package has already failed.

svn path=/main/trunk/; revision=11032

16 years agoDecrement the job count prior to merging a given package (after it's been
Zac Medico [Sun, 13 Jul 2008 01:10:45 +0000 (01:10 -0000)]
Decrement the job count prior to merging a given package (after it's been
built or extracted). This will allow additional jobs to spawn sooner in
some cases when there may be idle resources.

svn path=/main/trunk/; revision=11031

16 years ago* Wrap calls to PollScheduler._schedule_tasks() with a _schedule() method
Zac Medico [Sat, 12 Jul 2008 23:45:29 +0000 (23:45 -0000)]
* Wrap calls to PollScheduler._schedule_tasks() with a _schedule() method
  that automatically returns early from any recursive calls that the
  _schedule_tasks() call might trigger. This makes _schedule() safe to call from
  inside exit listeners.

* Call _schedule() inside Scheduler._job_exit() to trigger a new job when
  possible.

* Fix SequentialTaskQueue._task_exit() to avoid a potential KeyError if the
  task had already been removed from self.running_tasks for some reason.

svn path=/main/trunk/; revision=11030

16 years agoRename PollScheduler._schedule() to _schedule_wait().
Zac Medico [Sat, 12 Jul 2008 23:19:12 +0000 (23:19 -0000)]
Rename PollScheduler._schedule() to _schedule_wait().

svn path=/main/trunk/; revision=11029

16 years agoRemember to close file descriptors before returning early from can_poll_pty().
Zac Medico [Sat, 12 Jul 2008 22:54:35 +0000 (22:54 -0000)]
Remember to close file descriptors before returning early from can_poll_pty().

svn path=/main/trunk/; revision=11028

16 years agoAdd missing self.wait() calls when tasks return unsuccessfully.
Zac Medico [Sat, 12 Jul 2008 11:52:40 +0000 (11:52 -0000)]
Add missing self.wait() calls when tasks return unsuccessfully.

svn path=/main/trunk/; revision=11027

16 years agoSet self._mergelist inside Scheduler._calc_resume_list() since it needs to
Zac Medico [Sat, 12 Jul 2008 11:36:41 +0000 (11:36 -0000)]
Set self._mergelist inside Scheduler._calc_resume_list() since it needs to
be set when calling _set_digraph().

svn path=/main/trunk/; revision=11026

16 years agoImplement a runtime test that checks whether it's possible to use poll() to
Zac Medico [Sat, 12 Jul 2008 10:17:55 +0000 (10:17 -0000)]
Implement a runtime test that checks whether it's possible to use poll() to
read data from a pty device. The test only runs the first time that the
can_poll_pty() function is called (currently happens when the first
PollScheduler instance is created). The test is known to fail on Darwin and
we're not sure how many other kernels might have a similar problem, this
runtime test allows us to avoid having to maintain special cases for various
kernels.

svn path=/main/trunk/; revision=11025

16 years agoSplit out a portage._create_pty_or_pipe() function to avoid duplicate code.
Zac Medico [Sat, 12 Jul 2008 08:43:29 +0000 (08:43 -0000)]
Split out a portage._create_pty_or_pipe() function to avoid duplicate code.

svn path=/main/trunk/; revision=11024

16 years agoDerive a PtyReaderTestCase from PipeReaderTestCase. This test is expected to
Zac Medico [Sat, 12 Jul 2008 07:32:55 +0000 (07:32 -0000)]
Derive a PtyReaderTestCase from PipeReaderTestCase. This test is expected to
fail on some operating systems such as Darwin that do not support poll() on
pty devices.

svn path=/main/trunk/; revision=11023

16 years agoCreate a test case for the poll loop which uses the loop to read data from a
Zac Medico [Sat, 12 Jul 2008 06:22:26 +0000 (06:22 -0000)]
Create a test case for the poll loop which uses the loop to read data from a
pipe and assert that the data written to the pipe is identical to the data
read from the pipe. In order to implement this test, several useful classes
have been added:

 * PipeReader

     Reads output from one or more files and saves it in memory,
     for retrieval via the getvalue() method. This is driven by
     the scheduler's poll() loop, so it runs entirely within the
     current process.

 * QueueScheduler

     Add instances of SequentialTaskQueue and then call run().
     The run() method returns when no tasks remain.

 * TaskScheduler

     A simple way to handle scheduling of AsynchrousTask instances. Simply
     add tasks and call run(). The run() method returns when no tasks remain.

svn path=/main/trunk/; revision=11022

16 years ago* Add a digraph.get() method, analogous to dict.get(), used to simplify
Zac Medico [Sat, 12 Jul 2008 01:58:23 +0000 (01:58 -0000)]
* Add a digraph.get() method, analogous to dict.get(),  used to simplify
  Scheduler._pkg().

* Define digraph.__contains__ for containment tests via the "in" operator.

svn path=/main/trunk/; revision=11021

16 years agoFor subclasses of Task, explicitly assign __hash__ = Task.__hash__ for
Zac Medico [Sat, 12 Jul 2008 00:03:01 +0000 (00:03 -0000)]
For subclasses of Task, explicitly assign __hash__ = Task.__hash__ for
compatibility with python-2.6_beta. Thanks to Arfrever for reporting.

svn path=/main/trunk/; revision=11020

16 years agoFix SpawnProcess.start() to return early and notify exit listeners if the
Zac Medico [Fri, 11 Jul 2008 22:40:26 +0000 (22:40 -0000)]
Fix SpawnProcess.start() to return early and notify exit listeners if the
spawn call fails.

svn path=/main/trunk/; revision=11019

16 years agoFix a few typos found with pyflakes. Thanks to Cardoe for reporting.
Zac Medico [Fri, 11 Jul 2008 21:28:15 +0000 (21:28 -0000)]
Fix a few typos found with pyflakes. Thanks to Cardoe for reporting.

svn path=/main/trunk/; revision=11018

16 years agoCopy INHERITED/_eclasses_ handling from portdbapi._metadata_callback() to
Zac Medico [Fri, 11 Jul 2008 06:02:40 +0000 (06:02 -0000)]
Copy INHERITED/_eclasses_ handling from portdbapi._metadata_callback() to
portdbapi.aux_get() since it's still needed there or else aux_get won't
properly return INHERITED.

svn path=/main/trunk/; revision=11016

16 years agoCall settings.reset() in EbuildMetadataPhase.start() since that's what
Zac Medico [Fri, 11 Jul 2008 05:12:00 +0000 (05:12 -0000)]
Call settings.reset() in EbuildMetadataPhase.start() since that's what
portdbapi does too before generating metadata, and without it, a previous
CATEGORY can leak into the ebuild env and corrupt it (breaks lots of packages
that reference $CATEGORY).

svn path=/main/trunk/; revision=11015

16 years agoDocument the new --jobs and --load-average options for parallelization.
Zac Medico [Fri, 11 Jul 2008 04:16:17 +0000 (04:16 -0000)]
Document the new --jobs and --load-average options for parallelization.

svn path=/main/trunk/; revision=11014

16 years agoRefactor and simplify the main task scheduling and poll loops:
Zac Medico [Fri, 11 Jul 2008 02:38:35 +0000 (02:38 -0000)]
Refactor and simplify the main task scheduling and poll loops:

* Make output handlers unregister themselves and call wait() to notify
  exit listeners immediately. This makes the exit listeners more useful
  for scheduling tasks. This makes the poll loop nice an clean because
  it just calls the handlers and then the handlers can do the scheduling
  when necessary.

* Make SequentialTaskQueue.add() and addFront() trigger scheduling
  internally, so that it's more of a chain reaction than something that has
  to be done explicitly.

svn path=/main/trunk/; revision=11013

16 years agoAdd support for parallel --regen, using the existing --jobs and --load-average
Zac Medico [Thu, 10 Jul 2008 15:00:17 +0000 (15:00 -0000)]
Add support for parallel --regen, using the existing --jobs and --load-average
options:

 * Split out a PollLoop base class from Scheduler and derive a MetadataRegen
   class to handle parallel scheduling for --regen.

 * Add pordbapi._metadata_process() and _metadata_callback() methods to
   implement asynchronous metadata generation. One method returns an
   EbuildMetadataPhase instance to encapsulate the async task. The other
   method is called to save the metadata when the process completes
   successfully. These methods share code with aux_get to avoid duplicate
   code.

svn path=/main/trunk/; revision=11012

16 years agoWhen a package replaces another one, add the corresponding uninstall task
Zac Medico [Thu, 10 Jul 2008 10:33:03 +0000 (10:33 -0000)]
When a package replaces another one, add the corresponding uninstall task
for the replaced package to Scheduler._completed_tasks so that it's properly
accounted for in _dependent_on_scheduled_merges().

svn path=/main/trunk/; revision=11011

16 years agoBug #231382 - If the open() call in getconfig() raises an unexpected IOError,
Zac Medico [Thu, 10 Jul 2008 09:08:39 +0000 (09:08 -0000)]
Bug #231382 - If the open() call in getconfig() raises an unexpected IOError,
write a message to stderr showing the exact call that failed, including the
path of the file, in order to aid in debugging.

svn path=/main/trunk/; revision=11010

16 years agoMake digestcheck() skip the "checking ebuild checksums" part when fetching in
Zac Medico [Thu, 10 Jul 2008 04:41:27 +0000 (04:41 -0000)]
Make digestcheck() skip the "checking ebuild checksums" part when fetching in
the background.

svn path=/main/trunk/; revision=11009

16 years ago* Fix AsynchronousTask._wait_hook() to prevent potential recursive calls
Zac Medico [Thu, 10 Jul 2008 03:40:39 +0000 (03:40 -0000)]
* Fix AsynchronousTask._wait_hook() to prevent potential recursive calls
  triggered when calling exit listeners.

* Make SequentialTaskQueue use exit listeners to trigger scheduling as
  soon as a task exits. This makes it possible for the queues to
  automatically schedule new tasks while scheduler remains in it's
  poll() loop.

svn path=/main/trunk/; revision=11008

16 years ago* Handle errno.ECHILD and errno.ESRCH from waitpid() and kill() calls. These
Zac Medico [Thu, 10 Jul 2008 01:24:43 +0000 (01:24 -0000)]
* Handle errno.ECHILD and errno.ESRCH from waitpid() and kill() calls. These
  errors commonly occur when ^C is used to kill emerge.

* Remove spawned pids from portage.process.spawned_pids immediately in order
  to avoid a race condition for removing it.

svn path=/main/trunk/; revision=11007

16 years agoFix Scheduler._is_restart_necessary() to return False if the replacement
Zac Medico [Thu, 10 Jul 2008 00:58:08 +0000 (00:58 -0000)]
Fix Scheduler._is_restart_necessary() to return False if the replacement
version of portage is the same exact version as the currently running
instance.

svn path=/main/trunk/; revision=11006

16 years agoLike in Binpkg, cancel or wait for the prefetcher in EbuildBuild.
Zac Medico [Wed, 9 Jul 2008 23:53:10 +0000 (23:53 -0000)]
Like in Binpkg, cancel or wait for the prefetcher in EbuildBuild.

svn path=/main/trunk/; revision=11005

16 years agoBug #231358 - Fix 'NameError: global name 'p' is not defined' in
Zac Medico [Wed, 9 Jul 2008 23:20:32 +0000 (23:20 -0000)]
Bug #231358 - Fix 'NameError: global name 'p' is not defined' in
getmaskingstatus() that's triggered by a conditional in LICENSE.

svn path=/main/trunk/; revision=11004

16 years agoFor parallel-fetch prefetchers, pass fetchonly=1 into the EbuildFetcher
Zac Medico [Wed, 9 Jul 2008 21:51:25 +0000 (21:51 -0000)]
For parallel-fetch prefetchers, pass fetchonly=1 into the EbuildFetcher
constructor.

svn path=/main/trunk/; revision=11003

16 years agoUse the EbuildFetcher class to implement async fetch for the EbuildBuild
Zac Medico [Wed, 9 Jul 2008 21:39:39 +0000 (21:39 -0000)]
Use the EbuildFetcher class to implement async fetch for the EbuildBuild
class. When in background mode, fetch output is logged to emerge-fetch.log.
This solves a problem with calling doebuild(returnpid=1) and then doebuild
calling fetch() synchronously. In that case, doebuild would return an int
if fetch() failed (including when running pkg_nofetch), which would lead
to a TypeError since doebuild(returnpid=1) is supposed to return a list
containing a spawned pid.

svn path=/main/trunk/; revision=11002

16 years ago * Unify the scheduler's "build" and "extract" queues into a single
Zac Medico [Wed, 9 Jul 2008 12:33:08 +0000 (12:33 -0000)]
 * Unify the scheduler's "build" and "extract" queues into a single
   "jobs" queue.

 * Add support for logging fetches to /var/log/emerge-fetch.log when
   --jobs is enabled. Previously this log was only used for
   the parallel-fetch feature but now it's also used for --jobs.
   The scheduler's "prefetch" queue has been renamed to "fetch" since
   it's not exclusively used for parallel-fetch anymore.

 * Pass the "background" parameter from Binpkg in to the BinpkgFetcher
   instance, to send output to emerge-fetch.log instead of stdout.

svn path=/main/trunk/; revision=10999

16 years agoFix SubProcess._poll to handle a missing pid gracefully.
Zac Medico [Wed, 9 Jul 2008 12:03:53 +0000 (12:03 -0000)]
Fix SubProcess._poll to handle a missing pid gracefully.

svn path=/main/trunk/; revision=10998

16 years agoFix Scheduler._choose_pkg() to perform uninstallation actions (due to
Zac Medico [Wed, 9 Jul 2008 02:15:19 +0000 (02:15 -0000)]
Fix Scheduler._choose_pkg() to perform uninstallation actions (due to
blockers) in the correct order when building in parallel.

svn path=/main/trunk/; revision=10996

16 years agoFix _prune_digraph() to correctly distinguish "nomerge" nodes from
Zac Medico [Wed, 9 Jul 2008 01:46:24 +0000 (01:46 -0000)]
Fix _prune_digraph() to correctly distinguish "nomerge" nodes from
"uninstall" nodes.

svn path=/main/trunk/; revision=10995

16 years ago* Fix _choose_pkg() to pop the first node if there are no other jobs running.
Zac Medico [Tue, 8 Jul 2008 23:42:28 +0000 (23:42 -0000)]
* Fix _choose_pkg() to pop the first node if there are no other jobs running.
* Allow _choose_pkg() to choose an uninstall node when it reaches the front
  of the queue. TODO: fix order for uninstall operations.
* Fix _dependent_on_scheduled_merges() to handle direct circular deps
  correctly.

svn path=/main/trunk/; revision=10994

16 years agoMake the "tight loop" assertion trigger whenever there is no state change
Zac Medico [Tue, 8 Jul 2008 22:35:27 +0000 (22:35 -0000)]
Make the "tight loop" assertion trigger whenever there is no state change
detected for a given _schedule_main() call.

svn path=/main/trunk/; revision=10993

16 years agoIn scheduler._schedule_main(), only consider a poll() event to be a state
Zac Medico [Tue, 8 Jul 2008 22:28:41 +0000 (22:28 -0000)]
In scheduler._schedule_main(), only consider a poll() event to be a state
change if it results in a handler being unregistered.

svn path=/main/trunk/; revision=10992

16 years agoFix Scheduler._dependent_on_scheduled_merges() to properly distinguish
Zac Medico [Tue, 8 Jul 2008 22:19:21 +0000 (22:19 -0000)]
Fix Scheduler._dependent_on_scheduled_merges() to properly distinguish
"nomerge" nodes from "uninstall" nodes.

svn path=/main/trunk/; revision=10991

16 years agoRemove redundant call to _schedule_main().
Zac Medico [Tue, 8 Jul 2008 21:56:20 +0000 (21:56 -0000)]
Remove redundant call to _schedule_main().

svn path=/main/trunk/; revision=10990

16 years ago* Call self._wait() when unregistering output handlers, in order to avoid
Zac Medico [Tue, 8 Jul 2008 21:19:22 +0000 (21:19 -0000)]
* Call self._wait() when unregistering output handlers, in order to avoid
  triggering a tight loop. Also fix Subprocess._wait() to only schedule
  when self.registered is True.

* Add an assertion inside Scheduler._schedule_main() to try and detect tight
  loops like the one above.

* Fix typo in Scheduler._merge_exit().

svn path=/main/trunk/; revision=10989

16 years agoDerive PackageMerge from AsynchronousTask instead of CompositeTask since
Zac Medico [Tue, 8 Jul 2008 17:44:53 +0000 (17:44 -0000)]
Derive PackageMerge from AsynchronousTask instead of CompositeTask since
CompositeTask is designed for self._current_task to be set.

svn path=/main/trunk/; revision=10988

16 years agoRaise an AssertionError in CompositeTask._wait() if it's detected that
Zac Medico [Tue, 8 Jul 2008 17:26:46 +0000 (17:26 -0000)]
Raise an AssertionError in CompositeTask._wait() if it's detected that
self._current_task hasn't been properly updated after calling wait on
it.

svn path=/main/trunk/; revision=10987

16 years agoFix infinite loop in CompositeTask._wait().
Zac Medico [Tue, 8 Jul 2008 16:59:40 +0000 (16:59 -0000)]
Fix infinite loop in CompositeTask._wait().

svn path=/main/trunk/; revision=10986

16 years agoFix ValueError triggered when Scheduler._choose_pkg() doesn't find a package
Zac Medico [Tue, 8 Jul 2008 10:49:02 +0000 (10:49 -0000)]
Fix ValueError triggered when Scheduler._choose_pkg() doesn't find a package
and returns None.

svn path=/main/trunk/; revision=10985

16 years agoUse a normal list instead of a deque for Scheduler._pkg_queue since deque
Zac Medico [Tue, 8 Jul 2008 10:28:43 +0000 (10:28 -0000)]
Use a normal list instead of a deque for Scheduler._pkg_queue since deque
doesn't have a remove() method until python-2.5.

svn path=/main/trunk/; revision=10984

16 years agoImplement parallel build support by adding new --jobs and --load-average
Zac Medico [Tue, 8 Jul 2008 09:50:37 +0000 (09:50 -0000)]
Implement parallel build support by adding new --jobs and --load-average
options that are analogous to the corresponding `make` options. Input and
output handling still need work to make it look better and act more friendly
for things like interactive ebuilds that require input.

svn path=/main/trunk/; revision=10983

16 years ago* Add background support to the SpawProcess class.
Zac Medico [Tue, 8 Jul 2008 05:49:27 +0000 (05:49 -0000)]
* Add background support to the SpawProcess class.
* Make Binpkg pass the current background setting into subtasks.

svn path=/main/trunk/; revision=10982

16 years agoAdd AsynchronousTask.background attribute and add support for it to the
Zac Medico [Tue, 8 Jul 2008 05:15:55 +0000 (05:15 -0000)]
Add AsynchronousTask.background attribute and add support for it to the
EbuildPhase class.

svn path=/main/trunk/; revision=10981

16 years agoAlso add "background" attributes to the EbuildBuild and Binpkg classes.
Zac Medico [Tue, 8 Jul 2008 01:01:54 +0000 (01:01 -0000)]
Also add "background" attributes to the EbuildBuild and Binpkg classes.

svn path=/main/trunk/; revision=10980

16 years agoAdd a MergeListItem.background attribute for putting a task in the background
Zac Medico [Tue, 8 Jul 2008 00:53:33 +0000 (00:53 -0000)]
Add a MergeListItem.background attribute for putting a task in the background
when max_jobs > 1.

svn path=/main/trunk/; revision=10979

16 years agoAdd a missing check in Scheduler._main_loop() to ensure that max_jobs
Zac Medico [Tue, 8 Jul 2008 00:39:48 +0000 (00:39 -0000)]
Add a missing check in Scheduler._main_loop() to ensure that max_jobs
isn't exceeded.

svn path=/main/trunk/; revision=10978

16 years agoUse plain sequential search for the SRC_URI.mirror check since the regex
Zac Medico [Tue, 8 Jul 2008 00:25:56 +0000 (00:25 -0000)]
Use plain sequential search for the SRC_URI.mirror check since the regex
approach is actually much slower.

svn path=/main/trunk/; revision=10977

16 years agoAdd a Scheduler._digraph attribute for use in parallel build scheduling.
Zac Medico [Mon, 7 Jul 2008 23:32:11 +0000 (23:32 -0000)]
Add a Scheduler._digraph attribute for use in parallel build scheduling.

svn path=/main/trunk/; revision=10976

16 years agoRename SubProcess.reg_id to _reg_id since it no longer needs to be exposed.
Zac Medico [Mon, 7 Jul 2008 22:55:26 +0000 (22:55 -0000)]
Rename SubProcess.reg_id to _reg_id since it no longer needs to be exposed.

svn path=/main/trunk/; revision=10975

16 years ago* Add new CompositeTask._default_final_exit() method and use it as the new
Zac Medico [Mon, 7 Jul 2008 22:41:10 +0000 (22:41 -0000)]
* Add new CompositeTask._default_final_exit() method and use it as the new
  generic task exit callback.

* Remove erroneous self.wait() calls inside MergeListItem.start().

svn path=/main/trunk/; revision=10974

16 years agoAdd a new "SRC_URI.mirror" check which checks to if a uri listed in
Zac Medico [Mon, 7 Jul 2008 22:07:03 +0000 (22:07 -0000)]
Add a new "SRC_URI.mirror" check which checks to if a uri listed in
profiles/thirdpartymirrors is found in SRC_URI. Thanks to Betelgeuse for
the initial patch which has now been optimized with a regex.

svn path=/main/trunk/; revision=10973