Zac Medico [Tue, 17 May 2011 06:29:21 +0000 (23:29 -0700)]
--autounmask-write: only prompt when necessary
Zac Medico [Tue, 17 May 2011 06:10:38 +0000 (23:10 -0700)]
--autounmask-write: handle config file recursion
Zac Medico [Tue, 17 May 2011 04:30:08 +0000 (21:30 -0700)]
Document that --autounmask is enabled by default.
Sebastian Luther [Mon, 16 May 2011 20:06:40 +0000 (22:06 +0200)]
Enable --autounmask by default
Sebastian Luther [Mon, 16 May 2011 19:21:12 +0000 (21:21 +0200)]
--autounmask-write: respect --ask
Zac Medico [Mon, 16 May 2011 07:21:03 +0000 (00:21 -0700)]
BinpkgFetcher: handle AsynchronousLock failure
Zac Medico [Mon, 16 May 2011 07:10:07 +0000 (00:10 -0700)]
EbuildBuildDir: handle AsynchronousLock failure
Zac Medico [Mon, 16 May 2011 06:54:49 +0000 (23:54 -0700)]
_LockProcess: handle process failure more
Zac Medico [Mon, 16 May 2011 06:17:27 +0000 (23:17 -0700)]
_LockProcess: handle process failure if cancelled
Zac Medico [Mon, 16 May 2011 01:25:59 +0000 (18:25 -0700)]
preserve-libs: lock vardb for removeFromContents
Zac Medico [Sun, 15 May 2011 23:28:32 +0000 (16:28 -0700)]
Revert "counter_tick_core: don't lock if parallel-install"
This reverts commit
405ad9eed65393205ec28af8772f7ea45ce0371e.
The root problem is fixed by commit
8380ec9d7590a47b912e0d1e1db2ebb6049b903b.
Zac Medico [Sun, 15 May 2011 23:17:36 +0000 (16:17 -0700)]
treewalk: unlockdb/lockdb typo triggers EDEADLK
Zac Medico [Sun, 15 May 2011 22:29:15 +0000 (15:29 -0700)]
make.conf.txt: remove because unused/unmaintained
Arfrever Frehtes Taifersar Arahesis [Sun, 15 May 2011 21:24:31 +0000 (23:24 +0200)]
Fix some typos.
Sebastian Luther [Sun, 15 May 2011 19:05:19 +0000 (12:05 -0700)]
--autounmask: Treat missing keywords as masks
Sebastian Luther [Sun, 15 May 2011 19:03:20 +0000 (12:03 -0700)]
--autounmask: Allow package.mask changes
Sebastian Luther [Sun, 15 May 2011 19:01:03 +0000 (12:01 -0700)]
Implement --autounmask-write
Enabling this option together with --autounmask writes proposed
changes to config files, honoring CONFIG_PROTECT.
Zac Medico [Sun, 15 May 2011 09:30:15 +0000 (02:30 -0700)]
counter_tick_core: don't lock if parallel-install
This is the same as commit
461564ae94ff936918eeaa18493bc1da3846796f
but this time with comments that make sense.
Zac Medico [Sun, 15 May 2011 09:02:08 +0000 (02:02 -0700)]
EbuildMerge: inherit from CompositeTask
Zac Medico [Sun, 15 May 2011 08:14:09 +0000 (01:14 -0700)]
EbuildPhase: fix locking for setup phase
Zac Medico [Sun, 15 May 2011 08:03:14 +0000 (01:03 -0700)]
EbuildPhase: AsynchronousLock for ebuild-locks
Zac Medico [Sun, 15 May 2011 05:21:08 +0000 (22:21 -0700)]
Use vardbapi.lock() where applicable.
Zac Medico [Sun, 15 May 2011 04:39:26 +0000 (21:39 -0700)]
movefile: handle EEXIST when os.symlink fails
Zac Medico [Sun, 15 May 2011 04:29:48 +0000 (21:29 -0700)]
Revert "counter_tick_core: don't lock if parallel-install"
This reverts commit
461564ae94ff936918eeaa18493bc1da3846796f.
Disabling the lock here doesn't really make sense, and I'm still
triggering EDEADLK is some other places.
Zac Medico [Sun, 15 May 2011 02:55:24 +0000 (19:55 -0700)]
preserve-libs: use vardbapi _fs_lock/unlock
The preserve-libs code is too dependent on the filesystem state to
allow merging/unmerge/env_update to execute concurrently.
Zac Medico [Sun, 15 May 2011 02:43:12 +0000 (19:43 -0700)]
vardbapi: add reentrant _fs_lock/unlock methods
Zac Medico [Sun, 15 May 2011 01:56:39 +0000 (18:56 -0700)]
_LibGraphNode: re-use the _key attribute
This allows us to avoid repeating any previous stat calls, which helps
to avoid potential race conditions due to inconsistent stat results
when the file system is being modified concurrently.
Zac Medico [Sat, 14 May 2011 23:53:35 +0000 (16:53 -0700)]
counter_tick_core: don't lock if parallel-install
Hopefully this avoids the following exception:
File "/usr/lib/portage/pym/portage/locks.py", line 138, in lockfile
fcntl.lockf(myfd, fcntl.LOCK_EX)
IOError: [Errno 35] Resource deadlock avoided
Arfrever Frehtes Taifersar Arahesis [Sat, 14 May 2011 22:56:35 +0000 (00:56 +0200)]
Fix a typo in a comment.
Zac Medico [Sat, 14 May 2011 22:14:45 +0000 (15:14 -0700)]
find_unused_preserved_libs: fix unmerge handling
Since commit
c64d4abee145d083c70273be8fd23bd56dffe7ec (bug #286714),
preserved libs have not been properly garbage collected when packages
are unmerged without replacement, due to a difference in LinkageMap
state. It should behave correcly now.
Zac Medico [Sat, 14 May 2011 21:41:22 +0000 (14:41 -0700)]
prune_plib_registry: display preserved libs
Since commit
c64d4abee145d083c70273be8fd23bd56dffe7ec, sometimes we
preserve libs for unmerged packages here (bug #286714), so we should
display them like we do in _add_preserve_libs_to_contents().
Zac Medico [Sat, 14 May 2011 21:25:16 +0000 (14:25 -0700)]
treewalk: fix plib_collisions with same cpv
Sebastian Luther [Sat, 14 May 2011 20:44:22 +0000 (22:44 +0200)]
--autounmask: Use >= atoms if possible
If the package is the latest avaiable it now uses >=cat/pkg-ver instead
of =cat/pkg-ver. If the package is not the latest, but the latest in
its slot, >=cat/pkg-ver:slot is used. In all other case =cat/pkg-ver
is used. Rational is that it's more likely that newer versions, added
in the future, will work with the same configuration changes as the
currently latest version than not.
Zac Medico [Sat, 14 May 2011 08:45:06 +0000 (01:45 -0700)]
preserve-libs: preserve during uninstall
This will fix bug #286714. The emerge --depclean-lib-check option will
now be ignored when FEATURES=preserve-libs is enabled, since any
libraries that have consumers will simply be preserved.
Zac Medico [Thu, 12 May 2011 19:04:54 +0000 (12:04 -0700)]
cache.sqlite: handle readonly more
Zac Medico [Thu, 12 May 2011 19:02:27 +0000 (12:02 -0700)]
porttree: use pure volatile cache if necessary
Zac Medico [Thu, 12 May 2011 18:58:27 +0000 (11:58 -0700)]
cache.volatile: discard "perms" param
Zac Medico [Thu, 12 May 2011 18:42:08 +0000 (11:42 -0700)]
cache.sqlite: handle readonly
Zac Medico [Thu, 12 May 2011 17:51:08 +0000 (10:51 -0700)]
expand_new_virt: return early for non-virtual cat
Zac Medico [Thu, 12 May 2011 07:09:19 +0000 (00:09 -0700)]
expand_new_virt: skip match for non-virtual cat
Zac Medico [Thu, 12 May 2011 05:35:40 +0000 (22:35 -0700)]
treewalk: fix inverted counter init logic
Zac Medico [Thu, 12 May 2011 01:14:54 +0000 (18:14 -0700)]
portdbapi: tweak cache permission handling
This is a reponse to the following issue:
http://code.google.com/p/chromium-os/issues/detail?id=15234
Zac Medico [Wed, 11 May 2011 23:50:15 +0000 (16:50 -0700)]
EbuildPhase: extract binpkg env for pkg_pretend
This will fix bug #366939.
Zac Medico [Wed, 11 May 2011 20:24:22 +0000 (13:24 -0700)]
unmerge: use frozenset for syslist
Zac Medico [Wed, 11 May 2011 19:00:27 +0000 (12:00 -0700)]
depgraph: in complete mode respect --with-bdeps
If you want the graph as complete as possible, now you'll have to use
--with-bdeps=y together with --complete-graph.
Zac Medico [Wed, 11 May 2011 04:03:02 +0000 (21:03 -0700)]
dep_check_composite_db: return early from cp_list
Since this implementation is only intended to check for existence of
new-style virtuals, it's a waste of time to return more than one cpv.
Zac Medico [Tue, 10 May 2011 23:54:55 +0000 (16:54 -0700)]
depgraph: traverse ignored DEPEND if "complete"
David James [Tue, 10 May 2011 20:41:06 +0000 (13:41 -0700)]
Strip build-time dependencies when they're ignored and we're not in --rebuild mode.
This should improve depgraph calculation performance.
TEST=Ran test suite.
BUG=chromium-os:15144
Change-Id: I9e06817201d7a17ae9b44b424d01c1a2bd89210a
Zac Medico [Tue, 10 May 2011 19:52:36 +0000 (12:52 -0700)]
MergeProcess: lock vdb earlier when appropriate
Zac Medico [Tue, 10 May 2011 05:11:33 +0000 (22:11 -0700)]
depgraph: don't clear vardbapi cache in _load_vdb
Most of the memory is probably held on the heap by the installed
package instances anyway, and the cache is useful for being inherited
by subprocess in MergeProcess.
Zac Medico [Tue, 10 May 2011 05:05:40 +0000 (22:05 -0700)]
get_counter_tick_core: use cpv_all
It's more efficient to use cpv_all since cp_all calls cpv_all anyway,
and calls to cp_list induce additional stat calls.
David James [Tue, 10 May 2011 04:11:47 +0000 (21:11 -0700)]
Cache counter values, avoiding I/O when it doesn't change.
This improves merge times by up to 25%, since looping over the vardb for
each package install is slow.
TEST=Emerge a bunch of packages, notice 25% speed improvement.
BUG=chromium-os:15112
Change-Id: I51dd617219cd1820ceeb702291bd790990995be4
Zac Medico [Tue, 10 May 2011 01:03:27 +0000 (18:03 -0700)]
dep_check_composite_db: remove unused _dep_expand
This code has been unused since dep_check started passing
token_class=Atom to use_reduce in commit
c9f7930883d62fc26af72bff0c4623db0bbc8221. Since then, any atoms
without categories are dropped by dep_check because it returns
early after catching an InvalidDependString exception.
Zac Medico [Mon, 9 May 2011 22:34:06 +0000 (15:34 -0700)]
dep_check_composite_db: implement cp_list
This is used by dep_check since commit
6f9ee9c508c1506cdf8eb0dc46796dbe30f268db.
Zac Medico [Mon, 9 May 2011 20:41:56 +0000 (13:41 -0700)]
config: call _populate_treeVirtuals directly
Zac Medico [Mon, 9 May 2011 20:36:22 +0000 (13:36 -0700)]
cpv_expand: avoid old-style virtuals more
Zac Medico [Mon, 9 May 2011 20:30:04 +0000 (13:30 -0700)]
dep_check: avoid old-style virtuals code
Zac Medico [Mon, 9 May 2011 06:27:39 +0000 (23:27 -0700)]
display_news_notification: don't populate virts
This typically isn't useful anymore, since Gentoo has stopped using
old-style virtuals.
Zac Medico [Mon, 9 May 2011 06:23:07 +0000 (23:23 -0700)]
cpv_expand: populate old-style virts on demand
If old-style virtuals code is triggered here and we are given a
vartree, use it to avoid creating a temporary instance.
Zac Medico [Mon, 9 May 2011 06:08:43 +0000 (23:08 -0700)]
unmerge: skip getvirtuals() if cat is not virtual
This allows us to avoid triggering the old-style virtual code which
scans /var/db/pkg/*/*/PROVIDE.
Zac Medico [Mon, 9 May 2011 05:13:52 +0000 (22:13 -0700)]
unmerge: use expand_new_virt for sys pkg warnings
Zac Medico [Mon, 9 May 2011 04:51:47 +0000 (21:51 -0700)]
vardbapi: disable subprocess vdb cache updates
Since commit
7535cabdf2fab76fc55df83643157613dfd66be9,
vardbapi.flush_cache() is often called within subprocesses spawned
from MergeProcess. The _aux_cache_threshold doesn't work as designed
if the cache is flushed from a subprocess like this, can lead to the
vdb cache being flushed for every single merge. This is a waste of
disk IO, so disable vdb cache updates in subprocesses.
Zac Medico [Sun, 8 May 2011 21:36:48 +0000 (14:36 -0700)]
LinkageMapELF: remove unused imports
Zac Medico [Sun, 8 May 2011 21:31:11 +0000 (14:31 -0700)]
PreservedLibsRegistry: normalize counter as str
Zac Medico [Sun, 8 May 2011 20:46:40 +0000 (13:46 -0700)]
Add back cpv_exists check from previous commit.
Zac Medico [Sun, 8 May 2011 20:19:47 +0000 (13:19 -0700)]
Remove obsolete others_in_slot plibs code.
This code dates back to commit
79a2ade5a6cb5a6c2d7ace838a39a634265c5522
and the preserve-libs logic was different back then. At that time, the
dblink._preserve_libs() method would earlier register preserved libs
for the new package currently being merged, while the vardb entry was
still a -MERGING- node. Current code does not register preserve_paths
until later, so this others_in_slot code is obsolete.
Zac Medico [Sun, 8 May 2011 19:31:07 +0000 (12:31 -0700)]
treewalk: always initialize "needed"
Zac Medico [Sun, 8 May 2011 19:05:49 +0000 (12:05 -0700)]
merge_wait_queue: use deque.clear()
Zac Medico [Sun, 8 May 2011 18:24:40 +0000 (11:24 -0700)]
resume_depgraph: don't call break_refs in loop
The break_refs call was unnecessary and it broke the state of the
frozen_config instance, causing the following error:
File "pym/_emerge/depgraph.py", line 1290, in
_wrapped_add_pkg_dep_string
inst_pkgs = vardb.match_pkgs(atom)
AttributeError: 'vardbapi' object has no attribute 'match_pkgs'
David James [Sun, 8 May 2011 07:26:42 +0000 (00:26 -0700)]
Fix typos in unmerge function.
BUG=none
TEST=Try unmerging some packages.
Change-Id: Ib643ec95b8da14b49a6b519e445fe57f8995c52c
Zac Medico [Sun, 8 May 2011 07:09:37 +0000 (00:09 -0700)]
linkmap: use exclude_pkgs for the registry too
David James [Sun, 8 May 2011 06:08:01 +0000 (23:08 -0700)]
Use new vardbapi lock function in FakeVartree.
BUG=none
TEST=Merged some packages with new code. Ran test suite.
Change-Id: Ibb9380122631bf6c79e39691233cfd641931b75a
Zac Medico [Sun, 8 May 2011 05:58:33 +0000 (22:58 -0700)]
Be careful with system pkgs + parallel-install.
This should prevent FEATURES=parallel-install from interfering with
the fixes from bug #256616 and bug #259954.
Zac Medico [Sun, 8 May 2011 05:02:38 +0000 (22:02 -0700)]
linkmap: check for access before locking vardbapi
Zac Medico [Sun, 8 May 2011 04:46:08 +0000 (21:46 -0700)]
Use EROOT with VDB_PATH in recent changes.
Zac Medico [Sun, 8 May 2011 04:16:58 +0000 (21:16 -0700)]
Scheduler: add queue for ebuild-locks
Use a separate queue for ebuild-locks when the merge
queue allows more than 1 job (due to parallel-install),
since the portage.locks module does not behave as desired
if we try to lock the same file multiple times
concurrently from the same process.
Zac Medico [Sun, 8 May 2011 03:29:27 +0000 (20:29 -0700)]
Add FEATURES=parallel-install for finer locks.
Zac Medico [Sun, 8 May 2011 03:05:21 +0000 (20:05 -0700)]
vardbapi: acquire lock during counter_tick_core
Zac Medico [Sun, 8 May 2011 02:57:29 +0000 (19:57 -0700)]
Add vardbapi reentrant lock/unlock methods.
Zac Medico [Sun, 8 May 2011 04:34:25 +0000 (21:34 -0700)]
Rename FEATURES=no-ebuild-locks to ebuild-locks.
David James [Sat, 7 May 2011 04:53:31 +0000 (21:53 -0700)]
Use finer grained locks for install.
Narrow scope of merge locks to improve performance.
Instead of locking the DB for the entire package merge, just lock it
when we actually need to do so. Also add locks around conf_mem_file
updating and pkg_* phases.
Locking in pkg_* phases can be turned off with
FEATURES="no-ebuild-locks" if you use ebuilds that are careful not
to mess with each other during theses phases. The default is to leave
this locking enabled.
Given this new locking, I've improved the scheduler to run merge jobs
in parallel.
Time required for merging 348 packages with --usepkgonly:
- Before patch: 29m50s
- After patch: 10m2s
- After patch w/o locks: 7m9s
Change-Id: I63588c4cc59fa6fe2f8327ea1e4a9e71b241d4fe
Review URL: http://gerrit.chromium.org/gerrit/498
Zac Medico [Sat, 7 May 2011 21:59:54 +0000 (14:59 -0700)]
doebuild: use EbuildBuildDir for locking
EbuildBuildDir also performs safe removal of the category when it is
empty, so ebuild.sh/dyn_clean doesn't need to do it anymore. This
fixes a race condition if one process is trying to remove the category
directory while another one is trying to create it for
PORTAGE_BUILDDIR.
Zac Medico [Sat, 7 May 2011 17:25:20 +0000 (10:25 -0700)]
PreserveLibsRegistry: add lock/unlock assertions
Also, add comments to store() about unobvious interaction with
locking due to atomic replacement of the inode.
Zac Medico [Sat, 7 May 2011 03:28:21 +0000 (20:28 -0700)]
Remove redundant pruneNonExisting() call.
Since commit
f36b9fa38b5268c2a5579db62acec026625f84a9, the
PreservedLibsRegistry automatically prunes itself each time
that it is loaded.
Zac Medico [Sat, 7 May 2011 03:02:50 +0000 (20:02 -0700)]
preserve-libs: fix linkmap logic for new plibs
Together with commit
f36b9fa38b5268c2a5579db62acec026625f84a9,
hopefully this solves bug #366061.
David James [Thu, 5 May 2011 21:08:27 +0000 (14:08 -0700)]
Cleanup preserved lib locking in portage.
This change makes preserved lib modification atomic, and
prepares us for narrowing the scope of the merge lock.
BUG=chromium-os:14983
TEST=Ran test suite and some example emerges.
Change-Id: I39abb6a5ec72be3274e508ef807ac1d9e69db1a8
Review URL: http://gerrit.chromium.org/gerrit/417
Zac Medico [Thu, 5 May 2011 17:21:43 +0000 (10:21 -0700)]
depgraph: simplify break_refs
Zac Medico [Thu, 5 May 2011 15:58:22 +0000 (08:58 -0700)]
emerge: fix misspell suggestion with category
A name collision in the "cp" variable caused it to malfunction.
Zac Medico [Thu, 5 May 2011 15:15:20 +0000 (08:15 -0700)]
Update timestamps in headers of modified files.
Zac Medico [Thu, 5 May 2011 06:44:48 +0000 (23:44 -0700)]
repoman: make virtual.oldstyle a warning for now
This will give third-party repoman users some time to migrate their
repos away old-style virtuals.
Zac Medico [Wed, 4 May 2011 21:40:11 +0000 (14:40 -0700)]
add_pkg_deps: removal action discard DEPEND early
Zac Medico [Wed, 4 May 2011 18:18:34 +0000 (11:18 -0700)]
add_pkg_deps: use dep priority to ignore blockers
David James [Wed, 4 May 2011 17:35:02 +0000 (10:35 -0700)]
test_rebuild: test --rebuild-if-new-rev more
Zac Medico [Wed, 4 May 2011 17:20:58 +0000 (10:20 -0700)]
add_pkg_dep_string: respect traverse_ignored_deps
I don't think this makes any difference now, but it clarifies how
traverse_ignored_deps should behave.
David James [Wed, 4 May 2011 06:32:05 +0000 (23:32 -0700)]
emerge: replace --rebuild with finer grained opts
Replace --rebuild option with --rebuild-if-* options.
--rebuild-if-new-rev [ y | n ]
Rebuild packages when dependencies that are used at both build-time and
run-time are built, if the dependency is not already installed with the
same version and revision.
--rebuild-if-new-ver [ y | n ]
Rebuild packages when dependencies that are used at both build-time and
run-time are built, if the dependency is not already installed with the
same version. Revision numbers are ignored.
--rebuild-if-unbuilt [ y | n ]
Rebuild packages when dependencies that are used at both build-time and
run-time are built.
Change-Id: Ia50c1702bfe1b98a8d1891740e7bbb045921a905
Review URL: http://gerrit.chromium.org/gerrit/280
Zac Medico [Wed, 4 May 2011 16:36:29 +0000 (09:36 -0700)]
doins: add reference to bug #239529
The reasoning for the code from commit
55b3150af635a418ba3f1424132359c894db7ec4 is not really obvious, so
it's helpful to reference the corresponding bug report.
Mike Gilbert [Wed, 4 May 2011 03:28:21 +0000 (23:28 -0400)]
Fix Manifest.updateAllHashes()
David James [Tue, 3 May 2011 20:51:30 +0000 (13:51 -0700)]
Add --rebuild-ignore flag.
A space separated list of package names or slot atoms. Emerge will not rebuild
packages that depend on matching packages due to \fB\-\-rebuild\fR.
Change-Id: Ia58fe632ed06c97a22413da0341d7f8da2d65ba8
Review URL: http://gerrit.chromium.org/gerrit/209
David James [Tue, 3 May 2011 20:10:28 +0000 (13:10 -0700)]
rebuild_config: propagate runtime deps to parents
Update rebuild option to propagate runtime deps to parents.
Suggested by SebastianLuther@gmx.de
BUG=chromium-os:14858
TEST=Added unit test. Ran unit tests.
Change-Id: I7228a8558eddd1956c590de39430172476c66228
Review URL: http://gerrit.chromium.org/gerrit/202
Zac Medico [Tue, 3 May 2011 03:00:54 +0000 (20:00 -0700)]
cpv_expand: avoid old-style virtual code
This allows portageq calls to avoid loading vdb_metadata.pickle in
most cases, which greatly improves performance.