| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
| |
Change-Id: I46c55827df79f859330be9a290b22b9ada0b33f2
|
| |
|
|
| |
Change-Id: I1becfc6730a04bc694cf9140e58a4cca085d01d6
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Gitiles has special FilteredRepository wrapper that
allows to carefully hide refs based on the project's ACLs.
There is however an optimisation that skips the filtering
in case a user has READ permissions on every ACLs patterns.
When the target repository is All-Users, the optimisation
turns into a security issue because it allows seeing everything
that belongs to everyone:
- draft comments
- PII of all users
- external ids
- draft edits
Block Gitiles or any other part of Gerrit to abuse of this
power when the target repository is All-Users, where nobody
can be authorised to skip the ACLs evaluation.
Cover the additional special case of the All-Users project
access with two explicit positive and negative tests,
so that the security check is covered.
Bug: Issue 13621
Change-Id: Ia6ea1a9fd5473adff534204aea7d8f25324a45b7
|
| |
|
|
| |
Change-Id: Ib0811a75b49475508b0edba53be51a9d812a6e97
|
| |
|
|
| |
Change-Id: I17fa745d38b17b94a63652ff21ed9006541d1e1d
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This change fixes a misconception that leads to data being accessible
through Gerrit APIs that should be locked down.
Gerrit had two components for determining if a Git ref is visible to a
user: (Default)RefFilter and PermissionBackend#ForRef (ex RefControl).
The former was always capable of providing correct results for all refs.
The latter only had logic to decide if a Git ref is visible according to
the Gerrit READ permissions. This includes all refs under refs/heads as
well as any other ref that isn't a database ref or a Git tag. This
component was unware of Git tags and database references. Hence, when
asked for a database reference such as refs/changes/xx/yyyyxx/meta the
logic would allow access if the user has READ permissions on any of the
ref prefixes, such as the default "read refs/* Anonymous Users".
That is problematic, because it bypasses documented behavior [1] where
a user should only have access to a change if they can see the destination
ref. The same goes for other database references.
This change fixes the problem. It is intentionally kept to a minimally
invasive code change so that it's easier to backport it.
Add tests to assert the correct behavior. These tests would fail before
this fix. We have included them in this change to be able to backport
just a single commit.
[1] https://gerrit-review.googlesource.com/Documentation/access-control.html
Change-Id: Ice3a756cf573dd9b38e3f198ccc44899ccf65f75
|
| |\
| |
| |
| |
| |
| |
| |
| |
| | |
* stable-3.0:
Update git submodules
Set version to 2.16.24-SNAPSHOT
Set version to 2.16.23
Change-Id: Ia25617f76c94a6d796a2601163ed28e8fc6af791
|
| | |\
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
* stable-2.16:
Update git submodules
Set version to 2.16.24-SNAPSHOT
Set version to 2.16.23
Change-Id: I3ebaf0908c1b3e360fde02a96dc120dd48b096a9
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
* Update plugins/replication from branch 'stable-2.16'
to 8fdb0f9ac0a7f68b3f942cb4a9fd4c94e488ab57
- ReplicationStorageIT: Wait for all pushes without order
Some tests don't have a predefined order for which events will be
replicated first. Using a timeout based on a single replication event is
flawed when we don't know the expected order. Instead, use a timeout for
the group of events and ignore the order.
For two events replicating to a single remote with a single thread, we
expect the complete replication to take twice as long. Two events
replicating to two remotes will use one thread each and therefore not
take any longer than the single remote case.
Change-Id: Ieb21b7eee32105eab5b5a15a35159bb4a837e363
|
| | | |
| | |
| | |
| | | |
Change-Id: Ie3b33382fe2b8d64894f89afc25061ecd17ece90
|
| | | |
| | |
| | |
| | | |
Change-Id: Ia533bb65648b3799fc742ec982058e11712ac78e
|
| | | |
| | |
| | |
| | | |
Change-Id: I097dd88d906008d7af1ee9483f6b8ec96380ff72
|
| |\| |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
* stable-3.0:
Update git submodules
Documentation: Escape <= characters in user-search.txt
Documentation: Provide examples for advanced regexp searching
Change-Id: I48d09b04dbde42387a7378715adb54e4001c68dd
|
| | |\ \ |
|
| | | |\|
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
* stable-2.16:
Documentation: Escape <= characters in user-search.txt
Documentation: Provide examples for advanced regexp searching
Change-Id: Ie18c40745ebfaca7a6ca0dd672b6aca86bbf36be
|
| | | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
Without escaping '<=' is rendered as ⇐ by AsciiDoc.
Change-Id: I2223cca45f80c2aaee76d1e84c2de34e966d7620
|
| | | | |\
| | | | |
| | | | |
| | | | | |
stable-2.16
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
dk.brics regexp syntax reference [1] doesn't contain examples.
[1] https://www.brics.dk/automaton/doc/index.html?dk/brics/automaton/RegExp.html
Change-Id: I9be2a3e4f1f387ec17f1702831a9bbebc85585be
|
| | |/ / /
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
* Update plugins/replication from branch 'stable-3.0'
to 4604b01e43c3b4aee3d03cb84945b3238c150ede
- Merge branch 'stable-2.16' into stable-3.0
* stable-2.16:
ReplicationStorageIT: Wait for all pushes without order
ReplicationTasksStorage: Add multi-primary unit tests
Change-Id: I1d749621c189ee2e49f092ddc7558f83e508411f
- ReplicationStorageIT: Wait for all pushes without order
Some tests don't have a predefined order for which events will be
replicated first. Using a timeout based on a single replication event is
flawed when we don't know the expected order. Instead, use a timeout for
the group of events and ignore the order.
For two events replicating to a single remote with a single thread, we
expect the complete replication to take twice as long. Two events
replicating to two remotes will use one thread each and therefore not
take any longer than the single remote case.
Change-Id: Ieb21b7eee32105eab5b5a15a35159bb4a837e363
- Merge "ReplicationTasksStorage: Add multi-primary unit tests" into stable-2.16
- ReplicationTasksStorage: Add multi-primary unit tests
These tests examine the replication scenarios under multi-primary
setup making use of the api calls present in ReplicationTasksStorage
class similarly as done in single primary setup.
These tests ensure that the replication compatibility in multi-primary
setup is not broken.
Change-Id: I375b731829f3c0640d3a7a98635e1e5c526908ca
|
| | | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
* Update plugins/replication from branch 'stable-3.1'
to e09b5a08e183d4f92410bfa06289a1784cefb43b
- Merge branch 'stable-3.0' into stable-3.1
* stable-3.0:
ReplicationStorageIT: Wait for all pushes without order
ReplicationTasksStorage: Add multi-primary unit tests
Change-Id: I3961368f7bcf7d4aa923d07f7f89beeaaeb307d3
- Merge branch 'stable-2.16' into stable-3.0
* stable-2.16:
ReplicationStorageIT: Wait for all pushes without order
ReplicationTasksStorage: Add multi-primary unit tests
Change-Id: I1d749621c189ee2e49f092ddc7558f83e508411f
- ReplicationStorageIT: Wait for all pushes without order
Some tests don't have a predefined order for which events will be
replicated first. Using a timeout based on a single replication event is
flawed when we don't know the expected order. Instead, use a timeout for
the group of events and ignore the order.
For two events replicating to a single remote with a single thread, we
expect the complete replication to take twice as long. Two events
replicating to two remotes will use one thread each and therefore not
take any longer than the single remote case.
Change-Id: Ieb21b7eee32105eab5b5a15a35159bb4a837e363
- Merge "ReplicationTasksStorage: Add multi-primary unit tests" into stable-2.16
- ReplicationTasksStorage: Add multi-primary unit tests
These tests examine the replication scenarios under multi-primary
setup making use of the api calls present in ReplicationTasksStorage
class similarly as done in single primary setup.
These tests ensure that the replication compatibility in multi-primary
setup is not broken.
Change-Id: I375b731829f3c0640d3a7a98635e1e5c526908ca
|
| | | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
* Update plugins/replication from branch 'stable-3.1'
to f3fdc78f71dd4afcf30152e7b653050debf85bef
- Merge "ReplicationTasksStorage.Task: Add multi-primary unit tests" into stable-3.1
- ReplicationTasksStorage.Task: Add multi-primary unit tests
These tests examine the replication scenarios under
multi-primary setup making use of the api calls present
in ReplicationTasksStorage.Task class similarly as done in
single master setup.
These tests ensure that the replication compatability in
multi-primary setup is not broken.
Change-Id: I980e8286bf11d31c6ab89e49ef065fdde1118181
|
| | | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
* Update plugins/replication from branch 'stable-3.1'
to abf3bfd32057db1efab3222fb26a14733db90bd0
- Replication*IT: Share getRef method
This helper is common to a couple test classes, so share it.
Change-Id: I5839c31ad734c384e812e9e1c7bcba8ba05c23cc
- ReplicationFanoutIT: Share setReplicationDestination
Refactor setReplicationDestination to share the per-remote-file and
single replication config file implementations more.
Change-Id: Ic0a98ccf0f7703f14c01856a42b8a70e3d20aa8b
- ReplicationFanoutIT: Split shouldReplicateNewBranch tests
Split these into storage-based and e2e tests so that the storage tests
can be reliably verified through use of replicationDelay large enough
that task state on disk doesn't change during the tests.
Keep them all in ReplicationFanoutIT for now since the setup for these
tests is unique to that class.
Also remove the unnecessary cleanup of tasks.
Change-Id: I36e0a4affe1f5d1330ea27a496fd8ba295176763
- ReplicationFanoutIT: Remove generic waitUntil helper
Using a non-specific timeout is a bad pattern. Tests should pick a
timeout appropriate to the action being tested.
Change-Id: I69a7e469df1dc532af6a777ac47d89852091797e
- ReplicationFanoutIT: Inherit from ReplicationDaemon
Reduces duplication across the replication IT classes. More dedup is
possible with the helper methods, but leave that for a future change.
Change-Id: Iddd6dca9a4fe84b065954cd4dcec7289d7ed68a2
- ReplicationFanoutIT: Refactor setRemoteReplicationDestination
Simplify callers by providing a way to set the replicationDelay up
front.
Change-Id: I28cea83559aa1eb379ec2ff962a0beaf25fe4ca6
- ReplicationFanoutIT: Rename setReplicationDestination
Distinguish the methods that set the per-remote config files vs the
methods that set the global replication.config. This helps lead up to
ReplicationFanoutIT inheriting from ReplicationDaemon.
Change-Id: I6139d2dbde15c0b0449d7d7801c169253bc7449d
- ReplicationFanoutIT: Cleanup shouldCreateIndividualReplicationTasksForEveryRemoteUrlPair
Remove some dead code, use Integer.MAX_VALUE for the replicationDelay so
that tasks stay in the waiting/ area of storage for the entire test, and
use a dedicated listWaitingTasks() to show it only depends on tasks in
that state.
Change-Id: I0035a4edc656ed4833249322c45204124a66e20d
|
| |\| | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
* stable-3.0:
Update git submodules
Change-Id: I8d0c764c2d5e335d16051f8dd15b0e0256232516
|
| | |\| |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
* stable-2.16:
Update git submodules
Change-Id: I62a00efaf5ebe0555cb1db0b36e26d980c9fffb6
|
| | | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
* Update plugins/replication from branch 'stable-2.16'
to 53e083fd0f17d1403b4d150e66655907c1ea139d
- Merge "ReplicationTasksStorage: Add multi-primary unit tests" into stable-2.16
- ReplicationTasksStorage: Add multi-primary unit tests
These tests examine the replication scenarios under multi-primary
setup making use of the api calls present in ReplicationTasksStorage
class similarly as done in single primary setup.
These tests ensure that the replication compatibility in multi-primary
setup is not broken.
Change-Id: I375b731829f3c0640d3a7a98635e1e5c526908ca
|
| |\ \ \ \ |
|
| | |\| | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
* stable-3.0:
Update git submodules
Bump Bazel version to 3.7.0
Documentation: Update dev-release's rc information
Update git submodules
Fix tests for stable-2.16 branch
Remove generation for c.g.gwtexpui.* JavaDoc
Fetch JGit JavaDoc from archive.eclipse.org
Update git submodules
Change-Id: I8a200bc075fe209f27e3fec846fe2d397d133820
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
* Update plugins/replication from branch 'stable-3.0'
to 8650a78b2682c90247435cfd807f0d0bacc84f9c
- Move storage portion of replicateBranchDeletion ITs
All other ITs split e2e and storage tests on stable-2.16, so this change
only updates the new replicateBranchDeletion tests that were added in
stable-3.0.
The e2e check for if the destination branch is removed or not stays in
ReplicationIT and the check that a task is created in storage when the
branch delete API is invoked moves to ReplicationStorageIT.
This split allows the best practices for verifying e2e and storage to be
applied independently.
Change-Id: Iec7ee090bd614e3442b1f9cb454437c9e05290be
- Merge branch 'stable-2.16' into stable-3.0
* stable-2.16:
Refactor Replication*IT tests to share a base class
ReplicationIT: Add shouldMatch* e2e tests
ReplicationStorageIT: Move shouldMatch* tests from ReplicationIT
ReplicationStorageIT: Add shouldFire*ChangeRefs tests
Move storage-based ITs into ReplicationStorageIT
ReplicationQueue: Remove unused method
This change does not try to reimpose the breakdown of tests that was
done in 2.16. That will be done in follow up change(s) to improve
reviewability of this change.
Change-Id: I83202997610c5ad0d8849cb477ca36db8df760f5
- Refactor Replication*IT tests to share a base class
These classes have very similar setups and duplicate helper methods.
Improve maintainability by reducing the duplication.
ReplicationQueueIT is not modified because it is merged into
ReplicationIT on stable-3.0.
Change-Id: Ibc22ae4d0db2d09009f65c0e745f1095c67827ba
- ReplicationIT: Add shouldMatch* e2e tests
These new tests utilize creating a branch in a way that does not trigger
replication so that scheduleFullSync() is responsible for replicating
the update. In this way, the tests verify the destination receives the
update because scheduleFullSync() matched the given URI.
Change-Id: I4ae15d0301a308a12cbca3684915e89ca421e02f
- ReplicationStorageIT: Move shouldMatch* tests from ReplicationIT
These tests are focused on verifying storage, so they belong in
ReplicationStorageIT. Improve these tests to better verify storage
correctness by switching the 'now' parameter to false such that
replicationDelay is honored and follow the ReplicationStorageIT
pattern using a very long delay. These improvements make these tests
much more stable.
The tests improve the ref matching slightly by comparing to the
PushOne.ALL_REFS constant.
Also removes the disableDeleteForTesting flag as there are no users of
it now.
A later change can add ReplicationIT e2e tests for these use cases.
Change-Id: Iaa14a7429a40fb62325259efa1c7d7637deef95a
- ReplicationStorageIT: Add shouldFire*ChangeRefs tests
Copy the shouldFire*IncompleteUri tests as shouldFire*ChangeRefs to
fill a gap in test coverage.
Change-Id: Ia8df64a8574b776e6a9f7201c0862f1e6794687e
- Move storage-based ITs into ReplicationStorageIT
Tests in ReplicationStorageIT utilize very long replication delays such
that tasks are never expected to complete during the test. This allows
test writers to assume the task files are still there.
Refactor tests from ReplicationIT into ReplicationStorageIT and focus
them on verifying storage correctness. This is mostly a direct copy
except that shouldFirePendingOnlyToStoredUri gets renamed and split into
two tests. One that validates tasks are fired and another that validates
replication completes to the expected destinations. This split is
necessary because of the very long delay methodology mentioned above.
Code sharing between ReplicationIT and ReplicationStorageIT will be
improved in a later commit.
Change-Id: I41179c20a10354953cff3628368dfd5f910cc940
- ReplicationQueue: Remove unused method
And drop the misleading @VisibleForTesting annotation from the method
the removed method was wrapping. scheduleFullSync() is public so that
PushAll can call it.
Change-Id: I0139e653654fcaf20de68dddfb5ea85560a323d0
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
https://github.com/bazelbuild/bazel/releases/tag/3.7.0
Change-Id: I4ed5a44a16f57bc6d1a038fa419d20563b3d6fe8
|
| | | | | |
| | | | |
| | | | |
| | | | | |
Change-Id: I674cec44628c24002942a9f446bc13f3668fb1be
|
| | | |\| |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
* stable-2.16:
Update git submodules
Fix tests for stable-2.16 branch
Remove generation for c.g.gwtexpui.* JavaDoc
Fetch JGit JavaDoc from archive.eclipse.org
Update git submodules
Change-Id: I8cf99479382398812c8f37630e045bec131f97bc
|
| | | | |\ \
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | | |
* changes:
Fix tests for stable-2.16 branch
Remove generation for c.g.gwtexpui.* JavaDoc
Fetch JGit JavaDoc from archive.eclipse.org
|
| | | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | | |
Add the 'manual' tag to wct test_suite templates,
so it is excluded from bazel test //...
Change-Id: I73fdddc9c08eeaacff9401ea9531c95e6a782ced
(cherry picked from commit ae42cd00bdfa8a34e75c563b62f0151a561cc82b)
|
| | | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | | |
The JavaDoc for com.google.gwtexpui.* cannot be generated
because the source files are not accessible anymore.
Failing to generate the JavaDocs caused the Gerrit build to
fail with 'No source files for package com.google.gwtexpui...'.
Change-Id: Ie36e650962636813d8f9f615e495a980b7280420
|
| | | | | | |
| | | | | |
| | | | | |
| | | | | | |
Change-Id: I363ad0df632fdb25236b3d0a0c06fb15dbf8acf2
|
| | | | |/ /
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
* Update plugins/replication from branch 'stable-2.16'
to 4cb59f096b84f4369f62c8645db326c61826be79
- Refactor Replication*IT tests to share a base class
These classes have very similar setups and duplicate helper methods.
Improve maintainability by reducing the duplication.
ReplicationQueueIT is not modified because it is merged into
ReplicationIT on stable-3.0.
Change-Id: Ibc22ae4d0db2d09009f65c0e745f1095c67827ba
- ReplicationIT: Add shouldMatch* e2e tests
These new tests utilize creating a branch in a way that does not trigger
replication so that scheduleFullSync() is responsible for replicating
the update. In this way, the tests verify the destination receives the
update because scheduleFullSync() matched the given URI.
Change-Id: I4ae15d0301a308a12cbca3684915e89ca421e02f
- ReplicationStorageIT: Move shouldMatch* tests from ReplicationIT
These tests are focused on verifying storage, so they belong in
ReplicationStorageIT. Improve these tests to better verify storage
correctness by switching the 'now' parameter to false such that
replicationDelay is honored and follow the ReplicationStorageIT
pattern using a very long delay. These improvements make these tests
much more stable.
The tests improve the ref matching slightly by comparing to the
PushOne.ALL_REFS constant.
Also removes the disableDeleteForTesting flag as there are no users of
it now.
A later change can add ReplicationIT e2e tests for these use cases.
Change-Id: Iaa14a7429a40fb62325259efa1c7d7637deef95a
- ReplicationStorageIT: Add shouldFire*ChangeRefs tests
Copy the shouldFire*IncompleteUri tests as shouldFire*ChangeRefs to
fill a gap in test coverage.
Change-Id: Ia8df64a8574b776e6a9f7201c0862f1e6794687e
- Move storage-based ITs into ReplicationStorageIT
Tests in ReplicationStorageIT utilize very long replication delays such
that tasks are never expected to complete during the test. This allows
test writers to assume the task files are still there.
Refactor tests from ReplicationIT into ReplicationStorageIT and focus
them on verifying storage correctness. This is mostly a direct copy
except that shouldFirePendingOnlyToStoredUri gets renamed and split into
two tests. One that validates tasks are fired and another that validates
replication completes to the expected destinations. This split is
necessary because of the very long delay methodology mentioned above.
Code sharing between ReplicationIT and ReplicationStorageIT will be
improved in a later commit.
Change-Id: I41179c20a10354953cff3628368dfd5f910cc940
|
| | | | |/
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
* Update plugins/replication from branch 'stable-2.16'
to 64617a846c9fa06215031b2ad34a30d58003a732
- ReplicationQueue: Remove unused method
And drop the misleading @VisibleForTesting annotation from the method
the removed method was wrapping. scheduleFullSync() is public so that
PushAll can call it.
Change-Id: I0139e653654fcaf20de68dddfb5ea85560a323d0
|
| | | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
* Update plugins/replication from branch 'stable-3.1'
to 2be86dab82510bcda4740b9a003e9f64ad43033b
- Merge changes I0ef708ab,I81d27fd4 into stable-3.1
* changes:
Move shouldCleanupTasksAfterNewProjectReplication test
Merge branch 'stable-3.0' into stable-3.1
- Move shouldCleanupTasksAfterNewProjectReplication test
This test is focused on the storage level, so move it to
ReplicationStorageIT. Slightly improve it to use the new best practices
for specifying test timeouts.
Change-Id: I0ef708ab7813ee09d6f115d3151d2d12b9984a80
- Merge branch 'stable-3.0' into stable-3.1
* stable-3.0:
Move storage portion of replicateBranchDeletion ITs
Refactor Replication*IT tests to share a base class
ReplicationIT: Add shouldMatch* e2e tests
ReplicationStorageIT: Move shouldMatch* tests from ReplicationIT
ReplicationStorageIT: Add shouldFire*ChangeRefs tests
Move storage-based ITs into ReplicationStorageIT
ReplicationQueue: Remove unused method
This change does not try to reimpose the breakdown of tests that was
done in 3.0. That will be done in follow up change(s) to improve
reviewability of this change.
Change-Id: I81d27fd47da8eecad3aca36d8e6400679fb564a3
- Move storage portion of replicateBranchDeletion ITs
All other ITs split e2e and storage tests on stable-2.16, so this change
only updates the new replicateBranchDeletion tests that were added in
stable-3.0.
The e2e check for if the destination branch is removed or not stays in
ReplicationIT and the check that a task is created in storage when the
branch delete API is invoked moves to ReplicationStorageIT.
This split allows the best practices for verifying e2e and storage to be
applied independently.
Change-Id: Iec7ee090bd614e3442b1f9cb454437c9e05290be
- Merge branch 'stable-2.16' into stable-3.0
* stable-2.16:
Refactor Replication*IT tests to share a base class
ReplicationIT: Add shouldMatch* e2e tests
ReplicationStorageIT: Move shouldMatch* tests from ReplicationIT
ReplicationStorageIT: Add shouldFire*ChangeRefs tests
Move storage-based ITs into ReplicationStorageIT
ReplicationQueue: Remove unused method
This change does not try to reimpose the breakdown of tests that was
done in 2.16. That will be done in follow up change(s) to improve
reviewability of this change.
Change-Id: I83202997610c5ad0d8849cb477ca36db8df760f5
- Refactor Replication*IT tests to share a base class
These classes have very similar setups and duplicate helper methods.
Improve maintainability by reducing the duplication.
ReplicationQueueIT is not modified because it is merged into
ReplicationIT on stable-3.0.
Change-Id: Ibc22ae4d0db2d09009f65c0e745f1095c67827ba
- ReplicationIT: Add shouldMatch* e2e tests
These new tests utilize creating a branch in a way that does not trigger
replication so that scheduleFullSync() is responsible for replicating
the update. In this way, the tests verify the destination receives the
update because scheduleFullSync() matched the given URI.
Change-Id: I4ae15d0301a308a12cbca3684915e89ca421e02f
- ReplicationStorageIT: Move shouldMatch* tests from ReplicationIT
These tests are focused on verifying storage, so they belong in
ReplicationStorageIT. Improve these tests to better verify storage
correctness by switching the 'now' parameter to false such that
replicationDelay is honored and follow the ReplicationStorageIT
pattern using a very long delay. These improvements make these tests
much more stable.
The tests improve the ref matching slightly by comparing to the
PushOne.ALL_REFS constant.
Also removes the disableDeleteForTesting flag as there are no users of
it now.
A later change can add ReplicationIT e2e tests for these use cases.
Change-Id: Iaa14a7429a40fb62325259efa1c7d7637deef95a
- ReplicationStorageIT: Add shouldFire*ChangeRefs tests
Copy the shouldFire*IncompleteUri tests as shouldFire*ChangeRefs to
fill a gap in test coverage.
Change-Id: Ia8df64a8574b776e6a9f7201c0862f1e6794687e
- Move storage-based ITs into ReplicationStorageIT
Tests in ReplicationStorageIT utilize very long replication delays such
that tasks are never expected to complete during the test. This allows
test writers to assume the task files are still there.
Refactor tests from ReplicationIT into ReplicationStorageIT and focus
them on verifying storage correctness. This is mostly a direct copy
except that shouldFirePendingOnlyToStoredUri gets renamed and split into
two tests. One that validates tasks are fired and another that validates
replication completes to the expected destinations. This split is
necessary because of the very long delay methodology mentioned above.
Code sharing between ReplicationIT and ReplicationStorageIT will be
improved in a later commit.
Change-Id: I41179c20a10354953cff3628368dfd5f910cc940
- ReplicationQueue: Remove unused method
And drop the misleading @VisibleForTesting annotation from the method
the removed method was wrapping. scheduleFullSync() is public so that
PushAll can call it.
Change-Id: I0139e653654fcaf20de68dddfb5ea85560a323d0
|
| |/ / /
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
* Update plugins/replication from branch 'stable-3.1'
to 61321228045db5c612f321896c9bd07569c44820
- Merge "ReplicationTasksStorage: Add multi-primary unit tests" into stable-3.1
- ReplicationTasksStorage: Add multi-primary unit tests
These tests examine the replication scenarios under
multi-primary setup making use of the api calls present
in ReplicationTasksStorage class similarly as done in
single primary setup.
These tests ensure that the replication compatability in
multi-primary setup is not broken.
Change-Id: Ib2d0017c4d2ac3f4cfc7262b68b09a3a357e1337
|
| |\| |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
* stable-3.0:
Update git submodules
Update git submodules
Update git submodules
ReceiveCommits: nit - closing bracket in usage String
Update git submodules
MigrateToNoteDb: add option to force state change with skipped project
setNoteDbPrimary: update in chunks, one DB connection per chunk
GC repositories regularly during noteDb migration
NoteDbMigrator: set thread name to show project slice being processed
NoteDbMigrator: shuffle slices to reduce concurrency on same repo
NoteDbMigrator: Fix contention on single database connection
MigrateToNoteDb: implement option --verbose
NoteDbMigrator: log total progress of the migration every 1000 changes
Use multiple threads per project to migrate changes to noteDb
Update Jetty to 9.4.32.v20200930
Change-Id: If70aba6d3107b30978f4a9a4e294c38f99297f26
|
| | |\|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
* stable-2.16:
Update git submodules
Update git submodules
Update git submodules
MigrateToNoteDb: add option to force state change with skipped project
setNoteDbPrimary: update in chunks, one DB connection per chunk
GC repositories regularly during noteDb migration
NoteDbMigrator: set thread name to show project slice being processed
NoteDbMigrator: shuffle slices to reduce concurrency on same repo
NoteDbMigrator: Fix contention on single database connection
MigrateToNoteDb: implement option --verbose
NoteDbMigrator: log total progress of the migration every 1000 changes
Use multiple threads per project to migrate changes to noteDb
Change-Id: I4dfe8268c3bc362835fa92b71ab7e1f18e16308b
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
* Update plugins/replication from branch 'stable-2.16'
to 84d96eb953d51c97b2093d06597bc69812b812e7
- ReplicationIT: Remove unnecessary storage inspection
Integration tests shouldn't need to rely on inspecting the underlying
ReplicationTasksStorage layer(s). All of these tests already verify the
expected end result.
This leaves 4 tests that currently completely rely on inspecting the
task storage to verify the expected result. Those tests need further
improvement to decouple from the storage layer.
Change-Id: I029d63ce7d07414d9bf5d9290d556378beedcabf
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
* Update plugins/replication from branch 'stable-2.16'
to 5529649274286edbb7559a3af13724cdcb90f1c3
- ReplicationIT: Fix invalid replicationDelay setting
Setting config values for a remote in replication.config vs the remote's
own config file results in the replication.config values being ignored.
Fix this by setting the values in each remote's config file.
This test had delays added to avoid any flakiness, but the delays
weren't working because of this issue. While the test generally passes,
the delay makes it safer from races.
Change-Id: Idcdf5f07b3fc91724068ec6216527665c4a48bb3
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
* Update plugins/replication from branch 'stable-2.16'
to 882c6147720227c161a2fb573c79cfc683e70379
- Split replication plugins tests in two groups
Run unit-tests and integration tests in parallel by splitting
them into two separate tasks.
This also allows to potentially identify which group of tests
is flaky, because Bazel would flag one or the other in case of
instability.
Change-Id: I21f969a17e3653dfc5ab93d71cc6955024fc2d8f
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
In one of our sites we have a giant repository. When trying to migrate
the complete site using offline migration including this giant repo
the migration crawled at few change/sec since it seems to allocate so
much memory that the migration of the other 25k repositories can’t
really make progress when running the migration with 1 thread/core.
This also caused the JVM GC ratio to increase heavily (>60%). Reducing
the number of threads to 16 reduced the gc ratio to 15-20% but still
migration speed reached only 20 changes/sec.
Hence we migrate this giant repo on a staging copy of the site which
takes around 11 hours when only migrating the huge repository
using 16 threads. Then the meta refs from the giant repo are transferred
from the staging site via git fetch. This is possible since
the repository is read-only so we can be sure there are no new changes
on the production server since we migrated it on the staging server.
With this patch series we can migrate the other 25k repos / 4.5m changes
in a bit more than 1 hour (1200 changes/sec) if we skip the giant
repository in this migration run.
This change adds an option to the offline migration which allows to
finish the migration of the other repositories despite we set the option
to skip the giant repository which wasn't supported before. We must skip
it even when all the migration work was already done on the staging site
and the result transferred to the site where the migration of all the
other repositories is done since otherwise we are back to a very slow
20 changes/second.
An alternative approach could be to migrate this huge repository slowly
using online migration but this would have the disadvantage that this
would take much longer and affect the performance of the productive
server until this migration finished.
Change-Id: Ib78d257ce19bf8370ae0c259d887c600e7195dab
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Before this change we used one connection to migrate one change. We
observed an excessive number of DB connections using netstat during
the execution of the setNoteDbPrimary method. The number of connections
reached 27K and then opening a new connection started to fail and the
migration started to fail. I assume that this is caused by the
exhaustion of the local port range: we open/close connections in quick
succession and the operating system doesn't have enough time to release
local ports.
By updating a chunk of changes from a single thread, we make sure to use
only one DB connection for one chunk. This should reduce the rate at
which DB connections are open/closed and the overall number of
connections open during the migration.
Change-Id: Ie4a1b4d41b92824c87a0ae39b13a13d9ccb4ca3c
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
In order to ensure good performance run auto gc during the notedb
migration every new 10000 refs which are created during the migration of
a project. Auto gc will do garbage collection by default if it finds
more than 6700 loose objects or more than 50 pack files. This can be
changed by setting options gc.auto (default 6700) and gc.autoPackLimit
(default 50) [1].
[1] https://git-scm.com/docs/git-config#Documentation/git-config.txt-gcauto
Change-Id: If56219a1d256d6f1c84e6788f46668b481ff4718
|
| | | |
| | |
| | |
| | | |
Change-Id: If1d5c6a68b2dc15ee45ae9b61b8719d511663565
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Migrating many slices of the same repository concurrently increases
pressure on the thread trying to pack refs of the repository. Hence
shuffle all slices of all projects to decrease concurrency per
repository.
When testing migration of a large project (370k changes, >1m refs) using
80 threads I observed that there were always 20-30k loose refs despite
the fact that refs were constantly packed which took 1-3 minutes for
each repacking.
Change-Id: I39e1d99995d7e543cba8eedcd921706ca1655b5c
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
The migrator used a single database connection for all threads
rebuilding changes. Instead use one database connection per rebuild
thread.
Change-Id: If785208cc571421b0a2bac65b4970c24a4c33e1f
|
