-
Notifications
You must be signed in to change notification settings - Fork 2.2k
Remove custom sparse-checkouts for built-in checkout task #50402
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
1f30844
Begin to remove custom sparse checkout from standard check out of repo
alzimmermsft 4d450f7
Minor cleanups, remove tag fetching where it isn't necessary
alzimmermsft 844d963
Make checkout quiet
alzimmermsft 322c583
Attempt to fix tests
alzimmermsft 3140eea
Merge branch 'main' into ReplaceCustomSparseCheckout
alzimmermsft fc5ce88
Merge branch 'main' into ReplaceCustomSparseCheckout
alzimmermsft ede9e3d
Minor tyding
alzimmermsft File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,104 +1,103 @@ | ||
| <# | ||
| .SYNOPSIS | ||
| Invokes sparse checkout on the specified repositories. | ||
| Prepares or restores working-tree changes around a native sparse checkout. | ||
|
|
||
| .DESCRIPTION | ||
| Invokes sparse checkout on the specified repositories. | ||
| Combines the initial sparse checkout patterns with paths computed by Java's dependency discovery. | ||
| Tracked-file changes are saved outside the repository and restored after the Azure Pipelines checkout | ||
| step resets them. The checkout must use clean: false to preserve generated untracked files. | ||
|
|
||
| This script is special to Java as it uses layered sparse checkout to reduce the amount of code to checkout. | ||
| The first run of sparse checkout is inlined into YAML as there is a chicken and egg problem where the script | ||
| to perform sparse checkout won't be available until after the checkout step has completed. | ||
| .PARAMETER PathsJson | ||
| JSON representation of the additional paths to checkout. | ||
|
|
||
| This script is used to reduce the size of YAML files as this is only called when the initial checkout has | ||
| already been completed. | ||
| .PARAMETER ChangesPath | ||
| Absolute path outside the repository for the patch containing tracked-file changes. | ||
|
|
||
| .PARAMETER PathsJson | ||
| JSON representation of the paths to checkout. | ||
| .PARAMETER SourceVersion | ||
| The original source revision, which must still be checked out before restoring changes. | ||
|
|
||
| .PARAMETER RepositoriesJson | ||
| JSON representation of the repositories to checkout from. | ||
| .PARAMETER Restore | ||
| Restore the saved changes after the native checkout has completed. | ||
| #> | ||
|
|
||
| [CmdletBinding(DefaultParameterSetName = 'Prepare')] | ||
| param( | ||
| [Parameter(Mandatory = $true)] | ||
| [string]$PathsJson, | ||
| [Parameter(Mandatory = $true, ParameterSetName = 'Prepare')] | ||
| [string]$PathsJson, | ||
|
|
||
| [Parameter(Mandatory = $true)] | ||
| [string]$ChangesPath, | ||
|
|
||
| [Parameter(Mandatory = $true)] | ||
| [string]$RepositoriesJson | ||
| [Parameter(Mandatory = $true, ParameterSetName = 'Restore')] | ||
| [string]$SourceVersion, | ||
|
|
||
| [Parameter(Mandatory = $true, ParameterSetName = 'Restore')] | ||
| [switch]$Restore | ||
| ) | ||
|
|
||
| # Setting $PSNativeCommandArgumentPassing to 'Legacy' to use PowerShell | ||
| # 7.2 behavior for command argument passing. Newer behaviors will result | ||
| # in errors from git.exe. | ||
| $PSNativeCommandArgumentPassing = 'Legacy' | ||
| $ErrorActionPreference = 'Stop' | ||
|
|
||
| function SparseCheckout([Array]$paths, [Hashtable]$repository) | ||
| { | ||
| $dir = $repository.WorkingDirectory | ||
| if (!$dir) { | ||
| $dir = "./$($repository.Name)" | ||
| if ($Restore) { | ||
| $currentVersion = git rev-parse HEAD | ||
| if ($LASTEXITCODE -ne 0 -or $currentVersion -ne $SourceVersion) { | ||
| throw "The native checkout changed the source revision. Patch retained at $ChangesPath." | ||
| } | ||
| New-Item $dir -ItemType Directory -Force | Out-Null | ||
| Push-Location $dir | ||
|
|
||
| if (Test-Path .git/info/sparse-checkout) { | ||
| $hasInitialized = $true | ||
| Write-Host "Repository $($repository.Name) has already been initialized. Skipping this step." | ||
| } else { | ||
| Write-Host "Repository $($repository.Name) is being initialized." | ||
|
|
||
| if ($repository.Commitish -match '^refs/pull/\d+/merge$') { | ||
| Write-Host "git clone --no-checkout --filter=tree:0 -c remote.origin.fetch='+$($repository.Commitish):refs/remotes/origin/$($repository.Commitish)' https://github.com/$($repository.Name) ." | ||
| git clone --no-checkout --filter=tree:0 -c remote.origin.fetch=''+$($repository.Commitish):refs/remotes/origin/$($repository.Commitish)'' https://github.com/$($repository.Name) . | ||
| } else { | ||
| Write-Host "git clone --no-checkout --filter=tree:0 https://github.com/$($repository.Name) ." | ||
| git clone --no-checkout --filter=tree:0 https://github.com/$($repository.Name) . | ||
| if ((Get-Item -LiteralPath $ChangesPath).Length -gt 0) { | ||
| git apply --whitespace=nowarn -- $ChangesPath | ||
| if ($LASTEXITCODE -ne 0) { | ||
| throw "Restoring checkout changes failed with exit code $LASTEXITCODE. Patch retained at $ChangesPath." | ||
| } | ||
|
|
||
| # Turn off git GC for sparse checkout. Note: The devops checkout task does this by default | ||
| Write-Host "git config gc.auto 0" | ||
| git config gc.auto 0 | ||
|
|
||
| Write-Host "git sparse-checkout init" | ||
| git sparse-checkout init | ||
|
|
||
| # Set non-cone mode otherwise path filters will not work in git >= 2.37.0 | ||
| # See https://github.blog/2022-06-27-highlights-from-git-2-37/#tidbits | ||
| Write-Host "git sparse-checkout set --no-cone '/*' '!/*/' '/eng'" | ||
| git sparse-checkout set --no-cone '/*' '!/*/' '/eng' | ||
| } | ||
| Remove-Item -LiteralPath $ChangesPath | ||
| return | ||
| } | ||
|
|
||
| # Prevent wildcard expansion in Invoke-Expression (e.g. for checkout path '/*') | ||
| $quotedPaths = $paths | ForEach-Object { "'$_'" } | ||
| $gitsparsecmd = "git sparse-checkout add $quotedPaths" | ||
| Write-Host $gitsparsecmd | ||
| Invoke-Expression -Command $gitsparsecmd | ||
| Write-Output '##vso[task.setvariable variable=SparseCheckoutRequired]false' | ||
|
|
||
| Write-Host "Set sparse checkout paths to:" | ||
| Get-Content .git/info/sparse-checkout | ||
| # Paths may be sourced as a yaml object literal OR a dynamically generated variable json string. | ||
| # If the latter, convertToJson will wrap the 'string' in quotes, so remove them. | ||
| $paths = $PathsJson.Trim('"') | ConvertFrom-Json | ||
| if (@($paths).Count -eq 0) { | ||
| return | ||
| } | ||
|
|
||
| # sparse-checkout commands after initial checkout will auto-checkout again | ||
| if (!$hasInitialized) { | ||
| # Remove refs/heads/ prefix from branch names | ||
| $commitish = $repository.Commitish -replace '^refs/heads/', '' | ||
| $isWorkingTree = git rev-parse --is-inside-work-tree | ||
| if ($LASTEXITCODE -ne 0 -or $isWorkingTree -ne 'true') { | ||
| throw 'The repository is not an initialized Git working tree.' | ||
| } | ||
|
|
||
| # use -- to prevent git from interpreting the commitish as a path | ||
| Write-Host "git -c advice.detachedHead=false checkout $commitish --" | ||
| $isSparseCheckout = git config --type=bool --default=false --get core.sparseCheckout | ||
| if ($LASTEXITCODE -ne 0) { | ||
| throw 'Unable to determine sparse checkout mode.' | ||
| } | ||
| if ($isSparseCheckout -ne 'true') { | ||
| Write-Information 'The repository has a full checkout. Skipping expansion.' -InformationAction Continue | ||
| return | ||
| } | ||
|
|
||
| # This will use the default branch if repo.Commitish is empty | ||
| git -c advice.detachedHead=false checkout $commitish -- | ||
| } else { | ||
| Write-Host "Skipping checkout as repo has already been initialized" | ||
| $patternsPath = git rev-parse --git-path info/sparse-checkout | ||
| if ($LASTEXITCODE -ne 0) { | ||
| throw 'Unable to locate the initial sparse checkout patterns.' | ||
| } | ||
| $patterns = @(Get-Content -LiteralPath $patternsPath) + @($paths) | ||
| $quotedPatterns = foreach ($pattern in $patterns) { | ||
| if ($pattern -match "[`r`n]") { | ||
| throw 'Sparse checkout patterns cannot contain line breaks.' | ||
| } | ||
|
|
||
| Pop-Location | ||
| '"' + ($pattern -replace '(\\*)"', '$1$1\"' -replace '(\\+)$', '$1$1') + '"' | ||
| } | ||
|
|
||
| # Paths may be sourced as a yaml object literal OR a dynamically generated variable json string. | ||
| # If the latter, convertToJson will wrap the 'string' in quotes, so remove them. | ||
| $paths = $PathsJson.Trim('"') | ConvertFrom-Json | ||
| # Replace windows backslash paths, as Azure Pipelines default directories are sometimes formatted like 'D:\a\1\s' | ||
| $repositories = $RepositoriesJson -replace '\\', '/' | ConvertFrom-Json -AsHashtable | ||
| foreach ($repo in $repositories) { | ||
| SparseCheckout $paths $repo | ||
| New-Item -ItemType Directory -Path (Split-Path -Parent $ChangesPath) -Force | Out-Null | ||
| $SourceVersion = git rev-parse HEAD | ||
| if ($LASTEXITCODE -ne 0) { | ||
| throw 'Unable to determine the original source revision.' | ||
| } | ||
| git diff --binary --no-ext-diff --no-textconv --src-prefix=a/ --dst-prefix=b/ --output=$ChangesPath $SourceVersion -- | ||
| if ($LASTEXITCODE -ne 0) { | ||
| throw "Saving checkout changes failed with exit code $LASTEXITCODE." | ||
| } | ||
|
|
||
| $patternsValue = ($quotedPatterns -join ' ').Replace('%', '%AZP25') | ||
| Write-Output "##vso[task.setvariable variable=SparseCheckoutPatterns]$patternsValue" | ||
|
alzimmermsft marked this conversation as resolved.
|
||
| Write-Output "##vso[task.setvariable variable=SparseCheckoutSourceVersion]$SourceVersion" | ||
| Write-Output '##vso[task.setvariable variable=SparseCheckoutRequired]true' | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.