CI: Make the built packages available to be downloaded #6772

Smjert · 2020-11-27T18:31:16Z

Also add TGZ as a supported package format on macOS,
since it's already used on the CI.

NOTE, to download them:

You need to open the CI page which points to a specific build; you can reach that for instance from a PR build results in the Checks tab or on the bottom of the Conversation tab where all the checks results are shown.
You want to follow the Details link of the osquery check, then View more details on Azure Pipelines.
On the Azure Pipelines website, you should see the summary of the PR CI build, with all the platforms. On the top right you want to find the following area:

Instead of a 0 artifacts there's should be written something like 6 published, click on it.
A list of packages grouped by platform and build type should be shown now. You can download all the packages for a specific platform build type, or you can expand the platform build and select which package you want to download.

These are not meant to be used in production, but could be useful to test some features or debug issues that the CI build highlights.
The artifacts will remain available as long as the build run is available on the CI, which is 30 days.

theopolis · 2020-11-29T13:31:37Z

This is great! Heads up that there are still slight differences in the package names that CPack creates, and the ones we upload to S3.

I think we should try to do the rename in CPack and failing that, rename them in the pipeline scripts here.

I'll find the details shortly.

azure-pipelines.yml

directionless · 2020-12-01T00:19:11Z

Love it.

Also add TGZ as a supported package format on macOS, since it's already used on the CI.

Smjert · 2020-12-02T10:14:15Z

Rebased against master to resolve conflicts.

theopolis

Just a heads up about the differences between how we store files in S3 and what our CPack output creates.

theopolis · 2020-12-05T02:02:36Z

azure-pipelines.yml

      displayName: "Run DEB packaging"
      workingDirectory: $(BUILD_DIR)

    - script: |
        cmake -DPACKAGING_SYSTEM=RPM $(Build.SourcesDirectory)
        cmake --build . --target package -j 3
+        cp *.rpm $(Build.ArtifactStagingDirectory)


After building packages I run:

mv osquery-$VERSION-1.$ARCH.rpm osquery-$VERSION-1.linux.$ARCH.rpm mv osquery-debuginfo-$VERSION-1.$ARCH.rpm osquery-debuginfo-$VERSION-1.linux.$ARCH.rpm

theopolis · 2020-12-05T02:04:27Z

azure-pipelines.yml

@@ -100,21 +100,31 @@ jobs:
    - script: |
        cmake -DPACKAGING_SYSTEM=DEB $(Build.SourcesDirectory)
        cmake --build . --target package -j 3
+        cp *.deb $(Build.ArtifactStagingDirectory)


After building packages I run:

mv osquery_$VERSION-1.linux_$ARCH.deb osquery_$VERSION_1.linux.$ARCH.deb

theopolis · 2020-12-05T02:04:55Z

azure-pipelines.yml

@@ -100,21 +100,31 @@ jobs:
    - script: |
        cmake -DPACKAGING_SYSTEM=DEB $(Build.SourcesDirectory)
        cmake --build . --target package -j 3
+        cp *.deb $(Build.ArtifactStagingDirectory)
+        cp *.ddeb $(Build.ArtifactStagingDirectory)


After building packages I run:

mv osquery-dbgsym_$VERSION-1.linux_$ARCH.ddeb osquery-dbgsym_$VERSION_1.linux.$ARCH.deb

Smjert · 2020-12-05T12:34:57Z

Just a heads up about the differences between how we store files in S3 and what our CPack output creates.

Curiosity, I see that is how we are still doing things, but is there a reason why we are not following the standard naming convention for each packaging format?
The PR #6468 intention was to adhere to the standard naming given by the underlying tools.
If we don't want to follow the standard naming convention, I think the best course is to revert that PR then, instead of having to do an additional move step (although with the debug Deb package I think it will still happen because it takes the name from a variable that's common with the standard package, so the extension is .deb and it replaces it with -dbgsym.ddeb, which is not correct).

theopolis · 2020-12-05T14:43:52Z

I think the reason we have the current naming scheme is most likely we did not know better in 2015 and set a bad precedent.

We could start following the standard. The downside is that some people's scripts might break. And the "ddeb" to "deb" rename is due to a limitation in freight (the tool used to create an apt repo).

Smjert · 2020-12-05T19:45:35Z

I think the reason we have the current naming scheme is most likely we did not know better in 2015 and set a bad precedent.

We could start following the standard. The downside is that some people's scripts might break. And the "ddeb" to "deb" rename is due to a limitation in freight (the tool used to create an apt repo).

I see, so I would suggest to revert #6468 first then.
I need to re-check but I think the ddeb issue was the only one, so reverting that PR would require you to change then name of only one package.

theopolis · 2020-12-05T20:54:27Z

I'll investigate.

I think we can land this change and then follow up in subsequent PRs to revert and fix up the names.

…0 to master * commit '2685844e7a4b1dbddd786f0ab3c1d26656deff28': (62 commits) CHANGELOG 4.6.0 (osquery#6809) Merge pull request from GHSA-4g56-2482-x7q8 cpack: Use CPACK_RPM_PACKAGE_RELEASE (osquery#6805) CMake: add max version limit to 3.18.0 on Linux (osquery#6801) libs: Update sqlite to version 3.34.0 (osquery#6804) Add indexed column support to Windows users table (osquery#6782) file_events: Add configuration flag (osquery#6663) CI: Make the built packages available to be downloaded (osquery#6772) Remove a Buck leftover (osquery#6799) CI: Update brew scripts (osquery#6794) CI: Python installation improvements on Windows (osquery#6764) BPF: Fix the tests (osquery#6783) libs: Update bzip2 to version 1.0.8 (osquery#6786) Update cmake_minum_required to 3.17.5 and pin version in CI (osquery#6770) CMake: Remove unused LTCG flag (osquery#6769) Remove boost workaround introduced in osquery#5591 for string_view (osquery#6771) libs: Update BPF libraries to support LLVM 11 (osquery#6775) BPF: Fix wrong d_type compare in fs classes (osquery#6774) startup_info: Fix memory leak in the dbus classes (osquery#6773) Change urls for submodules gpg-error, libgcrypt, libcap (osquery#6768) ...

Smjert added packaging CI/CD Anything about the Continuous Integration or Continuous Deployment tool used by the repository labels Nov 27, 2020

directionless reviewed Dec 1, 2020

View reviewed changes

azure-pipelines.yml Outdated Show resolved Hide resolved

CI: Make the built packages available to be downloaded

910083e

Also add TGZ as a supported package format on macOS, since it's already used on the CI.

Smjert force-pushed the stefano/ci/publish-packages-as-build-artifacts branch from 39d7af5 to 910083e Compare December 2, 2020 10:13

theopolis approved these changes Dec 4, 2020

View reviewed changes

theopolis reviewed Dec 5, 2020

View reviewed changes

theopolis merged commit 370908d into osquery:master Dec 12, 2020

theopolis mentioned this pull request Dec 14, 2020

cpack: Use CPACK_RPM_PACKAGE_RELEASE #6805

Merged

Smjert deleted the stefano/ci/publish-packages-as-build-artifacts branch December 21, 2020 18:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: Make the built packages available to be downloaded #6772

CI: Make the built packages available to be downloaded #6772

Smjert commented Nov 27, 2020 •

edited

theopolis commented Nov 29, 2020

directionless commented Dec 1, 2020

Smjert commented Dec 2, 2020

theopolis left a comment

theopolis Dec 5, 2020

theopolis Dec 5, 2020

theopolis Dec 5, 2020

Smjert commented Dec 5, 2020 •

edited

theopolis commented Dec 5, 2020

Smjert commented Dec 5, 2020 •

edited

theopolis commented Dec 5, 2020

CI: Make the built packages available to be downloaded #6772

CI: Make the built packages available to be downloaded #6772

Conversation

Smjert commented Nov 27, 2020 • edited

theopolis commented Nov 29, 2020

directionless commented Dec 1, 2020

Smjert commented Dec 2, 2020

theopolis left a comment

Choose a reason for hiding this comment

theopolis Dec 5, 2020

Choose a reason for hiding this comment

theopolis Dec 5, 2020

Choose a reason for hiding this comment

theopolis Dec 5, 2020

Choose a reason for hiding this comment

Smjert commented Dec 5, 2020 • edited

theopolis commented Dec 5, 2020

Smjert commented Dec 5, 2020 • edited

theopolis commented Dec 5, 2020

Smjert commented Nov 27, 2020 •

edited

Smjert commented Dec 5, 2020 •

edited

Smjert commented Dec 5, 2020 •

edited