Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cpack: Update build-id and create debug RPMs #5936

Merged
merged 1 commit into from
Oct 31, 2019

Conversation

theopolis
Copy link
Member

This commit adds -debuginfo package builds for the RPM generator. This also updates the --build-id linker flag to use SHA1 build IDs.

@theopolis
Copy link
Member Author

@Smjert thoughts on how to fix

[100%] Built target tests_integration_tables-test
Run CPack packaging tool...
CPack: Create package using RPM
CPack: Install projects
CPack: - Run preinstall target for: osquery
CPack: - Install project: osquery
CPack: Create package
CMake Warning (dev) at /usr/local/share/cmake-3.14/Modules/Internal/CPack/CPackRPM.cmake:135 (message):
  CPackRPM:Warning: Path /etc/init.d/osqueryd is not on one of the
  relocatable paths! Package will be partially relocatable.
Call Stack (most recent call first):
  /usr/local/share/cmake-3.14/Modules/Internal/CPack/CPackRPM.cmake:1001 (cpack_rpm_prepare_relocation_paths)
  /usr/local/share/cmake-3.14/Modules/Internal/CPack/CPackRPM.cmake:1870 (cpack_rpm_generate_package)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at /usr/local/share/cmake-3.14/Modules/Internal/CPack/CPackRPM.cmake:135 (message):
  CPackRPM:Warning: Path /etc/sysconfig/osqueryd is not on one of the
  relocatable paths! Package will be partially relocatable.
Call Stack (most recent call first):
  /usr/local/share/cmake-3.14/Modules/Internal/CPack/CPackRPM.cmake:1001 (cpack_rpm_prepare_relocation_paths)
  /usr/local/share/cmake-3.14/Modules/Internal/CPack/CPackRPM.cmake:1870 (cpack_rpm_generate_package)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- CPackRPM:Debug: Using CPACK_RPM_ROOTDIR=/__w/1/b/build/_CPack_Packages/Linux/RPM
CMake Error at /usr/local/share/cmake-3.14/Modules/Internal/CPack/CPackRPM.cmake:611 (message):
  CPackRPM: source dir path '/__w/1/s' is shorter than debuginfo sources dir
  path '/usr/src/debug/osquery/src_0'! Source dir path must be longer than
  debuginfo sources dir path.  Set CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX
  variable to a shorter value or make source dir path longer.  Required for
  debuginfo packaging.  See documentation of CPACK_RPM_DEBUGINFO_PACKAGE
  variable for details.
Call Stack (most recent call first):
  /usr/local/share/cmake-3.14/Modules/Internal/CPack/CPackRPM.cmake:1376 (cpack_rpm_debugsymbol_check)
  /usr/local/share/cmake-3.14/Modules/Internal/CPack/CPackRPM.cmake:1870 (cpack_rpm_generate_package)

@theopolis
Copy link
Member Author

We may have to set a long directory path for the builds, similar to https://github.com/apple/foundationdb/blob/master/build/cmake/docker-compose.yml#L20

@theopolis
Copy link
Member Author

Dang, I need to apply the padding for the source (of course!). This might be a little more challenging.

@Smjert
Copy link
Member

Smjert commented Oct 27, 2019

Looking at what CPackRPM.cmake does, it uses the debugedit tool to modify the executable to substitute the wanted source dir path with the original.
Since the tool cannot make the executable bigger and has to avoid overwriting other fields, the original source dir path has to be longer and this is also why we can't modify that after the fact; we need to build from a long source directory from the start.
Therefore, in a step after the one that creates the build directory, I suggest to do:

mkdir $(Build.BinariesDirectory)/really/long/path/needed/by/rpm/debuginfo/osquery
ln -s $(Build.SourcesDirectory) $(Build.BinariesDirectory)/really/long/path/needed/by/rpm/debuginfo/osquery

Then in the configure step change $(Build.SourcesDirectory) to $(Build.BinariesDirectory)/really/long/path/needed/by/rpm/debuginfo/osquery/s

And so on for all the steps that mention $(Build.SourcesDirectory).

Moreover, I see that it will require a tool, eu-strip which is not currently installed in the Docker image, so the build will fail. I'll open a PR with the change.

EDIT: I've edited the base path where the long source dir is created because the Agent might not be able to create it there if it's using a container.

@theopolis
Copy link
Member Author

Ok, I've reset the PR to remove the changes to Azure.

@Smjert
Copy link
Member

Smjert commented Oct 29, 2019

Sorry, I think I recalled the issue wrong; it's the submodule that cannot find its correct path.
So I think the issue is caused by the fact that the /tmp/usr/blah directory is inside the folder where the repository is cloned (which is $(Build.SourcesDirectory)).

EDIT: Just to clarify, what the CI is doing here is

git clone git@github.com:osquery/osquery $(Build.SourcesDirectory)

Not

cd $(Build.SourcesDirectory)
git clone git@github.com:osquery/osquery

Which would create the osquery folder under $(Build.SourcesDirectory)

I suggest creating that path and the link starting from the $(Build.BinariesDirectory).
Or we could just change the path where the first clone happens: https://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema#checkout

@theopolis
Copy link
Member Author

I'm checking to see if the checkout approach works. If this is OK then I will update the yml with a bit more buffer to the pathname (and include a comment).

@Smjert
Copy link
Member

Smjert commented Oct 30, 2019

I'm checking to see if the checkout approach works. If this is OK then I will update the yml with a bit more buffer to the pathname (and include a comment).

Seems to work; It indeed needs more padding

@Smjert
Copy link
Member

Smjert commented Oct 30, 2019

Interesting... beyond the openssl macOS failure, it seems that it's also trying to include sources from the build folder.. which it kind of make sense because we have links and some generated files there.
So, we have to increase the length of that path too.

@Smjert
Copy link
Member

Smjert commented Oct 30, 2019

Also, since this now depends on the presence of elfutils package, we might want to add that package on the building doc.

@theopolis
Copy link
Member Author

I'm skipping the documentation about package building dependencies for now since they are required by CPack. I am nearly 100% I know what dependencies are needed, as I can read the CPack generator code but that seems a overkill for the information we provide.

@theopolis theopolis merged commit 9a33687 into osquery:master Oct 31, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants