Improve vulnerability scanner local component test execution #23517

pereyra-m · 2024-05-18T23:27:15Z

Related issue
Closes #23472

Description

This PR improves the current tests in many ways:

It allows the use of a pre-decompressed content. Skips the decompression test if the feed folder is found
It fixes a bug that left the DBs blocked for the following tests if one of them fails
Creates a helpers.py file to include all common methods
Creates a conftest.py file to include all common fixtures
Adds documentation to run the tests locally
Documents the methods and fixtures
Adds the folder argument to run only a single test
Adds a header for .py files
Creates specific configuration files for the QA tests to avoid using the same files than the test tool
Fixes some Pylint warnings
Improves the timeout logic to decrease the total run time in case a log is missing
General logs improvement to make them more clear and useful

The commit from #23457 is taken.

Tests

Manual tests:

Running a single test skips the rest

root@fb659c773250:/workspaces/wazuh/src# GITHUB_WORKSPACE=/workspaces/wazuh/src/ python3 -m pytest  -vv wazuh_modules/vulnerabiliE
y_scanner/qa/test_efficacy_log.py --log-cli-level=DEBUG --folder=001
====================================================== test session starts =======================================================
platform linux -- Python 3.10.12, pytest-7.2.2, pluggy-1.5.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /workspaces/wazuh/src
collected 14 items                                                                                                               

wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log0] SKIPPED (Skipping test 000) [  7%]
wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log1] 
--------------------------------------------------------- live log setup ---------------------------------------------------------
INFO     test_efficacy_log:test_efficacy_log.py:171 The decompressed content will be used
DEBUG    test_efficacy_log:test_efficacy_log.py:203 Running test wazuh_modules/vulnerability_scanner/qa/test_data/001
DEBUG    test_efficacy_log:test_efficacy_log.py:226 Waiting for the process to be initialized
DEBUG    test_efficacy_log:helpers.py:55 Found '1' matches of a total of '1' expected of regex 'Vulnerability scanner module started'.
INFO     test_efficacy_log:test_efficacy_log.py:229 Process initialized
DEBUG    test_efficacy_log:test_efficacy_log.py:240 Expected lines: ["Match found, the package 'systemd', is vulnerable to 'CVE-2013-4392'. Current version: '247.3-7+deb11u4' (less than '' or equal to '247.3-7+deb11u4'). - Agent '' (ID: '001', Version: '').", "Match found, the package 'systemd', is vulnerable to 'CVE-2020-13529'. Current version: '247.3-7+deb11u4' (less than '' or equal to '247.3-7+deb11u4'). - Agent '' (ID: '001', Version: '').", "Match found, the package 'systemd', is vulnerable to 'CVE-2023-31437'. Current version: '247.3-7+deb11u4' (less than '' or equal to '247.3-7+deb11u4'). - Agent '' (ID: '001', Version: '').", "Match found, the package 'systemd', is vulnerable to 'CVE-2023-31438'. Current version: '247.3-7+deb11u4' (less than '' or equal to '247.3-7+deb11u4'). - Agent '' (ID: '001', Version: '').", "Match found, the package 'systemd', is vulnerable to 'CVE-2023-31439'. Current version: '247.3-7+deb11u4' (less than '' or equal to '247.3-7+deb11u4'). - Agent '' (ID: '001', Version: '').", 'Deleting package agent key: 001_ec465b7eb5fa011a336e95614072e4c7f1a65a53']
DEBUG    test_efficacy_log:test_efficacy_log.py:242 Quantity expected lines: 6
DEBUG    test_efficacy_log:test_efficacy_log.py:114 Running test wazuh_modules/vulnerability_scanner/qa/test_data/001/input_001.json
DEBUG    test_efficacy_log:test_efficacy_log.py:97 Is deltas
DEBUG    test_efficacy_log:test_efficacy_log.py:135 Sending flatbuffer data
DEBUG    test_efficacy_log:test_efficacy_log.py:114 Running test wazuh_modules/vulnerability_scanner/qa/test_data/001/input_002.json
DEBUG    test_efficacy_log:test_efficacy_log.py:97 Is deltas
DEBUG    test_efficacy_log:test_efficacy_log.py:135 Sending flatbuffer data
DEBUG    test_efficacy_log:test_efficacy_log.py:114 Running test wazuh_modules/vulnerability_scanner/qa/test_data/001/input_003.json
DEBUG    test_efficacy_log:test_efficacy_log.py:97 Is deltas
DEBUG    test_efficacy_log:test_efficacy_log.py:135 Sending flatbuffer data
DEBUG    test_efficacy_log:helpers.py:55 Found '1' matches of a total of '3' expected of regex 'Event type: (.*) processed'.
DEBUG    test_efficacy_log:helpers.py:58 Waiting for regex: 'Event type: (.*) processed'
DEBUG    test_efficacy_log:helpers.py:55 Found '3' matches of a total of '3' expected of regex 'Event type: (.*) processed'.
INFO     test_efficacy_log:test_efficacy_log.py:260 Scan finished, all events were processed
DEBUG    test_efficacy_log:test_efficacy_log.py:265 Waiting for log line: Match found, the package 'systemd', is vulnerable to 'CVE-2013-4392'. Current version: '247.3-7+deb11u4' (less than '' or equal to '247.3-7+deb11u4'). - Agent '' (ID: '001', Version: '').
INFO     test_efficacy_log:helpers.py:28 Found log line: wazuh-modulesd:vulnerability-scanner:packageScanner.hpp:471 versionMatch : Match found, the package 'systemd', is vulnerable to 'CVE-2013-4392'. Current version: '247.3-7+deb11u4' (less than '' or equal to '247.3-7+deb11u4'). - Agent '' (ID: '001', Version: '').

INFO     test_efficacy_log:helpers.py:28 Found log line: wazuh-modulesd:vulnerability-scanner:packageScanner.hpp:471 versionMatch : Match found, the package 'systemd', is vulnerable to 'CVE-2020-13529'. Current version: '247.3-7+deb11u4' (less than '' or equal to '247.3-7+deb11u4'). - Agent '' (ID: '001', Version: '').

INFO     test_efficacy_log:helpers.py:28 Found log line: wazuh-modulesd:vulnerability-scanner:packageScanner.hpp:471 versionMatch : Match found, the package 'systemd', is vulnerable to 'CVE-2023-31437'. Current version: '247.3-7+deb11u4' (less than '' or equal to '247.3-7+deb11u4'). - Agent '' (ID: '001', Version: '').

INFO     test_efficacy_log:helpers.py:28 Found log line: wazuh-modulesd:vulnerability-scanner:packageScanner.hpp:471 versionMatch : Match found, the package 'systemd', is vulnerable to 'CVE-2023-31438'. Current version: '247.3-7+deb11u4' (less than '' or equal to '247.3-7+deb11u4'). - Agent '' (ID: '001', Version: '').

INFO     test_efficacy_log:helpers.py:28 Found log line: wazuh-modulesd:vulnerability-scanner:packageScanner.hpp:471 versionMatch : Match found, the package 'systemd', is vulnerable to 'CVE-2023-31439'. Current version: '247.3-7+deb11u4' (less than '' or equal to '247.3-7+deb11u4'). - Agent '' (ID: '001', Version: '').

INFO     test_efficacy_log:helpers.py:28 Found log line: wazuh-modulesd:vulnerability-scanner:eventDeleteInventory.hpp:77 handleRequest : Deleting package agent key: 001_ec465b7eb5fa011a336e95614072e4c7f1a65a53

DEBUG    test_efficacy_log:test_efficacy_log.py:276 Waiting for the process to finish
PASSED                                                                                                                     [ 14%]
wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log2] SKIPPED (Skipping test 002) [ 21%]
wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log3] SKIPPED (Skipping test 003) [ 28%]
wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log4] SKIPPED (Skipping test 004) [ 35%]
wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log5] SKIPPED (Skipping test 005) [ 42%]
wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log6] SKIPPED (Skipping test 006) [ 50%]
wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log7] SKIPPED (Skipping test 007) [ 57%]
wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log8] SKIPPED (Skipping test 008) [ 64%]
wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log9] SKIPPED (Skipping test 009) [ 71%]
wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log10] SKIPPED (Skipping test 010) [ 78%]
wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log11] SKIPPED (Skipping test 011) [ 85%]
wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log12] SKIPPED (Skipping test 012) [ 92%]
wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log13] SKIPPED (Skipping test 013) [100%]

================================================= 1 passed, 13 skipped in 3.48s ==================================================

Run a single test without content fails

root@fb659c773250:/workspaces/wazuh/src# GITHUB_WORKSPACE=/workspaces/wazuh/src/ python3 -m pytest  -vv wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py --log-cli-level=DEBUG --folder=001
====================================================== test session starts =======================================================
platform linux -- Python 3.10.12, pytest-7.2.2, pluggy-1.5.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /workspaces/wazuh/src
collected 14 items                                                                                                               

wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log0] SKIPPED (Skipping test 000) [  7%]
wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log1] ERROR      [ 14%]
wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log2] SKIPPED (Skipping test 002) [ 21%]
wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log3] SKIPPED (Skipping test 003) [ 28%]
wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log4] SKIPPED (Skipping test 004) [ 35%]
wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log5] SKIPPED (Skipping test 005) [ 42%]
wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log6] SKIPPED (Skipping test 006) [ 50%]
wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log7] SKIPPED (Skipping test 007) [ 57%]
wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log8] SKIPPED (Skipping test 008) [ 64%]
wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log9] SKIPPED (Skipping test 009) [ 71%]
wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log10] SKIPPED (Skipping test 010) [ 78%]
wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log11] SKIPPED (Skipping test 011) [ 85%]
wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log12] SKIPPED (Skipping test 012) [ 92%]
wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log13] SKIPPED (Skipping test 013) [100%]

wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py:176: Failed
==================================================== short test summary info =====================================================
ERROR wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log1] - Failed: The test can't continue because there isn't a decompressed content in queue folder
================================================== 13 skipped, 1 error in 1.39s ==================================================

The decompression test is skipped if the content is found

root@fb659c773250:/workspaces/wazuh/src# GITHUB_WORKSPACE=/workspaces/wazuh/src/ python3 -m pytest  -vv wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py --log-cli-level=DEBUG --folder=000
====================================================== test session starts =======================================================
platform linux -- Python 3.10.12, pytest-7.2.2, pluggy-1.5.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /workspaces/wazuh/src
collected 14 items                                                                                                               

wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log0] SKIPPED (The decompression test is skipped because there is a compressed content in queue folder) [  7%]

The failure of a test doesn't impact others

------------------------------------------------------- Captured log call --------------------------------------------------------
ERROR    test_efficacy_log:test_efficacy_log.py:302 Log entry not found: Fake line
---------------------------------------------------- Captured stdout teardown ----------------------------------------------------
Copying /workspaces/wazuh/src/log.out to /workspaces/wazuh/src//qa_logs/log.out.test_false_negatives[run_process_and_monitor_log1]
==================================================== short test summary info =====================================================
FAILED wazuh_modules/vulnerability_scanner/qa/test_efficacy_log.py::test_false_negatives[run_process_and_monitor_log1] - AssertionError: The test failed because some expected lines were not found
============================================ 1 failed, 12 passed, 1 skipped in 59.19s ============================================

Review logs syntax and correct language

sebasfalcone · 2024-05-20T17:06:25Z

@pereyra-m Changes LGTM!

I've changed the PR target to 4.9.0, please do the corresponding rebase

- Documenting methods - Allowing a single test execution - Adding conftest and helpers files - Fixing Pylint warnings - Terminating the test tool process if an assert fails

sebasfalcone

Great work @pereyra-m!

Once 4.8.0 is merged into 4.9.0 we are going to apply this changes

pereyra-m requested a review from sebasfalcone May 18, 2024 23:27

pereyra-m self-assigned this May 18, 2024

pereyra-m linked an issue May 18, 2024 that may be closed by this pull request

Improve the new vulnerability scanner efficacy tests to make them more flexible #23472

Open

3 tasks

sebasfalcone marked this pull request as ready for review May 20, 2024 12:27

sebasfalcone changed the base branch from 4.8.0 to 4.9.0 May 20, 2024 17:03

pereyra-m added 2 commits May 20, 2024 19:57

Adding more details for the tests in case of failure

e08a178

Improving local test execution:

babbef3

- Documenting methods - Allowing a single test execution - Adding conftest and helpers files - Fixing Pylint warnings - Terminating the test tool process if an assert fails

pereyra-m force-pushed the enhancement/23472-improve-qa-tests branch from 3517496 to babbef3 Compare May 20, 2024 20:14

pereyra-m marked this pull request as draft May 20, 2024 20:16

sebasfalcone approved these changes May 21, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve vulnerability scanner local component test execution #23517

Improve vulnerability scanner local component test execution #23517

pereyra-m commented May 18, 2024 •

edited

sebasfalcone commented May 20, 2024

sebasfalcone left a comment

Improve vulnerability scanner local component test execution #23517

Are you sure you want to change the base?

Improve vulnerability scanner local component test execution #23517

Conversation

pereyra-m commented May 18, 2024 • edited

Description

Tests

sebasfalcone commented May 20, 2024

sebasfalcone left a comment

Choose a reason for hiding this comment

pereyra-m commented May 18, 2024 •

edited