Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make AWS kinesis status logging configurable #6135

Merged
merged 1 commit into from
Mar 16, 2020
Merged

Make AWS kinesis status logging configurable #6135

merged 1 commit into from
Mar 16, 2020

Conversation

nickcollier
Copy link
Contributor

@nickcollier nickcollier commented Dec 17, 2019 •
edited

The AWS kineis logger plugin is sent two types of log record: results logs which are produced by scheduled queries and status logs which are produced by glog. When the logger_min_status or verbose options are used the volume of status logs can cause undesirable bandwidth usage.

Logger plugins can opt out of being sent status logs by overriding the base LoggerPlugin usesLogStatus method. The AWS kinesis logger plugin currently opts in for status logs and this cannot be configured.

This PR adds a new configuration option aws_kinesis_disable_log_status that when set to true will prevent status logs being sent to kinesis. The default value of this option is false so when not specified the previous behaviour is retained. The options can be set either as a CLI flag or in the configuration.

@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Dec 17, 2019 •
edited

CLA Check
The committers are authorized under a signed CLA.

@nickcollier nickcollier force-pushed the make_kinesis_status_logging_configurable branch from e1bf667 to d213e15 Compare December 17, 2019 14:42
@nickcollier nickcollier force-pushed the make_kinesis_status_logging_configurable branch from d213e15 to 09159aa Compare December 17, 2019 15:53
@theopolis theopolis added AWS Related to Amazon Web Services tables logging labels Dec 19, 2019
@theopolis
Copy link
Member

Thanks for the PR @nickcollier, do you mind including a description of why this is useful. Is there a specific problem this is solving?

Do you mind including a test plan too? For example, steps I can use to reproduce testing enabling/disabling the status logging.

We should also look closer and understand if this can be toggled on/off via configuration updates. I cannot remember if this check is performed once or each time a status log occurs.

@nickcollier
Copy link
Contributor Author

@theopolis here are the steps I used to test this change - please let me know if you want any more details.

  1. Create a kinesis stream
aws configure
aws kinesis create-stream --stream-name osquery-test-stream --shard-count 1
  1. Configure osquery options
  • logger_plugin=aws_kinesis
  • aws_kinesis_stream=osquery-test-stream
  • aws_access_key_id=REDACTED
  • aws_secret_access_key=REDACTED
  • verbose
  1. Add a scheduled query
    "schedule": {
       "test_query": {
            "query": "select * from time",
            "interval": 10
        }
    }
  1. Check stream results and see a combination of status and result log types reported back
> SHARD_ITERATOR=$(aws kinesis get-shard-iterator --stream-name osquery-test-stream --shard-id shardId-000000000000 --shard-iterator-type LATEST --query 'ShardIterator')

> aws kinesis get-records --shard-iterator $SHARD_ITERATOR --query 'Records[0].Data' --output text | base64 -d
{hostIdentifier":"UK-GN-44077",calendarTime":"Fri Dec 20 14:31:11 2019 UTC",unixTime":"1576852271",severity":"0",filename":"aws_log_forwarder.h",line":"216",message":"aws_kinesis:Successfully sent 1 out of 1 log records",version":"4.0.1",log_type":"status"}

> aws kinesis get-records --shard-iterator $SHARD_ITERATOR --query 'Records[1].Data' --output text | base64 -d
{"name":"test_query","hostIdentifier":"UK-GN-44077","calendarTime":"Fri Dec 20 14:31:25 2019 UTC","unixTime":"1576852285","epoch":"0","counter":"10","logNumericsAsNumbers":"false","columns":{"datetime":"2019-12-20T14:02:40Z","day":"20","hour":"14","iso_8601":"2019-12-20T14:02:40Z","local_time":"1576850560","local_timezone":"UTC","minutes":"2","month":"12","seconds":"40","timestamp":"Fri Dec 20 14:02:40 2019 UTC","timezone":"UTC","unix_time":"1576850560","weekday":"Friday","win_timestamp":"132213241603576009","year":"2019"},"action":"removed","log_type":"result"}

> aws kinesis get-records --shard-iterator $SHARD_ITERATOR --query 'Records[2].Data' --output text | base64 -d
{"name":"test_query","hostIdentifier":"UK-GN-44077","calendarTime":"Fri Dec 20 14:31:25 2019 UTC","unixTime":"1576852285","epoch":"0","counter":"10","logNumericsAsNumbers":"false","columns":{"datetime":"2019-12-20T14:31:25Z","day":"20","hour":"14","iso_8601":"2019-12-20T14:31:25Z","local_time":"1576852285","local_timezone":"UTC","minutes":"31","month":"12","seconds":"25","timestamp":"Fri Dec 20 14:31:25 2019 UTC","timezone":"UTC","unix_time":"1576852285","weekday":"Friday","win_timestamp":"132213258857639047","year":"2019"},"action":"added","log_type":"result"}
  1. Update osquery configuration
  • aws_kinesis_disable_log_status=true
  1. Check stream results and see a contain only result logs
> SHARD_ITERATOR=$(aws kinesis get-shard-iterator --stream-name osquery-test-stream --shard-id shardId-000000000000 --shard-iterator-type LATEST --query 'ShardIterator')

> aws kinesis get-records --shard-iterator $SHARD_ITERATOR --query 'Records[0].Data' --output text | base64 -d
{"name":"test_query","hostIdentifier":"UK-GN-44077","calendarTime":"Fri Dec 20 15:29:30 2019 UTC","unixTime":"1576855770","epoch":"0","counter":"323","logNumericsAsNumbers":"false","columns":{"datetime":"2019-12-20T15:29:25Z","day":"20","hour":"15","iso_8601":"2019-12-20T15:29:25Z","local_time":"1576855765","local_timezone":"UTC","minutes":"29","month":"12","seconds":"25","timestamp":"Fri Dec 20 15:29:25 2019 UTC","timezone":"UTC","unix_time":"1576855765","weekday":"Friday","win_timestamp":"132213293659101666","year":"2019"},"action":"removed","log_type":"result"}

> aws kinesis get-records --shard-iterator $SHARD_ITERATOR --query 'Records[1].Data' --output text | base64 -d
{"name":"test_query","hostIdentifier":"UK-GN-44077","calendarTime":"Fri Dec 20 15:29:30 2019 UTC","unixTime":"1576855770","epoch":"0","counter":"323","logNumericsAsNumbers":"false","columns":{"datetime":"2019-12-20T15:29:30Z","day":"20","hour":"15","iso_8601":"2019-12-20T15:29:30Z","local_time":"1576855770","local_timezone":"UTC","minutes":"29","month":"12","seconds":"30","timestamp":"Fri Dec 20 15:29:30 2019 UTC","timezone":"UTC","unix_time":"1576855770","weekday":"Friday","win_timestamp":"132213293709168865","year":"2019"},"action":"added","log_type":"result"}

> aws kinesis get-records --shard-iterator $SHARD_ITERATOR --query 'Records[3].Data' --output text | base64 -d
{"name":"test_query","hostIdentifier":"UK-GN-44077","calendarTime":"Fri Dec 20 15:29:35 2019 UTC","unixTime":"1576855775","epoch":"0","counter":"324","logNumericsAsNumbers":"false","columns":{"datetime":"2019-12-20T15:29:35Z","day":"20","hour":"15","iso_8601":"2019-12-20T15:29:35Z","local_time":"1576855775","local_timezone":"UTC","minutes":"29","month":"12","seconds":"35","timestamp":"Fri Dec 20 15:29:35 2019 UTC","timezone":"UTC","unix_time":"1576855775","weekday":"Friday","win_timestamp":"132213293759204983","year":"2019"},"action":"added","log_type":"result"}

@Smjert
Copy link
Member

Smjert commented Jan 3, 2020

@theopolis usesLogStatus it's called in the LoggerPlugin::call function, here:

osquery/osquery/core/plugins/logger.cpp

Line 64 in 21fa6d9

features |= (usesLogStatus()) ? LOGGER_FEATURE_LOGSTATUS : 0;

So it gets called every line that has to be logged.

@nickcollier Please remember to sign our CLA too.

@nickcollier
Copy link
Contributor Author

@Smjert I've signed the CLA now

@Smjert Smjert closed this Mar 2, 2020
@Smjert Smjert reopened this Mar 2, 2020
@theopolis theopolis merged commit 09c5246 into osquery:master Mar 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@theopolis theopolis theopolis approved these changes

Assignees
No one assigned
Labels
AWS Related to Amazon Web Services tables logging
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@nickcollier @theopolis @Smjert