Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

out_cloudwatch_logs: support log_group_class option #8832

Merged
merged 1 commit into from
May 29, 2024
Merged

out_cloudwatch_logs: support log_group_class option #8832

merged 1 commit into from
May 29, 2024

Conversation

PettitWesley
Copy link
Contributor

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@PettitWesley
out_cloudwatch_logs: support log_group_class option
1bd17d8 
Signed-off-by: Wesley Pettit <wppttt@amazon.com>
@PettitWesley PettitWesley mentioned this pull request May 17, 2024
Closed
7 tasks
@PettitWesley
Copy link
Contributor Author

147dda285fd4:bottlerocket-eks wppttt$ kubectl -n amazon-cloudwatch logs fluent-bit-fbf8584f8-vvsnl
Fluent Bit v1.9.10

  • Copyright (C) 2015-2022 The Fluent Bit Authors
  • Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
  • https://fluentbit.io

[2024/05/17 03:47:44] [ info] [fluent bit] version=1.9.10, commit=7eb6b397eb, pid=1
[2024/05/17 03:47:44] [ info] [storage] version=1.4.0, type=memory+filesystem, sync=normal, checksum=disabled, max_chunks_up=128
[2024/05/17 03:47:44] [ info] [storage] backlog input plugin: storage_backlog.8
[2024/05/17 03:47:44] [ info] [cmetrics] version=0.3.7
[2024/05/17 03:47:44] [ info] [input:tail:tail.0] multiline core started
[2024/05/17 03:47:44] [ info] [input:tail:tail.1] multiline core started
[2024/05/17 03:47:44] [ info] [input:tail:tail.2] multiline core started
[2024/05/17 03:47:44] [ info] [input:systemd:systemd.3] seek_cursor=s=fd652eba798a44cc9ecbea19cdd2ac3e;i=1af... OK
[2024/05/17 03:47:44] [ info] [input:tail:tail.4] multiline core started
[2024/05/17 03:47:44] [ info] [input:storage_backlog:storage_backlog.8] queue memory limit: 4.8M
[2024/05/17 03:47:44] [ info] [filter:kubernetes:kubernetes.0] https=1 host=127.0.0.1 port=10250
[2024/05/17 03:47:44] [ info] [filter:kubernetes:kubernetes.0] token updated
[2024/05/17 03:47:44] [ info] [filter:kubernetes:kubernetes.0] local POD info OK
[2024/05/17 03:47:44] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with Kubelet...
[2024/05/17 03:47:44] [ info] [filter:kubernetes:kubernetes.0] connectivity OK
[2024/05/17 03:47:44] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] Configured log_group_class INFREQUENT_ACCESS will only apply to log groups created by Fluent Bit. Look for the Created log group info level message emitted when a group does not already exist and is created.
[2024/05/17 03:47:44] [ warn] [output:cloudwatch_logs:cloudwatch_logs.1] Configured log_group_class INFREQUENT_ACCESS will only apply to log groups created by Fluent Bit. Look for the Created log group info level message emitted when a group does not already exist and is created.
[2024/05/17 03:47:44] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] worker #0 started
[2024/05/17 03:47:44] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] worker #0 started
[2024/05/17 03:47:44] [ info] [output:cloudwatch_logs:cloudwatch_logs.2] worker #0 started
[2024/05/17 03:47:44] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2024/05/17 03:47:44] [ info] [sp] stream processor started
[2024/05/17 03:47:44] [ info] [input:tail:tail.0] inotify_fs_add(): inode=42307 watch_fd=1 name=/var/log/containers/coredns-68c6b7b454-r6t8q_kube-system_coredns-1709ec5295314385997364635ee603e6d6a559f6cfb70bf3f3e2c431355833c5.log
[2024/05/17 03:47:44] [ info] [input:tail:tail.0] inotify_fs_add(): inode=167812073 watch_fd=2 name=/var/log/containers/coredns-68c6b7b454-tc5mh_kube-system_coredns-2cd2d04c3f2bc1460ac0e86dbd1c3b3b3f8ba0828fc97cc122b491ce897b457e.log
[2024/05/17 03:47:44] [ info] [input:tail:tail.0] inotify_fs_add(): inode=201708930 watch_fd=3 name=/var/log/containers/eks-pod-identity-agent-q5vxk_kube-system_eks-pod-identity-agent-init-809c5a47b01cca41b44dc18b54b81ab3f3753d1caae2c0c0ca1f6439b5fa4ae3.log
[2024/05/17 03:47:44] [ info] [input:tail:tail.4] inotify_fs_add(): inode=218165991 watch_fd=1 name=/var/log/containers/aws-node-jhslt_kube-system_aws-eks-nodeagent-f083e3e56bc143254ca23011138731a632725afa821c9fc1f442f4e0901eef29.log
[2024/05/17 03:47:44] [ info] [input:tail:tail.4] inotify_fs_add(): inode=16777403 watch_fd=2 name=/var/log/containers/aws-node-jhslt_kube-system_aws-node-7cd53aacd33b6193b69e8085093fe6876e3d0dcda16f4003813528a7c378059e.log
[2024/05/17 03:47:44] [ info] [input:tail:tail.4] inotify_fs_add(): inode=117531748 watch_fd=3 name=/var/log/containers/aws-node-jhslt_kube-system_aws-vpc-cni-init-e5fe7f223d165f4b1c55d1bcc3b8e9f47aa1a475f45d8b3e83c2a1070e82f941.log
[2024/05/17 03:47:44] [ info] [input:tail:tail.0] inotify_fs_add(): inode=185075710 watch_fd=4 name=/var/log/containers/eks-pod-identity-agent-q5vxk_kube-system_eks-pod-identity-agent-59bc7f4cc8194e140161525d8c2db02008ca5718770d120cfe7d48f6a1203a21.log
[2024/05/17 03:47:44] [ info] [input:tail:tail.1] inotify_fs_add(): inode=218429523 watch_fd=1 name=/var/log/containers/fluent-bit-fbf8584f8-vvsnl_amazon-cloudwatch_fluent-bit-1deecf439298817933984983865ffed777bf4347c4b8d42a1562f215cc12468a.log
[2024/05/17 03:47:44] [ info] [input:tail:tail.4] inotify_fs_add(): inode=100789792 watch_fd=4 name=/var/log/containers/kube-proxy-pgx7n_kube-system_kube-proxy-5a8416e3845cdf77cbdc4789db4e1494b6959b912afb07c6d39158e1d41dbc61.log
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.eks-pod-identity-agent-q5vxk_kube-system_eks-pod-identity-agent-59bc7f4cc8194e140161525d8c2db02008ca5718770d120cfe7d48f6a1203a21.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-dataplane.systemd.containerd.service in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.fluent-bit-fbf8584f8-vvsnl_amazon-cloudwatch_fluent-bit-1deecf439298817933984983865ffed777bf4347c4b8d42a1562f215cc12468a.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-dataplane.systemd.kubelet.service in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-dataplane.tail.var.log.containers.kube-proxy-pgx7n_kube-system_kube-proxy-5a8416e3845cdf77cbdc4789db4e1494b6959b912afb07c6d39158e1d41dbc61.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane not found. Will attempt to create it.
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application not found. Will attempt to create it.
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane not found. Will attempt to create it.
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane not found. Will attempt to create it.
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application not found. Will attempt to create it.
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.1] CreateLogGroup API responded with error='OperationAbortedException', message='A conflicting operation is currently in progress against this resource. Please try again.'
[2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.1] Failed to create log group
[2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.1] Failed to send events
[2024/05/17 03:47:49] [ warn] [engine] failed to flush chunk '1-1715917664.974275925.flb', retry in 10 seconds: task_id=4, input=tail.4 > output=cloudwatch_logs.1 (out_id=1)
[2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.1] CreateLogGroup API responded with error='OperationAbortedException', message='A conflicting operation is currently in progress against this resource. Please try again.'
[2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.1] Failed to create log group
[2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.1] Failed to send events
[2024/05/17 03:47:49] [ warn] [engine] failed to flush chunk '1-1715917664.948458095.flb', retry in 10 seconds: task_id=3, input=systemd.3 > output=cloudwatch_logs.1 (out_id=1)
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Created log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane with storage class INFREQUENT_ACCESS
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-dataplane.systemd.containerd.service in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Created log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application with storage class INFREQUENT_ACCESS
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.fluent-bit-fbf8584f8-vvsnl_amazon-cloudwatch_fluent-bit-1deecf439298817933984983865ffed777bf4347c4b8d42a1562f215cc12468a.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application not found. Will attempt to create it.
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:49] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application already exists; Fluent Bit did not create this group in this execution. Fluent Bit therefore was unable verify or set INFREQUENT_ACCESS storage. Check CloudWatch Console or API for the groups storage class status.
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.eks-pod-identity-agent-q5vxk_kube-system_eks-pod-identity-agent-59bc7f4cc8194e140161525d8c2db02008ca5718770d120cfe7d48f6a1203a21.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Created log stream ip-192-168-34-117.ca-central-1.compute.internal-dataplane.systemd.containerd.service
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application not found. Will attempt to create it.
[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:49] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application already exists; Fluent Bit did not create this group in this execution. Fluent Bit therefore was unable verify or set INFREQUENT_ACCESS storage. Check CloudWatch Console or API for the groups storage class status.
[2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Failed to send events
[2024/05/17 03:47:49] [ warn] [engine] failed to flush chunk '1-1715917664.940218327.flb', retry in 7 seconds: task_id=1, input=tail.1 > output=cloudwatch_logs.0 (out_id=0)
[2024/05/17 03:47:49] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application already exists; Fluent Bit did not create this group in this execution. Fluent Bit therefore was unable verify or set INFREQUENT_ACCESS storage. Check CloudWatch Console or API for the groups storage class status.
[2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Failed to send events
[2024/05/17 03:47:49] [ warn] [engine] failed to flush chunk '1-1715917664.929877801.flb', retry in 6 seconds: task_id=0, input=tail.0 > output=cloudwatch_logs.0 (out_id=0)
[2024/05/17 03:47:54] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.fluent-bit-fbf8584f8-vvsnl_amazon-cloudwatch_fluent-bit-1deecf439298817933984983865ffed777bf4347c4b8d42a1562f215cc12468a.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:54] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application not found. Will attempt to create it.
[2024/05/17 03:47:54] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:54] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application already exists; Fluent Bit did not create this group in this execution. Fluent Bit therefore was unable verify or set INFREQUENT_ACCESS storage. Check CloudWatch Console or API for the groups storage class status.
[2024/05/17 03:47:54] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.fluent-bit-fbf8584f8-vvsnl_amazon-cloudwatch_fluent-bit-1deecf439298817933984983865ffed777bf4347c4b8d42a1562f215cc12468a.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:54] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application not found. Will attempt to create it.
[2024/05/17 03:47:54] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:54] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application already exists; Fluent Bit did not create this group in this execution. Fluent Bit therefore was unable verify or set INFREQUENT_ACCESS storage. Check CloudWatch Console or API for the groups storage class status.
[2024/05/17 03:47:54] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Failed to send events
[2024/05/17 03:47:54] [ warn] [engine] failed to flush chunk '1-1715917669.177582045.flb', retry in 10 seconds: task_id=2, input=tail.1 > output=cloudwatch_logs.0 (out_id=0)
[2024/05/17 03:47:55] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.eks-pod-identity-agent-q5vxk_kube-system_eks-pod-identity-agent-59bc7f4cc8194e140161525d8c2db02008ca5718770d120cfe7d48f6a1203a21.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:55] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Created log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.eks-pod-identity-agent-q5vxk_kube-system_eks-pod-identity-agent-59bc7f4cc8194e140161525d8c2db02008ca5718770d120cfe7d48f6a1203a21.log
[2024/05/17 03:47:55] [ info] [engine] flush chunk '1-1715917664.929877801.flb' succeeded at retry 1: task_id=0, input=tail.0 > output=cloudwatch_logs.0 (out_id=0)
[2024/05/17 03:47:56] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.fluent-bit-fbf8584f8-vvsnl_amazon-cloudwatch_fluent-bit-1deecf439298817933984983865ffed777bf4347c4b8d42a1562f215cc12468a.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:56] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Created log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.fluent-bit-fbf8584f8-vvsnl_amazon-cloudwatch_fluent-bit-1deecf439298817933984983865ffed777bf4347c4b8d42a1562f215cc12468a.log
[2024/05/17 03:47:56] [ info] [engine] flush chunk '1-1715917664.940218327.flb' succeeded at retry 1: task_id=1, input=tail.1 > output=cloudwatch_logs.0 (out_id=0)
[2024/05/17 03:47:59] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-dataplane.tail.var.log.containers.kube-proxy-pgx7n_kube-system_kube-proxy-5a8416e3845cdf77cbdc4789db4e1494b6959b912afb07c6d39158e1d41dbc61.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane
[2024/05/17 03:47:59] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-dataplane.systemd.kubelet.service in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane
[2024/05/17 03:47:59] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Created log stream ip-192-168-34-117.ca-central-1.compute.internal-dataplane.systemd.kubelet.service
[2024/05/17 03:47:59] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Created log stream ip-192-168-34-117.ca-central-1.compute.internal-dataplane.tail.var.log.containers.kube-proxy-pgx7n_kube-system_kube-proxy-5a8416e3845cdf77cbdc4789db4e1494b6959b912afb07c6d39158e1d41dbc61.log
[2024/05/17 03:47:59] [ info] [engine] flush chunk '1-1715917664.948458095.flb' succeeded at retry 1: task_id=3, input=systemd.3 > output=cloudwatch_logs.1 (out_id=1)
[2024/05/17 03:47:59] [ info] [engine] flush chunk '1-1715917664.974275925.flb' succeeded at retry 1: task_id=4, input=tail.4 > output=cloudwatch_logs.1 (out_id=1)
[2024/05/17 03:48:04] [ info] [engine] flush chunk '1-1715917669.177582045.flb' succeeded at retry 1: task_id=2, input=tail.1 > output=cloudwatch_logs.0 (out_id=0)

@PettitWesley PettitWesley mentioned this pull request May 17, 2024
Open
6 tasks
@PettitWesley
Copy link
Contributor Author

OperationAbortedException with log_group_class: Race Condition between Fluent Bit instances

Below is normal even if the log_group_class was successfully set.

[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.1] CreateLogGroup API responded with error='OperationAbortedException', message='A conflicting operation is currently in progress against this resource. Please try again.'
[2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.1] Failed to create log group
[2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.1] Failed to send events

[2024/05/17 03:47:54] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application already exists; Fluent Bit did not create this group in this execution. Fluent Bit therefore was unable verify or set INFREQUENT_ACCESS storage. Check CloudWatch Console or API for the groups storage class status.

A k8s cluster usually has > 1 node and Fluent Bit is run as a daemonset. On startup, every daemon will try to set the logGroupClass, but only one can succeed and create the group. Thus, the above messages will be printed by individual Fluent Bit instances that failed.

The succeeding instance would print:

[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Created log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application with storage class INFREQUENT_ACCESS

Search CloudWatch Log Insights for the above to determine if one instance succeeded in setting the log group class because it created the group.

Workaround: Create Log Group before deployment

The best practice workaround is to create log groups before deployment; generally at cluster creation time. Each Fluent Bit instance will not need to create the group. You can disable group creation with auto_create_group Off in your Fluent Bit config.

@PettitWesley
Copy link
Contributor Author 

[OUTPUT]
        Name                cloudwatch_logs
        Match               dataplane.*
        region              ${AWS_REGION}
        log_group_name      standard-access-test16/containerinsights/${CLUSTER_NAME}/dataplane
        log_stream_prefix   ${HOST_NAME}-
        auto_create_group   true
        extra_user_agent    container-insights
        log_group_class     STANDARD

@edsiper edsiper added this to the Fluent Bit v3.0.7 milestone May 29, 2024
@edsiper
Copy link
Member

edsiper commented May 29, 2024

@PettitWesley thank you, can you pls add here the PR for docs ? (cc: @lecaros)

@edsiper edsiper merged commit 1059bee into fluent:master May 29, 2024
45 checks passed
@PettitWesley PettitWesley deleted the cw-log-group-class3.0 branch May 31, 2024 20:22
@PettitWesley PettitWesley restored the cw-log-group-class3.0 branch May 31, 2024 20:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
docs-required
Projects
None yet
Milestone
Fluent Bit v3.0.7
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@PettitWesley @edsiper