cloudwatch_out not working - logs never show up in CloudWatch

[deweller]

Put simply, I just can't get this to work.

Here is my config:

  <match applog.**>
    @type cloudwatch_logs
    log_group_name application-logs
    use_tag_as_stream true
    auto_create_stream true
    message_keys message
  </match>

The log group and the stream are created as expected, but no entries ever show up.

There is no information in the fluentd logs.

Does this plugin work? Is anyone else experiencing this issue? How can I debug this?

Thanks.

[tkob]

I'm experiencing the same problem (streams created, no log events appear). The config is similar (use_tag_as_stream=true, auto_create_stream=true, but no message_keys).

When fluentd is launched with -v option, debug messages saying "Calling PutLogEvents API" are emitted. The weird thing is that the debug messages themselves are written to "fluent.debug" stream.

[tkob]

For my case, I found the cause:

• Fluentd receives from remote syslog server which uses JST(+09:00) timezone, while the fluentd server uses UTC(+00:00).
• RFC3164 timestamps do not have timezone.
• By default, fluentd syslog input plugin uses remote timestamps as the time of events. This causes timestamp discrepancy (+00:00 vs. +09:00).
• AWS PutLogEvents skips events which are more than 2 hours in the future.
• Current version of this plugin does not report skipped log events.

I do not know whether this applies to @deweller's case, but similar cause is very probable, I think.

My workaround is to edit regexp of syslog input format so that it does not use remote timestamp as event's timestamp.

[uover82]

Hi,
Is this a current issue? I'm experiencing something similar - worker log data w/ no noticeable errors, yet no CloudWatch logs. If so, could fluent.conf configuration (localtime, timezone, etc) be a viable alternative solution?
Thanks

[adamcheney]

Yeah, I'm also experiencing the same issue. I'll take a look at the timestamps thing, but in our case (we're based in NZ) UTC timestamps will be around 13 hours behind. Otherwise, same symptoms as the original poster - log group and log stream created without problems but no actual log events; trace level logging shows:

2019-03-28 01:41:49 +0000 [trace]: #0 executing sync write chunk="5851da4c65301cbeb92c43f8e2e5f113"
2019-03-28 01:41:52 +0000 [debug]: #0 Called PutLogEvents API group="k8s-workloads" stream="kubexray" events_count=5 events_bytesize=6465 sequence_token=nil thread=70351681997220 request_sec=0.246174792
2019-03-28 01:41:52 +0000 [trace]: #0 write operation done, committing chunk="5851da4c65301cbeb92c43f8e2e5f113"

[Watson1978]

Thanks for the report, I believe that identifies the root cause correctly.

The symptom here (the log group and log stream are created, but no log events ever appear, and nothing is printed to the Fluentd log) is almost always caused by AWS silently rejecting the events on the PutLogEvents side, not by the plugin failing to send them.

PutLogEvents rejects events whose timestamp is more than 2 hours in the future (tooNewLogEventStartIndex), more than 14 days in the past (tooOldLogEventEndIndex), or older than the log group's retention period (expiredLogEventEndIndex). Rejected events are only reported back in the API response's rejectedLogEventsInfo; the request itself still succeeds, so from Fluentd's point of view everything looks fine — which is exactly why there is "no information in the fluentd logs."

As @tkob points out, the most common trigger is a timestamp/timezone mismatch — for example RFC3164 syslog timestamps that carry no timezone, or a source clock that is ahead of the Fluentd host — which pushes the event time more than 2 hours into the future and gets it dropped.

On current versions of the plugin you can confirm this directly by enabling log_rejected_request, which logs exactly these rejections:

<match applog.**>
  @type cloudwatch_logs
  log_group_name application-logs
  use_tag_as_stream true
  auto_create_stream true
  message_keys message
  log_rejected_request true
</match>

If the output shows tooNewLogEventStartIndex or tooOldLogEventEndIndex under rejected_log_events_info, it is the timestamp issue described above, and correcting the source timestamp/timezone (so the parsed event time is accurate) resolves it.

Since the root cause is understood and log_rejected_request now makes it diagnosable, I'll close this as resolved. If anyone still sees events being dropped without any rejectedLogEventsInfo in the output, please open a new issue with the log_rejected_request true log and your Fluentd / plugin versions. Thanks!