Skip to content

Feature request: Include Unwritten Log Entries On Error #519

Closed
3 of 3 issues completed
Closed
@jasonwadsworth

Description

@jasonwadsworth

Description of the feature request

It would be great if the logger could collect some unwritten logs (maybe a configurable amount that defaults to 10 or something) and if there is an error logged it would also write any logs that were not already written.

Problem statement

I've always hated that the debug logs are never available when I need them. They are typically turned off in a production system, but when there is an error I'd like to have information from those logs. I don't want them most of the time, so really only when an error occurs. Sampling isn't a good option because the error may be intermittent and the logs will not be present for the relevant time.

Summary of the feature

Basically, at the point when the decision is made whether to log it would either log (current functionality) or store the log. If there was an error log it would log the error as well as everything stored (and flush the store).

Code examples

Benefits for you and the wider AWS community

Faster time to resolution on errors

Describe alternatives you've considered

I've looked at doing something like this myself but haven't tried it yet. As I mentioned, sampling doesn't accomplish what I'm looking for.

Additional context

Related issues, RFCs

Sub-issues

Sub-issues

3 of 3 Issues completed

Activity

added
triageThis item has not been triaged by a maintainer, please wait
on Jan 27, 2022
flochaz

flochaz commented on Jan 28, 2022

@flochaz
Contributor

Thx for your suggestion ! Interesting idea that can definitely help reducing logging cost while having a verbose output on error. we will look into it .

ijemmy

ijemmy commented on Feb 14, 2022

@ijemmy
Contributor

I like this idea very much. It addresses my main use case for sampling rate. But it's more cost-efficient and we don't miss anything.

I suppose usage will be like this?

import { Logger } from '@aws-lambda-powertools/logger';
const logger = new Logger({
   logAllLevelsOnError: true,
});

In this case, the client needs to make sure that they catch any error and print out the log to get it flushed out.

It can be used with Middy's error-logger, to ensure that uncaught error are always handled.

jasonwadsworth

jasonwadsworth commented on Feb 23, 2022

@jasonwadsworth
Author

In this case, the client needs to make sure that they catch any error and print out the log to get it flushed out.

Yes, it is definitely dependent on the code catching and logging an error.

Came across this repo, which is doing what I'm talking about here. Doesn't seem too bad in this example, but I haven't looked at the logging code in this repo to see how complex it might be here.

added and removed
triageThis item has not been triaged by a maintainer, please wait
on Feb 28, 2022
ijemmy

ijemmy commented on Mar 15, 2022

@ijemmy
Contributor

There is a blog about this. Useful for implementation reference: https://dev.to/aws-builders/saving-on-aws-lambda-amazon-cloudwatch-logs-costs-51od

Notice the clear() function or the logs can leak to the next Lambda call.

changed the title [-](logger): Include Unwritten Log Entries On Error[/-] [+]Feature (logger): Include Unwritten Log Entries On Error[/+] on Apr 29, 2022
buggy

buggy commented on Aug 17, 2022

@buggy

I really like this idea.

Could I suggest using levelOnError instead of logAllLevelsOnError? This would provide additional flexibility by allowing the developer to set the log level on error instead of assuming they want everything.

Use case:

  1. Set level to WARN and levelOnError to INFO.
  2. Use WARN for warnings that should always be logged
  3. Use INFO for additional information (like the event) that will allow debugging when an error occurs
  4. Use DEBUG for low level debugging (i.e. logging each iteration of a loop)

Why?
Frequently I only need a little extra information, like the event, to be logged on error. By allowing the developer to set the log level on error I can use INFO for the stuff I want to see when something goes wrong but still have DEBUG littered through my code for when I need to do development / debugging. This is particularly helpful if you only keep the last N items to be written on error because I want some things I log (INFO) but not others (DEBUG).

added
triageThis item has not been triaged by a maintainer, please wait
need-customer-feedbackRequires more customers feedback before making or revisiting a decision
on-holdThis item is on-hold and will be revisited in the future
and removed
triageThis item has not been triaged by a maintainer, please wait
on Nov 9, 2022

22 remaining items

added
confirmedThe scope is clear, ready for implementation
and removed
need-more-informationRequires more information before making any calls
on Feb 10, 2025
self-assigned this
on Feb 10, 2025
dreamorosi

dreamorosi commented on Feb 10, 2025

@dreamorosi
Contributor

We have just concluded a RFC for a version of this feature that we believe will unlock this use case, you can find more details here: #3410 (start from this comment).

We'll work on this feature in the next 2-3 weeks.

moved this from Backlog to Working on it in Powertools for AWS Lambda (TypeScript)on Feb 11, 2025
dreamorosi

dreamorosi commented on Mar 7, 2025

@dreamorosi
Contributor

This was released in v2.16.0

moved this from Coming soon to Shipped in Powertools for AWS Lambda (TypeScript)on Mar 7, 2025
github-actions

github-actions commented on Mar 7, 2025

@github-actions
Contributor

⚠️ COMMENT VISIBILITY WARNING ⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

added
completedThis item is complete and has been merged/shipped
and removed
confirmedThe scope is clear, ready for implementation
on Mar 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

completedThis item is complete and has been merged/shippedfeature-requestThis item refers to a feature request for an existing or new utilityloggerThis item relates to the Logger Utility

Type

No type

Projects

Status

Shipped

Milestone

No milestone

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @ijemmy@buggy@flochaz@heitorlessa@jasonwadsworth

      Issue actions

        Feature request: Include Unwritten Log Entries On Error · Issue #519 · aws-powertools/powertools-lambda-typescript