feat(ourlogs): Adjust 'log' protocol per sdk feedback

[k-fish]

Summary

Now that we've mostly finalized the logs protocol with sdks (see develop doc for more info), we want to update Relay to allow the 'log' item type to be sent in this format.

SDK's will likely primarily use the log ItemType instead of otel_log since we don't need to change timestamp conventions, can send a simplified level (see OurLogLevel added in this PR).

Schema changes

We've deprecated some fields in the protocol that only exist for OTEL:

• severity_number and severity_text, we're coercing these to level, but we're keeping the original severity text and number as attributes as OTel allows custom severity text.
• observed_timestamp_nanos is always set by relay regardless of what is sent because Relay is the 'collector'. We have to leave this as an attribute as well since it's being used by the existing consumer for origin timestamp.
• timestamp_nanos becomes timestamp: Timestamp
• trace_flags, this is unused, and the consumer doesn't even store it in the table. Will decide what to do with this later.

Future work

• The ourlog_merge_otel function can be trimmed down since we won't need to fill in deprecated fields to send the same data to the kafka consumer.
• We may need to transform the OurLog protocol from json received from sdks to a generic EAP "trace items" kafka message that is essentially a couple fields (eg. traceid) + a KVMap for attributes.

[Dav1dde]

Just some tiny nits

[Dav1dde]

The implementation has a differing implementation for FromValue and ToValue with the implicit assumption that the only serialization happens for Kafka. But an event can flow through multiple Relay instances, this will not work. FromValue and ToValue must agree on the format. I recommend updating the consumers first.

On that note, the Attribute implementation is not forward compatible, but it is also very hard to implement. I'll work on this in a separate PR which you then can depend on.

[Dav1dde]

#4626 implements support for #[metastructure(flatten)] which we can use to model the attributes, this allows for a layout like:

#[derive(Debug, Empty, FromValue, IntoValue, ProcessValue)]
pub struct Attribute {
    #[metastructure(flatten)]
    pub value: AttributeValue,

    /// Additional arbitrary fields for forwards compatibility.
    #[metastructure(additional_properties)]
    pub other: Object<Value>,
}

#[derive(Debug, Empty, FromValue, IntoValue, ProcessValue)]
pub struct AttributeValue {
    #[metastructure(field = "type", required = true, trim = "false")]
    ty: Annotated<String>,
    #[metastructure(required = true pii = "true")]
    value: Annotated<Value>,
}

I experimented with a strongly typed variant in #4627, which 'validates' when converting via FromValue but this runs into a lot of problems (described a bit in that PR). So I think we should not go that approach and use the schema sketched here.

Instead we can invest into not making AttributeValue's fields public and only provide accessors (getters/setters) with strong validation (we can do that in a follow-up).

We will need an additional normalization and validation step during processing.

In any Relay we can validate the validity of the ty field with provided value, does the actual type match the type provided in value.

A lot of that logic I already started implementing (also in #4627), which may be of use:

        match (value.ty, value.value) {
            (Annotated(Some(ty), ty_meta), Annotated(Some(value), mut value_meta)) => {
                match (ty, value) {
                    (AttributeType::Bool, Value::Bool(v)) => Self::Bool(v),
                    (AttributeType::Int, Value::I64(v)) => Self::I64(v),
                    (AttributeType::Int, Value::U64(v)) => Self::U64(v),
                    (AttributeType::Float, Value::F64(v)) => Self::F64(v),
                    (AttributeType::Float, Value::I64(v)) => Self::F64(v as f64),
                    (AttributeType::Float, Value::U64(v)) => Self::F64(v as f64),
                    (AttributeType::String, Value::String(v)) => Self::String(v),
                    (ty @ AttributeType::Unknown(_), value) => Self::Other(AttributeValueRaw {
                        ty: Annotated(Some(ty), ty_meta),
                        value: Annotated(Some(value), value_meta),
                    }),
                    (ty, value) => {
                        value_meta.add_error(ErrorKind::InvalidData);
                        value_meta.set_original_value(Some(value));
                        Self::Other(AttributeValueRaw {
                            ty: Annotated(Some(ty), ty_meta),
                            value: Annotated(None, value_meta),
                        })
                    }
                }
            }
            (mut ty, mut value) => {
                if ty.is_empty() {
                    ty.meta_mut().add_error(ErrorKind::MissingAttribute);
                }
                if value.is_empty() {
                    value.meta_mut().add_error(ErrorKind::MissingAttribute);
                }
                Self::Other(AttributeValueRaw { ty, value })
            }

Then as an additional step we need a processing step, which removes unknown type/value combinations and annotates the value as InvalidData. This step can only run when Relay is ran in processing mode!

It is important to note, that only a processing Relay can make the decision whether a provided type is unknown or not.

[cleptric]

Just some naming updates

[cleptric]

Protocol in tests looks good to me 👍

[Dav1dde]

When parsing, we default other to info, is 25 correct here or should it be rather treated as info?

[k-fish]

You're right, actually 0 is default and is essentially 'unmapped'.

[Dav1dde]

As per discussion, this should still not be used by SDKs, there are more changes planned to the envelope protocol, just not to the log item.

https://www.notion.so/sentry/Performance-Envelope-Containers-1d28b10e4b5d80b3b1c3e661f5eda6b4