Possible resolutions I can see:

1. Attempt to decode the string as UTF8; if it fails then decode again as latin-1 (which will give codepoints u80 to uFF)
2. Only decode the string as UTF8 if it begins with BOM; otherwise decode as latin-1
3. Decode as UTF8 and replace invalid characters with RuneError (uFFFD)
4. Decode as UTF8 and replace invalid characters with surrogate escape uDC80 to uDCFF
5. Decode as UTF8 and replace invalid characters with some ASCII sequence such as \ x N N or < N N > (the latter is what less does)

2 is fully reversible. The problem with 2 is that there might be some devices out there already sending UTF8 inside syslog but not declaring it with a BOM, and these would be mis-decoded (and this would be a change in behaviour).

The problem with 3 is that there is data loss.

The problem with 4 is that surrogate escapes are not widely used, and are technically still invalid unicode and hence cannot safely be carried in JSON.

5 is not safely reversible, unless you also replace a bare \ with \\ in all messages.

My inclination is a combination of 1 and 2:

• If the message starts with BOM, then decode as UTF8 and replace invalid runes with uFFFD
• If the message does not start with BOM, then decode as UTF8, but if it fails anywhere, re-decode the entire message as latin-1

EDIT: reference https://en.wikipedia.org/wiki/UTF-8#Invalid_byte_sequences

Until this is fixed, here is a workaround when using rsyslog: you can use the mmutf8fix module to replace invalid utf8 characters with a single ASCII character (unfortunately the Unicode replacement character is not possible).

Put this before the action which forwards to promtail:

module(load="mmutf8fix")
action(type="mmutf8fix" replacementChar="?")

This module is available as part of rsyslog from the Adiscon v8-stable repository, in separate package "rsyslog-mmutf8fix".

Thank you Brian for the workaround and possible solutions, we'll look into it !

🥇

We started seeing this too. Thanks for the workaround!

The problem is related to influxdata/go-syslog#21 and I started fixing upstream: influxdata/go-syslog#35.

The idea is to relax the parser to accept any encoding and add an option to enforce a compliant MSG. This would stop promtail from dropping the connection when receiving a non UTF8 message and pass it as-is.

The encoding could then be fixed or reencoded with a fallback encoding in the syslog receiver part of promtail or more generically in a later pipeline stage (new transform stage?).

Something like:

scrape_configs:
- job_name: syslog
  syslog:
    listen_address: 127.0.0.1:5140
  relabel_configs:
  - source_labels: [__syslog_message_hostname]
    target_label: host

  pipeline_stages:
  - match:
    selector: '{host=~"ex-.*"}'
    stages:
    - encoding:
        reencode_with_fallback: iso-latin-1