GitLab

A queue name is easily changed within the WebUI and maybe the admin forget to
also update the OverrideOutgoingMailFrom setting. It more reliable to use the
queue id instead the queue name as the hash key.

A reasonable use case for rt is to comment on a ticket (or
"correspond", which is just a comment that is also sent to the ticket
requestor).  In the web form for commenting, there is the ability to
update the status of the ticket at the same time.

This feature should be exposed in the command line client as well.

Bug-Debian: http://bugs.debian.org/760292

As input/output encoding has been cleaned up significantly, stop hiding
"Wide character in..." warnings, as they may conceal true problems.

Tests may produce characters (e.g. ok(1, "¡Oy!") ) which need to be
encoded before being output.  Use the suggested technique from the
Test::More POD to encode these strings.

Currently, any string which has the "UTF-8" flag is encoded as UTF-8
before being sent to the browser.  This requires that any output which
is binary, or has already been encoded to bytes, _not_ have the flag
accidentally set.

It also requires that all output character strings have the "UTF-8" flag
enabled; while necessary for codepoints > 255, it is not strictly
required for codepoints between 127 and 255.  As RT now consistently
uses Encode::decode() to produce character strings, which sets the
"UTF-8" flag even for characters in that range, this is likely safe.

The most correct fix would be to explicitly flag output that needs to be
encoded.  However, doing so in a backwards compatible manner is
extremely difficult; as is_utf8 is unlikely to be incorrect in this
context, the small potential additional correctness is deemed unworth
the cost of requiring all external modules to flag their binary (or
character) output as such.

This does require a version dump of the DBIx::SearchBuilder dependency
for PostgreSQL installs, as DBIx::SearchBuilder previous to that version
manually forced the "UTF8" off on all bound values before executing
statements.  When the character/byte distinction has been intentionally
made, manually (and unpredictably) applying an additional layer of UTF-8
encoding is incorrect.

RT::Interface::Web::Handler ensures that any code which contains
characters is encoded in UTF-8 before being sent to the browser.  As
such, the explicit encoding here is unnecessary.

929b4231 specifically documented _EncodeLOB and _DecodeLOB to take
bytes; enforce that by checking the arguments they are passed using
RT::Util::assert_bytes.

There is no need to check is_utf8 on the arguments passed to DecodeArgs,
as it is the first and only line of decoding of arguments over HTTP.

c95221e4 moved this comment from html/autohandler, but associated it
with the code it was above rather than the code above it.  Move it to
where it belongs, and update it slightly.

66930fd8 switched from an explicit _utf8_off to an explicit _utf8_on, in
an attempt to switch from splitting on bytes to splitting on characters.
However, the "UTF8" flag does not magically determine if a string is
bytes or characters.  Instead, only consistency in calling convention
can do so.  All callsites of RT::Interface::Email::EncodeToMIME and
RT::Action::SendEmail::MIMEEncodeString now pass character strings; all
that _utf8_on can do is incorrectly "decode" those strings as UTF-8 if
they happen to not have the "UTF8" flag set.

While there ay have been bugs surrounding encodings in LWP or
WWW::Mechanize previously, the ->content method correctly returns
characters and not bytes for all modern versions.  Remove the explicit
adjustment.

The dashboard mailer code replicates many of the original steps of Mason
parameter parsing -- but omitted the important step of decoding the
bytes once they had been un-URI-encoded.

All remaining cases of "use utf8" lie in the testsuite.  As "use utf8"
changes the semantics of dealing with Unicode strings, remove it to
allow programmers to always assume that literals are interpreted as
bytestrings, not characters.  To do otherwise means that one must always
ask if "use utf8" is in scope before performing operations on any
literals; instead, simply make the encodings and decodings explicit.

Note that wide characters may appear in editors, and that the encoding
of the characters on _disk_ will always be UTF-8.  The removal of "use
utf8" merely means that perl will generate a two-byte string from "é",
and not a one-character string.

"use utf8" causes the sourcecode (including all strings) to be
interpreted by perl as characters encoded in UTF-8, not bytes.  In
lib/RT/I18N/fr.pm, this was being used to substitude codepoint 160
(NO-BREAK SPACE, U+00A0) for commas.  The fact that the space character
was not 0x20, but rather 0xA0, was mostly hidden by use of "use utf8".

Remove the "use utf8" and make the replacement character clear.

This is not only for code consistency, but also for consistency of
output.  Encode::encode_utf8(...) is equivalent to
Encode::encode("utf8",...) which is the non-"strict" form of UTF-8.
Strict UTF-8 encoding differs in that (from `perldoc Encode`):

    ...its range is much narrower (0 ..  0x10_FFFF to cover only 21 bits
    instead of 32 or 64 bits) and some sequences are not allowed, like
    those used in surrogate pairs, the 31 non-character code points
    0xFDD0 .. 0xFDEF, the last two code points in any plane (0xXX_FFFE
    and 0xXX_FFFF), all non-shortest encodings, etc.

RT deals with interchange with databases, email, and other systems.  In
dealing with encodings, it should ensure that it does not produce byte
sequences that are invalid according to official Unicode standards.

See the prior commit for reasoning, which applies just as much to the
header as the body.

Use the newly-added RT::Util::assert_bytes function to verify that the
body is indeed bytes, and not characters.

We also remove the _utf8_off call -- because, contrary to what the
comment implies, the presence or absence of the "UTF8" flag does _not_
determine if a string is "encoded as octets and not as characters"; it
merely states that the string is capable of holding codepoints > 255.
If it happens to not contain any, the _utf8_off does nothing.  If it
does, it effectively encodes all codepoints > 127 in UTF-8.

Given the premise that the string contains bytes in some (probably
non-UTF-8) encoding, re-encoding some bytes of it as UTF-8 cannot
possibly produce valid output.  The flaw in this situation cannot be
fixed by a simple _utf8_off, but instead must be fixed by ensuring that
the body always contains bytes, not wide characters -- as it now does,
thanks to the prior commits.  The call to RT::Util::assert_bytes serves
as an additional safeguard against backsliding on that assumption.

Note that it is impossible to verify that an input is characters; here,
we can only validate if it _could_ be bytes.

First, any string with the "UTF8" flag off cannot contain codepoints
above 255, and as such is safe.  Additionally, if the "UTF8" flag is on,
having no codepoints above 127 means the bytes are unambigious.  Having
codepoints above 255 is guaranteedly a sign that the input is not a byte
string.

This leaves only the case of a string with the "UTF8" flag on, and
codepoints above 127 but below 255.  The "UTF8" flag is a sign that they
were _likely_ touched by character data at some point.  In such cases we
warn, suggesting that the bytes have the "UTF8" flag disabled by means
of utf8::downgrade, if they are indeed bytes.

This helper method is used in a number of places in
RT::Action::SendEmail, often without remembering that it should be
passed bytes, not characters.  Change it to always take characters, and
modify the two callsite which (correctly) passed it bytes to no longer
do so.

Placing wide characters into MIME::Entity objects can lead to
double-encoding, as discovered most recently in d469cacc.  Explicitly
decode all headers as UTF-8 when retrieving them with ->get(), and
encode them as UTF-8 before updating them with ->set() or ->replace().
This also applies to headers passed to ->build().  The only exceptions
to this are fixed strings in the source (which, in the absence of "use
utf8", are always bytes).

While the majority of these headers will never have wide characters in
them, always decoding and encoding ensures the proper disipline to
guarantee that strings with the "UTF8" flag do not get placed in a
header, which can cause double-encoding.

Placing wide characters into MIME::Entity objects can lead to
double-encoding.  Always treat them as byte stores, encoding as UTF-8
and noting their character set.

In the case of Approvals/index.html, there was no need for an explicit
MIME::Entity object; ->Correspond creates one as needed from a "Content"
argument.

The comment and code were added in RT 2.1.38, which only required perl
5.6.1; the perl version was increased to 5.8.3 to cover a large number
of encoding bugs, such as the one this comment was likely alluding to.

Ensure that we always send UTF-8 encoded bytes to loggers, and not wide
characters.  This is correctly done via an explicit call to
Encode::encode, and not via checks of utf8::is_utf8 (which may be false
for character strings with codepoints > 127 but < 256), and not via
_utf8_off (which would fail similarly for such characters).

t/data/emails/text-html-in-russian was removed because the original
purpose of the test was removed in 46fd04d9, after 90f9c190 stopped
attaching text/html incoming mail to autoreplies.

This adds a row limit to the Classes and Templates list for the first time.

It also changes the row limit for Groups and Users lists from 100 to 50.

The rows limit for the Membership page for Groups and Users is changed from
20 to 50. This e.g. helps adding groups to users as it avoids jumping through
pages if you have many groups.

dec257a4 fixed RT::CustomField->Create() to accept LinkValueTo and
IncludeContentForValue, which are stored in attributes.  However, it
also ensured that those attributes are created even if they're empty.
This means every CF in the system, regardless of if it uses them, adds
two attributes.

Adjust _URLTemplate to remove attributes that are now empty, and more
explicitly return undef when the attribute does not exist.

Fixes: #22191

This makes Due on ticket display consistent with ticket lists (collections).

This treats Due the same as DueRelative in ticket lists.

There's a callback to add data at the end of the Links portlet, but
nothing to allow you to change what's display.  This provides a callback
that would let you remove (for example) the Parents from the listing, so
that you can list them in a separate portlet later, with a custom
display.