FlexibleStringExpander is still using too much memory

[Adam Heath-2]

Recently, Adrian decided to alter FSE to use less memory.  He did this
by modifying the class to store a single char[], which represented the
text contained by the FSE instance.  However, this change had the
opposite effect.

Taken by itself, it made the class take up less memory, this is true.
 However, when considering that there could be sub-classes that have
to parse sub-parts of the original string, each sub-instance would
have it's very own copy of the internal chars that were in the parent
string.

Additionally, when the cache is in use, the memory doubles, because
the original String key, containing the original array of chars, is
kept around.

Switching each instance back to String usage won't help; while calling
String.subString may allow the internal array to be shared, it's not
guaranteed.  And, we'll still have the hit of the stored String's
internal hashCode variable(4 bytes), not to mention the additional
pointer to the string(another 4 bytes).

There are additional memory usage issues, when the sub classes taking
the original string/array, then concatenating the open/close brackets,
thereby causing a new String to be created.

I'm working on fixing all these issues, before I continue on fixing 3486.

[Adrian Crum-2]

--- On Sun, 2/21/10, Adam Heath <[hidden email]> wrote:

> From: Adam Heath <[hidden email]>
> Subject: FlexibleStringExpander is still using too much memory
> To: [hidden email]
> Date: Sunday, February 21, 2010, 12:18 PM
> Recently, Adrian decided to alter FSE
> to use less memory.  He did this
> by modifying the class to store a single char[], which
> represented the
> text contained by the FSE instance.  However, this
> change had the
> opposite effect.
>
> Taken by itself, it made the class take up less memory,
> this is true.
>  However, when considering that there could be sub-classes
> that have
> to parse sub-parts of the original string, each
> sub-instance would
> have it's very own copy of the internal chars that were in
> the parent
> string.
>
> Additionally, when the cache is in use, the memory doubles,
> because
> the original String key, containing the original array of
> chars, is
> kept around.
>
> Switching each instance back to String usage won't help;
> while calling
> String.subString may allow the internal array to be shared,
> it's not
> guaranteed.  And, we'll still have the hit of the
> stored String's
> internal hashCode variable(4 bytes), not to mention the
> additional
> pointer to the string(another 4 bytes).
>
> There are additional memory usage issues, when the sub
> classes taking
> the original string/array, then concatenating the
> open/close brackets,
> thereby causing a new String to be created.
>
> I'm working on fixing all these issues, before I continue
> on fixing 3486.

The specific memory use I tried to address was when an expression wasn't an expression - it was just a plain string. In those cases instead of taking up the space of a String object, you were taking up the space of the String object, plus a List, plus the FSE.

Those FSE-wrapped String constants are then kept in memory via the various model classes. To test this concept, I created a counter to count how many non-expression FSEs were created, then visited the artifact info page. More than 78,000 instances were created.

I might be missing something, or maybe I'm misunderstanding what you're saying, but it seems like you're saying there are memory issues with the cases where there *is* an expression (the ones that are cached). Those instances total a few thousand, and as you mentioned previously, they aren't taking up much memory.

-Adrian