Javascript include renderer

[Scott Gray-2]

Hi All,

I've just finished a freemarker transform that validates and combines javascript resources and then renders the script tags, I'm interested to know if it's something you want in the project.

Features:
- Makes sure that each script actually exists before telling the browser about it, removes any possibility of the browser 404'ing on javascripts.
- Automatically combines scripts into a single file to reduce the number of http requests the browser has to make
- Define groupings that control how the scripts are combined, e.g. if you have some scripts that are used on every page then you can combine them into their own group to take the most advantage of browser caching.  You simply use the "#" sign to append a group name to the end of each script path:
    <set field="layoutSettings.javaScripts[+0]" value="/images/jquery/plugins/datetimepicker/jquery-ui-timepicker-addon-0.9.1.min.js#core" global="true"/>
    <set field="layoutSettings.javaScripts[+0]" value="/images/jquery/ui/js/jquery-ui-1.8.6.custom.min.js#core" global="true"/>
- Use the #solo group name to indicate scripts that shouldn't be combined, useful if you're using some sort of CDN for your jQuery lib or whatever.
- Caches the results so that the processing for each set of resources happens only once

The idea is to remove the need for a balancing act between lots of smaller scripts but too many requests or a few large scripts with lots of unused code.  I figured since it changes something that hasn't changed in years I'd check before committing.  I could also just commit the transform but not use it OOTB.

I'm also planning something similar for CSS files but it's a little more complicated because of the @import statement that can appear in them.

Thanks
Scott

HotWax Media
http://www.hotwaxmedia.com

[Adam Heath-2]

Scott Gray wrote:

> Hi All,
>
> I've just finished a freemarker transform that validates and combines javascript resources and then renders the script tags, I'm interested to know if it's something you want in the project.
>
> Features:
> - Makes sure that each script actually exists before telling the browser about it, removes any possibility of the browser 404'ing on javascripts.
> - Automatically combines scripts into a single file to reduce the number of http requests the browser has to make
> - Define groupings that control how the scripts are combined, e.g. if you have some scripts that are used on every page then you can combine them into their own group to take the most advantage of browser caching.  You simply use the "#" sign to append a group name to the end of each script path:
>     <set field="layoutSettings.javaScripts[+0]" value="/images/jquery/plugins/datetimepicker/jquery-ui-timepicker-addon-0.9.1.min.js#core" global="true"/>
>     <set field="layoutSettings.javaScripts[+0]" value="/images/jquery/ui/js/jquery-ui-1.8.6.custom.min.js#core" global="true"/>
> - Use the #solo group name to indicate scripts that shouldn't be combined, useful if you're using some sort of CDN for your jQuery lib or whatever.
> - Caches the results so that the processing for each set of resources happens only once
>
> The idea is to remove the need for a balancing act between lots of smaller scripts but too many requests or a few large scripts with lots of unused code.  I figured since it changes something that hasn't changed in years I'd check before committing.  I could also just commit the transform but not use it OOTB.
>
> I'm also planning something similar for CSS files but it's a little more complicated because of the @import statement that can appear in them.

Be very careful about combining lots of javascript files into one.
You *must* do proper If-Modified-Since handling, or you will greatly
increase the bandwidth usage of client browsers.

Without a proper ETag, and If-Modified-Since handling, the browser
will download the combined javascript file repeatedly, on each page
load.  The solution, is a bit convoluted.
==
allETags = []
greatestLastModified = 0
for(file in files) {
  perFileETag = perFileLastModifiedTime + '-' + perFileSize
  allETags.add(perFileETag)
  if (perFileLastModifiedTime > greatestLastModified) {
    greatestLastModified = perFileLastModified
  }
}
etag = allEtags.join(':')

response.setHeader('ETag', etag)
response.setLastModifiedTime(greatestLastModified)
==
and...

in IMS mode, the previous ETag will be given.  Split that on ':',
iterate each part, comparing the lastModifiedTime and size to the
current list of files.

IMS mode is also different in HEAD and GET mode.

I suppose I should pull out the Binary handler in webslinger, as it
has support for all of this.

[Scott Gray-2]

On 20/12/2010, at 10:04 PM, Adam Heath wrote:

> Scott Gray wrote:
>> Hi All,
>>
>> I've just finished a freemarker transform that validates and combines javascript resources and then renders the script tags, I'm interested to know if it's something you want in the project.
>>
>> Features:
>> - Makes sure that each script actually exists before telling the browser about it, removes any possibility of the browser 404'ing on javascripts.
>> - Automatically combines scripts into a single file to reduce the number of http requests the browser has to make
>> - Define groupings that control how the scripts are combined, e.g. if you have some scripts that are used on every page then you can combine them into their own group to take the most advantage of browser caching.  You simply use the "#" sign to append a group name to the end of each script path:
>>    <set field="layoutSettings.javaScripts[+0]" value="/images/jquery/plugins/datetimepicker/jquery-ui-timepicker-addon-0.9.1.min.js#core" global="true"/>
>>    <set field="layoutSettings.javaScripts[+0]" value="/images/jquery/ui/js/jquery-ui-1.8.6.custom.min.js#core" global="true"/>
>> - Use the #solo group name to indicate scripts that shouldn't be combined, useful if you're using some sort of CDN for your jQuery lib or whatever.
>> - Caches the results so that the processing for each set of resources happens only once
>>
>> The idea is to remove the need for a balancing act between lots of smaller scripts but too many requests or a few large scripts with lots of unused code.  I figured since it changes something that hasn't changed in years I'd check before committing.  I could also just commit the transform but not use it OOTB.
>>
>> I'm also planning something similar for CSS files but it's a little more complicated because of the @import statement that can appear in them.
>
> Be very careful about combining lots of javascript files into one.
> You *must* do proper If-Modified-Since handling, or you will greatly
> increase the bandwidth usage of client browsers.
>
> Without a proper ETag, and If-Modified-Since handling, the browser
> will download the combined javascript file repeatedly, on each page
> load.  The solution, is a bit convoluted.
> ==
> allETags = []
> greatestLastModified = 0
> for(file in files) {
>  perFileETag = perFileLastModifiedTime + '-' + perFileSize
>  allETags.add(perFileETag)
>  if (perFileLastModifiedTime > greatestLastModified) {
>    greatestLastModified = perFileLastModified
>  }
> }
> etag = allEtags.join(':')
>
> response.setHeader('ETag', etag)
> response.setLastModifiedTime(greatestLastModified)
> ==
> and...
>
> in IMS mode, the previous ETag will be given.  Split that on ':',
> iterate each part, comparing the lastModifiedTime and size to the
> current list of files.
>
> IMS mode is also different in HEAD and GET mode.
>
> I suppose I should pull out the Binary handler in webslinger, as it
> has support for all of this.
>

The combined files are written to runtime/tempfiles (tempfiles being a webapp mount point) and only regenerated when the cache entry is removed, in production you wouldn't set an expireTime so they'd essentially only be regenerated if the instance is restarted.  No special handling should be necessary, the app or http server would set the headers in the same way it does for the static javascript files.

Regards
Scott

[Adam Heath-2]

Scott Gray wrote:

> On 20/12/2010, at 10:04 PM, Adam Heath wrote:
>
>> Be very careful about combining lots of javascript files into one.
>> You *must* do proper If-Modified-Since handling, or you will greatly
>> increase the bandwidth usage of client browsers.
>>
>> Without a proper ETag, and If-Modified-Since handling, the browser
>> will download the combined javascript file repeatedly, on each page
>> load.  The solution, is a bit convoluted.
>> ==
>> allETags = []
>> greatestLastModified = 0
>> for(file in files) {
>>  perFileETag = perFileLastModifiedTime + '-' + perFileSize
>>  allETags.add(perFileETag)
>>  if (perFileLastModifiedTime > greatestLastModified) {
>>    greatestLastModified = perFileLastModified
>>  }
>> }
>> etag = allEtags.join(':')
>>
>> response.setHeader('ETag', etag)
>> response.setLastModifiedTime(greatestLastModified)
>> ==
>> and...
>>
>> in IMS mode, the previous ETag will be given.  Split that on ':',
>> iterate each part, comparing the lastModifiedTime and size to the
>> current list of files.
>>
>> IMS mode is also different in HEAD and GET mode.
>>
>> I suppose I should pull out the Binary handler in webslinger, as it
>> has support for all of this.
>>
>
> The combined files are written to runtime/tempfiles (tempfiles being a webapp mount point) and only regenerated when the cache entry is removed, in production you wouldn't set an expireTime so they'd essentially only be regenerated if the instance is restarted.  No special handling should be necessary, the app or http server would set the headers in the same way it does for the static javascript files.

Why rewrite the combined files into tempfiles at all?  Just iterate
over them each time during serve.  No need to use extra disk space.

Would the cache have to cleared anytime one of the member files is
modified on disk?  If so, then ew.  That's very sucky.

Then, if the multi-binary code were extracted and reused, and a member
file was very large, the copying to a on-disk cache entry is another
hit to be concerned about.  Being able to have read-only access for
various subsystems is a nice goal to have.

Webslinger already supports multi-binary javascript and css(but not
@import; multi-css is only used in the base code we have, which don't
use @import).  We developed this code in response to the firefox yslow
plugin.  You will find that in multi-binary mode, @import should not
really be used, as the multi mode is more efficient.

Our etag output is also more efficient than both apache and catalina.
 They use $lastModified-$size, but use a base-10 encoding.  We use a
base-64 encoding, to reduce the size of the response headers(and
subsequent request header in IMS mode).

The reason for encoding the lastModified time for each file into the
etag, btw, is so that you can detect when a file was reverted to
something previous.  This could happen by checking out an old version,
with a rcs that restores timestamps, or by just copying a previously
saved version from another location.

I've thought about doing something similiar to gwt, in that it uses a
hashsum(can't recall if it is sha1 or md5, or something else) for each
static file.  If a multi-binary renderer can place the combined url
into the page(the webslinger version doesn't support this), then
adding some hash->list-of-files wouldn't be hard to do.

[Scott Gray-2]

On 20/12/2010, at 10:36 PM, Adam Heath wrote:

> Scott Gray wrote:
>> On 20/12/2010, at 10:04 PM, Adam Heath wrote:
>>
>>> Be very careful about combining lots of javascript files into one.
>>> You *must* do proper If-Modified-Since handling, or you will greatly
>>> increase the bandwidth usage of client browsers.
>>>
>>> Without a proper ETag, and If-Modified-Since handling, the browser
>>> will download the combined javascript file repeatedly, on each page
>>> load.  The solution, is a bit convoluted.
>>> ==
>>> allETags = []
>>> greatestLastModified = 0
>>> for(file in files) {
>>> perFileETag = perFileLastModifiedTime + '-' + perFileSize
>>> allETags.add(perFileETag)
>>> if (perFileLastModifiedTime > greatestLastModified) {
>>>   greatestLastModified = perFileLastModified
>>> }
>>> }
>>> etag = allEtags.join(':')
>>>
>>> response.setHeader('ETag', etag)
>>> response.setLastModifiedTime(greatestLastModified)
>>> ==
>>> and...
>>>
>>> in IMS mode, the previous ETag will be given.  Split that on ':',
>>> iterate each part, comparing the lastModifiedTime and size to the
>>> current list of files.
>>>
>>> IMS mode is also different in HEAD and GET mode.
>>>
>>> I suppose I should pull out the Binary handler in webslinger, as it
>>> has support for all of this.
>>>
>>
>> The combined files are written to runtime/tempfiles (tempfiles being a webapp mount point) and only regenerated when the cache entry is removed, in production you wouldn't set an expireTime so they'd essentially only be regenerated if the instance is restarted.  No special handling should be necessary, the app or http server would set the headers in the same way it does for the static javascript files.
>
> Why rewrite the combined files into tempfiles at all?  Just iterate
> over them each time during serve.  No need to use extra disk space.
>
> Would the cache have to cleared anytime one of the member files is
> modified on disk?  If so, then ew.  That's very sucky.

Well you would use a short expireTime for development, same as for a million other things in OFBiz.  I'll post the code to jira if anyone wants to use it while they wait for the superior solution.

Regards
Scott