Slim-down continue [was Re: svn commit: r1340414 - ...]

[Jacques Le Roux]

Quick questions (seems that there is a lack of internal documentation to say the least), there are not only addressed to Adam...

About javacc dependency do we really need it *OOTB*?
Can't we use rather externally maintained ant targets like in ant-contrib? For instance pleaser read
http://markmail.org/message/lidw73cuzyfr6cic

What is framework\sql used for *OOTB*?

What is the jjtree task used for *OOTB*?

In other words what are the benefits for the Apache OFBiz projet from those marvellous pieces of code (a "bit" undocumented though)

Jacques
PS : you can refer to https://issues.apache.org/jira/browse/OFBIZ-4833: "Housekeeping of jar files"

From: "Jacques Le Roux" <[hidden email]>

> From: "Adam Heath" <[hidden email]>
>> And, this commit was done wrong in the first place.  If you remove a library, remove it's entry from *all* locations, including
>> NOTICE, LICENSE, build.xml(s).  Do a grep on it's name.
>
> Indeed
>
>> In addition, each library removal should have been done as a *separate* change, not all bundled like this.  Separate allows for
>> debugging things when something goes wrong in the future, and one has to revert back to previous versions to figure it out(also
>> known as git bisect).
>
> Right
>
> Jacques
>
>> On 05/19/2012 07:24 PM, Scott Gray wrote:
>>> +1, I'm really sick and tired of this Jacques.  The lack of care shown in obvious places like this makes me very concerned about
>>> all of your commits in general, I really don't want to have to start closely reviewing every single one.  Given that you are
>>> OFBiz's most prolific committer by a large margin, you're taking time away from the other committers by forcing us to review all
>>> of your work and that's just not fair.  I regularly see basic problems with your commits but I just don't have the energy to
>>> respond most of the time unless it's obviously going to break something.
>>>
>>> Before committing please just stop and ask yourself a few questions:
>>> - Do I fully understand the problem I'm about to try and solve?
>>> - Have a taken all practical steps to ensure this change won't break anything?
>>> - Is this the best solution to the problem?
>>>
>>> If the answer to any of the above is no then just stop, wait, think and discuss.
>>>
>>> Thanks
>>> Scott
>>>
>>> On 20/05/2012, at 12:52 AM, Adrian Crum wrote:
>>>
>>>> Jacques,
>>>>
>>>> I am going to repeat what Scott said the other day:
>>>>
>>>> Before you commit, run ant clean-all load-demo run-tests.
>>>>
>>>> This lack of care in the commit process needs to stop.
>>>>
>>>> -Adrian
>>>>
>>>> On 5/19/2012 1:41 PM, Adrian Crum wrote:
>>>>> The project will not compile after this change.
>>>>>
>>>>> -Adrian
>>>>>
>>>>> On 5/19/2012 11:57 AM, [hidden email] wrote:
>>>>>> Author: jleroux
>>>>>> Date: Sat May 19 10:57:40 2012
>>>>>> New Revision: 1340414
>>>>>>
>>>>>> URL: http://svn.apache.org/viewvc?rev=1340414&view=rev
>>>>>> Log:
>>>>>> Following
>>>>>> https://issues.apache.org/jira/browse/OFBIZ-4833?focusedCommentId=13275249&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13275249
>>>>>> Moves to Attic:
>>>>>>      framework/base/lib/ant/ant-nodeps-1.8.1.jar =>   can be removed IMO. It was added with ant 1.8 by Hans, not sure why (no
>>>>>> comment about need), seems to contains no needed stuff
>>>>>>      framework/base/lib/javacc can be removed IMO. The Java Compiler Compiler tm (JavaCC tm) - The Java Parser Generator, not
>>>>>> used OOTB, was added by Marco, not sure why (no comment about need)
>>>>>>      framework/base/lib/Tidy.jar was used at bottom of eCommerce pages before, no longer needed
>>>>>>      framework/base/lib/ant-trax-1.8.0.jar can be removed IMO. It was added with ant 1.8 by Hans, not sure why (no comment
>>>>>> about need), contains XSLT task
>>>>>>      framework/base/lib/commons/commons-vfs-20070730.jar certainly added with Webslinger can be removed now
>>>>>>
>>>>>> Removed:
>>>>>>      ofbiz/trunk/framework/base/lib/Tidy.jar
>>>>>>      ofbiz/trunk/framework/base/lib/ant-trax-1.8.0.jar
>>>>>>      ofbiz/trunk/framework/base/lib/ant/ant-nodeps-1.8.1.jar
>>>>>>      ofbiz/trunk/framework/base/lib/commons/commons-vfs-20070730.jar
>>>>>>      ofbiz/trunk/framework/base/lib/javacc/
>>>>>>      ofbiz/trunk/lib/
>>>>>>
>>>
>>
>

[Adrian Crum-3]

On 5/20/2012 10:01 AM, Jacques Le Roux wrote:
> Quick questions (seems that there is a lack of internal documentation
> to say the least), there are not only addressed to Adam...
>
> About javacc dependency do we really need it *OOTB*?
> Can't we use rather externally maintained ant targets like in
> ant-contrib? For instance pleaser read
> http://markmail.org/message/lidw73cuzyfr6cic
>
> What is framework\sql used for *OOTB*?

 From what I understand, the sql component parses SQL statements into
OFBiz entity conditions, and then executes the statement using the
entity engine. I don't think it is used OOTB for anything, but it could
be useful for integration with third-party libraries that take SQL
statements - like Jasper Reports.

-Adrian

[Adam Heath-2]

On 05/24/2012 08:15 AM, Adrian Crum wrote:

> On 5/20/2012 10:01 AM, Jacques Le Roux wrote:
>> Quick questions (seems that there is a lack of internal
>> documentation to say the least), there are not only addressed to
>> Adam...
>>
>> About javacc dependency do we really need it *OOTB*?
>> Can't we use rather externally maintained ant targets like in
>> ant-contrib? For instance pleaser read
>> http://markmail.org/message/lidw73cuzyfr6cic
>>
>> What is framework\sql used for *OOTB*?
>
> From what I understand, the sql component parses SQL statements into
> OFBiz entity conditions, and then executes the statement using the
> entity engine. I don't think it is used OOTB for anything, but it
> could be useful for integration with third-party libraries that take
> SQL statements - like Jasper Reports.

You can also parse *raw* where clauses.

==
import org.ofbiz.sql.Parser;
import org.ofbiz.entity.sql.EntityPlanner;

String sqlConditionString = "contactMechId = ?contactMechId";

def sqlPlanner = new EntityPlanner();
def reader = new StringReader(sqlCondition);
def sqlCondition = new Parser(reader).EntityCondition();

while (true) {
        def condition = sqlPlanner.getConditionPlanner().parse(sqlCondition,
[contactMechId: '1']);
        delegator.findList(entityName, condition, ....)
}
==

I suppose I should place some of that in a util class(SQLUtil comes to
mind, it was never finished).

I support (), and, or, in, between, it's rather complete.

[Adam Heath-2]

On 05/24/2012 10:18 AM, Adam Heath wrote:

> On 05/24/2012 08:15 AM, Adrian Crum wrote:
>> On 5/20/2012 10:01 AM, Jacques Le Roux wrote:
>>> Quick questions (seems that there is a lack of internal
>>> documentation to say the least), there are not only addressed to
>>> Adam...
>>>
>>> About javacc dependency do we really need it *OOTB*?
>>> Can't we use rather externally maintained ant targets like in
>>> ant-contrib? For instance pleaser read
>>> http://markmail.org/message/lidw73cuzyfr6cic
>>>
>>> What is framework\sql used for *OOTB*?
>>
>> From what I understand, the sql component parses SQL statements into
>> OFBiz entity conditions, and then executes the statement using the
>> entity engine. I don't think it is used OOTB for anything, but it
>> could be useful for integration with third-party libraries that take
>> SQL statements - like Jasper Reports.
>
> You can also parse *raw* where clauses.
>
> ==
> import org.ofbiz.sql.Parser;
> import org.ofbiz.entity.sql.EntityPlanner;
>
> String sqlConditionString = "contactMechId = ?contactMechId";
>
> def sqlPlanner = new EntityPlanner();
> def reader = new StringReader(sqlCondition);
> def sqlCondition = new Parser(reader).EntityCondition();
>
> while (true) {
> def condition = sqlPlanner.getConditionPlanner().parse(sqlCondition,
> [contactMechId: '1']);
> delegator.findList(entityName, condition, ....)
> }
> ==
>
> I suppose I should place some of that in a util class(SQLUtil comes to
> mind, it was never finished).
>
> I support (), and, or, in, between, it's rather complete.

The idea was that you would parse the sqlCondition once, in a static
{} block somewhere, then at runtime, just build that map.  This
builds-upon the map/list idea inherent in ofbiz.

I also had plans that you could store sql strings into a properties
file somewhere, so that they could possibly be changed by the end-user
without having to recompile.

I need to revisit the "SELECT a.partyId, b.* EXCLUDE(partyId) FROM
Party a LEFT JOIN PartyContactMech b USING (partyId)", now that ofbiz
better supports conditions on the join clauses, esp. when combining
views into other views.

[Jacques Le Roux]

Thanks Adrian

Jacques

From: "Adrian Crum" <[hidden email]>

> On 5/20/2012 10:01 AM, Jacques Le Roux wrote:
>> Quick questions (seems that there is a lack of internal documentation
>> to say the least), there are not only addressed to Adam...
>>
>> About javacc dependency do we really need it *OOTB*?
>> Can't we use rather externally maintained ant targets like in
>> ant-contrib? For instance pleaser read
>> http://markmail.org/message/lidw73cuzyfr6cic
>>
>> What is framework\sql used for *OOTB*?
>
> From what I understand, the sql component parses SQL statements into
> OFBiz entity conditions, and then executes the statement using the
> entity engine. I don't think it is used OOTB for anything, but it could
> be useful for integration with third-party libraries that take SQL
> statements - like Jasper Reports.
>
> -Adrian
>

[Jacques Le Roux]

From: "Adam Heath" <[hidden email]>

> On 05/24/2012 10:18 AM, Adam Heath wrote:
>> On 05/24/2012 08:15 AM, Adrian Crum wrote:
>>> On 5/20/2012 10:01 AM, Jacques Le Roux wrote:
>>>> Quick questions (seems that there is a lack of internal
>>>> documentation to say the least), there are not only addressed to
>>>> Adam...
>>>>
>>>> About javacc dependency do we really need it *OOTB*?
>>>> Can't we use rather externally maintained ant targets like in
>>>> ant-contrib? For instance pleaser read
>>>> http://markmail.org/message/lidw73cuzyfr6cic
>>>>
>>>> What is framework\sql used for *OOTB*?
>>>
>>> From what I understand, the sql component parses SQL statements into
>>> OFBiz entity conditions, and then executes the statement using the
>>> entity engine. I don't think it is used OOTB for anything, but it
>>> could be useful for integration with third-party libraries that take
>>> SQL statements - like Jasper Reports.
>>
>> You can also parse *raw* where clauses.
>>
>> ==
>> import org.ofbiz.sql.Parser;
>> import org.ofbiz.entity.sql.EntityPlanner;
>>
>> String sqlConditionString = "contactMechId = ?contactMechId";
>>
>> def sqlPlanner = new EntityPlanner();
>> def reader = new StringReader(sqlCondition);
>> def sqlCondition = new Parser(reader).EntityCondition();
>>
>> while (true) {
>> def condition = sqlPlanner.getConditionPlanner().parse(sqlCondition,
>> [contactMechId: '1']);
>> delegator.findList(entityName, condition, ....)
>> }
>> ==
>>
>> I suppose I should place some of that in a util class(SQLUtil comes to
>> mind, it was never finished).
>>
>> I support (), and, or, in, between, it's rather complete.
>
> The idea was that you would parse the sqlCondition once, in a static
> {} block somewhere, then at runtime, just build that map.  This
> builds-upon the map/list idea inherent in ofbiz.
>
> I also had plans that you could store sql strings into a properties
> file somewhere, so that they could possibly be changed by the end-user
> without having to recompile.
>
> I need to revisit the "SELECT a.partyId, b.* EXCLUDE(partyId) FROM
> Party a LEFT JOIN PartyContactMech b USING (partyId)", now that ofbiz
> better supports conditions on the join clauses, esp. when combining
> views into other views.

Thanks for the explanation,

So should we not rather create a Jira with all the needed in a patch until this is finished?
Or maybe a branch would be easier?

Still with the slim-down idea in mind and as objective.

Jacques

[Adam Heath-2]

On 05/24/2012 04:05 PM, Jacques Le Roux wrote:

> From: "Adam Heath" <[hidden email]>
>> On 05/24/2012 10:18 AM, Adam Heath wrote:
>>> On 05/24/2012 08:15 AM, Adrian Crum wrote:
>>>> On 5/20/2012 10:01 AM, Jacques Le Roux wrote:
>>>>> Quick questions (seems that there is a lack of internal
>>>>> documentation to say the least), there are not only addressed to
>>>>> Adam...
>>>>>
>>>>> About javacc dependency do we really need it *OOTB*?
>>>>> Can't we use rather externally maintained ant targets like in
>>>>> ant-contrib? For instance pleaser read
>>>>> http://markmail.org/message/lidw73cuzyfr6cic
>>>>>
>>>>> What is framework\sql used for *OOTB*?
>>>>
>>>> From what I understand, the sql component parses SQL statements into
>>>> OFBiz entity conditions, and then executes the statement using the
>>>> entity engine. I don't think it is used OOTB for anything, but it
>>>> could be useful for integration with third-party libraries that take
>>>> SQL statements - like Jasper Reports.
>>>
>>> You can also parse *raw* where clauses.
>>>
>>> ==
>>> import org.ofbiz.sql.Parser;
>>> import org.ofbiz.entity.sql.EntityPlanner;
>>>
>>> String sqlConditionString = "contactMechId = ?contactMechId";
>>>
>>> def sqlPlanner = new EntityPlanner();
>>> def reader = new StringReader(sqlCondition);
>>> def sqlCondition = new Parser(reader).EntityCondition();
>>>
>>> while (true) {
>>> def condition = sqlPlanner.getConditionPlanner().parse(sqlCondition,
>>> [contactMechId: '1']);
>>> delegator.findList(entityName, condition, ....)
>>> }
>>> ==
>>>
>>> I suppose I should place some of that in a util class(SQLUtil comes to
>>> mind, it was never finished).
>>>
>>> I support (), and, or, in, between, it's rather complete.
>>
>> The idea was that you would parse the sqlCondition once, in a static
>> {} block somewhere, then at runtime, just build that map. This
>> builds-upon the map/list idea inherent in ofbiz.
>>
>> I also had plans that you could store sql strings into a properties
>> file somewhere, so that they could possibly be changed by the end-user
>> without having to recompile.
>>
>> I need to revisit the "SELECT a.partyId, b.* EXCLUDE(partyId) FROM
>> Party a LEFT JOIN PartyContactMech b USING (partyId)", now that ofbiz
>> better supports conditions on the join clauses, esp. when combining
>> views into other views.
>
> Thanks for the explanation,
>
> So should we not rather create a Jira with all the needed in a patch
> until this is finished?
> Or maybe a branch would be easier?
>
> Still with the slim-down idea in mind and as objective.

I like the slimdown, but tbh, I would like to see the framework/sql
stuff used more than it is(0 right now).  Andrew Zeneski was an original
requestor for something that parsed sql into EntityCondition.  I took
his suggestion, but went further, to allow CREATE VIEW AS SELECT to work.

I've noticed that there aren't that many view definitions in ofbiz.  As
I've been deprecating all this code recently, I've noticed java code
doing nested loop kinda-stuff, instead of just doing a view.  I'm
guessing because view-xml is verbose and not how people actually think.

However, with what I committed, you can define the view using a SQL
syntax, which is then backed by a DynamicViewEntity.

I've seen rather impressive speedups just rewriting code to a single SQL
query, instead of java loops; the database can be rather efficient.  So
making view writing simpler is a laudable goal.

[Jacques Le Roux]

From: "Adam Heath" <[hidden email]>

> On 05/24/2012 04:05 PM, Jacques Le Roux wrote:
>> From: "Adam Heath" <[hidden email]>
>>> On 05/24/2012 10:18 AM, Adam Heath wrote:
>>>> On 05/24/2012 08:15 AM, Adrian Crum wrote:
>>>>> On 5/20/2012 10:01 AM, Jacques Le Roux wrote:
>>>>>> Quick questions (seems that there is a lack of internal
>>>>>> documentation to say the least), there are not only addressed to
>>>>>> Adam...
>>>>>>
>>>>>> About javacc dependency do we really need it *OOTB*?
>>>>>> Can't we use rather externally maintained ant targets like in
>>>>>> ant-contrib? For instance pleaser read
>>>>>> http://markmail.org/message/lidw73cuzyfr6cic
>>>>>>
>>>>>> What is framework\sql used for *OOTB*?
>>>>>
>>>>> From what I understand, the sql component parses SQL statements into
>>>>> OFBiz entity conditions, and then executes the statement using the
>>>>> entity engine. I don't think it is used OOTB for anything, but it
>>>>> could be useful for integration with third-party libraries that take
>>>>> SQL statements - like Jasper Reports.
>>>>
>>>> You can also parse *raw* where clauses.
>>>>
>>>> ==
>>>> import org.ofbiz.sql.Parser;
>>>> import org.ofbiz.entity.sql.EntityPlanner;
>>>>
>>>> String sqlConditionString = "contactMechId = ?contactMechId";
>>>>
>>>> def sqlPlanner = new EntityPlanner();
>>>> def reader = new StringReader(sqlCondition);
>>>> def sqlCondition = new Parser(reader).EntityCondition();
>>>>
>>>> while (true) {
>>>> def condition = sqlPlanner.getConditionPlanner().parse(sqlCondition,
>>>> [contactMechId: '1']);
>>>> delegator.findList(entityName, condition, ....)
>>>> }
>>>> ==
>>>>
>>>> I suppose I should place some of that in a util class(SQLUtil comes to
>>>> mind, it was never finished).
>>>>
>>>> I support (), and, or, in, between, it's rather complete.
>>>
>>> The idea was that you would parse the sqlCondition once, in a static
>>> {} block somewhere, then at runtime, just build that map. This
>>> builds-upon the map/list idea inherent in ofbiz.
>>>
>>> I also had plans that you could store sql strings into a properties
>>> file somewhere, so that they could possibly be changed by the end-user
>>> without having to recompile.
>>>
>>> I need to revisit the "SELECT a.partyId, b.* EXCLUDE(partyId) FROM
>>> Party a LEFT JOIN PartyContactMech b USING (partyId)", now that ofbiz
>>> better supports conditions on the join clauses, esp. when combining
>>> views into other views.
>>
>> Thanks for the explanation,
>>
>> So should we not rather create a Jira with all the needed in a patch
>> until this is finished?
>> Or maybe a branch would be easier?
>>
>> Still with the slim-down idea in mind and as objective.
>
> I like the slimdown, but tbh, I would like to see the framework/sql stuff used more than it is(0 right now).  Andrew Zeneski was
> an original requestor for something that parsed sql into EntityCondition.  I took his suggestion, but went further, to allow
> CREATE VIEW AS SELECT to work.
>
> I've noticed that there aren't that many view definitions in ofbiz.  As I've been deprecating all this code recently, I've noticed
> java code doing nested loop kinda-stuff, instead of just doing a view.  I'm guessing because view-xml is verbose and not how
> people actually think.
>
> However, with what I committed, you can define the view using a SQL syntax, which is then backed by a DynamicViewEntity.
>
> I've seen rather impressive speedups just rewriting code to a single SQL query, instead of java loops; the database can be rather
> efficient.  So making view writing simpler is a laudable goal.

Great, but still, why not a branch as long as it's not finished?

Also something which I think is pretty neat in the principle (I still did not review the code) and would speed up views:
https://issues.apache.org/jira/browse/OFBIZ-4041

Jacques

[Jacques Le Roux]

From: "Jacques Le Roux" <[hidden email]>

> From: "Adam Heath" <[hidden email]>
>> On 05/24/2012 04:05 PM, Jacques Le Roux wrote:
>>> From: "Adam Heath" <[hidden email]>
>>>> On 05/24/2012 10:18 AM, Adam Heath wrote:
>>>>> On 05/24/2012 08:15 AM, Adrian Crum wrote:
>>>>>> On 5/20/2012 10:01 AM, Jacques Le Roux wrote:
>>>>>>> Quick questions (seems that there is a lack of internal
>>>>>>> documentation to say the least), there are not only addressed to
>>>>>>> Adam...
>>>>>>>
>>>>>>> About javacc dependency do we really need it *OOTB*?
>>>>>>> Can't we use rather externally maintained ant targets like in
>>>>>>> ant-contrib? For instance pleaser read
>>>>>>> http://markmail.org/message/lidw73cuzyfr6cic
>>>>>>>
>>>>>>> What is framework\sql used for *OOTB*?
>>>>>>
>>>>>> From what I understand, the sql component parses SQL statements into
>>>>>> OFBiz entity conditions, and then executes the statement using the
>>>>>> entity engine. I don't think it is used OOTB for anything, but it
>>>>>> could be useful for integration with third-party libraries that take
>>>>>> SQL statements - like Jasper Reports.
>>>>>
>>>>> You can also parse *raw* where clauses.
>>>>>
>>>>> ==
>>>>> import org.ofbiz.sql.Parser;
>>>>> import org.ofbiz.entity.sql.EntityPlanner;
>>>>>
>>>>> String sqlConditionString = "contactMechId = ?contactMechId";
>>>>>
>>>>> def sqlPlanner = new EntityPlanner();
>>>>> def reader = new StringReader(sqlCondition);
>>>>> def sqlCondition = new Parser(reader).EntityCondition();
>>>>>
>>>>> while (true) {
>>>>> def condition = sqlPlanner.getConditionPlanner().parse(sqlCondition,
>>>>> [contactMechId: '1']);
>>>>> delegator.findList(entityName, condition, ....)
>>>>> }
>>>>> ==
>>>>>
>>>>> I suppose I should place some of that in a util class(SQLUtil comes to
>>>>> mind, it was never finished).
>>>>>
>>>>> I support (), and, or, in, between, it's rather complete.
>>>>
>>>> The idea was that you would parse the sqlCondition once, in a static
>>>> {} block somewhere, then at runtime, just build that map. This
>>>> builds-upon the map/list idea inherent in ofbiz.
>>>>
>>>> I also had plans that you could store sql strings into a properties
>>>> file somewhere, so that they could possibly be changed by the end-user
>>>> without having to recompile.
>>>>
>>>> I need to revisit the "SELECT a.partyId, b.* EXCLUDE(partyId) FROM
>>>> Party a LEFT JOIN PartyContactMech b USING (partyId)", now that ofbiz
>>>> better supports conditions on the join clauses, esp. when combining
>>>> views into other views.
>>>
>>> Thanks for the explanation,
>>>
>>> So should we not rather create a Jira with all the needed in a patch
>>> until this is finished?
>>> Or maybe a branch would be easier?
>>>
>>> Still with the slim-down idea in mind and as objective.
>>
>> I like the slimdown, but tbh, I would like to see the framework/sql stuff used more than it is(0 right now).  Andrew Zeneski was
>> an original requestor for something that parsed sql into EntityCondition.  I took his suggestion, but went further, to allow
>> CREATE VIEW AS SELECT to work.
>>
>> I've noticed that there aren't that many view definitions in ofbiz.  As I've been deprecating all this code recently, I've
>> noticed java code doing nested loop kinda-stuff, instead of just doing a view.  I'm guessing because view-xml is verbose and not
>> how people actually think.
>>
>> However, with what I committed, you can define the view using a SQL syntax, which is then backed by a DynamicViewEntity.
>>
>> I've seen rather impressive speedups just rewriting code to a single SQL query, instead of java loops; the database can be rather
>> efficient.  So making view writing simpler is a laudable goal.
>
> Great, but still, why not a branch as long as it's not finished?
>
> Also something which I think is pretty neat in the principle (I still did not review the code) and would speed up views:
> https://issues.apache.org/jira/browse/OFBIZ-4041
>
> Jacques

BTW another stuff that could be part of this branch https://issues.apache.org/jira/browse/OFBIZ-3575

Jacques

[Pierre Smits]

May be doing a little example code that shows how it works and the
performance impact can help developers with implementation in various areas.

Regards,

Pierre

2012/5/25 Adam Heath <[hidden email]>

> On 05/24/2012 04:05 PM, Jacques Le Roux wrote:
>
>> From: "Adam Heath" <[hidden email]>
>>
>>> On 05/24/2012 10:18 AM, Adam Heath wrote:
>>>
>>>> On 05/24/2012 08:15 AM, Adrian Crum wrote:
>>>>
>>>>> On 5/20/2012 10:01 AM, Jacques Le Roux wrote:
>>>>>
>>>>>> Quick questions (seems that there is a lack of internal
>>>>>> documentation to say the least), there are not only addressed to
>>>>>> Adam...
>>>>>>
>>>>>> About javacc dependency do we really need it *OOTB*?
>>>>>> Can't we use rather externally maintained ant targets like in
>>>>>> ant-contrib? For instance pleaser read
>>>>>> http://markmail.org/message/**lidw73cuzyfr6cic<http://markmail.org/message/lidw73cuzyfr6cic>
>>>>>>
>>>>>> What is framework\sql used for *OOTB*?
>>>>>>
>>>>>
>>>>> From what I understand, the sql component parses SQL statements into
>>>>> OFBiz entity conditions, and then executes the statement using the
>>>>> entity engine. I don't think it is used OOTB for anything, but it
>>>>> could be useful for integration with third-party libraries that take
>>>>> SQL statements - like Jasper Reports.
>>>>>
>>>>
>>>> You can also parse *raw* where clauses.
>>>>
>>>> ==
>>>> import org.ofbiz.sql.Parser;
>>>> import org.ofbiz.entity.sql.**EntityPlanner;
>>>>
>>>> String sqlConditionString = "contactMechId = ?contactMechId";
>>>>
>>>> def sqlPlanner = new EntityPlanner();
>>>> def reader = new StringReader(sqlCondition);
>>>> def sqlCondition = new Parser(reader).**EntityCondition();
>>>>
>>>> while (true) {
>>>> def condition = sqlPlanner.**getConditionPlanner().parse(**
>>>> sqlCondition,
>>>> [contactMechId: '1']);
>>>> delegator.findList(entityName, condition, ....)
>>>> }
>>>> ==
>>>>
>>>> I suppose I should place some of that in a util class(SQLUtil comes to
>>>> mind, it was never finished).
>>>>
>>>> I support (), and, or, in, between, it's rather complete.
>>>>
>>>
>>> The idea was that you would parse the sqlCondition once, in a static
>>> {} block somewhere, then at runtime, just build that map. This
>>> builds-upon the map/list idea inherent in ofbiz.
>>>
>>> I also had plans that you could store sql strings into a properties
>>> file somewhere, so that they could possibly be changed by the end-user
>>> without having to recompile.
>>>
>>> I need to revisit the "SELECT a.partyId, b.* EXCLUDE(partyId) FROM
>>> Party a LEFT JOIN PartyContactMech b USING (partyId)", now that ofbiz
>>> better supports conditions on the join clauses, esp. when combining
>>> views into other views.
>>>
>>
>> Thanks for the explanation,
>>
>> So should we not rather create a Jira with all the needed in a patch
>> until this is finished?
>> Or maybe a branch would be easier?
>>
>> Still with the slim-down idea in mind and as objective.
>>
>
> I like the slimdown, but tbh, I would like to see the framework/sql stuff
> used more than it is(0 right now).  Andrew Zeneski was an original
> requestor for something that parsed sql into EntityCondition.  I took his
> suggestion, but went further, to allow CREATE VIEW AS SELECT to work.
>
> I've noticed that there aren't that many view definitions in ofbiz.  As
> I've been deprecating all this code recently, I've noticed java code doing
> nested loop kinda-stuff, instead of just doing a view.  I'm guessing
> because view-xml is verbose and not how people actually think.
>
> However, with what I committed, you can define the view using a SQL
> syntax, which is then backed by a DynamicViewEntity.
>
> I've seen rather impressive speedups just rewriting code to a single SQL
> query, instead of java loops; the database can be rather efficient.  So
> making view writing simpler is a laudable goal.
>

[Adam Heath-2]

On 05/25/2012 12:26 AM, Jacques Le Roux wrote:

> From: "Jacques Le Roux" <[hidden email]>
>> From: "Adam Heath" <[hidden email]>
>>> On 05/24/2012 04:05 PM, Jacques Le Roux wrote:
>>>> From: "Adam Heath" <[hidden email]>
>>>>> On 05/24/2012 10:18 AM, Adam Heath wrote:
>>>>> The idea was that you would parse the sqlCondition once, in a static
>>>>> {} block somewhere, then at runtime, just build that map. This
>>>>> builds-upon the map/list idea inherent in ofbiz.
>>>>>
>>>>> I also had plans that you could store sql strings into a properties
>>>>> file somewhere, so that they could possibly be changed by the end-user
>>>>> without having to recompile.
>>>>>
>>>>> I need to revisit the "SELECT a.partyId, b.* EXCLUDE(partyId) FROM
>>>>> Party a LEFT JOIN PartyContactMech b USING (partyId)", now that ofbiz
>>>>> better supports conditions on the join clauses, esp. when combining
>>>>> views into other views.
>>>>
>>>> Thanks for the explanation,
>>>>
>>>> So should we not rather create a Jira with all the needed in a patch
>>>> until this is finished?
>>>> Or maybe a branch would be easier?
>>>>
>>>> Still with the slim-down idea in mind and as objective.
>>>
>>> I like the slimdown, but tbh, I would like to see the framework/sql
>>> stuff used more than it is(0 right now). Andrew Zeneski was an
>>> original requestor for something that parsed sql into
>>> EntityCondition. I took his suggestion, but went further, to allow
>>> CREATE VIEW AS SELECT to work.
>>>
>>> I've noticed that there aren't that many view definitions in ofbiz.
>>> As I've been deprecating all this code recently, I've noticed java
>>> code doing nested loop kinda-stuff, instead of just doing a view. I'm
>>> guessing because view-xml is verbose and not how people actually think.
>>>
>>> However, with what I committed, you can define the view using a SQL
>>> syntax, which is then backed by a DynamicViewEntity.
>>>
>>> I've seen rather impressive speedups just rewriting code to a single
>>> SQL query, instead of java loops; the database can be rather
>>> efficient. So making view writing simpler is a laudable goal.
>>
>> Great, but still, why not a branch as long as it's not finished?
>>
>> Also something which I think is pretty neat in the principle (I still
>> did not review the code) and would speed up views:
>> https://issues.apache.org/jira/browse/OFBIZ-4041
>>
>> Jacques
>
> BTW another stuff that could be part of this branch
> https://issues.apache.org/jira/browse/OFBIZ-3575

Ok, I suppose.  This weekend I'll create such a branch to
fix/improve the view system.  This will also attempt to fix the reverse
cache clearing issues.

[Jacques Le Roux]

From: "Adam Heath" <[hidden email]>

> On 05/25/2012 12:26 AM, Jacques Le Roux wrote:
>> From: "Jacques Le Roux" <[hidden email]>
>>> From: "Adam Heath" <[hidden email]>
>>>> On 05/24/2012 04:05 PM, Jacques Le Roux wrote:
>>>>> From: "Adam Heath" <[hidden email]>
>>>>>> On 05/24/2012 10:18 AM, Adam Heath wrote:
>>>>>> The idea was that you would parse the sqlCondition once, in a static
>>>>>> {} block somewhere, then at runtime, just build that map. This
>>>>>> builds-upon the map/list idea inherent in ofbiz.
>>>>>>
>>>>>> I also had plans that you could store sql strings into a properties
>>>>>> file somewhere, so that they could possibly be changed by the end-user
>>>>>> without having to recompile.
>>>>>>
>>>>>> I need to revisit the "SELECT a.partyId, b.* EXCLUDE(partyId) FROM
>>>>>> Party a LEFT JOIN PartyContactMech b USING (partyId)", now that ofbiz
>>>>>> better supports conditions on the join clauses, esp. when combining
>>>>>> views into other views.
>>>>>
>>>>> Thanks for the explanation,
>>>>>
>>>>> So should we not rather create a Jira with all the needed in a patch
>>>>> until this is finished?
>>>>> Or maybe a branch would be easier?
>>>>>
>>>>> Still with the slim-down idea in mind and as objective.
>>>>
>>>> I like the slimdown, but tbh, I would like to see the framework/sql
>>>> stuff used more than it is(0 right now). Andrew Zeneski was an
>>>> original requestor for something that parsed sql into
>>>> EntityCondition. I took his suggestion, but went further, to allow
>>>> CREATE VIEW AS SELECT to work.
>>>>
>>>> I've noticed that there aren't that many view definitions in ofbiz.
>>>> As I've been deprecating all this code recently, I've noticed java
>>>> code doing nested loop kinda-stuff, instead of just doing a view. I'm
>>>> guessing because view-xml is verbose and not how people actually think.
>>>>
>>>> However, with what I committed, you can define the view using a SQL
>>>> syntax, which is then backed by a DynamicViewEntity.
>>>>
>>>> I've seen rather impressive speedups just rewriting code to a single
>>>> SQL query, instead of java loops; the database can be rather
>>>> efficient. So making view writing simpler is a laudable goal.
>>>
>>> Great, but still, why not a branch as long as it's not finished?
>>>
>>> Also something which I think is pretty neat in the principle (I still
>>> did not review the code) and would speed up views:
>>> https://issues.apache.org/jira/browse/OFBIZ-4041
>>>
>>> Jacques
>>
>> BTW another stuff that could be part of this branch
>> https://issues.apache.org/jira/browse/OFBIZ-3575
>
> Ok, I suppose.  This weekend I'll create such a branch to
> fix/improve the view system.  This will also attempt to fix the reverse
> cache clearing issues.

Count me on it... if I can help you...

Jacques

[Adam Heath-2]

On 05/25/2012 01:22 AM, Adam Heath wrote:

> On 05/25/2012 12:26 AM, Jacques Le Roux wrote:
>> From: "Jacques Le Roux" <[hidden email]>
>>> From: "Adam Heath" <[hidden email]>
>>>> On 05/24/2012 04:05 PM, Jacques Le Roux wrote:
>>>>> From: "Adam Heath" <[hidden email]>
>>>>>> On 05/24/2012 10:18 AM, Adam Heath wrote:
>>>>>> The idea was that you would parse the sqlCondition once, in a static
>>>>>> {} block somewhere, then at runtime, just build that map. This
>>>>>> builds-upon the map/list idea inherent in ofbiz.
>>>>>>
>>>>>> I also had plans that you could store sql strings into a properties
>>>>>> file somewhere, so that they could possibly be changed by the
>>>>>> end-user
>>>>>> without having to recompile.
>>>>>>
>>>>>> I need to revisit the "SELECT a.partyId, b.* EXCLUDE(partyId) FROM
>>>>>> Party a LEFT JOIN PartyContactMech b USING (partyId)", now that ofbiz
>>>>>> better supports conditions on the join clauses, esp. when combining
>>>>>> views into other views.
>>>>>
>>>>> Thanks for the explanation,
>>>>>
>>>>> So should we not rather create a Jira with all the needed in a patch
>>>>> until this is finished?
>>>>> Or maybe a branch would be easier?
>>>>>
>>>>> Still with the slim-down idea in mind and as objective.
>>>>
>>>> I like the slimdown, but tbh, I would like to see the framework/sql
>>>> stuff used more than it is(0 right now). Andrew Zeneski was an
>>>> original requestor for something that parsed sql into
>>>> EntityCondition. I took his suggestion, but went further, to allow
>>>> CREATE VIEW AS SELECT to work.
>>>>
>>>> I've noticed that there aren't that many view definitions in ofbiz.
>>>> As I've been deprecating all this code recently, I've noticed java
>>>> code doing nested loop kinda-stuff, instead of just doing a view. I'm
>>>> guessing because view-xml is verbose and not how people actually think.
>>>>
>>>> However, with what I committed, you can define the view using a SQL
>>>> syntax, which is then backed by a DynamicViewEntity.
>>>>
>>>> I've seen rather impressive speedups just rewriting code to a single
>>>> SQL query, instead of java loops; the database can be rather
>>>> efficient. So making view writing simpler is a laudable goal.
>>>
>>> Great, but still, why not a branch as long as it's not finished?
>>>
>>> Also something which I think is pretty neat in the principle (I still
>>> did not review the code) and would speed up views:
>>> https://issues.apache.org/jira/browse/OFBIZ-4041
>>>
>>> Jacques
>>
>> BTW another stuff that could be part of this branch
>> https://issues.apache.org/jira/browse/OFBIZ-3575
>
> Ok, I suppose. This weekend I'll create such a branch to
> fix/improve the view system. This will also attempt to fix the reverse
> cache clearing issues.

branches/improved-entityengine-features-20120528

Also see 1343540, which adds a README that has some things that we might
want to implement in the branch.

[Jacques Le Roux]

Adam Heath wrote:

> On 05/25/2012 01:22 AM, Adam Heath wrote:
>> On 05/25/2012 12:26 AM, Jacques Le Roux wrote:
>>> From: "Jacques Le Roux" <[hidden email]>
>>>> From: "Adam Heath" <[hidden email]>
>>>>> On 05/24/2012 04:05 PM, Jacques Le Roux wrote:
>>>>>> From: "Adam Heath" <[hidden email]>
>>>>>>> On 05/24/2012 10:18 AM, Adam Heath wrote:
>>>>>>> The idea was that you would parse the sqlCondition once, in a static
>>>>>>> {} block somewhere, then at runtime, just build that map. This
>>>>>>> builds-upon the map/list idea inherent in ofbiz.
>>>>>>>
>>>>>>> I also had plans that you could store sql strings into a properties
>>>>>>> file somewhere, so that they could possibly be changed by the
>>>>>>> end-user
>>>>>>> without having to recompile.
>>>>>>>
>>>>>>> I need to revisit the "SELECT a.partyId, b.* EXCLUDE(partyId) FROM
>>>>>>> Party a LEFT JOIN PartyContactMech b USING (partyId)", now that ofbiz
>>>>>>> better supports conditions on the join clauses, esp. when combining
>>>>>>> views into other views.
>>>>>>
>>>>>> Thanks for the explanation,
>>>>>>
>>>>>> So should we not rather create a Jira with all the needed in a patch
>>>>>> until this is finished?
>>>>>> Or maybe a branch would be easier?
>>>>>>
>>>>>> Still with the slim-down idea in mind and as objective.
>>>>>
>>>>> I like the slimdown, but tbh, I would like to see the framework/sql
>>>>> stuff used more than it is(0 right now). Andrew Zeneski was an
>>>>> original requestor for something that parsed sql into
>>>>> EntityCondition. I took his suggestion, but went further, to allow
>>>>> CREATE VIEW AS SELECT to work.
>>>>>
>>>>> I've noticed that there aren't that many view definitions in ofbiz.
>>>>> As I've been deprecating all this code recently, I've noticed java
>>>>> code doing nested loop kinda-stuff, instead of just doing a view. I'm
>>>>> guessing because view-xml is verbose and not how people actually think.
>>>>>
>>>>> However, with what I committed, you can define the view using a SQL
>>>>> syntax, which is then backed by a DynamicViewEntity.
>>>>>
>>>>> I've seen rather impressive speedups just rewriting code to a single
>>>>> SQL query, instead of java loops; the database can be rather
>>>>> efficient. So making view writing simpler is a laudable goal.
>>>>
>>>> Great, but still, why not a branch as long as it's not finished?
>>>>
>>>> Also something which I think is pretty neat in the principle (I still
>>>> did not review the code) and would speed up views:
>>>> https://issues.apache.org/jira/browse/OFBIZ-4041
>>>>
>>>> Jacques
>>>
>>> BTW another stuff that could be part of this branch
>>> https://issues.apache.org/jira/browse/OFBIZ-3575
>>
>> Ok, I suppose. This weekend I'll create such a branch to
>> fix/improve the view system. This will also attempt to fix the reverse
>> cache clearing issues.
>
> branches/improved-entityengine-features-20120528
>
> Also see 1343540, which adds a README that has some things that we might
> want to implement in the branch.

Indeed, I did not thought there were so much pending already

Jacques

[Jacopo Cappellato-4]

On May 29, 2012, at 6:49 AM, Adam Heath wrote:

>>>>
>>>> Great, but still, why not a branch as long as it's not finished?
>>>>
>>>> Jacques
>
> branches/improved-entityengine-features-20120528
>
> Also see 1343540, which adds a README that has some things that we might want to implement in the branch.

Thanks Adam; do we really need a branch for this?

Jacopo

[Jacques Le Roux]

Jacopo Cappellato wrote:

> On May 29, 2012, at 6:49 AM, Adam Heath wrote:
>
>>>>>
>>>>> Great, but still, why not a branch as long as it's not finished?
>>>>>
>>>>> Jacques
>>
>> branches/improved-entityengine-features-20120528
>>
>> Also see 1343540, which adds a README that has some things that we might want to implement in the branch.
>
> Thanks Adam; do we really need a branch for this?
>
> Jacopo

I suggested a branch to avoid possible pitfalls during implementation of the new sql stuff wich is a WIP. Then adam collected all
more or les related pending issues

Jacques

[Adrian Crum-3]

On 5/29/2012 5:49 AM, Adam Heath wrote:

> On 05/25/2012 01:22 AM, Adam Heath wrote:
>> On 05/25/2012 12:26 AM, Jacques Le Roux wrote:
>>> From: "Jacques Le Roux" <[hidden email]>
>>>> From: "Adam Heath" <[hidden email]>
>>>>> On 05/24/2012 04:05 PM, Jacques Le Roux wrote:
>>>>>> From: "Adam Heath" <[hidden email]>
>>>>>>> On 05/24/2012 10:18 AM, Adam Heath wrote:
>>>>>>> The idea was that you would parse the sqlCondition once, in a
>>>>>>> static
>>>>>>> {} block somewhere, then at runtime, just build that map. This
>>>>>>> builds-upon the map/list idea inherent in ofbiz.
>>>>>>>
>>>>>>> I also had plans that you could store sql strings into a properties
>>>>>>> file somewhere, so that they could possibly be changed by the
>>>>>>> end-user
>>>>>>> without having to recompile.
>>>>>>>
>>>>>>> I need to revisit the "SELECT a.partyId, b.* EXCLUDE(partyId) FROM
>>>>>>> Party a LEFT JOIN PartyContactMech b USING (partyId)", now that
>>>>>>> ofbiz
>>>>>>> better supports conditions on the join clauses, esp. when combining
>>>>>>> views into other views.
>>>>>>
>>>>>> Thanks for the explanation,
>>>>>>
>>>>>> So should we not rather create a Jira with all the needed in a patch
>>>>>> until this is finished?
>>>>>> Or maybe a branch would be easier?
>>>>>>
>>>>>> Still with the slim-down idea in mind and as objective.
>>>>>
>>>>> I like the slimdown, but tbh, I would like to see the framework/sql
>>>>> stuff used more than it is(0 right now). Andrew Zeneski was an
>>>>> original requestor for something that parsed sql into
>>>>> EntityCondition. I took his suggestion, but went further, to allow
>>>>> CREATE VIEW AS SELECT to work.
>>>>>
>>>>> I've noticed that there aren't that many view definitions in ofbiz.
>>>>> As I've been deprecating all this code recently, I've noticed java
>>>>> code doing nested loop kinda-stuff, instead of just doing a view. I'm
>>>>> guessing because view-xml is verbose and not how people actually
>>>>> think.
>>>>>
>>>>> However, with what I committed, you can define the view using a SQL
>>>>> syntax, which is then backed by a DynamicViewEntity.
>>>>>
>>>>> I've seen rather impressive speedups just rewriting code to a single
>>>>> SQL query, instead of java loops; the database can be rather
>>>>> efficient. So making view writing simpler is a laudable goal.
>>>>
>>>> Great, but still, why not a branch as long as it's not finished?
>>>>
>>>> Also something which I think is pretty neat in the principle (I still
>>>> did not review the code) and would speed up views:
>>>> https://issues.apache.org/jira/browse/OFBIZ-4041
>>>>
>>>> Jacques
>>>
>>> BTW another stuff that could be part of this branch
>>> https://issues.apache.org/jira/browse/OFBIZ-3575
>>
>> Ok, I suppose. This weekend I'll create such a branch to
>> fix/improve the view system. This will also attempt to fix the reverse
>> cache clearing issues.
>
> branches/improved-entityengine-features-20120528
>
> Also see 1343540, which adds a README that has some things that we
> might want to implement in the branch.

Adam,

I commented on the commit, but you didn't reply, so I will try again in
this thread.

If you don't mind, I would like to clean up the EntityConditionVisitor
interface so it looks more like a conventional visitor pattern.

Also, I was wondering if we could add some timing metrics to the entity
engine. Maybe keep an average query time per entity, and throw an
exception when the average exceeds a configurable threshold. This would
facilitate server overload management.

-Adrian

[Adam Heath-2]

On 05/30/2012 02:46 AM, Adrian Crum wrote:

> On 5/29/2012 5:49 AM, Adam Heath wrote:
>> On 05/25/2012 01:22 AM, Adam Heath wrote:
>>> On 05/25/2012 12:26 AM, Jacques Le Roux wrote:
>>>> From: "Jacques Le Roux" <[hidden email]>
>>>>> From: "Adam Heath" <[hidden email]>
>>>>>> On 05/24/2012 04:05 PM, Jacques Le Roux wrote:
>>>>>>> From: "Adam Heath" <[hidden email]>
>>>>>>>> On 05/24/2012 10:18 AM, Adam Heath wrote:
>>>>>>>> The idea was that you would parse the sqlCondition once, in a
>>>>>>>> static
>>>>>>>> {} block somewhere, then at runtime, just build that map. This
>>>>>>>> builds-upon the map/list idea inherent in ofbiz.
>>>>>>>>
>>>>>>>> I also had plans that you could store sql strings into a
>>>>>>>> properties
>>>>>>>> file somewhere, so that they could possibly be changed by the
>>>>>>>> end-user
>>>>>>>> without having to recompile.
>>>>>>>>
>>>>>>>> I need to revisit the "SELECT a.partyId, b.* EXCLUDE(partyId)
>>>>>>>> FROM
>>>>>>>> Party a LEFT JOIN PartyContactMech b USING (partyId)", now
>>>>>>>> that ofbiz
>>>>>>>> better supports conditions on the join clauses, esp. when
>>>>>>>> combining
>>>>>>>> views into other views.
>>>>>>>
>>>>>>> Thanks for the explanation,
>>>>>>>
>>>>>>> So should we not rather create a Jira with all the needed in a
>>>>>>> patch
>>>>>>> until this is finished?
>>>>>>> Or maybe a branch would be easier?
>>>>>>>
>>>>>>> Still with the slim-down idea in mind and as objective.
>>>>>>
>>>>>> I like the slimdown, but tbh, I would like to see the framework/sql
>>>>>> stuff used more than it is(0 right now). Andrew Zeneski was an
>>>>>> original requestor for something that parsed sql into
>>>>>> EntityCondition. I took his suggestion, but went further, to allow
>>>>>> CREATE VIEW AS SELECT to work.
>>>>>>
>>>>>> I've noticed that there aren't that many view definitions in ofbiz.
>>>>>> As I've been deprecating all this code recently, I've noticed java
>>>>>> code doing nested loop kinda-stuff, instead of just doing a
>>>>>> view. I'm
>>>>>> guessing because view-xml is verbose and not how people actually
>>>>>> think.
>>>>>>
>>>>>> However, with what I committed, you can define the view using a SQL
>>>>>> syntax, which is then backed by a DynamicViewEntity.
>>>>>>
>>>>>> I've seen rather impressive speedups just rewriting code to a
>>>>>> single
>>>>>> SQL query, instead of java loops; the database can be rather
>>>>>> efficient. So making view writing simpler is a laudable goal.
>>>>>
>>>>> Great, but still, why not a branch as long as it's not finished?
>>>>>
>>>>> Also something which I think is pretty neat in the principle (I
>>>>> still
>>>>> did not review the code) and would speed up views:
>>>>> https://issues.apache.org/jira/browse/OFBIZ-4041
>>>>>
>>>>> Jacques
>>>>
>>>> BTW another stuff that could be part of this branch
>>>> https://issues.apache.org/jira/browse/OFBIZ-3575
>>>
>>> Ok, I suppose. This weekend I'll create such a branch to
>>> fix/improve the view system. This will also attempt to fix the reverse
>>> cache clearing issues.
>>
>> branches/improved-entityengine-features-20120528
>>
>> Also see 1343540, which adds a README that has some things that we
>> might want to implement in the branch.
>
> Adam,
>
> I commented on the commit, but you didn't reply, so I will try again
> in this thread.

I saw, but didn't really think it needed a comment.  I guess I should
have given an affirmative response, instead of no response.  I assumed
that no response is not negative(yeah, that's a double negative).

>
> If you don't mind, I would like to clean up the EntityConditionVisitor
> interface so it looks more like a conventional visitor pattern.

I'm down to whatever other people for, within reason.  Tbh, I'm
currently tweaking EntityFieldValue, which is used by
ModelViewEntity.ViewEntityCondition, in a hackish way; it means I'm
having to tweak the visitor pattern, which sucks.

I'm the one who added the visitor pattern all those ages ago(goes back
to svn.ofbiz.org timeframe); I'd be fine with ripping it completely
out, as we(brainfood) don't use it anywhere.

> Also, I was wondering if we could add some timing metrics to the
> entity engine. Maybe keep an average query time per entity, and throw
> an exception when the average exceeds a configurable threshold. This
> would facilitate server overload management.

There is already code that does that, when a query takes a long time.
 If you do some stats(make it per-entity), make certain to use
AtomicInteger(or other AtomicFoo class).  You don't need to use
ConcurrentMap, as the list of entities to gather stats against is
static.  Just created the map at startup, or even store the stats in
ModelEntity.

Maybe the following will do what you want.  It's non-blocking
concurrent, makes use of work-borrowing type stuff(the double loop
update of 2 variables).

class Statistics {
 class Stat {
  AtomicReference<Long[]> countDurationAvgRef;
  Queue<Long> window;

  void add(long nanos) {
   window.add(nanos);
   while (true) {
    Long[] oldValues = countDurationAvgRef.get();
    long newCount = oldValues[0] + 1;
    Long[] newValues = new Long[] {
     newCount,
     oldValues[1] + nanos,
     oldValues[2] + (nanos / newCount)
    };
    if (countDurationAvgRef.compareAndSet(oldValues, newValues)) {
     break;
    }
   }
   while (true) {
    Long[] oldValues = countDurationAvgRef.get();
    long oldCount = oldValues[0];
    if (oldCount <= maxSize) {
     break;
    }
    long oldNanos = window.remove(0);
    Long[] newValues = new Long[] {
     oldCount - 1,
     oldValues[1] - oldNanos,
     oldValues[2] - (oldNanos / oldCount)
    };
    if (countDurationAvgRef.compareAndSet(oldValues, newValues)) {
     break;
    }
    window.add(0, oldNanos);
   }
  }
 }
 void addTiming(String name, long duration, TimeUnit unit) {
  long nanos = TimeUnit.NANOSECONDS.convert(duration, unit);
  if (duration@unit > name@warnLevel) {
   Debug.logWarning();
  }
  Stat stat = stats.get(name);
  if (stat == null) {
   Stat newStat = new Stat();
   stat = stats.putIfAbsent(name, newStat);
   if (stat == null) stat = newStat;
  }
  stat.add(nanos);
 }
}

[Adrian Crum-3]

On 5/30/2012 4:30 PM, Adam Heath wrote:

> On 05/30/2012 02:46 AM, Adrian Crum wrote:
>> On 5/29/2012 5:49 AM, Adam Heath wrote:
>>> On 05/25/2012 01:22 AM, Adam Heath wrote:
>>>> On 05/25/2012 12:26 AM, Jacques Le Roux wrote:
>>>>> From: "Jacques Le Roux"<[hidden email]>
>>>>>> From: "Adam Heath"<[hidden email]>
>>>>>>> On 05/24/2012 04:05 PM, Jacques Le Roux wrote:
>>>>>>>> From: "Adam Heath"<[hidden email]>
>>>>>>>>> On 05/24/2012 10:18 AM, Adam Heath wrote:
>>>>>>>>> The idea was that you would parse the sqlCondition once, in a
>>>>>>>>> static
>>>>>>>>> {} block somewhere, then at runtime, just build that map. This
>>>>>>>>> builds-upon the map/list idea inherent in ofbiz.
>>>>>>>>>
>>>>>>>>> I also had plans that you could store sql strings into a
>>>>>>>>> properties
>>>>>>>>> file somewhere, so that they could possibly be changed by the
>>>>>>>>> end-user
>>>>>>>>> without having to recompile.
>>>>>>>>>
>>>>>>>>> I need to revisit the "SELECT a.partyId, b.* EXCLUDE(partyId)
>>>>>>>>> FROM
>>>>>>>>> Party a LEFT JOIN PartyContactMech b USING (partyId)", now
>>>>>>>>> that ofbiz
>>>>>>>>> better supports conditions on the join clauses, esp. when
>>>>>>>>> combining
>>>>>>>>> views into other views.
>>>>>>>> Thanks for the explanation,
>>>>>>>>
>>>>>>>> So should we not rather create a Jira with all the needed in a
>>>>>>>> patch
>>>>>>>> until this is finished?
>>>>>>>> Or maybe a branch would be easier?
>>>>>>>>
>>>>>>>> Still with the slim-down idea in mind and as objective.
>>>>>>> I like the slimdown, but tbh, I would like to see the framework/sql
>>>>>>> stuff used more than it is(0 right now). Andrew Zeneski was an
>>>>>>> original requestor for something that parsed sql into
>>>>>>> EntityCondition. I took his suggestion, but went further, to allow
>>>>>>> CREATE VIEW AS SELECT to work.
>>>>>>>
>>>>>>> I've noticed that there aren't that many view definitions in ofbiz.
>>>>>>> As I've been deprecating all this code recently, I've noticed java
>>>>>>> code doing nested loop kinda-stuff, instead of just doing a
>>>>>>> view. I'm
>>>>>>> guessing because view-xml is verbose and not how people actually
>>>>>>> think.
>>>>>>>
>>>>>>> However, with what I committed, you can define the view using a SQL
>>>>>>> syntax, which is then backed by a DynamicViewEntity.
>>>>>>>
>>>>>>> I've seen rather impressive speedups just rewriting code to a
>>>>>>> single
>>>>>>> SQL query, instead of java loops; the database can be rather
>>>>>>> efficient. So making view writing simpler is a laudable goal.
>>>>>> Great, but still, why not a branch as long as it's not finished?
>>>>>>
>>>>>> Also something which I think is pretty neat in the principle (I
>>>>>> still
>>>>>> did not review the code) and would speed up views:
>>>>>> https://issues.apache.org/jira/browse/OFBIZ-4041
>>>>>>
>>>>>> Jacques
>>>>> BTW another stuff that could be part of this branch
>>>>> https://issues.apache.org/jira/browse/OFBIZ-3575
>>>> Ok, I suppose. This weekend I'll create such a branch to
>>>> fix/improve the view system. This will also attempt to fix the reverse
>>>> cache clearing issues.
>>> branches/improved-entityengine-features-20120528
>>>
>>> Also see 1343540, which adds a README that has some things that we
>>> might want to implement in the branch.
>> Adam,
>>
>> I commented on the commit, but you didn't reply, so I will try again
>> in this thread.
> I saw, but didn't really think it needed a comment.  I guess I should
> have given an affirmative response, instead of no response.  I assumed
> that no response is not negative(yeah, that's a double negative).
>
>> If you don't mind, I would like to clean up the EntityConditionVisitor
>> interface so it looks more like a conventional visitor pattern.
> I'm down to whatever other people for, within reason.  Tbh, I'm
> currently tweaking EntityFieldValue, which is used by
> ModelViewEntity.ViewEntityCondition, in a hackish way; it means I'm
> having to tweak the visitor pattern, which sucks.
>
> I'm the one who added the visitor pattern all those ages ago(goes back
> to svn.ofbiz.org timeframe); I'd be fine with ripping it completely
> out, as we(brainfood) don't use it anywhere.

I need to reacquaint myself with the entity engine code. I was thinking
the visitor pattern could be used to construct the SQL string instead of
the complicated if-then-else code spread across a number of classes. We
could use different visitors for different databases.

 From my perspective, the entity engine implementation seems a bit
tangled, and I was trying to come up with a strategy to simplify things.


>> Also, I was wondering if we could add some timing metrics to the
>> entity engine. Maybe keep an average query time per entity, and throw
>> an exception when the average exceeds a configurable threshold. This
>> would facilitate server overload management.
> There is already code that does that, when a query takes a long time.

I don't see any code that does that.

>   If you do some stats(make it per-entity), make certain to use
> AtomicInteger(or other AtomicFoo class).  You don't need to use
> ConcurrentMap, as the list of entities to gather stats against is
> static.  Just created the map at startup, or even store the stats in
> ModelEntity.
>
> Maybe the following will do what you want.  It's non-blocking
> concurrent, makes use of work-borrowing type stuff(the double loop
> update of 2 variables).
>
> class Statistics {
>   class Stat {
>    AtomicReference<Long[]>  countDurationAvgRef;
>    Queue<Long>  window;
>
>    void add(long nanos) {
>     window.add(nanos);
>     while (true) {
>      Long[] oldValues = countDurationAvgRef.get();
>      long newCount = oldValues[0] + 1;
>      Long[] newValues = new Long[] {
>       newCount,
>       oldValues[1] + nanos,
>       oldValues[2] + (nanos / newCount)
>      };
>      if (countDurationAvgRef.compareAndSet(oldValues, newValues)) {
>       break;
>      }
>     }
>     while (true) {
>      Long[] oldValues = countDurationAvgRef.get();
>      long oldCount = oldValues[0];
>      if (oldCount<= maxSize) {
>       break;
>      }
>      long oldNanos = window.remove(0);
>      Long[] newValues = new Long[] {
>       oldCount - 1,
>       oldValues[1] - oldNanos,
>       oldValues[2] - (oldNanos / oldCount)
>      };
>      if (countDurationAvgRef.compareAndSet(oldValues, newValues)) {
>       break;
>      }
>      window.add(0, oldNanos);
>     }
>    }
>   }
>   void addTiming(String name, long duration, TimeUnit unit) {
>    long nanos = TimeUnit.NANOSECONDS.convert(duration, unit);
>    if (duration@unit>  name@warnLevel) {
>     Debug.logWarning();
>    }
>    Stat stat = stats.get(name);
>    if (stat == null) {
>     Stat newStat = new Stat();
>     stat = stats.putIfAbsent(name, newStat);
>     if (stat == null) stat = newStat;
>    }
>    stat.add(nanos);
>   }
> }

[Adam Heath-2]

On 05/30/2012 11:47 AM, Adrian Crum wrote:

>
> On 5/30/2012 4:30 PM, Adam Heath wrote:
>> On 05/30/2012 02:46 AM, Adrian Crum wrote:
>>> On 5/29/2012 5:49 AM, Adam Heath wrote:
>>>> On 05/25/2012 01:22 AM, Adam Heath wrote:
>>>>> On 05/25/2012 12:26 AM, Jacques Le Roux wrote:
>>>>>> From: "Jacques Le Roux"<[hidden email]>
>>>>>>> From: "Adam Heath"<[hidden email]>
>>>>>>>> On 05/24/2012 04:05 PM, Jacques Le Roux wrote:
>>>>>>>>> From: "Adam Heath"<[hidden email]>
>>>>>>>>>> On 05/24/2012 10:18 AM, Adam Heath wrote:
>>>>>>>>>> The idea was that you would parse the sqlCondition once, in a
>>>>>>>>>> static
>>>>>>>>>> {} block somewhere, then at runtime, just build that map. This
>>>>>>>>>> builds-upon the map/list idea inherent in ofbiz.
>>>>>>>>>>
>>>>>>>>>> I also had plans that you could store sql strings into a
>>>>>>>>>> properties
>>>>>>>>>> file somewhere, so that they could possibly be changed by the
>>>>>>>>>> end-user
>>>>>>>>>> without having to recompile.
>>>>>>>>>>
>>>>>>>>>> I need to revisit the "SELECT a.partyId, b.* EXCLUDE(partyId)
>>>>>>>>>> FROM
>>>>>>>>>> Party a LEFT JOIN PartyContactMech b USING (partyId)", now
>>>>>>>>>> that ofbiz
>>>>>>>>>> better supports conditions on the join clauses, esp. when
>>>>>>>>>> combining
>>>>>>>>>> views into other views.
>>>>>>>>> Thanks for the explanation,
>>>>>>>>>
>>>>>>>>> So should we not rather create a Jira with all the needed in a
>>>>>>>>> patch
>>>>>>>>> until this is finished?
>>>>>>>>> Or maybe a branch would be easier?
>>>>>>>>>
>>>>>>>>> Still with the slim-down idea in mind and as objective.
>>>>>>>> I like the slimdown, but tbh, I would like to see the
>>>>>>>> framework/sql
>>>>>>>> stuff used more than it is(0 right now). Andrew Zeneski was an
>>>>>>>> original requestor for something that parsed sql into
>>>>>>>> EntityCondition. I took his suggestion, but went further, to
>>>>>>>> allow
>>>>>>>> CREATE VIEW AS SELECT to work.
>>>>>>>>
>>>>>>>> I've noticed that there aren't that many view definitions in
>>>>>>>> ofbiz.
>>>>>>>> As I've been deprecating all this code recently, I've noticed
>>>>>>>> java
>>>>>>>> code doing nested loop kinda-stuff, instead of just doing a
>>>>>>>> view. I'm
>>>>>>>> guessing because view-xml is verbose and not how people actually
>>>>>>>> think.
>>>>>>>>
>>>>>>>> However, with what I committed, you can define the view using
>>>>>>>> a SQL
>>>>>>>> syntax, which is then backed by a DynamicViewEntity.
>>>>>>>>
>>>>>>>> I've seen rather impressive speedups just rewriting code to a
>>>>>>>> single
>>>>>>>> SQL query, instead of java loops; the database can be rather
>>>>>>>> efficient. So making view writing simpler is a laudable goal.
>>>>>>> Great, but still, why not a branch as long as it's not finished?
>>>>>>>
>>>>>>> Also something which I think is pretty neat in the principle (I
>>>>>>> still
>>>>>>> did not review the code) and would speed up views:
>>>>>>> https://issues.apache.org/jira/browse/OFBIZ-4041
>>>>>>>
>>>>>>> Jacques
>>>>>> BTW another stuff that could be part of this branch
>>>>>> https://issues.apache.org/jira/browse/OFBIZ-3575
>>>>> Ok, I suppose. This weekend I'll create such a branch to
>>>>> fix/improve the view system. This will also attempt to fix the
>>>>> reverse
>>>>> cache clearing issues.
>>>> branches/improved-entityengine-features-20120528
>>>>
>>>> Also see 1343540, which adds a README that has some things that we
>>>> might want to implement in the branch.
>>> Adam,
>>>
>>> I commented on the commit, but you didn't reply, so I will try again
>>> in this thread.
>> I saw, but didn't really think it needed a comment.  I guess I should
>> have given an affirmative response, instead of no response.  I assumed
>> that no response is not negative(yeah, that's a double negative).
>>
>>> If you don't mind, I would like to clean up the EntityConditionVisitor
>>> interface so it looks more like a conventional visitor pattern.
>> I'm down to whatever other people for, within reason.  Tbh, I'm
>> currently tweaking EntityFieldValue, which is used by
>> ModelViewEntity.ViewEntityCondition, in a hackish way; it means I'm
>> having to tweak the visitor pattern, which sucks.
>>
>> I'm the one who added the visitor pattern all those ages ago(goes back
>> to svn.ofbiz.org timeframe); I'd be fine with ripping it completely
>> out, as we(brainfood) don't use it anywhere.
>
> I need to reacquaint myself with the entity engine code. I was
> thinking the visitor pattern could be used to construct the SQL string
> instead of the complicated if-then-else code spread across a number of
> classes. We could use different visitors for different databases.

Yes, that was my original thought.  However, I don't think it's that
simple anymore.  I've got a much better understanding of the entity
system since I wrote the visitor stuff.

I'd actually like to see my generic sql code be used to represent
entitymodel, then add sql<->sql conversion code.  I have an xslt that
can read *all* entitymodel.xml and convert it to entitymodel.sql, or
entitymodel.java.  I used the former to verify that my sql parsing was
featureful enough.

> From my perspective, the entity engine implementation seems a bit
> tangled, and I was trying to come up with a strategy to simplify things.

It's not too bad, imho; just has some issues with view abstraction
that can't be currently represented.  It's getting much closer to not
having to do raw sql anymore thru jdbc.

>>> Also, I was wondering if we could add some timing metrics to the
>>> entity engine. Maybe keep an average query time per entity, and throw
>>> an exception when the average exceeds a configurable threshold. This
>>> would facilitate server overload management.
>> There is already code that does that, when a query takes a long time.
>
> I don't see any code that does that.

I've seen lines in the log file where queries take too long.
GenericDAO.selectListIteratorByCondition() has an example for that,
but it's not done in select().