FW: Mirgration to OFBiz newer version

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

FW: Mirgration to OFBiz newer version

SkipDever

-----Original Message-----
From: Skip [mailto:[hidden email]]
Sent: Friday, September 13, 2013 1:16 AM
To: Pierre Smits
Subject: RE: Mirgration to OFBiz newer version


Pierre

As I write this, it is it is just past midnight here and I am importing data
into the new system from the old for what I hope will be the final time.
This is a large data set and takes some 9 hours to get finished.  I have
been working on this for two months now and just today feel confident enough
to put it into production.

I have tried to share some of the pitfalls I ran into, but let me recap some
of the major ones.  But be sure to check the message board history for much
more.

1.  There is a major change in the way passwords are computed.  Because we
have external users that we cannot force to change their passwords en-mass,
I put the password computations back the way they were so they accept both
new and old passwords.  Check out HashCrypt.getDigestHash(), and modified
checkPassword() in LoginServices to call a new getUnsaltedPasswordHash()
which for the most part, just calls HashCrypt.getDigestHash() if the
comparePassword() failed.  Note that I commented out all this code in
HashCrypt and pasted in the stuff from the same file in 9.x code because it
will produce a different hash.

2.  When importing data from your old system, make sure you set
entity-eca-enabled="false" in your entityengine.xml for your delegators
before you begin importing and then remove it after.  Otherwise, you end up
with a lot of crap.

3.  There are lots of new ecas that did not exist before.  This caused me no
end of headaches.  Make sure you compare your old xxxseca.xml with the new
and comment out those you do not want.

4.  Date formats are now yyyy-mm-dd instead of localized.  Not a big deal,
but I got lots of complaints during beta testing.  Still have not resolved
this because there is a lot of javascript code to change to put it back the
way it was.

5.  Lots of changes in widgets xls definitions.  After you get it fired up,
visit all your custom screens and check the console.log.  It will have tons
of errors that need cleaned up.

6.  Any custom java code that you have that used Double database fields now
must use BigDecimal for most of it.  Ditto for .bsh scripts.

7.  The bsh interpreter that ships with 12.x doesnt work.  Restore the old
version unless you are going to rewrite all you bsh scripts into .groovy.  I
am using bsh-2.04b.jar but this puts a bunch of "Experimental..." crap in
the log.  But, at least it works until I get that figured out.

8.  To import your data do an export all, delete JobSandbox.xml, Comm*.xml,
ProductKeyword.xml, RuntimeData.xml, UserLoginHistory.xml and Workflow*.xml.
Then, on a clean database, do ant load-seed, then import your exported
files, being sure to turn off entity-eca-enabled.

8.1.  Several table structions have been modified.  ProductAverateCost now
has a new productAverateCostTypeId (set to SIMPLE_AVG_COST), that has to be
added after the export to make it import OK.  None of the other additions
caused import problems.

9.  Things are about 20% slower.

10.  I did my best to keep my ofbiz code untouched, but still, have a couple
hundred modifitions in the framework and application trees (mostly in
applications related to invoices and billing accounts).

11.  Because I did not want to migrate all my applications over to the new
MacroScreenViewHandler, in each of my custom controller.xml, I made a
macroScreen handler and left screen as ScreenWidgetViewHandler.  Then, as I
changed the ftl to use it, I could just change screen to macroScreen.  The
new MacroScreenViewHandler is completely integrated with jQuery, you can
create some really killer end user screens now with very little code.  Its
really easy to modify the css to get it to look just like you like.  The ftl
code behind MacroScreenViewHandler needs some work though to make it the
best it can be.

I do not need multi-tenat.  As such, if I had to do this over again, I would
just have replaced the widget code with the new.  That is all rather clever.
The rest of the changes are mostly useless in my opinion.  Much needless
crap and pointless refactorings in my view.  There is a big rewrite to the
entity engine code and all your custom java files will have to be changed.
Deletator instead of GenericDelegator, EntityCondition.makeCondition instead
of new EntityExpr, and lots stuff like that.  I had about a thousand java
files that I had to change and it is mostly search and replace, but a pain
in the butt for no value.  Even though there is mostly no difference, they
didn't just deprecate, the removed functions all together.  I added lots of
the deprecated functions back in to make my life easier.

Hope this helps

Skip




  -----Original Message-----
  From: Pierre Smits [mailto:[hidden email]]
  Sent: Friday, September 13, 2013 12:09 AM
  To: [hidden email]
  Subject: Mirgration to OFBiz newer version


  Hi Skip,


  I trust your migration project is going well.


  Currently I am facing the same situation as I need to migrate several
customers to our new multi-tenant environment.
  As I am in the planning phase, I was wondering if your would be willing to
share your best practices and do's and don'ts with me regarding the data
migration.


  With kind regards,


  Pierre Smits


  ORRTIZ.COM
  Services & Solutions for Cloud-
  Based Manufacturing, Professional
  Services and Retail & Trade
  http://www.orrtiz.com
Reply | Threaded
Open this post in threaded view
|

Re: FW: Mirgration to OFBiz newer version

Adrian Crum-3
Skip,

It's good to hear you finally got everything working okay.

I'm sure your frustration has influenced your opinions about some of the
changes in OFBiz.

A lot of time has passed since version 9, and the changes you
encountered were discussed in depth, old APIs were deprecated, and
eventually obsolete stuff was removed. We don't have the resources to
keep deprecated code in the project forever. I hope you understand that.

 From my perspective, refactorings are not pointless - each one had a
very good reason behind it. I don't know of any committer who has enough
free time to develop, test, and commit pointless changes. So, if there
is a specific change that has you puzzled, I would encourage you to
discuss it on the dev mailing list.

Adrian Crum
Sandglass Software
www.sandglass-software.com

On 9/13/2013 1:17 AM, Skip wrote:

>
> -----Original Message-----
> From: Skip [mailto:[hidden email]]
> Sent: Friday, September 13, 2013 1:16 AM
> To: Pierre Smits
> Subject: RE: Mirgration to OFBiz newer version
>
>
> Pierre
>
> As I write this, it is it is just past midnight here and I am importing data
> into the new system from the old for what I hope will be the final time.
> This is a large data set and takes some 9 hours to get finished.  I have
> been working on this for two months now and just today feel confident enough
> to put it into production.
>
> I have tried to share some of the pitfalls I ran into, but let me recap some
> of the major ones.  But be sure to check the message board history for much
> more.
>
> 1.  There is a major change in the way passwords are computed.  Because we
> have external users that we cannot force to change their passwords en-mass,
> I put the password computations back the way they were so they accept both
> new and old passwords.  Check out HashCrypt.getDigestHash(), and modified
> checkPassword() in LoginServices to call a new getUnsaltedPasswordHash()
> which for the most part, just calls HashCrypt.getDigestHash() if the
> comparePassword() failed.  Note that I commented out all this code in
> HashCrypt and pasted in the stuff from the same file in 9.x code because it
> will produce a different hash.
>
> 2.  When importing data from your old system, make sure you set
> entity-eca-enabled="false" in your entityengine.xml for your delegators
> before you begin importing and then remove it after.  Otherwise, you end up
> with a lot of crap.
>
> 3.  There are lots of new ecas that did not exist before.  This caused me no
> end of headaches.  Make sure you compare your old xxxseca.xml with the new
> and comment out those you do not want.
>
> 4.  Date formats are now yyyy-mm-dd instead of localized.  Not a big deal,
> but I got lots of complaints during beta testing.  Still have not resolved
> this because there is a lot of javascript code to change to put it back the
> way it was.
>
> 5.  Lots of changes in widgets xls definitions.  After you get it fired up,
> visit all your custom screens and check the console.log.  It will have tons
> of errors that need cleaned up.
>
> 6.  Any custom java code that you have that used Double database fields now
> must use BigDecimal for most of it.  Ditto for .bsh scripts.
>
> 7.  The bsh interpreter that ships with 12.x doesnt work.  Restore the old
> version unless you are going to rewrite all you bsh scripts into .groovy.  I
> am using bsh-2.04b.jar but this puts a bunch of "Experimental..." crap in
> the log.  But, at least it works until I get that figured out.
>
> 8.  To import your data do an export all, delete JobSandbox.xml, Comm*.xml,
> ProductKeyword.xml, RuntimeData.xml, UserLoginHistory.xml and Workflow*.xml.
> Then, on a clean database, do ant load-seed, then import your exported
> files, being sure to turn off entity-eca-enabled.
>
> 8.1.  Several table structions have been modified.  ProductAverateCost now
> has a new productAverateCostTypeId (set to SIMPLE_AVG_COST), that has to be
> added after the export to make it import OK.  None of the other additions
> caused import problems.
>
> 9.  Things are about 20% slower.
>
> 10.  I did my best to keep my ofbiz code untouched, but still, have a couple
> hundred modifitions in the framework and application trees (mostly in
> applications related to invoices and billing accounts).
>
> 11.  Because I did not want to migrate all my applications over to the new
> MacroScreenViewHandler, in each of my custom controller.xml, I made a
> macroScreen handler and left screen as ScreenWidgetViewHandler.  Then, as I
> changed the ftl to use it, I could just change screen to macroScreen.  The
> new MacroScreenViewHandler is completely integrated with jQuery, you can
> create some really killer end user screens now with very little code.  Its
> really easy to modify the css to get it to look just like you like.  The ftl
> code behind MacroScreenViewHandler needs some work though to make it the
> best it can be.
>
> I do not need multi-tenat.  As such, if I had to do this over again, I would
> just have replaced the widget code with the new.  That is all rather clever.
> The rest of the changes are mostly useless in my opinion.  Much needless
> crap and pointless refactorings in my view.  There is a big rewrite to the
> entity engine code and all your custom java files will have to be changed.
> Deletator instead of GenericDelegator, EntityCondition.makeCondition instead
> of new EntityExpr, and lots stuff like that.  I had about a thousand java
> files that I had to change and it is mostly search and replace, but a pain
> in the butt for no value.  Even though there is mostly no difference, they
> didn't just deprecate, the removed functions all together.  I added lots of
> the deprecated functions back in to make my life easier.
>
> Hope this helps
>
> Skip
>
>
>
>
>    -----Original Message-----
>    From: Pierre Smits [mailto:[hidden email]]
>    Sent: Friday, September 13, 2013 12:09 AM
>    To: [hidden email]
>    Subject: Mirgration to OFBiz newer version
>
>
>    Hi Skip,
>
>
>    I trust your migration project is going well.
>
>
>    Currently I am facing the same situation as I need to migrate several
> customers to our new multi-tenant environment.
>    As I am in the planning phase, I was wondering if your would be willing to
> share your best practices and do's and don'ts with me regarding the data
> migration.
>
>
>    With kind regards,
>
>
>    Pierre Smits
>
>
>    ORRTIZ.COM
>    Services & Solutions for Cloud-
>    Based Manufacturing, Professional
>    Services and Retail & Trade
>    http://www.orrtiz.com
>
Reply | Threaded
Open this post in threaded view
|

Re: FW: Mirgration to OFBiz newer version

Jacques Le Roux
Administrator
To reassure you Skip, OFBiz is more mature now, there should be less changes the next time you will do it.
Except, if we want to take advantage of Java 8 : http://www.techempower.com/blog/2013/03/26/everything-about-java-8/
Of course, updating more often should help...

Jacques

Adrian Crum wrote:

> Skip,
>
> It's good to hear you finally got everything working okay.
>
> I'm sure your frustration has influenced your opinions about some of the
> changes in OFBiz.
>
> A lot of time has passed since version 9, and the changes you
> encountered were discussed in depth, old APIs were deprecated, and
> eventually obsolete stuff was removed. We don't have the resources to
> keep deprecated code in the project forever. I hope you understand that.
>
> From my perspective, refactorings are not pointless - each one had a
> very good reason behind it. I don't know of any committer who has enough
> free time to develop, test, and commit pointless changes. So, if there
> is a specific change that has you puzzled, I would encourage you to
> discuss it on the dev mailing list.
>
> Adrian Crum
> Sandglass Software
> www.sandglass-software.com
>
> On 9/13/2013 1:17 AM, Skip wrote:
>>
>> -----Original Message-----
>> From: Skip [mailto:[hidden email]]
>> Sent: Friday, September 13, 2013 1:16 AM
>> To: Pierre Smits
>> Subject: RE: Mirgration to OFBiz newer version
>>
>>
>> Pierre
>>
>> As I write this, it is it is just past midnight here and I am importing data
>> into the new system from the old for what I hope will be the final time.
>> This is a large data set and takes some 9 hours to get finished.  I have
>> been working on this for two months now and just today feel confident enough
>> to put it into production.
>>
>> I have tried to share some of the pitfalls I ran into, but let me recap some
>> of the major ones.  But be sure to check the message board history for much
>> more.
>>
>> 1.  There is a major change in the way passwords are computed.  Because we
>> have external users that we cannot force to change their passwords en-mass,
>> I put the password computations back the way they were so they accept both
>> new and old passwords.  Check out HashCrypt.getDigestHash(), and modified
>> checkPassword() in LoginServices to call a new getUnsaltedPasswordHash()
>> which for the most part, just calls HashCrypt.getDigestHash() if the
>> comparePassword() failed.  Note that I commented out all this code in
>> HashCrypt and pasted in the stuff from the same file in 9.x code because it
>> will produce a different hash.
>>
>> 2.  When importing data from your old system, make sure you set
>> entity-eca-enabled="false" in your entityengine.xml for your delegators
>> before you begin importing and then remove it after.  Otherwise, you end up
>> with a lot of crap.
>>
>> 3.  There are lots of new ecas that did not exist before.  This caused me no
>> end of headaches.  Make sure you compare your old xxxseca.xml with the new
>> and comment out those you do not want.
>>
>> 4.  Date formats are now yyyy-mm-dd instead of localized.  Not a big deal,
>> but I got lots of complaints during beta testing.  Still have not resolved
>> this because there is a lot of javascript code to change to put it back the
>> way it was.
>>
>> 5.  Lots of changes in widgets xls definitions.  After you get it fired up,
>> visit all your custom screens and check the console.log.  It will have tons
>> of errors that need cleaned up.
>>
>> 6.  Any custom java code that you have that used Double database fields now
>> must use BigDecimal for most of it.  Ditto for .bsh scripts.
>>
>> 7.  The bsh interpreter that ships with 12.x doesnt work.  Restore the old
>> version unless you are going to rewrite all you bsh scripts into .groovy.  I
>> am using bsh-2.04b.jar but this puts a bunch of "Experimental..." crap in
>> the log.  But, at least it works until I get that figured out.
>>
>> 8.  To import your data do an export all, delete JobSandbox.xml, Comm*.xml,
>> ProductKeyword.xml, RuntimeData.xml, UserLoginHistory.xml and Workflow*.xml.
>> Then, on a clean database, do ant load-seed, then import your exported
>> files, being sure to turn off entity-eca-enabled.
>>
>> 8.1.  Several table structions have been modified.  ProductAverateCost now
>> has a new productAverateCostTypeId (set to SIMPLE_AVG_COST), that has to be
>> added after the export to make it import OK.  None of the other additions
>> caused import problems.
>>
>> 9.  Things are about 20% slower.
>>
>> 10.  I did my best to keep my ofbiz code untouched, but still, have a couple
>> hundred modifitions in the framework and application trees (mostly in
>> applications related to invoices and billing accounts).
>>
>> 11.  Because I did not want to migrate all my applications over to the new
>> MacroScreenViewHandler, in each of my custom controller.xml, I made a
>> macroScreen handler and left screen as ScreenWidgetViewHandler.  Then, as I
>> changed the ftl to use it, I could just change screen to macroScreen.  The
>> new MacroScreenViewHandler is completely integrated with jQuery, you can
>> create some really killer end user screens now with very little code.  Its
>> really easy to modify the css to get it to look just like you like.  The ftl
>> code behind MacroScreenViewHandler needs some work though to make it the
>> best it can be.
>>
>> I do not need multi-tenat.  As such, if I had to do this over again, I would
>> just have replaced the widget code with the new.  That is all rather clever.
>> The rest of the changes are mostly useless in my opinion.  Much needless
>> crap and pointless refactorings in my view.  There is a big rewrite to the
>> entity engine code and all your custom java files will have to be changed.
>> Deletator instead of GenericDelegator, EntityCondition.makeCondition instead
>> of new EntityExpr, and lots stuff like that.  I had about a thousand java
>> files that I had to change and it is mostly search and replace, but a pain
>> in the butt for no value.  Even though there is mostly no difference, they
>> didn't just deprecate, the removed functions all together.  I added lots of
>> the deprecated functions back in to make my life easier.
>>
>> Hope this helps
>>
>> Skip
>>
>>
>>
>>
>>    -----Original Message-----
>>    From: Pierre Smits [mailto:[hidden email]]
>>    Sent: Friday, September 13, 2013 12:09 AM
>>    To: [hidden email]
>>    Subject: Mirgration to OFBiz newer version
>>
>>
>>    Hi Skip,
>>
>>
>>    I trust your migration project is going well.
>>
>>
>>    Currently I am facing the same situation as I need to migrate several
>> customers to our new multi-tenant environment.
>>    As I am in the planning phase, I was wondering if your would be willing to
>> share your best practices and do's and don'ts with me regarding the data
>> migration.
>>
>>
>>    With kind regards,
>>
>>
>>    Pierre Smits
>>
>>
>>    ORRTIZ.COM
>>    Services & Solutions for Cloud-
>>    Based Manufacturing, Professional
>>    Services and Retail & Trade
>>    http://www.orrtiz.com
Reply | Threaded
Open this post in threaded view
|

Re: FW: Mirgration to OFBiz newer version

Pierre Smits
Jacques,

Updating (or migrating) is determined by various factors. Each update
brings risks and cost. Especially with a system in production. The
(perceived value of the) improvements must weigh against the cost involved
in implementing.

I thank Skip for sharing the experience.  I will surely use it as part of
my migration plans. As far as I can go back it is the first time that it is
done. Sharing implementation and upgrade experiences builds confidence. It
should be encouraged.

What worries me though is the experienced decrease in performance.

With kind regards,

Pierre Smits

*ORRTIZ.COM <http://www.orrtiz.com>*
Services & Solutions for Cloud-
Based Manufacturing, Professional
Services and Retail & Trade
http://www.orrtiz.com
Reply | Threaded
Open this post in threaded view
|

Re: FW: Mirgration to OFBiz newer version

Ted Byers
In reply to this post by Jacques Le Roux
Something my team is beginning to do with our system, is to create a
pair of scripts, one of which applies any changes to our database
(particularly focussing on ALTER TABLE statements, to add new fields
along with rational default values and constraints), and another to
deal with changes to our code (all written in Perl+SQL+Javascript - so
you wouldn't be interested in the details).  If the update process is
automated, it really doesn't matter how many changes there are,
because the script handles it all.  Of course, this doesn't help in
situations in which one takes the approach of storing specific data as
a hash of the data rather than the raw data, and then one decides to
use a different hash algorithm to make the hash (passwords are an
obvious example).  But in such cases, one ought to do a couple things:
1) provide an option to allow use of the original algorithm, and 2)
provide an option to let the user use the new algorithm when the old
password expires (or one can automate that, but that requires a little
extra code).  This isn't all that new an idea as I recall using diff
on C/C++ files what seems like eons ago.  But perhaps it is time that
such an idea was applied to OFBiz?  Of course, that means that someone
needs to take responsibility for maintaining such a pair of scripts.

Just a thought...

Ted

On Sat, Sep 14, 2013 at 7:56 AM, Jacques Le Roux
<[hidden email]> wrote:
> To reassure you Skip, OFBiz is more mature now, there should be less changes the next time you will do it.
> Except, if we want to take advantage of Java 8 : http://www.techempower.com/blog/2013/03/26/everything-about-java-8/
> Of course, updating more often should help...
>
> Jacques
Reply | Threaded
Open this post in threaded view
|

Re: FW: Mirgration to OFBiz newer version

Jacques Le Roux
Administrator
In reply to this post by Pierre Smits
Pierre Smits wrote:

> Jacques,
>
> Updating (or migrating) is determined by various factors. Each update
> brings risks and cost. Especially with a system in production. The
> (perceived value of the) improvements must weigh against the cost involved
> in implementing.
>
> I thank Skip for sharing the experience.  I will surely use it as part of
> my migration plans. As far as I can go back it is the first time that it is
> done. Sharing implementation and upgrade experiences builds confidence. It
> should be encouraged.
>
> What worries me though is the experienced decrease in performance.

This would need much more measures to know what we are talking about...
 
> With kind regards,
>
> Pierre Smits
Reply | Threaded
Open this post in threaded view
|

Re: FW: Mirgration to OFBiz newer version

Jacques Le Roux
Administrator
In reply to this post by Ted Byers
Thanks for the proposition Ted,

Unfortunately, we are not missing good ideas, but good wills...

Jacques

Ted Byers wrote:

> Something my team is beginning to do with our system, is to create a
> pair of scripts, one of which applies any changes to our database
> (particularly focussing on ALTER TABLE statements, to add new fields
> along with rational default values and constraints), and another to
> deal with changes to our code (all written in Perl+SQL+Javascript - so
> you wouldn't be interested in the details).  If the update process is
> automated, it really doesn't matter how many changes there are,
> because the script handles it all.  Of course, this doesn't help in
> situations in which one takes the approach of storing specific data as
> a hash of the data rather than the raw data, and then one decides to
> use a different hash algorithm to make the hash (passwords are an
> obvious example).  But in such cases, one ought to do a couple things:
> 1) provide an option to allow use of the original algorithm, and 2)
> provide an option to let the user use the new algorithm when the old
> password expires (or one can automate that, but that requires a little
> extra code).  This isn't all that new an idea as I recall using diff
> on C/C++ files what seems like eons ago.  But perhaps it is time that
> such an idea was applied to OFBiz?  Of course, that means that someone
> needs to take responsibility for maintaining such a pair of scripts.
>
> Just a thought...
>
> Ted
>
> On Sat, Sep 14, 2013 at 7:56 AM, Jacques Le Roux
> <[hidden email]> wrote:
>> To reassure you Skip, OFBiz is more mature now, there should be less changes the next time you will do it.
>> Except, if we want to take advantage of Java 8 : http://www.techempower.com/blog/2013/03/26/everything-about-java-8/
>> Of course, updating more often should help...
>>
>> Jacques
Reply | Threaded
Open this post in threaded view
|

Re: FW: Mirgration to OFBiz newer version

Jacques Le Roux
Administrator
In reply to this post by SkipDever
Or rather we have more good ideas than good wills

Jacques Le Roux wrote:

> Thanks for the proposition Ted,
>
> Unfortunately, we are not missing good ideas, but good wills...
>
> Jacques
>
> Ted Byers wrote:
>> Something my team is beginning to do with our system, is to create a
>> pair of scripts, one of which applies any changes to our database
>> (particularly focussing on ALTER TABLE statements, to add new fields
>> along with rational default values and constraints), and another to
>> deal with changes to our code (all written in Perl+SQL+Javascript - so
>> you wouldn't be interested in the details).  If the update process is
>> automated, it really doesn't matter how many changes there are,
>> because the script handles it all.  Of course, this doesn't help in
>> situations in which one takes the approach of storing specific data as
>> a hash of the data rather than the raw data, and then one decides to
>> use a different hash algorithm to make the hash (passwords are an
>> obvious example).  But in such cases, one ought to do a couple things:
>> 1) provide an option to allow use of the original algorithm, and 2)
>> provide an option to let the user use the new algorithm when the old
>> password expires (or one can automate that, but that requires a little
>> extra code).  This isn't all that new an idea as I recall using diff
>> on C/C++ files what seems like eons ago.  But perhaps it is time that
>> such an idea was applied to OFBiz?  Of course, that means that someone
>> needs to take responsibility for maintaining such a pair of scripts.
>>
>> Just a thought...
>>
>> Ted
>>
>> On Sat, Sep 14, 2013 at 7:56 AM, Jacques Le Roux
>> <[hidden email]> wrote:
>>> To reassure you Skip, OFBiz is more mature now, there should be less changes the next time you will do it.
>>> Except, if we want to take advantage of Java 8 : http://www.techempower.com/blog/2013/03/26/everything-about-java-8/
>>> Of course, updating more often should help...
>>>
>>> Jacques
Reply | Threaded
Open this post in threaded view
|

RE: FW: Mirgration to OFBiz newer version

SkipDever
In reply to this post by Jacques Le Roux
Jacques

Being nearly done with this and my frustrations long since passed, I have
only a couple of things to say about this for future development.  I have
been writing developer software for multiple decades and found from
experience that backward compatibility is your most important concern.

No one should ever change stuff just because they do not like wording or
style.  For example, there were wholesale changes to the xsd for widgets.
There was no real benefit to the change other than readability.  It should
never have been done.  Instead, both words should have been supported.  For
me, this required days and days of search and replace for no value.

Another is the replacement of new EntityExp() with
EntityCondition.makeCondition() etc.  Someone decided that they liked this
better. That is the only possible reason for the change because the
underling code could have supported either or both APIs.  This kind of stuff
is unforgivable in my view for developer software and that is what ofbiz is.
It has very little value OOTB and requires developers to give it life.  Some
of us have thousands of other files written to these apis and to force us to
change them for no real value is frustrating to the maximum.

Finally, the change in the password hash is the most agreggious example in
my view.  It would have been a simple matter to use both (or in this case
all three hashing methods).  When a password changes, use the newest, best
method, but allow the existing hashes stored in the database until changed.
I modified the ofbiz code to do exactly that and it works fine and took me
all of an hour once I spend several hours tracking down the problem.

As Ofbiz contributors, I am sure you guys wear two hats.  One as an Ofbiz
contributor and one as a developer using Ofbiz for end users who actually
pay you.  As you make changes to Ofbiz code to implement something a
customer really wants, remember to not screw the hundreds (thousands ?) of
other Ofbiz developers who depend on it like you do.

Backward compatibility is or should be your most important concern.  Just
take a look at the compile log for Ofbiz and see how many deprecated Java
methods we still use.  Then, look at how long they have been deprecated.
Some for decades.

Thanks for all you do.

Skip

-----Original Message-----
From: Jacques Le Roux [mailto:[hidden email]]
Sent: Saturday, September 14, 2013 10:01 AM
To: [hidden email]
Subject: Re: FW: Mirgration to OFBiz newer version


Pierre Smits wrote:
> Jacques,
>
> Updating (or migrating) is determined by various factors. Each update
> brings risks and cost. Especially with a system in production. The
> (perceived value of the) improvements must weigh against the cost involved
> in implementing.
>
> I thank Skip for sharing the experience.  I will surely use it as part of
> my migration plans. As far as I can go back it is the first time that it
is
> done. Sharing implementation and upgrade experiences builds confidence. It
> should be encouraged.
>
> What worries me though is the experienced decrease in performance.

This would need much more measures to know what we are talking about...

> With kind regards,
>
> Pierre Smits

Reply | Threaded
Open this post in threaded view
|

Re: FW: Mirgration to OFBiz newer version

Jacques Le Roux
Administrator
Skip,

I know what you are talking about, I had my share with a part of Opentaps adaptation for https://issues.apache.org/jira/browse/OFBIZ-4247
So believe me, I'm very cautious when deprecating stuff.

BTW you said that "Things are about 20% slower." Where, in which areas, did you cross this? How did you measure?

Jacques

Skip wrote:

> Jacques
>
> Being nearly done with this and my frustrations long since passed, I have
> only a couple of things to say about this for future development.  I have
> been writing developer software for multiple decades and found from
> experience that backward compatibility is your most important concern.
>
> No one should ever change stuff just because they do not like wording or
> style.  For example, there were wholesale changes to the xsd for widgets.
> There was no real benefit to the change other than readability.  It should
> never have been done.  Instead, both words should have been supported.  For
> me, this required days and days of search and replace for no value.
>
> Another is the replacement of new EntityExp() with
> EntityCondition.makeCondition() etc.  Someone decided that they liked this
> better. That is the only possible reason for the change because the
> underling code could have supported either or both APIs.  This kind of stuff
> is unforgivable in my view for developer software and that is what ofbiz is.
> It has very little value OOTB and requires developers to give it life.  Some
> of us have thousands of other files written to these apis and to force us to
> change them for no real value is frustrating to the maximum.
>
> Finally, the change in the password hash is the most agreggious example in
> my view.  It would have been a simple matter to use both (or in this case
> all three hashing methods).  When a password changes, use the newest, best
> method, but allow the existing hashes stored in the database until changed.
> I modified the ofbiz code to do exactly that and it works fine and took me
> all of an hour once I spend several hours tracking down the problem.
>
> As Ofbiz contributors, I am sure you guys wear two hats.  One as an Ofbiz
> contributor and one as a developer using Ofbiz for end users who actually
> pay you.  As you make changes to Ofbiz code to implement something a
> customer really wants, remember to not screw the hundreds (thousands ?) of
> other Ofbiz developers who depend on it like you do.
>
> Backward compatibility is or should be your most important concern.  Just
> take a look at the compile log for Ofbiz and see how many deprecated Java
> methods we still use.  Then, look at how long they have been deprecated.
> Some for decades.
>
> Thanks for all you do.
>
> Skip
>
> -----Original Message-----
> From: Jacques Le Roux [mailto:[hidden email]]
> Sent: Saturday, September 14, 2013 10:01 AM
> To: [hidden email]
> Subject: Re: FW: Mirgration to OFBiz newer version
>
>
> Pierre Smits wrote:
>> Jacques,
>>
>> Updating (or migrating) is determined by various factors. Each update
>> brings risks and cost. Especially with a system in production. The
>> (perceived value of the) improvements must weigh against the cost involved
>> in implementing.
>>
>> I thank Skip for sharing the experience.  I will surely use it as part of
>> my migration plans. As far as I can go back it is the first time that it is
>> done. Sharing implementation and upgrade experiences builds confidence. It
>> should be encouraged.
>>
>> What worries me though is the experienced decrease in performance.
>
> This would need much more measures to know what we are talking about...
>
>> With kind regards,
>>
>> Pierre Smits
Reply | Threaded
Open this post in threaded view
|

RE: FW: Mirgration to OFBiz newer version

SkipDever
Jacques

Goto ...webtools/control/entitymaint in both 12.x and 9.x.  In 9.x, the page
loads instantly, in 12.04, it takes several seconds.  Its anecdotally like
1000% slower.  Thats the worst case I have seen.

I have hundreds of custom services, one of which runs before a custom
inventory screen loads.  I noticed this screen taking a long time to load
and timed the service.  It was taking about 2 seconds longer to run.  It is
basically a database lookup for unshipped orders, outstanding backorders,
incoming inventory that needs packaged, etc.  Nothing at all extraordinary.
After timing it 10 times using System.currentMillsecs(), I noted it was
taking about 18% more time on the same dataset.  In this case, I actually
put a profiler on both of them trying to see something different in the call
trees (BTW, jProfiler is killer for this purpose).  Unfortunately, I do not
remember what was there except for a notably high amount of time in the
logging code called from the delegator code tree even though logging was set
to error and there were no errors.  Someday, I'll get back to exploring this
more.  Note: this is with the 9.x code running Minerva and the 12.04 code
running DBCP.  If I put them both on Minverva, 12.04 is something 35%
slower.  I did not test the other way, i.e. DBCP on both because I am not
sure it would run at all on 9.x code.

Second, I have some custom rest services attached to external POJO screens.
These services get a session, login, sometimes get a shopping cart,
sometimes lookup prices and stock, etc.  These felt slow as well, so I timed
one of them(again using System.currentMillsecs()) and found the REST calls
return 20% slower.  This is essentially just the amount of time it takes on
a local machine to get a JSON response from http://localhost:8080/someurl.
You can't really see the time difference here unless you do it a hundred
times because the time for each is so small, but for these POJO apps, that
is exactly what they do, login, lookup a customer, get some inventory
counts, etc, all in rapid succession.

One more thing, in my 9.x code, I used Minerva because because DBCP was
causing some errors.  When I tried the DB stuff with Minerva in the 12.04
code, it was like 35% slower (both systems running Minerva).   After
discussing it with you here on the ML, I moved to DBCP and speed improved to
a generally acceptable level (except as noted above).  A months beta testing
(with users doing mostly the same thing on old and new systems) shows there
have been no issues with DBCP, as you suggested would be the case some time
back.

Even though 12.04 is significanly slower, I am moving ahead with its
adoption because the widget code is failing periodically in 9.x because of
locking issues and I love the MacroScreenRenderer stuff.  I suspect that
lots of the slowness is because incoming data is scrubbed for security and
thats a good thing.

Too bad about the Opentaps project.  Their user interface in the 9.x
timeframe was lightyears ahead of ofbiz in my view (and still is for that
matter).  There is lots of good stuff in there that would be useful in
ofbiz. I was particularly impressed with their dojo based pagination. I too
adopted some code from them, but contributed lots back as well.

Hope this helps.

Skip

-----Original Message-----
From: Jacques Le Roux [mailto:[hidden email]]
Sent: Monday, September 16, 2013 10:42 PM
To: [hidden email]
Subject: Re: FW: Mirgration to OFBiz newer version


Skip,

I know what you are talking about, I had my share with a part of Opentaps
adaptation for https://issues.apache.org/jira/browse/OFBIZ-4247
So believe me, I'm very cautious when deprecating stuff.

BTW you said that "Things are about 20% slower." Where, in which areas, did
you cross this? How did you measure?

Jacques

Skip wrote:

> Jacques
>
> Being nearly done with this and my frustrations long since passed, I have
> only a couple of things to say about this for future development.  I have
> been writing developer software for multiple decades and found from
> experience that backward compatibility is your most important concern.
>
> No one should ever change stuff just because they do not like wording or
> style.  For example, there were wholesale changes to the xsd for widgets.
> There was no real benefit to the change other than readability.  It should
> never have been done.  Instead, both words should have been supported.
For
> me, this required days and days of search and replace for no value.
>
> Another is the replacement of new EntityExp() with
> EntityCondition.makeCondition() etc.  Someone decided that they liked this
> better. That is the only possible reason for the change because the
> underling code could have supported either or both APIs.  This kind of
stuff
> is unforgivable in my view for developer software and that is what ofbiz
is.
> It has very little value OOTB and requires developers to give it life.
Some
> of us have thousands of other files written to these apis and to force us
to
> change them for no real value is frustrating to the maximum.
>
> Finally, the change in the password hash is the most agreggious example in
> my view.  It would have been a simple matter to use both (or in this case
> all three hashing methods).  When a password changes, use the newest, best
> method, but allow the existing hashes stored in the database until
changed.

> I modified the ofbiz code to do exactly that and it works fine and took me
> all of an hour once I spend several hours tracking down the problem.
>
> As Ofbiz contributors, I am sure you guys wear two hats.  One as an Ofbiz
> contributor and one as a developer using Ofbiz for end users who actually
> pay you.  As you make changes to Ofbiz code to implement something a
> customer really wants, remember to not screw the hundreds (thousands ?) of
> other Ofbiz developers who depend on it like you do.
>
> Backward compatibility is or should be your most important concern.  Just
> take a look at the compile log for Ofbiz and see how many deprecated Java
> methods we still use.  Then, look at how long they have been deprecated.
> Some for decades.
>
> Thanks for all you do.
>
> Skip
>
> -----Original Message-----
> From: Jacques Le Roux [mailto:[hidden email]]
> Sent: Saturday, September 14, 2013 10:01 AM
> To: [hidden email]
> Subject: Re: FW: Mirgration to OFBiz newer version
>
>
> Pierre Smits wrote:
>> Jacques,
>>
>> Updating (or migrating) is determined by various factors. Each update
>> brings risks and cost. Especially with a system in production. The
>> (perceived value of the) improvements must weigh against the cost
involved
>> in implementing.
>>
>> I thank Skip for sharing the experience.  I will surely use it as part of
>> my migration plans. As far as I can go back it is the first time that it
is
>> done. Sharing implementation and upgrade experiences builds confidence.
It
>> should be encouraged.
>>
>> What worries me though is the experienced decrease in performance.
>
> This would need much more measures to know what we are talking about...
>
>> With kind regards,
>>
>> Pierre Smits

Reply | Threaded
Open this post in threaded view
|

Re: FW: Mirgration to OFBiz newer version

Adrian Crum-3
On 9/17/2013 10:52 AM, Skip wrote:

> I have hundreds of custom services, one of which runs before a custom
> inventory screen loads.  I noticed this screen taking a long time to load
> and timed the service.  It was taking about 2 seconds longer to run.  It is
> basically a database lookup for unshipped orders, outstanding backorders,
> incoming inventory that needs packaged, etc.  Nothing at all extraordinary.
> After timing it 10 times using System.currentMillsecs(), I noted it was
> taking about 18% more time on the same dataset.  In this case, I actually
> put a profiler on both of them trying to see something different in the call
> trees (BTW, jProfiler is killer for this purpose).  Unfortunately, I do not
> remember what was there except for a notably high amount of time in the
> logging code called from the delegator code tree even though logging was set
> to error and there were no errors.  Someday, I'll get back to exploring this
> more.

A better analysis would be helpful. I checked GenericDelegator.java and
there are very few logging calls outside exceptions or error conditions.

Adrian Crum
Sandglass Software
www.sandglass-software.com
Reply | Threaded
Open this post in threaded view
|

RE: FW: Mirgration to OFBiz newer version

SkipDever
Adrian

I agree, a better analysis would be helpful.  Sadly, I did not save the work
I did and do not now have the time.  These logging calls were not in
delegator code itself I suspect, but in code several levels deep called by
it.  I remember questioning why there was so much time spend in the logging
stuff when there was nothing to log.  This is complex stuff.  I spent a day
or so on it and if I could have seen something simple to fix, I would have
and shared it as I did with lots of the other stuff I did.

Skip

-----Original Message-----
From: Adrian Crum [mailto:[hidden email]]
Sent: Tuesday, September 17, 2013 11:43 AM
To: [hidden email]
Subject: Re: FW: Mirgration to OFBiz newer version


On 9/17/2013 10:52 AM, Skip wrote:
> I have hundreds of custom services, one of which runs before a custom
> inventory screen loads.  I noticed this screen taking a long time to load
> and timed the service.  It was taking about 2 seconds longer to run.  It
is
> basically a database lookup for unshipped orders, outstanding backorders,
> incoming inventory that needs packaged, etc.  Nothing at all
extraordinary.
> After timing it 10 times using System.currentMillsecs(), I noted it was
> taking about 18% more time on the same dataset.  In this case, I actually
> put a profiler on both of them trying to see something different in the
call
> trees (BTW, jProfiler is killer for this purpose).  Unfortunately, I do
not
> remember what was there except for a notably high amount of time in the
> logging code called from the delegator code tree even though logging was
set
> to error and there were no errors.  Someday, I'll get back to exploring
this
> more.

A better analysis would be helpful. I checked GenericDelegator.java and
there are very few logging calls outside exceptions or error conditions.

Adrian Crum
Sandglass Software
www.sandglass-software.com

Reply | Threaded
Open this post in threaded view
|

Re: Mirgration to OFBiz newer version

Jacopo Cappellato-4
In reply to this post by SkipDever
On Sep 17, 2013, at 7:52 PM, Skip <[hidden email]> wrote:

> Unfortunately, I do not
> remember what was there except for a notably high amount of time in the
> logging code called from the delegator code tree even though logging was set
> to error and there were no errors

If you are using Freemarker, you may want to check if in your log4j.xml you have the configuration committed in rev. 1308355 (and backported to releases).

Jacopo
Reply | Threaded
Open this post in threaded view
|

Re: FW: Mirgration to OFBiz newer version

Jacques Le Roux
Administrator
In reply to this post by SkipDever
Skip wrote:
> Jacques
>
> Goto ...webtools/control/entitymaint in both 12.x and 9.x.  In 9.x, the page
> loads instantly, in 12.04, it takes several seconds.  Its anecdotally like
> 1000% slower.  Thats the worst case I have seen.

I just tested with OOTB clean instances using Derby with "ant load-demo start" or "ant run-install run"
R09.04 ~ 5'
R10.04 ~ 4'
R11.04 ~ 3'
R12.04 ~ 3'

So I don't find the same as you. I beleive it's almost the same duration for all releases, between 3 to 5' on my machine depending on the environment conditions.
 
Jacques