-----Original Message----- From: Skip [mailto:[hidden email]] Sent: Friday, September 13, 2013 1:16 AM To: Pierre Smits Subject: RE: Mirgration to OFBiz newer version Pierre As I write this, it is it is just past midnight here and I am importing data into the new system from the old for what I hope will be the final time. This is a large data set and takes some 9 hours to get finished. I have been working on this for two months now and just today feel confident enough to put it into production. I have tried to share some of the pitfalls I ran into, but let me recap some of the major ones. But be sure to check the message board history for much more. 1. There is a major change in the way passwords are computed. Because we have external users that we cannot force to change their passwords en-mass, I put the password computations back the way they were so they accept both new and old passwords. Check out HashCrypt.getDigestHash(), and modified checkPassword() in LoginServices to call a new getUnsaltedPasswordHash() which for the most part, just calls HashCrypt.getDigestHash() if the comparePassword() failed. Note that I commented out all this code in HashCrypt and pasted in the stuff from the same file in 9.x code because it will produce a different hash. 2. When importing data from your old system, make sure you set entity-eca-enabled="false" in your entityengine.xml for your delegators before you begin importing and then remove it after. Otherwise, you end up with a lot of crap. 3. There are lots of new ecas that did not exist before. This caused me no end of headaches. Make sure you compare your old xxxseca.xml with the new and comment out those you do not want. 4. Date formats are now yyyy-mm-dd instead of localized. Not a big deal, but I got lots of complaints during beta testing. Still have not resolved this because there is a lot of javascript code to change to put it back the way it was. 5. Lots of changes in widgets xls definitions. After you get it fired up, visit all your custom screens and check the console.log. It will have tons of errors that need cleaned up. 6. Any custom java code that you have that used Double database fields now must use BigDecimal for most of it. Ditto for .bsh scripts. 7. The bsh interpreter that ships with 12.x doesnt work. Restore the old version unless you are going to rewrite all you bsh scripts into .groovy. I am using bsh-2.04b.jar but this puts a bunch of "Experimental..." crap in the log. But, at least it works until I get that figured out. 8. To import your data do an export all, delete JobSandbox.xml, Comm*.xml, ProductKeyword.xml, RuntimeData.xml, UserLoginHistory.xml and Workflow*.xml. Then, on a clean database, do ant load-seed, then import your exported files, being sure to turn off entity-eca-enabled. 8.1. Several table structions have been modified. ProductAverateCost now has a new productAverateCostTypeId (set to SIMPLE_AVG_COST), that has to be added after the export to make it import OK. None of the other additions caused import problems. 9. Things are about 20% slower. 10. I did my best to keep my ofbiz code untouched, but still, have a couple hundred modifitions in the framework and application trees (mostly in applications related to invoices and billing accounts). 11. Because I did not want to migrate all my applications over to the new MacroScreenViewHandler, in each of my custom controller.xml, I made a macroScreen handler and left screen as ScreenWidgetViewHandler. Then, as I changed the ftl to use it, I could just change screen to macroScreen. The new MacroScreenViewHandler is completely integrated with jQuery, you can create some really killer end user screens now with very little code. Its really easy to modify the css to get it to look just like you like. The ftl code behind MacroScreenViewHandler needs some work though to make it the best it can be. I do not need multi-tenat. As such, if I had to do this over again, I would just have replaced the widget code with the new. That is all rather clever. The rest of the changes are mostly useless in my opinion. Much needless crap and pointless refactorings in my view. There is a big rewrite to the entity engine code and all your custom java files will have to be changed. Deletator instead of GenericDelegator, EntityCondition.makeCondition instead of new EntityExpr, and lots stuff like that. I had about a thousand java files that I had to change and it is mostly search and replace, but a pain in the butt for no value. Even though there is mostly no difference, they didn't just deprecate, the removed functions all together. I added lots of the deprecated functions back in to make my life easier. Hope this helps Skip -----Original Message----- From: Pierre Smits [mailto:[hidden email]] Sent: Friday, September 13, 2013 12:09 AM To: [hidden email] Subject: Mirgration to OFBiz newer version Hi Skip, I trust your migration project is going well. Currently I am facing the same situation as I need to migrate several customers to our new multi-tenant environment. As I am in the planning phase, I was wondering if your would be willing to share your best practices and do's and don'ts with me regarding the data migration. With kind regards, Pierre Smits ORRTIZ.COM Services & Solutions for Cloud- Based Manufacturing, Professional Services and Retail & Trade http://www.orrtiz.com |
Skip,
It's good to hear you finally got everything working okay. I'm sure your frustration has influenced your opinions about some of the changes in OFBiz. A lot of time has passed since version 9, and the changes you encountered were discussed in depth, old APIs were deprecated, and eventually obsolete stuff was removed. We don't have the resources to keep deprecated code in the project forever. I hope you understand that. From my perspective, refactorings are not pointless - each one had a very good reason behind it. I don't know of any committer who has enough free time to develop, test, and commit pointless changes. So, if there is a specific change that has you puzzled, I would encourage you to discuss it on the dev mailing list. Adrian Crum Sandglass Software www.sandglass-software.com On 9/13/2013 1:17 AM, Skip wrote: > > -----Original Message----- > From: Skip [mailto:[hidden email]] > Sent: Friday, September 13, 2013 1:16 AM > To: Pierre Smits > Subject: RE: Mirgration to OFBiz newer version > > > Pierre > > As I write this, it is it is just past midnight here and I am importing data > into the new system from the old for what I hope will be the final time. > This is a large data set and takes some 9 hours to get finished. I have > been working on this for two months now and just today feel confident enough > to put it into production. > > I have tried to share some of the pitfalls I ran into, but let me recap some > of the major ones. But be sure to check the message board history for much > more. > > 1. There is a major change in the way passwords are computed. Because we > have external users that we cannot force to change their passwords en-mass, > I put the password computations back the way they were so they accept both > new and old passwords. Check out HashCrypt.getDigestHash(), and modified > checkPassword() in LoginServices to call a new getUnsaltedPasswordHash() > which for the most part, just calls HashCrypt.getDigestHash() if the > comparePassword() failed. Note that I commented out all this code in > HashCrypt and pasted in the stuff from the same file in 9.x code because it > will produce a different hash. > > 2. When importing data from your old system, make sure you set > entity-eca-enabled="false" in your entityengine.xml for your delegators > before you begin importing and then remove it after. Otherwise, you end up > with a lot of crap. > > 3. There are lots of new ecas that did not exist before. This caused me no > end of headaches. Make sure you compare your old xxxseca.xml with the new > and comment out those you do not want. > > 4. Date formats are now yyyy-mm-dd instead of localized. Not a big deal, > but I got lots of complaints during beta testing. Still have not resolved > this because there is a lot of javascript code to change to put it back the > way it was. > > 5. Lots of changes in widgets xls definitions. After you get it fired up, > visit all your custom screens and check the console.log. It will have tons > of errors that need cleaned up. > > 6. Any custom java code that you have that used Double database fields now > must use BigDecimal for most of it. Ditto for .bsh scripts. > > 7. The bsh interpreter that ships with 12.x doesnt work. Restore the old > version unless you are going to rewrite all you bsh scripts into .groovy. I > am using bsh-2.04b.jar but this puts a bunch of "Experimental..." crap in > the log. But, at least it works until I get that figured out. > > 8. To import your data do an export all, delete JobSandbox.xml, Comm*.xml, > ProductKeyword.xml, RuntimeData.xml, UserLoginHistory.xml and Workflow*.xml. > Then, on a clean database, do ant load-seed, then import your exported > files, being sure to turn off entity-eca-enabled. > > 8.1. Several table structions have been modified. ProductAverateCost now > has a new productAverateCostTypeId (set to SIMPLE_AVG_COST), that has to be > added after the export to make it import OK. None of the other additions > caused import problems. > > 9. Things are about 20% slower. > > 10. I did my best to keep my ofbiz code untouched, but still, have a couple > hundred modifitions in the framework and application trees (mostly in > applications related to invoices and billing accounts). > > 11. Because I did not want to migrate all my applications over to the new > MacroScreenViewHandler, in each of my custom controller.xml, I made a > macroScreen handler and left screen as ScreenWidgetViewHandler. Then, as I > changed the ftl to use it, I could just change screen to macroScreen. The > new MacroScreenViewHandler is completely integrated with jQuery, you can > create some really killer end user screens now with very little code. Its > really easy to modify the css to get it to look just like you like. The ftl > code behind MacroScreenViewHandler needs some work though to make it the > best it can be. > > I do not need multi-tenat. As such, if I had to do this over again, I would > just have replaced the widget code with the new. That is all rather clever. > The rest of the changes are mostly useless in my opinion. Much needless > crap and pointless refactorings in my view. There is a big rewrite to the > entity engine code and all your custom java files will have to be changed. > Deletator instead of GenericDelegator, EntityCondition.makeCondition instead > of new EntityExpr, and lots stuff like that. I had about a thousand java > files that I had to change and it is mostly search and replace, but a pain > in the butt for no value. Even though there is mostly no difference, they > didn't just deprecate, the removed functions all together. I added lots of > the deprecated functions back in to make my life easier. > > Hope this helps > > Skip > > > > > -----Original Message----- > From: Pierre Smits [mailto:[hidden email]] > Sent: Friday, September 13, 2013 12:09 AM > To: [hidden email] > Subject: Mirgration to OFBiz newer version > > > Hi Skip, > > > I trust your migration project is going well. > > > Currently I am facing the same situation as I need to migrate several > customers to our new multi-tenant environment. > As I am in the planning phase, I was wondering if your would be willing to > share your best practices and do's and don'ts with me regarding the data > migration. > > > With kind regards, > > > Pierre Smits > > > ORRTIZ.COM > Services & Solutions for Cloud- > Based Manufacturing, Professional > Services and Retail & Trade > http://www.orrtiz.com > |
Administrator
|
To reassure you Skip, OFBiz is more mature now, there should be less changes the next time you will do it.
Except, if we want to take advantage of Java 8 : http://www.techempower.com/blog/2013/03/26/everything-about-java-8/ Of course, updating more often should help... Jacques Adrian Crum wrote: > Skip, > > It's good to hear you finally got everything working okay. > > I'm sure your frustration has influenced your opinions about some of the > changes in OFBiz. > > A lot of time has passed since version 9, and the changes you > encountered were discussed in depth, old APIs were deprecated, and > eventually obsolete stuff was removed. We don't have the resources to > keep deprecated code in the project forever. I hope you understand that. > > From my perspective, refactorings are not pointless - each one had a > very good reason behind it. I don't know of any committer who has enough > free time to develop, test, and commit pointless changes. So, if there > is a specific change that has you puzzled, I would encourage you to > discuss it on the dev mailing list. > > Adrian Crum > Sandglass Software > www.sandglass-software.com > > On 9/13/2013 1:17 AM, Skip wrote: >> >> -----Original Message----- >> From: Skip [mailto:[hidden email]] >> Sent: Friday, September 13, 2013 1:16 AM >> To: Pierre Smits >> Subject: RE: Mirgration to OFBiz newer version >> >> >> Pierre >> >> As I write this, it is it is just past midnight here and I am importing data >> into the new system from the old for what I hope will be the final time. >> This is a large data set and takes some 9 hours to get finished. I have >> been working on this for two months now and just today feel confident enough >> to put it into production. >> >> I have tried to share some of the pitfalls I ran into, but let me recap some >> of the major ones. But be sure to check the message board history for much >> more. >> >> 1. There is a major change in the way passwords are computed. Because we >> have external users that we cannot force to change their passwords en-mass, >> I put the password computations back the way they were so they accept both >> new and old passwords. Check out HashCrypt.getDigestHash(), and modified >> checkPassword() in LoginServices to call a new getUnsaltedPasswordHash() >> which for the most part, just calls HashCrypt.getDigestHash() if the >> comparePassword() failed. Note that I commented out all this code in >> HashCrypt and pasted in the stuff from the same file in 9.x code because it >> will produce a different hash. >> >> 2. When importing data from your old system, make sure you set >> entity-eca-enabled="false" in your entityengine.xml for your delegators >> before you begin importing and then remove it after. Otherwise, you end up >> with a lot of crap. >> >> 3. There are lots of new ecas that did not exist before. This caused me no >> end of headaches. Make sure you compare your old xxxseca.xml with the new >> and comment out those you do not want. >> >> 4. Date formats are now yyyy-mm-dd instead of localized. Not a big deal, >> but I got lots of complaints during beta testing. Still have not resolved >> this because there is a lot of javascript code to change to put it back the >> way it was. >> >> 5. Lots of changes in widgets xls definitions. After you get it fired up, >> visit all your custom screens and check the console.log. It will have tons >> of errors that need cleaned up. >> >> 6. Any custom java code that you have that used Double database fields now >> must use BigDecimal for most of it. Ditto for .bsh scripts. >> >> 7. The bsh interpreter that ships with 12.x doesnt work. Restore the old >> version unless you are going to rewrite all you bsh scripts into .groovy. I >> am using bsh-2.04b.jar but this puts a bunch of "Experimental..." crap in >> the log. But, at least it works until I get that figured out. >> >> 8. To import your data do an export all, delete JobSandbox.xml, Comm*.xml, >> ProductKeyword.xml, RuntimeData.xml, UserLoginHistory.xml and Workflow*.xml. >> Then, on a clean database, do ant load-seed, then import your exported >> files, being sure to turn off entity-eca-enabled. >> >> 8.1. Several table structions have been modified. ProductAverateCost now >> has a new productAverateCostTypeId (set to SIMPLE_AVG_COST), that has to be >> added after the export to make it import OK. None of the other additions >> caused import problems. >> >> 9. Things are about 20% slower. >> >> 10. I did my best to keep my ofbiz code untouched, but still, have a couple >> hundred modifitions in the framework and application trees (mostly in >> applications related to invoices and billing accounts). >> >> 11. Because I did not want to migrate all my applications over to the new >> MacroScreenViewHandler, in each of my custom controller.xml, I made a >> macroScreen handler and left screen as ScreenWidgetViewHandler. Then, as I >> changed the ftl to use it, I could just change screen to macroScreen. The >> new MacroScreenViewHandler is completely integrated with jQuery, you can >> create some really killer end user screens now with very little code. Its >> really easy to modify the css to get it to look just like you like. The ftl >> code behind MacroScreenViewHandler needs some work though to make it the >> best it can be. >> >> I do not need multi-tenat. As such, if I had to do this over again, I would >> just have replaced the widget code with the new. That is all rather clever. >> The rest of the changes are mostly useless in my opinion. Much needless >> crap and pointless refactorings in my view. There is a big rewrite to the >> entity engine code and all your custom java files will have to be changed. >> Deletator instead of GenericDelegator, EntityCondition.makeCondition instead >> of new EntityExpr, and lots stuff like that. I had about a thousand java >> files that I had to change and it is mostly search and replace, but a pain >> in the butt for no value. Even though there is mostly no difference, they >> didn't just deprecate, the removed functions all together. I added lots of >> the deprecated functions back in to make my life easier. >> >> Hope this helps >> >> Skip >> >> >> >> >> -----Original Message----- >> From: Pierre Smits [mailto:[hidden email]] >> Sent: Friday, September 13, 2013 12:09 AM >> To: [hidden email] >> Subject: Mirgration to OFBiz newer version >> >> >> Hi Skip, >> >> >> I trust your migration project is going well. >> >> >> Currently I am facing the same situation as I need to migrate several >> customers to our new multi-tenant environment. >> As I am in the planning phase, I was wondering if your would be willing to >> share your best practices and do's and don'ts with me regarding the data >> migration. >> >> >> With kind regards, >> >> >> Pierre Smits >> >> >> ORRTIZ.COM >> Services & Solutions for Cloud- >> Based Manufacturing, Professional >> Services and Retail & Trade >> http://www.orrtiz.com |
Jacques,
Updating (or migrating) is determined by various factors. Each update brings risks and cost. Especially with a system in production. The (perceived value of the) improvements must weigh against the cost involved in implementing. I thank Skip for sharing the experience. I will surely use it as part of my migration plans. As far as I can go back it is the first time that it is done. Sharing implementation and upgrade experiences builds confidence. It should be encouraged. What worries me though is the experienced decrease in performance. With kind regards, Pierre Smits *ORRTIZ.COM <http://www.orrtiz.com>* Services & Solutions for Cloud- Based Manufacturing, Professional Services and Retail & Trade http://www.orrtiz.com |
In reply to this post by Jacques Le Roux
Something my team is beginning to do with our system, is to create a
pair of scripts, one of which applies any changes to our database (particularly focussing on ALTER TABLE statements, to add new fields along with rational default values and constraints), and another to deal with changes to our code (all written in Perl+SQL+Javascript - so you wouldn't be interested in the details). If the update process is automated, it really doesn't matter how many changes there are, because the script handles it all. Of course, this doesn't help in situations in which one takes the approach of storing specific data as a hash of the data rather than the raw data, and then one decides to use a different hash algorithm to make the hash (passwords are an obvious example). But in such cases, one ought to do a couple things: 1) provide an option to allow use of the original algorithm, and 2) provide an option to let the user use the new algorithm when the old password expires (or one can automate that, but that requires a little extra code). This isn't all that new an idea as I recall using diff on C/C++ files what seems like eons ago. But perhaps it is time that such an idea was applied to OFBiz? Of course, that means that someone needs to take responsibility for maintaining such a pair of scripts. Just a thought... Ted On Sat, Sep 14, 2013 at 7:56 AM, Jacques Le Roux <[hidden email]> wrote: > To reassure you Skip, OFBiz is more mature now, there should be less changes the next time you will do it. > Except, if we want to take advantage of Java 8 : http://www.techempower.com/blog/2013/03/26/everything-about-java-8/ > Of course, updating more often should help... > > Jacques |
Administrator
|
In reply to this post by Pierre Smits
Pierre Smits wrote:
> Jacques, > > Updating (or migrating) is determined by various factors. Each update > brings risks and cost. Especially with a system in production. The > (perceived value of the) improvements must weigh against the cost involved > in implementing. > > I thank Skip for sharing the experience. I will surely use it as part of > my migration plans. As far as I can go back it is the first time that it is > done. Sharing implementation and upgrade experiences builds confidence. It > should be encouraged. > > What worries me though is the experienced decrease in performance. This would need much more measures to know what we are talking about... > With kind regards, > > Pierre Smits |
Administrator
|
In reply to this post by Ted Byers
Thanks for the proposition Ted,
Unfortunately, we are not missing good ideas, but good wills... Jacques Ted Byers wrote: > Something my team is beginning to do with our system, is to create a > pair of scripts, one of which applies any changes to our database > (particularly focussing on ALTER TABLE statements, to add new fields > along with rational default values and constraints), and another to > deal with changes to our code (all written in Perl+SQL+Javascript - so > you wouldn't be interested in the details). If the update process is > automated, it really doesn't matter how many changes there are, > because the script handles it all. Of course, this doesn't help in > situations in which one takes the approach of storing specific data as > a hash of the data rather than the raw data, and then one decides to > use a different hash algorithm to make the hash (passwords are an > obvious example). But in such cases, one ought to do a couple things: > 1) provide an option to allow use of the original algorithm, and 2) > provide an option to let the user use the new algorithm when the old > password expires (or one can automate that, but that requires a little > extra code). This isn't all that new an idea as I recall using diff > on C/C++ files what seems like eons ago. But perhaps it is time that > such an idea was applied to OFBiz? Of course, that means that someone > needs to take responsibility for maintaining such a pair of scripts. > > Just a thought... > > Ted > > On Sat, Sep 14, 2013 at 7:56 AM, Jacques Le Roux > <[hidden email]> wrote: >> To reassure you Skip, OFBiz is more mature now, there should be less changes the next time you will do it. >> Except, if we want to take advantage of Java 8 : http://www.techempower.com/blog/2013/03/26/everything-about-java-8/ >> Of course, updating more often should help... >> >> Jacques |
Administrator
|
In reply to this post by SkipDever
Or rather we have more good ideas than good wills
Jacques Le Roux wrote: > Thanks for the proposition Ted, > > Unfortunately, we are not missing good ideas, but good wills... > > Jacques > > Ted Byers wrote: >> Something my team is beginning to do with our system, is to create a >> pair of scripts, one of which applies any changes to our database >> (particularly focussing on ALTER TABLE statements, to add new fields >> along with rational default values and constraints), and another to >> deal with changes to our code (all written in Perl+SQL+Javascript - so >> you wouldn't be interested in the details). If the update process is >> automated, it really doesn't matter how many changes there are, >> because the script handles it all. Of course, this doesn't help in >> situations in which one takes the approach of storing specific data as >> a hash of the data rather than the raw data, and then one decides to >> use a different hash algorithm to make the hash (passwords are an >> obvious example). But in such cases, one ought to do a couple things: >> 1) provide an option to allow use of the original algorithm, and 2) >> provide an option to let the user use the new algorithm when the old >> password expires (or one can automate that, but that requires a little >> extra code). This isn't all that new an idea as I recall using diff >> on C/C++ files what seems like eons ago. But perhaps it is time that >> such an idea was applied to OFBiz? Of course, that means that someone >> needs to take responsibility for maintaining such a pair of scripts. >> >> Just a thought... >> >> Ted >> >> On Sat, Sep 14, 2013 at 7:56 AM, Jacques Le Roux >> <[hidden email]> wrote: >>> To reassure you Skip, OFBiz is more mature now, there should be less changes the next time you will do it. >>> Except, if we want to take advantage of Java 8 : http://www.techempower.com/blog/2013/03/26/everything-about-java-8/ >>> Of course, updating more often should help... >>> >>> Jacques |
In reply to this post by Jacques Le Roux
Jacques
Being nearly done with this and my frustrations long since passed, I have only a couple of things to say about this for future development. I have been writing developer software for multiple decades and found from experience that backward compatibility is your most important concern. No one should ever change stuff just because they do not like wording or style. For example, there were wholesale changes to the xsd for widgets. There was no real benefit to the change other than readability. It should never have been done. Instead, both words should have been supported. For me, this required days and days of search and replace for no value. Another is the replacement of new EntityExp() with EntityCondition.makeCondition() etc. Someone decided that they liked this better. That is the only possible reason for the change because the underling code could have supported either or both APIs. This kind of stuff is unforgivable in my view for developer software and that is what ofbiz is. It has very little value OOTB and requires developers to give it life. Some of us have thousands of other files written to these apis and to force us to change them for no real value is frustrating to the maximum. Finally, the change in the password hash is the most agreggious example in my view. It would have been a simple matter to use both (or in this case all three hashing methods). When a password changes, use the newest, best method, but allow the existing hashes stored in the database until changed. I modified the ofbiz code to do exactly that and it works fine and took me all of an hour once I spend several hours tracking down the problem. As Ofbiz contributors, I am sure you guys wear two hats. One as an Ofbiz contributor and one as a developer using Ofbiz for end users who actually pay you. As you make changes to Ofbiz code to implement something a customer really wants, remember to not screw the hundreds (thousands ?) of other Ofbiz developers who depend on it like you do. Backward compatibility is or should be your most important concern. Just take a look at the compile log for Ofbiz and see how many deprecated Java methods we still use. Then, look at how long they have been deprecated. Some for decades. Thanks for all you do. Skip -----Original Message----- From: Jacques Le Roux [mailto:[hidden email]] Sent: Saturday, September 14, 2013 10:01 AM To: [hidden email] Subject: Re: FW: Mirgration to OFBiz newer version Pierre Smits wrote: > Jacques, > > Updating (or migrating) is determined by various factors. Each update > brings risks and cost. Especially with a system in production. The > (perceived value of the) improvements must weigh against the cost involved > in implementing. > > I thank Skip for sharing the experience. I will surely use it as part of > my migration plans. As far as I can go back it is the first time that it is > done. Sharing implementation and upgrade experiences builds confidence. It > should be encouraged. > > What worries me though is the experienced decrease in performance. This would need much more measures to know what we are talking about... > With kind regards, > > Pierre Smits |
Administrator
|
Skip,
I know what you are talking about, I had my share with a part of Opentaps adaptation for https://issues.apache.org/jira/browse/OFBIZ-4247 So believe me, I'm very cautious when deprecating stuff. BTW you said that "Things are about 20% slower." Where, in which areas, did you cross this? How did you measure? Jacques Skip wrote: > Jacques > > Being nearly done with this and my frustrations long since passed, I have > only a couple of things to say about this for future development. I have > been writing developer software for multiple decades and found from > experience that backward compatibility is your most important concern. > > No one should ever change stuff just because they do not like wording or > style. For example, there were wholesale changes to the xsd for widgets. > There was no real benefit to the change other than readability. It should > never have been done. Instead, both words should have been supported. For > me, this required days and days of search and replace for no value. > > Another is the replacement of new EntityExp() with > EntityCondition.makeCondition() etc. Someone decided that they liked this > better. That is the only possible reason for the change because the > underling code could have supported either or both APIs. This kind of stuff > is unforgivable in my view for developer software and that is what ofbiz is. > It has very little value OOTB and requires developers to give it life. Some > of us have thousands of other files written to these apis and to force us to > change them for no real value is frustrating to the maximum. > > Finally, the change in the password hash is the most agreggious example in > my view. It would have been a simple matter to use both (or in this case > all three hashing methods). When a password changes, use the newest, best > method, but allow the existing hashes stored in the database until changed. > I modified the ofbiz code to do exactly that and it works fine and took me > all of an hour once I spend several hours tracking down the problem. > > As Ofbiz contributors, I am sure you guys wear two hats. One as an Ofbiz > contributor and one as a developer using Ofbiz for end users who actually > pay you. As you make changes to Ofbiz code to implement something a > customer really wants, remember to not screw the hundreds (thousands ?) of > other Ofbiz developers who depend on it like you do. > > Backward compatibility is or should be your most important concern. Just > take a look at the compile log for Ofbiz and see how many deprecated Java > methods we still use. Then, look at how long they have been deprecated. > Some for decades. > > Thanks for all you do. > > Skip > > -----Original Message----- > From: Jacques Le Roux [mailto:[hidden email]] > Sent: Saturday, September 14, 2013 10:01 AM > To: [hidden email] > Subject: Re: FW: Mirgration to OFBiz newer version > > > Pierre Smits wrote: >> Jacques, >> >> Updating (or migrating) is determined by various factors. Each update >> brings risks and cost. Especially with a system in production. The >> (perceived value of the) improvements must weigh against the cost involved >> in implementing. >> >> I thank Skip for sharing the experience. I will surely use it as part of >> my migration plans. As far as I can go back it is the first time that it is >> done. Sharing implementation and upgrade experiences builds confidence. It >> should be encouraged. >> >> What worries me though is the experienced decrease in performance. > > This would need much more measures to know what we are talking about... > >> With kind regards, >> >> Pierre Smits |
Jacques
Goto ...webtools/control/entitymaint in both 12.x and 9.x. In 9.x, the page loads instantly, in 12.04, it takes several seconds. Its anecdotally like 1000% slower. Thats the worst case I have seen. I have hundreds of custom services, one of which runs before a custom inventory screen loads. I noticed this screen taking a long time to load and timed the service. It was taking about 2 seconds longer to run. It is basically a database lookup for unshipped orders, outstanding backorders, incoming inventory that needs packaged, etc. Nothing at all extraordinary. After timing it 10 times using System.currentMillsecs(), I noted it was taking about 18% more time on the same dataset. In this case, I actually put a profiler on both of them trying to see something different in the call trees (BTW, jProfiler is killer for this purpose). Unfortunately, I do not remember what was there except for a notably high amount of time in the logging code called from the delegator code tree even though logging was set to error and there were no errors. Someday, I'll get back to exploring this more. Note: this is with the 9.x code running Minerva and the 12.04 code running DBCP. If I put them both on Minverva, 12.04 is something 35% slower. I did not test the other way, i.e. DBCP on both because I am not sure it would run at all on 9.x code. Second, I have some custom rest services attached to external POJO screens. These services get a session, login, sometimes get a shopping cart, sometimes lookup prices and stock, etc. These felt slow as well, so I timed one of them(again using System.currentMillsecs()) and found the REST calls return 20% slower. This is essentially just the amount of time it takes on a local machine to get a JSON response from http://localhost:8080/someurl. You can't really see the time difference here unless you do it a hundred times because the time for each is so small, but for these POJO apps, that is exactly what they do, login, lookup a customer, get some inventory counts, etc, all in rapid succession. One more thing, in my 9.x code, I used Minerva because because DBCP was causing some errors. When I tried the DB stuff with Minerva in the 12.04 code, it was like 35% slower (both systems running Minerva). After discussing it with you here on the ML, I moved to DBCP and speed improved to a generally acceptable level (except as noted above). A months beta testing (with users doing mostly the same thing on old and new systems) shows there have been no issues with DBCP, as you suggested would be the case some time back. Even though 12.04 is significanly slower, I am moving ahead with its adoption because the widget code is failing periodically in 9.x because of locking issues and I love the MacroScreenRenderer stuff. I suspect that lots of the slowness is because incoming data is scrubbed for security and thats a good thing. Too bad about the Opentaps project. Their user interface in the 9.x timeframe was lightyears ahead of ofbiz in my view (and still is for that matter). There is lots of good stuff in there that would be useful in ofbiz. I was particularly impressed with their dojo based pagination. I too adopted some code from them, but contributed lots back as well. Hope this helps. Skip -----Original Message----- From: Jacques Le Roux [mailto:[hidden email]] Sent: Monday, September 16, 2013 10:42 PM To: [hidden email] Subject: Re: FW: Mirgration to OFBiz newer version Skip, I know what you are talking about, I had my share with a part of Opentaps adaptation for https://issues.apache.org/jira/browse/OFBIZ-4247 So believe me, I'm very cautious when deprecating stuff. BTW you said that "Things are about 20% slower." Where, in which areas, did you cross this? How did you measure? Jacques Skip wrote: > Jacques > > Being nearly done with this and my frustrations long since passed, I have > only a couple of things to say about this for future development. I have > been writing developer software for multiple decades and found from > experience that backward compatibility is your most important concern. > > No one should ever change stuff just because they do not like wording or > style. For example, there were wholesale changes to the xsd for widgets. > There was no real benefit to the change other than readability. It should > never have been done. Instead, both words should have been supported. > me, this required days and days of search and replace for no value. > > Another is the replacement of new EntityExp() with > EntityCondition.makeCondition() etc. Someone decided that they liked this > better. That is the only possible reason for the change because the > underling code could have supported either or both APIs. This kind of stuff > is unforgivable in my view for developer software and that is what ofbiz is. > It has very little value OOTB and requires developers to give it life. Some > of us have thousands of other files written to these apis and to force us to > change them for no real value is frustrating to the maximum. > > Finally, the change in the password hash is the most agreggious example in > my view. It would have been a simple matter to use both (or in this case > all three hashing methods). When a password changes, use the newest, best > method, but allow the existing hashes stored in the database until changed. > I modified the ofbiz code to do exactly that and it works fine and took me > all of an hour once I spend several hours tracking down the problem. > > As Ofbiz contributors, I am sure you guys wear two hats. One as an Ofbiz > contributor and one as a developer using Ofbiz for end users who actually > pay you. As you make changes to Ofbiz code to implement something a > customer really wants, remember to not screw the hundreds (thousands ?) of > other Ofbiz developers who depend on it like you do. > > Backward compatibility is or should be your most important concern. Just > take a look at the compile log for Ofbiz and see how many deprecated Java > methods we still use. Then, look at how long they have been deprecated. > Some for decades. > > Thanks for all you do. > > Skip > > -----Original Message----- > From: Jacques Le Roux [mailto:[hidden email]] > Sent: Saturday, September 14, 2013 10:01 AM > To: [hidden email] > Subject: Re: FW: Mirgration to OFBiz newer version > > > Pierre Smits wrote: >> Jacques, >> >> Updating (or migrating) is determined by various factors. Each update >> brings risks and cost. Especially with a system in production. The >> (perceived value of the) improvements must weigh against the cost >> in implementing. >> >> I thank Skip for sharing the experience. I will surely use it as part of >> my migration plans. As far as I can go back it is the first time that it is >> done. Sharing implementation and upgrade experiences builds confidence. It >> should be encouraged. >> >> What worries me though is the experienced decrease in performance. > > This would need much more measures to know what we are talking about... > >> With kind regards, >> >> Pierre Smits |
On 9/17/2013 10:52 AM, Skip wrote:
> I have hundreds of custom services, one of which runs before a custom > inventory screen loads. I noticed this screen taking a long time to load > and timed the service. It was taking about 2 seconds longer to run. It is > basically a database lookup for unshipped orders, outstanding backorders, > incoming inventory that needs packaged, etc. Nothing at all extraordinary. > After timing it 10 times using System.currentMillsecs(), I noted it was > taking about 18% more time on the same dataset. In this case, I actually > put a profiler on both of them trying to see something different in the call > trees (BTW, jProfiler is killer for this purpose). Unfortunately, I do not > remember what was there except for a notably high amount of time in the > logging code called from the delegator code tree even though logging was set > to error and there were no errors. Someday, I'll get back to exploring this > more. A better analysis would be helpful. I checked GenericDelegator.java and there are very few logging calls outside exceptions or error conditions. Adrian Crum Sandglass Software www.sandglass-software.com |
Adrian
I agree, a better analysis would be helpful. Sadly, I did not save the work I did and do not now have the time. These logging calls were not in delegator code itself I suspect, but in code several levels deep called by it. I remember questioning why there was so much time spend in the logging stuff when there was nothing to log. This is complex stuff. I spent a day or so on it and if I could have seen something simple to fix, I would have and shared it as I did with lots of the other stuff I did. Skip -----Original Message----- From: Adrian Crum [mailto:[hidden email]] Sent: Tuesday, September 17, 2013 11:43 AM To: [hidden email] Subject: Re: FW: Mirgration to OFBiz newer version On 9/17/2013 10:52 AM, Skip wrote: > I have hundreds of custom services, one of which runs before a custom > inventory screen loads. I noticed this screen taking a long time to load > and timed the service. It was taking about 2 seconds longer to run. It is > basically a database lookup for unshipped orders, outstanding backorders, > incoming inventory that needs packaged, etc. Nothing at all extraordinary. > After timing it 10 times using System.currentMillsecs(), I noted it was > taking about 18% more time on the same dataset. In this case, I actually > put a profiler on both of them trying to see something different in the call > trees (BTW, jProfiler is killer for this purpose). Unfortunately, I do not > remember what was there except for a notably high amount of time in the > logging code called from the delegator code tree even though logging was set > to error and there were no errors. Someday, I'll get back to exploring this > more. A better analysis would be helpful. I checked GenericDelegator.java and there are very few logging calls outside exceptions or error conditions. Adrian Crum Sandglass Software www.sandglass-software.com |
In reply to this post by SkipDever
On Sep 17, 2013, at 7:52 PM, Skip <[hidden email]> wrote:
> Unfortunately, I do not > remember what was there except for a notably high amount of time in the > logging code called from the delegator code tree even though logging was set > to error and there were no errors If you are using Freemarker, you may want to check if in your log4j.xml you have the configuration committed in rev. 1308355 (and backported to releases). Jacopo |
Administrator
|
In reply to this post by SkipDever
Skip wrote:
> Jacques > > Goto ...webtools/control/entitymaint in both 12.x and 9.x. In 9.x, the page > loads instantly, in 12.04, it takes several seconds. Its anecdotally like > 1000% slower. Thats the worst case I have seen. I just tested with OOTB clean instances using Derby with "ant load-demo start" or "ant run-install run" R09.04 ~ 5' R10.04 ~ 4' R11.04 ~ 3' R12.04 ~ 3' So I don't find the same as you. I beleive it's almost the same duration for all releases, between 3 to 5' on my machine depending on the environment conditions. Jacques |
Free forum by Nabble | Edit this page |