Bringing up version 12.04

[SkipDever]

I am bringing up version 12.04 from 9.x.  I am looking at the
entityengine.xml file and note that my old version uses the
MinervaConnectionFactory.  That is because way back then, the
DBCPConnectionFactory was giving me LOTS of problems.

Can anyone tell me the difference between the two and what would the
advantage be using the default DBCPConnectionFactory?

Thanks

Skip

[Jacques Le Roux]

If you are OK with Minerva you could keep it for now and wait a move to Atomikos https://issues.apache.org/jira/browse/OFBIZ-5129, jumping above DBCP
I don't remember new features brought by DBCP over Minerva. Maybe there were better performances or capabilities, but actually I don't know any available comparisons.
Some minor features were dropped when switching to DBCP http://svn.apache.org/viewvc/ofbiz/trunk/framework/entity/dtd/entity-config.xsd?r1=1482911&r2=1482910&pathrev=1482911
Minerva was mostly replaced by DBCP because the Minerva version OFBiz used was patched by the OFBiz team and the Minerva team never integrated these patches.
Both Minerva and DBCP are nowaydays not maintained, or, for DBCP, not for a long moment at least.

Note: we also spoked about
http://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html 
http://markmail.org/message/mc6334bscnaek3qk

HTH

Jacques

From: "Skip" <[hidden email]>

>I am bringing up version 12.04 from 9.x.  I am looking at the
> entityengine.xml file and note that my old version uses the
> MinervaConnectionFactory.  That is because way back then, the
> DBCPConnectionFactory was giving me LOTS of problems.
>
> Can anyone tell me the difference between the two and what would the
> advantage be using the default DBCPConnectionFactory?
>
> Thanks
>
> Skip
>

[Adrian Crum-3]

 From what I recall, Minerva had a problem with memory leaks. Also, I
believe we have a Jira issue that fixes that.

DBCP is being maintained. I am a member of the Apache Commons dev team
and I can assure you there is plenty of activity in DBCP.

-Adrian

On 5/16/2013 8:49 AM, Jacques Le Roux wrote:

> If you are OK with Minerva you could keep it for now and wait a move to Atomikos https://issues.apache.org/jira/browse/OFBIZ-5129, jumping above DBCP
> I don't remember new features brought by DBCP over Minerva. Maybe there were better performances or capabilities, but actually I don't know any available comparisons.
> Some minor features were dropped when switching to DBCP http://svn.apache.org/viewvc/ofbiz/trunk/framework/entity/dtd/entity-config.xsd?r1=1482911&r2=1482910&pathrev=1482911
> Minerva was mostly replaced by DBCP because the Minerva version OFBiz used was patched by the OFBiz team and the Minerva team never integrated these patches.
> Both Minerva and DBCP are nowaydays not maintained, or, for DBCP, not for a long moment at least.
>
> Note: we also spoked about
> http://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html
> http://markmail.org/message/mc6334bscnaek3qk
>
> HTH
>
> Jacques
>
> From: "Skip" <[hidden email]>
>> I am bringing up version 12.04 from 9.x.  I am looking at the
>> entityengine.xml file and note that my old version uses the
>> MinervaConnectionFactory.  That is because way back then, the
>> DBCPConnectionFactory was giving me LOTS of problems.
>>
>> Can anyone tell me the difference between the two and what would the
>> advantage be using the default DBCPConnectionFactory?
>>
>> Thanks
>>
>> Skip
>>

[Jacques Le Roux]

From: "Adrian Crum" <[hidden email]>
> From what I recall, Minerva had a problem with memory leaks. Also, I
> believe we have a Jira issue that fixes that.
>
> DBCP is being maintained. I am a member of the Apache Commons dev team
> and I can assure you there is plenty of activity in DBCP.

Actually I participated to the DBCP effort as well https://issues.apache.org/jira/issues/?jql=project%20%3D%20OFBIZ%20AND%20text%20~%20%22dbcp%22

BTW just noticed https://issues.apache.org/jira/browse/OFBIZ-4283

I read that DBCP was stagnant from the Tomcat team, http://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html#Introduction point 5, which is indeed maybe biased

Still, I agree moving to Atomikos seems the way.

Jacques
 

> -Adrian
>
> On 5/16/2013 8:49 AM, Jacques Le Roux wrote:
>> If you are OK with Minerva you could keep it for now and wait a move to Atomikos https://issues.apache.org/jira/browse/OFBIZ-5129, jumping above DBCP
>> I don't remember new features brought by DBCP over Minerva. Maybe there were better performances or capabilities, but actually I don't know any available comparisons.
>> Some minor features were dropped when switching to DBCP http://svn.apache.org/viewvc/ofbiz/trunk/framework/entity/dtd/entity-config.xsd?r1=1482911&r2=1482910&pathrev=1482911
>> Minerva was mostly replaced by DBCP because the Minerva version OFBiz used was patched by the OFBiz team and the Minerva team never integrated these patches.
>> Both Minerva and DBCP are nowaydays not maintained, or, for DBCP, not for a long moment at least.
>>
>> Note: we also spoked about
>> http://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html
>> http://markmail.org/message/mc6334bscnaek3qk
>>
>> HTH
>>
>> Jacques
>>
>> From: "Skip" <[hidden email]>
>>> I am bringing up version 12.04 from 9.x.  I am looking at the
>>> entityengine.xml file and note that my old version uses the
>>> MinervaConnectionFactory.  That is because way back then, the
>>> DBCPConnectionFactory was giving me LOTS of problems.
>>>
>>> Can anyone tell me the difference between the two and what would the
>>> advantage be using the default DBCPConnectionFactory?
>>>
>>> Thanks
>>>
>>> Skip
>>>
>

[SkipDever]

This document details the requirements to upgrade from previous versions of
Ofbiz (10.0 and back) to the 12.04 version.

I upgraded from 9.x to 12.04, so my experiences are probably severe.

It is assumed that you have an existing database.  If you don't then none of
this matters much to you except perhaps for the bsh script comments below.

First, this version changes the field type definitions in
framework/entity/fieldtypes.  For example, NUMERIC NUMERIC(18,0) was changed
to NUMERIC(20,0).  This causes LOTS of warnings, thousands of them.  What I
did was simply copy the old field type definition file from my old
installation.  I am using postgres, so I copied fieldtypepostgres.xml to my
new installation.

Problem solved.

Next, I had LOTS of errors about duplicate fields in view-entity.  The old
entity engine did not complain about these.  The new one does.  These are
your errors and should be fixed and are usually associated with alias-all.
I could not get "ant load-extseed" to begin the compile phase until I had
them all fixed even though they were listed as [INFO].

extend-entity is broken for views.  I am pretty sure about this.  If you
have any, simply modify the extended view with your modifications.  Expect
this to be fixed at some point.

Next, if you use Hibernate, the lucene library used will not work. Remove
the lucene-3.5 and replace it with lucene-core-2.4.1.jar from your previous
installation.  I have read that the latest hibernate works with the latest
lucene, but I have not tested this yet.

Then, there are lots of compile errors about new EntityExpr() and new
EntityConditionList().  I did some research and these were deprecated at
least in version 10 and then removed some time later.  I fixed all my java
code to use the recommended EntityCondition.makeCondition, but after getting
it to compile, I ended up with several thousand bsh scripts that used the
deprecated constructors.  So, I recommend that you just copy back in the
deprecated constructors for both classes.  That is what I did. I have not
yet completely tested all this stuff, but it appears to work fine.



Next comes a HUGE problem with password encryption.  The old getDigestHash()
has been deprecated.  Worse, they changed the algorithm in subtle ways that
makes it produce hashes that are not the same as the old way.  This means
that NONE of your passwords will ever work.  A bummer if you have lots of
customers.  To fix this, I did the following:

First, I deleted the old getDigestHash() functions in HashCrypt.java and
replaced them with the ones from the 9.x build.  Next, I modified
LoginServices by adding the function:

    public static String getUnsaltedPasswordHash(String str) {
        String hashType =
UtilProperties.getPropertyValue("security.properties",
"password.encrypt.hash.type");

        if (hashType == null || hashType.length() == 0) {
            Debug.logWarning("Password encrypt hash type is not specified in
security.properties, use SHA", module);
            hashType = "SHA";
        }

        return HashCrypt.getDigestHash(str, hashType);
    }

Finally, around line 983, I modified checkPassword() to look like this:
    private static boolean checkPassword(String oldPassword, boolean
useEncryption, String currentPassword) {
        boolean passwordMatches = false;
        if (oldPassword != null) {
            if (useEncryption) {
                passwordMatches = HashCrypt.comparePassword(oldPassword,
getHashType(), currentPassword);
                //Added by skipd if hashcrypt does not work, try the old way
                if(!passwordMatches) {
                String oldTypePassword =
getUnsaltedPasswordHash(currentPassword);
                passwordMatches = oldPassword.equals(oldTypePassword);
                }
            } else {
                passwordMatches = oldPassword.equals(currentPassword);
            }
        }
        if (!passwordMatches &&
"true".equals(UtilProperties.getPropertyValue("security.properties",
"password.accept.encrypted.and.plain"))) {
            passwordMatches = currentPassword.equals(oldPassword);
        }
        return passwordMatches;
    }

Note the addition of the call to getUnsaltedPasswordHash();

Whow, now my old passwords work just fine as well as the new ones.  I am
going to send emails to my customers to ask them to change their password at
some point so we are utilizing the new and more secure salted ones, but for
the time being, they can still use the applications I wrote.

At this point, I can now log into ofbiz webapps and it all looks ok.  But,
when I start running any of my apps, they ALL crash with bsh script errors.

This used to work:

userLogin = request.getAttribute("userLogin");
if(userLogin != null)
        do something

Now, even if userLogin does not exist, if(userLogin != null) return true
because userLogin is void.  void for kryst sake, not null.

Worse yet, this code used to work

user = null;
...
context.put("user", user);

Now, it fails on the context.put("user", user); line saying that user is
undefined.  Jezzus, I can see user two lines up, are you stupid (or so I was
thinking).

Gads, I was getting frustrated at this point.  If you realize the
implications of the above, you realize that you are forced to learn a new
language and rewrite all you bsh scripts. The prospect of re-writting
literally thousands of bsh scripts was not making me happy.  Instead, I
loaded the old bash engine which thankfully is still part of 12.04.  To do
it, I modified ModelWidgetAction.java around line 399 modifying
Script.runAction around line 422 adding this code:

            //Modified by skipd to run the old bsh engine if this is a bsh
script
            } else if(location.endsWith(".bsh")){
                BshUtil.runBshAtLocation(this.location, context);
            } else {
                ScriptUtil.executeScript(this.location, this.method,
context);
            }

This causes the old bsh engine to run your bsh scripts and they now work
just fine and the groovy files still use the ScriptUtil class to run them.

Finally, things are beginning to work.  I still have lots of work to do
adding back in the modifications I have made over the years, but at least I
expected that.

Hope this helps someone else.

Skip

Now a rant.  First I want to say that 12.04 is amazingly well written.
However...

It is my hope that in the future, you NEVER remove deprecated stuff.  Leave
it deprecated unless there is a compelling reason to remove it.  The rewrite
of getDigestHash() was particularly unforgivable in my view.  Remember that
there are hundreds of people with thousands of applications that are used by
millions of our customers.  Our customers are not going to be happy spending
thousands of dollars getting us to re-write our code to a replaced or
modified api, especially when there was no good reason to do so.  This
removal of new EntityExpr() really got my goat because it was doing no harm
remaining in place.  There should be COMPELLING reasons to remove stuff or
alter APIs in any way.

Skip

[Adrian Crum-3]

Skip,

It's good to hear you are making progress.

I can sympathize with your frustration about the password encryption
change - I have a client who is facing a similar situation. The old way
of converting the hash to a string was incompatible with other
encryption tools, so integrating OFBiz with external systems was
difficult. The new password hashing code fixed that, but as you
discovered, it can be a pain getting existing passwords converted over
to the new version.

Regarding deprecation: The development team had a lengthy discussion
about that years ago. In the past, we would try very hard to keep
everything backward-compatible. But over time, that caused the project
to stagnate. (The creation of the Moqui project was partially motivated
by the inability to innovate because of backward-compatibility issues.)
So, starting around Release 11, we decided to relax the constraints a
bit and allow non-backward-compatible changes between major release
versions. That change brought innovation back in the project, but at the
cost of major version incompatibility. My recommendations would be to
subscribe to the dev mailing list so you can stay current on upcoming
changes, and also try to upgrade more frequently.

-Adrian

On 5/18/2013 10:16 PM, Skip wrote:

> This document details the requirements to upgrade from previous versions of
> Ofbiz (10.0 and back) to the 12.04 version.
>
> I upgraded from 9.x to 12.04, so my experiences are probably severe.
>
> It is assumed that you have an existing database.  If you don't then none of
> this matters much to you except perhaps for the bsh script comments below.
>
> First, this version changes the field type definitions in
> framework/entity/fieldtypes.  For example, NUMERIC NUMERIC(18,0) was changed
> to NUMERIC(20,0).  This causes LOTS of warnings, thousands of them.  What I
> did was simply copy the old field type definition file from my old
> installation.  I am using postgres, so I copied fieldtypepostgres.xml to my
> new installation.
>
> Problem solved.
>
> Next, I had LOTS of errors about duplicate fields in view-entity.  The old
> entity engine did not complain about these.  The new one does.  These are
> your errors and should be fixed and are usually associated with alias-all.
> I could not get "ant load-extseed" to begin the compile phase until I had
> them all fixed even though they were listed as [INFO].
>
> extend-entity is broken for views.  I am pretty sure about this.  If you
> have any, simply modify the extended view with your modifications.  Expect
> this to be fixed at some point.
>
> Next, if you use Hibernate, the lucene library used will not work. Remove
> the lucene-3.5 and replace it with lucene-core-2.4.1.jar from your previous
> installation.  I have read that the latest hibernate works with the latest
> lucene, but I have not tested this yet.
>
> Then, there are lots of compile errors about new EntityExpr() and new
> EntityConditionList().  I did some research and these were deprecated at
> least in version 10 and then removed some time later.  I fixed all my java
> code to use the recommended EntityCondition.makeCondition, but after getting
> it to compile, I ended up with several thousand bsh scripts that used the
> deprecated constructors.  So, I recommend that you just copy back in the
> deprecated constructors for both classes.  That is what I did. I have not
> yet completely tested all this stuff, but it appears to work fine.
>
>
>
> Next comes a HUGE problem with password encryption.  The old getDigestHash()
> has been deprecated.  Worse, they changed the algorithm in subtle ways that
> makes it produce hashes that are not the same as the old way.  This means
> that NONE of your passwords will ever work.  A bummer if you have lots of
> customers.  To fix this, I did the following:
>
> First, I deleted the old getDigestHash() functions in HashCrypt.java and
> replaced them with the ones from the 9.x build.  Next, I modified
> LoginServices by adding the function:
>
>      public static String getUnsaltedPasswordHash(String str) {
>          String hashType =
> UtilProperties.getPropertyValue("security.properties",
> "password.encrypt.hash.type");
>
>          if (hashType == null || hashType.length() == 0) {
>              Debug.logWarning("Password encrypt hash type is not specified in
> security.properties, use SHA", module);
>              hashType = "SHA";
>          }
>
>          return HashCrypt.getDigestHash(str, hashType);
>      }
>
> Finally, around line 983, I modified checkPassword() to look like this:
>      private static boolean checkPassword(String oldPassword, boolean
> useEncryption, String currentPassword) {
>          boolean passwordMatches = false;
>          if (oldPassword != null) {
>              if (useEncryption) {
>                  passwordMatches = HashCrypt.comparePassword(oldPassword,
> getHashType(), currentPassword);
>                  //Added by skipd if hashcrypt does not work, try the old way
>                  if(!passwordMatches) {
>                   String oldTypePassword =
> getUnsaltedPasswordHash(currentPassword);
>                   passwordMatches = oldPassword.equals(oldTypePassword);
>                  }
>              } else {
>                  passwordMatches = oldPassword.equals(currentPassword);
>              }
>          }
>          if (!passwordMatches &&
> "true".equals(UtilProperties.getPropertyValue("security.properties",
> "password.accept.encrypted.and.plain"))) {
>              passwordMatches = currentPassword.equals(oldPassword);
>          }
>          return passwordMatches;
>      }
>
> Note the addition of the call to getUnsaltedPasswordHash();
>
> Whow, now my old passwords work just fine as well as the new ones.  I am
> going to send emails to my customers to ask them to change their password at
> some point so we are utilizing the new and more secure salted ones, but for
> the time being, they can still use the applications I wrote.
>
> At this point, I can now log into ofbiz webapps and it all looks ok.  But,
> when I start running any of my apps, they ALL crash with bsh script errors.
>
> This used to work:
>
> userLogin = request.getAttribute("userLogin");
> if(userLogin != null)
> do something
>
> Now, even if userLogin does not exist, if(userLogin != null) return true
> because userLogin is void.  void for kryst sake, not null.
>
> Worse yet, this code used to work
>
> user = null;
> ...
> context.put("user", user);
>
> Now, it fails on the context.put("user", user); line saying that user is
> undefined.  Jezzus, I can see user two lines up, are you stupid (or so I was
> thinking).
>
> Gads, I was getting frustrated at this point.  If you realize the
> implications of the above, you realize that you are forced to learn a new
> language and rewrite all you bsh scripts. The prospect of re-writting
> literally thousands of bsh scripts was not making me happy.  Instead, I
> loaded the old bash engine which thankfully is still part of 12.04.  To do
> it, I modified ModelWidgetAction.java around line 399 modifying
> Script.runAction around line 422 adding this code:
>
>              //Modified by skipd to run the old bsh engine if this is a bsh
> script
>              } else if(location.endsWith(".bsh")){
>                  BshUtil.runBshAtLocation(this.location, context);
>              } else {
>                  ScriptUtil.executeScript(this.location, this.method,
> context);
>              }
>
> This causes the old bsh engine to run your bsh scripts and they now work
> just fine and the groovy files still use the ScriptUtil class to run them.
>
> Finally, things are beginning to work.  I still have lots of work to do
> adding back in the modifications I have made over the years, but at least I
> expected that.
>
> Hope this helps someone else.
>
> Skip
>
> Now a rant.  First I want to say that 12.04 is amazingly well written.
> However...
>
> It is my hope that in the future, you NEVER remove deprecated stuff.  Leave
> it deprecated unless there is a compelling reason to remove it.  The rewrite
> of getDigestHash() was particularly unforgivable in my view.  Remember that
> there are hundreds of people with thousands of applications that are used by
> millions of our customers.  Our customers are not going to be happy spending
> thousands of dollars getting us to re-write our code to a replaced or
> modified api, especially when there was no good reason to do so.  This
> removal of new EntityExpr() really got my goat because it was doing no harm
> remaining in place.  There should be COMPELLING reasons to remove stuff or
> alter APIs in any way.
>
> Skip
>