Dev - Import Dir (XML-s) consume JDBC connections and break ofbiz

[Marcell Gal]

Hi Guys,

We've seen a strange error, below are my notes.
Any ideas ?

  Marcell

'''Import XML-s from Dir'''

* use the "delete after import" flag so we know what was imported

It starts importing, but unfortunately after a few files, something bad
happens (the pool is consumed? why isn't it waiting?), and from than it's all
hell:

Message: Unable to esablish a connection with the database. Connection was
null!
---- stack
trace ---------------------------------------------------------------
org.ofbiz.entity.GenericDataSourceException: Unable to esablish a connection
with the database. Connection was null!
org.ofbiz.entity.jdbc.SQLProcessor.getConnection(SQLProcessor.java:276)
org.ofbiz.entity.jdbc.SQLProcessor.prepareStatement(SQLProcessor.java:374)


Remember this is NOT a "localpostgres" configuration problem!
It worked fine during install-seed and
admin could log in successfully (and a few files imported).
After this error, even login through the web admin interface becomes
impossible (any database operation I guess).

* With FK's on (and "create dummy FK flag" on) it fails _very_ soon
* without FK-s (and "create dummy FK flag" off) it can import quite many (of
our small) files

After ofbiz breaks, restart ofbiz (absolutely necessary!) and the import can
be continued (with lower number of files than earlier, because of the "delete
xml files after import").

The same happens with postgres 8.1.x and 7.4.9:
15688[       DatabaseUtil.java:766:INFO ] Database Product Version is 7.4.9
15690[       DatabaseUtil.java:774:INFO ] Database Driver Name is PostgreSQL
Native Driver
15692[       DatabaseUtil.java:775:INFO ] Database Driver Version is
PostgreSQL 7.4.5 JDBC3 with SSL (build 216)
15693[       DatabaseUtil.java:776:INFO ] Database Driver JDBC Version is 3.0

Naturally, jdbc3 driver matching postgres was used. SUN JDK 1.4.2.
We used svn trunk revision 7471. Tried on 2 different linux hosts.

The imported data is not much by any means, appr 2 Mbyte total. XML-s exported
by an earlier trunk revision, and Visits and similar stripped.
The following files didn't make in at all, because NullPointerException:
* InvoiceItemTypeMap.xml
* ProductPrice.xml
* UserLoginHistory.xml
* ProductGlAccount.xml
* SimpleSalesTaxLookup.xml
* WorkEffortGoodStandard.xml
This is OK for us, (prices will be adjusted after the import anyway).
 
_______________________________________________
Dev mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/dev

[David E. Jones]

Were you using the 7 series JDBC drivers with an 8 series version of Postgres? I've heard of problems like this from that sort, and there is quite a bit of confirmation that this is an unstable and not recommended combination for Postgres. You definitely want the 8 series drivers AND an 8 series database. The 7 series drivers (aside from the ones that we hacked) did not support database cursors and so would use a lot more memory than necessary, and for large data sets often caused out of memory problems... That is separate from the stability issues of using a PG7 JDBC driver with a PG8 database, which is a problem on its own.

-David


Marcell Gal wrote:

> Hi Guys,
>
> We've seen a strange error, below are my notes.
> Any ideas ?
>
>   Marcell
>
> '''Import XML-s from Dir'''
>
> * use the "delete after import" flag so we know what was imported
>
> It starts importing, but unfortunately after a few files, something bad
> happens (the pool is consumed? why isn't it waiting?), and from than it's all
> hell:
>
> Message: Unable to esablish a connection with the database. Connection was
> null!
> ---- stack
> trace ---------------------------------------------------------------
> org.ofbiz.entity.GenericDataSourceException: Unable to esablish a connection
> with the database. Connection was null!
> org.ofbiz.entity.jdbc.SQLProcessor.getConnection(SQLProcessor.java:276)
> org.ofbiz.entity.jdbc.SQLProcessor.prepareStatement(SQLProcessor.java:374)
>
>
> Remember this is NOT a "localpostgres" configuration problem!
> It worked fine during install-seed and
> admin could log in successfully (and a few files imported).
> After this error, even login through the web admin interface becomes
> impossible (any database operation I guess).
>
> * With FK's on (and "create dummy FK flag" on) it fails _very_ soon
> * without FK-s (and "create dummy FK flag" off) it can import quite many (of
> our small) files
>
> After ofbiz breaks, restart ofbiz (absolutely necessary!) and the import can
> be continued (with lower number of files than earlier, because of the "delete
> xml files after import").
>
> The same happens with postgres 8.1.x and 7.4.9:
> 15688[       DatabaseUtil.java:766:INFO ] Database Product Version is 7.4.9
> 15690[       DatabaseUtil.java:774:INFO ] Database Driver Name is PostgreSQL
> Native Driver
> 15692[       DatabaseUtil.java:775:INFO ] Database Driver Version is
> PostgreSQL 7.4.5 JDBC3 with SSL (build 216)
> 15693[       DatabaseUtil.java:776:INFO ] Database Driver JDBC Version is 3.0
>
> Naturally, jdbc3 driver matching postgres was used. SUN JDK 1.4.2.
> We used svn trunk revision 7471. Tried on 2 different linux hosts.
>
> The imported data is not much by any means, appr 2 Mbyte total. XML-s exported
> by an earlier trunk revision, and Visits and similar stripped.
> The following files didn't make in at all, because NullPointerException:
> * InvoiceItemTypeMap.xml
> * ProductPrice.xml
> * UserLoginHistory.xml
> * ProductGlAccount.xml
> * SimpleSalesTaxLookup.xml
> * WorkEffortGoodStandard.xml
> This is OK for us, (prices will be adjusted after the import anyway).
>  
> _______________________________________________
> Dev mailing list
> [hidden email]
> http://lists.ofbiz.org/mailman/listinfo/dev

 
_______________________________________________
Dev mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/dev