java.lang.OutOfMemoryError - still, please some advice?
Locked
12
12
Re: java.lang.OutOfMemoryError - still, please some advice?
 
|
Thanks BJ -
It works fine using psql (and with mixed versions) - I can do select *'s to my heart's content. It also works with pgAdmin - albiet slowwwwwwly. And, it works with PHP - we have some PHP code that already does these queries. So, I'm guessing its not a Postgres problem, per sa. But then, I'm not a DB expert. I am also able to do this: EntityFindOptions findOptions = new EntityFindOptions(); findOptions.setDistinct(false); // this doesnt seem to work when used with a non-primary key field findOptions.setFetchSize(100); // need to set fetch size or it will never return // findOptions.setResultSetType(EntityFindOptions.TYPE_SCROLL_INSENSITIVE); // need this for partial list reads // findOptions.setResultSetType(EntityFindOptions.CONCUR_READ_ONLY); boolean beganTransaction = false; try { GenericValue thisCdr = null; String calling_number = null; eli = delegator.findListIteratorByCondition("cdrdata", null, null, null, null, findOptions); List searchSet = eli.getPartialList(1,100); Will return 100 values, but if I add this: findOptions.setResultSetType(EntityFindOptions.TYPE_SCROLL_INSENSITIVE); // need this for partial list reads I get timeouts/out of memory errors. Which means that I can't actually run through my result set - except for the first 100 returned records. Ruth BJ Freeman wrote: > couple of things to help on the db side > http://secure.linuxports.com/pgsql-novice/2002-04/msg00182.php > On Tue, 2006-08-01 at 14:51 +0000, Karl O. Pinc wrote: > > > > Whatever happened to the rpm packaging of the 7.x libpq? > > > (Which enabled programs linked against the old libraries > > > to be used with a 8.x. postgresql.) > > http://developer.postgresql.org/~devrim/rpms/compat/ > > > > Ruth Hoffman sent the following on 8/1/2006 9:12 AM: > >> Hi David: >> Ok, thats fair. >> Thanks for your help. >> Ruth >> >> David E Jones wrote: >> >>> >>> Sounds like you have some digging to do then. >>> >>> Please understand this is most likely a database configuration >>> problem, not a problem that can be fixed with code or other changes >>> in OFBiz. >>> >>> The best thing to do is review the JDBC driver documentation to see >>> if special parameters are needed, or in general do some debugging >>> to see exactly what is going on. >>> >>> Unless someone else has any tips from recent experience for you, >>> I'm not sure how much help this mailing list can be... >>> >>> Best of luck, >>> -David >>> >>> >>> On Aug 1, 2006, at 9:20 AM, Ruth Hoffman wrote: >>> >>>> Thanks David: >>>> Now I have the same client version as the server version (here's >>>> the log file): >>>> >>>> 2006-08-01 10:14:06,553 (main) [ DatabaseUtil.java:765:INFO ] >>>> Database Product Name is PostgreSQL >>>> 2006-08-01 10:14:06,555 (main) [ DatabaseUtil.java:766:INFO ] >>>> Database Product Version is 8.1.4 >>>> 2006-08-01 10:14:06,557 (main) [ DatabaseUtil.java:774:INFO ] >>>> Database Driver Name is PostgreSQL Native Driver >>>> 2006-08-01 10:14:06,561 (main) [ DatabaseUtil.java:775:INFO ] >>>> Database Driver Version is PostgreSQL 8.1 JDBC3 with SSL (build >>>> 407) >>>> >>>> And I just verified that: Webtools/Entity Reference; Webtools/XML >>>> export & programmatic EntityListIterator...all cause either an out >>>> of memory error or, a transaction time out error. >>>> >>>> TIA >>>> Ruth >>>> >>>> >>>> >>>> David E Jones wrote: >>>> >>>>> >>>>> On Aug 1, 2006, at 8:50 AM, Ruth Hoffman wrote: >>>>> >>>>>> David: >>>>>> Please clarify, if you could - are you saying that PostgresSQL >>>>>> v. 8.1.4 (as a client) is not compatible with PostgresSQL 8.0? >>>>>> (as installed on my remote db) >>>>> >>>>> >>>>> >>>>> >>>>> I'm saying look at the pgsql web site because there are issues >>>>> with using the wrong version of the JDBC driver for a given >>>>> version of the database. >>>>> >>>>> In other words, there are two different things: the database, and >>>>> the JDBC driver. Each has its own version. If the JDBC driver >>>>> version does not work with the version of the database it is >>>>> talking to, things will not work. >>>>> >>>>>> To your knowledge, has anyone ever successfully used the Entity >>>>>> Engine with remote (not on the same server) database tables of >>>>>> over 1.5Milliion records? >>>>> >>>>> >>>>> >>>>> >>>>> Yes, there are many OFBiz installations with single tables even >>>>> larger than this and they work fine. >>>>> >>>>> -David >>>>> >>>>> >>>>> >>>>> >>> >>> >> > |
Re: java.lang.OutOfMemoryError - still, please some advice?
|
|
if all those were using the same driver, and DB then it most definitely
would point to ofbiz. however if those use different drivers, then that add another variable to the equation. You have gone deeper into the code than I have. it will take me while to figure out exactly what you have tested and what the results should be. will get back to you shortly. Ruth Hoffman sent the following on 8/1/2006 9:44 AM: > Thanks BJ - > It works fine using psql (and with mixed versions) - I can do select *'s > to my heart's content. It also works with pgAdmin - albiet slowwwwwwly. > And, it works with PHP - we have some PHP code that already does these > queries. So, I'm guessing its not a Postgres problem, per sa. But then, > I'm not a DB expert. > > I am also able to do this: > > EntityFindOptions findOptions = new EntityFindOptions(); > findOptions.setDistinct(false); // this doesnt > seem to work when used with a non-primary key field > findOptions.setFetchSize(100); // need to set fetch > size or it will never return > // > findOptions.setResultSetType(EntityFindOptions.TYPE_SCROLL_INSENSITIVE); > // need this for partial list reads > // findOptions.setResultSetType(EntityFindOptions.CONCUR_READ_ONLY); > boolean beganTransaction = false; > try { > GenericValue thisCdr = null; > String calling_number = null; > eli = delegator.findListIteratorByCondition("cdrdata", > null, null, null, null, findOptions); List > searchSet = eli.getPartialList(1,100); > > Will return 100 values, but if I add this: > > > findOptions.setResultSetType(EntityFindOptions.TYPE_SCROLL_INSENSITIVE); > // need this for partial list reads > > I get timeouts/out of memory errors. Which means that I can't actually > run through my result set - except for the first 100 returned records. > > Ruth > > > > BJ Freeman wrote: > >> couple of things to help on the db side >> http://secure.linuxports.com/pgsql-novice/2002-04/msg00182.php >> On Tue, 2006-08-01 at 14:51 +0000, Karl O. Pinc wrote: >> >> > > Whatever happened to the rpm packaging of the 7.x libpq? >> > > (Which enabled programs linked against the old libraries >> > > to be used with a 8.x. postgresql.) >> >> http://developer.postgresql.org/~devrim/rpms/compat/ >> >> >> >> Ruth Hoffman sent the following on 8/1/2006 9:12 AM: >> >>> Hi David: >>> Ok, thats fair. >>> Thanks for your help. >>> Ruth >>> >>> David E Jones wrote: >>> >>>> >>>> Sounds like you have some digging to do then. >>>> >>>> Please understand this is most likely a database configuration >>>> problem, not a problem that can be fixed with code or other changes >>>> in OFBiz. >>>> >>>> The best thing to do is review the JDBC driver documentation to see >>>> if special parameters are needed, or in general do some debugging >>>> to see exactly what is going on. >>>> >>>> Unless someone else has any tips from recent experience for you, >>>> I'm not sure how much help this mailing list can be... >>>> >>>> Best of luck, >>>> -David >>>> >>>> >>>> On Aug 1, 2006, at 9:20 AM, Ruth Hoffman wrote: >>>> >>>>> Thanks David: >>>>> Now I have the same client version as the server version (here's >>>>> the log file): >>>>> >>>>> 2006-08-01 10:14:06,553 (main) [ DatabaseUtil.java:765:INFO ] >>>>> Database Product Name is PostgreSQL >>>>> 2006-08-01 10:14:06,555 (main) [ DatabaseUtil.java:766:INFO ] >>>>> Database Product Version is 8.1.4 >>>>> 2006-08-01 10:14:06,557 (main) [ DatabaseUtil.java:774:INFO ] >>>>> Database Driver Name is PostgreSQL Native Driver >>>>> 2006-08-01 10:14:06,561 (main) [ DatabaseUtil.java:775:INFO ] >>>>> Database Driver Version is PostgreSQL 8.1 JDBC3 with SSL (build >>>>> 407) >>>>> >>>>> And I just verified that: Webtools/Entity Reference; Webtools/XML >>>>> export & programmatic EntityListIterator...all cause either an out >>>>> of memory error or, a transaction time out error. >>>>> >>>>> TIA >>>>> Ruth >>>>> >>>>> >>>>> >>>>> David E Jones wrote: >>>>> >>>>>> >>>>>> On Aug 1, 2006, at 8:50 AM, Ruth Hoffman wrote: >>>>>> >>>>>>> David: >>>>>>> Please clarify, if you could - are you saying that PostgresSQL >>>>>>> v. 8.1.4 (as a client) is not compatible with PostgresSQL 8.0? >>>>>>> (as installed on my remote db) >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> I'm saying look at the pgsql web site because there are issues >>>>>> with using the wrong version of the JDBC driver for a given >>>>>> version of the database. >>>>>> >>>>>> In other words, there are two different things: the database, and >>>>>> the JDBC driver. Each has its own version. If the JDBC driver >>>>>> version does not work with the version of the database it is >>>>>> talking to, things will not work. >>>>>> >>>>>>> To your knowledge, has anyone ever successfully used the Entity >>>>>>> Engine with remote (not on the same server) database tables of >>>>>>> over 1.5Milliion records? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Yes, there are many OFBiz installations with single tables even >>>>>> larger than this and they work fine. >>>>>> >>>>>> -David >>>>>> >>>>>> >>>>>> >>>>>> >>>> >>>> >>> >> > |
Re: java.lang.OutOfMemoryError - still, please some advice?
|
|
Hi BJ:
Yes, I tested all with the same drivers/same target database. I tested the Webtools stuff on both versions 7759 & 427128. I tested the findListIteratorByCondition only on version 7759. I'd be happy to help...just tell me where to start looking.. Ruth BJ Freeman wrote: > if all those were using the same driver, and DB then it most definitely > would point to ofbiz. > however if those use different drivers, then that add another variable > to the equation. > > You have gone deeper into the code than I have. > it will take me while to figure out exactly what you have tested and > what the results should be. > > will get back to you shortly. > > > Ruth Hoffman sent the following on 8/1/2006 9:44 AM: > >> Thanks BJ - >> It works fine using psql (and with mixed versions) - I can do select >> *'s to my heart's content. It also works with pgAdmin - albiet >> slowwwwwwly. And, it works with PHP - we have some PHP code that >> already does these queries. So, I'm guessing its not a Postgres >> problem, per sa. But then, I'm not a DB expert. >> >> I am also able to do this: >> >> EntityFindOptions findOptions = new EntityFindOptions(); >> findOptions.setDistinct(false); // this doesnt >> seem to work when used with a non-primary key field >> findOptions.setFetchSize(100); // need to set >> fetch size or it will never return >> // >> findOptions.setResultSetType(EntityFindOptions.TYPE_SCROLL_INSENSITIVE); >> // need this for partial list reads >> // >> findOptions.setResultSetType(EntityFindOptions.CONCUR_READ_ONLY); >> boolean beganTransaction = false; >> try { >> GenericValue thisCdr = null; >> String calling_number = null; >> eli = >> delegator.findListIteratorByCondition("cdrdata", null, null, null, >> null, findOptions); List searchSet = >> eli.getPartialList(1,100); >> >> Will return 100 values, but if I add this: >> >> >> findOptions.setResultSetType(EntityFindOptions.TYPE_SCROLL_INSENSITIVE); >> // need this for partial list reads >> >> I get timeouts/out of memory errors. Which means that I can't >> actually run through my result set - except for the first 100 >> returned records. >> >> Ruth >> >> >> >> BJ Freeman wrote: >> >>> couple of things to help on the db side >>> http://secure.linuxports.com/pgsql-novice/2002-04/msg00182.php >>> On Tue, 2006-08-01 at 14:51 +0000, Karl O. Pinc wrote: >>> >>> > > Whatever happened to the rpm packaging of the 7.x libpq? >>> > > (Which enabled programs linked against the old libraries >>> > > to be used with a 8.x. postgresql.) >>> >>> http://developer.postgresql.org/~devrim/rpms/compat/ >>> >>> >>> >>> Ruth Hoffman sent the following on 8/1/2006 9:12 AM: >>> >>>> Hi David: >>>> Ok, thats fair. >>>> Thanks for your help. >>>> Ruth >>>> >>>> David E Jones wrote: >>>> >>>>> >>>>> Sounds like you have some digging to do then. >>>>> >>>>> Please understand this is most likely a database configuration >>>>> problem, not a problem that can be fixed with code or other >>>>> changes in OFBiz. >>>>> >>>>> The best thing to do is review the JDBC driver documentation to >>>>> see if special parameters are needed, or in general do some >>>>> debugging to see exactly what is going on. >>>>> >>>>> Unless someone else has any tips from recent experience for you, >>>>> I'm not sure how much help this mailing list can be... >>>>> >>>>> Best of luck, >>>>> -David >>>>> >>>>> >>>>> On Aug 1, 2006, at 9:20 AM, Ruth Hoffman wrote: >>>>> >>>>>> Thanks David: >>>>>> Now I have the same client version as the server version (here's >>>>>> the log file): >>>>>> >>>>>> 2006-08-01 10:14:06,553 (main) [ >>>>>> DatabaseUtil.java:765:INFO ] >>>>>> Database Product Name is PostgreSQL >>>>>> 2006-08-01 10:14:06,555 (main) [ >>>>>> DatabaseUtil.java:766:INFO ] >>>>>> Database Product Version is 8.1.4 >>>>>> 2006-08-01 10:14:06,557 (main) [ >>>>>> DatabaseUtil.java:774:INFO ] >>>>>> Database Driver Name is PostgreSQL Native Driver >>>>>> 2006-08-01 10:14:06,561 (main) [ >>>>>> DatabaseUtil.java:775:INFO ] >>>>>> Database Driver Version is PostgreSQL 8.1 JDBC3 with SSL >>>>>> (build 407) >>>>>> >>>>>> And I just verified that: Webtools/Entity Reference; >>>>>> Webtools/XML export & programmatic EntityListIterator...all >>>>>> cause either an out of memory error or, a transaction time out >>>>>> error. >>>>>> >>>>>> TIA >>>>>> Ruth >>>>>> >>>>>> >>>>>> >>>>>> David E Jones wrote: >>>>>> >>>>>>> >>>>>>> On Aug 1, 2006, at 8:50 AM, Ruth Hoffman wrote: >>>>>>> >>>>>>>> David: >>>>>>>> Please clarify, if you could - are you saying that PostgresSQL >>>>>>>> v. 8.1.4 (as a client) is not compatible with PostgresSQL >>>>>>>> 8.0? (as installed on my remote db) >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> I'm saying look at the pgsql web site because there are issues >>>>>>> with using the wrong version of the JDBC driver for a given >>>>>>> version of the database. >>>>>>> >>>>>>> In other words, there are two different things: the database, >>>>>>> and the JDBC driver. Each has its own version. If the JDBC >>>>>>> driver version does not work with the version of the database >>>>>>> it is talking to, things will not work. >>>>>>> >>>>>>>> To your knowledge, has anyone ever successfully used the >>>>>>>> Entity Engine with remote (not on the same server) database >>>>>>>> tables of over 1.5Milliion records? >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Yes, there are many OFBiz installations with single tables >>>>>>> even larger than this and they work fine. >>>>>>> >>>>>>> -David >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>> >>>>> >>>> >>> >> > > |
Re: java.lang.OutOfMemoryError - still, please some advice?
|
|
I am walking down the ofbiz code till I get to the actual routine that
structures the call to the DB. one thing I notice is findOptions.useCache(false);//default is true. I have not gone far enough to be certain, but I am guessing this means cache in memory, or use the cache which is memory based. See the other email about excucutefind. Note: till I get all the way thru, I really won't understand this enough go be a novice.. LOL. Ruth Hoffman sent the following on 8/1/2006 11:03 AM: > Hi BJ: > Yes, I tested all with the same drivers/same target database. I tested > the Webtools stuff on both versions 7759 & 427128. I tested the > findListIteratorByCondition only on version 7759. > > I'd be happy to help...just tell me where to start looking.. > Ruth > > BJ Freeman wrote: > >> if all those were using the same driver, and DB then it most definitely >> would point to ofbiz. >> however if those use different drivers, then that add another variable >> to the equation. >> >> You have gone deeper into the code than I have. >> it will take me while to figure out exactly what you have tested and >> what the results should be. >> >> will get back to you shortly. >> >> >> Ruth Hoffman sent the following on 8/1/2006 9:44 AM: >> >>> Thanks BJ - >>> It works fine using psql (and with mixed versions) - I can do select >>> *'s to my heart's content. It also works with pgAdmin - albiet >>> slowwwwwwly. And, it works with PHP - we have some PHP code that >>> already does these queries. So, I'm guessing its not a Postgres >>> problem, per sa. But then, I'm not a DB expert. >>> >>> I am also able to do this: >>> >>> EntityFindOptions findOptions = new EntityFindOptions(); >>> findOptions.setDistinct(false); // this doesnt >>> seem to work when used with a non-primary key field >>> findOptions.setFetchSize(100); // need to set >>> fetch size or it will never return >>> // >>> findOptions.setResultSetType(EntityFindOptions.TYPE_SCROLL_INSENSITIVE); >>> // need this for partial list reads >>> // >>> findOptions.setResultSetType(EntityFindOptions.CONCUR_READ_ONLY); >>> boolean beganTransaction = false; >>> try { >>> GenericValue thisCdr = null; >>> String calling_number = null; >>> eli = >>> delegator.findListIteratorByCondition("cdrdata", null, null, null, >>> null, findOptions); List searchSet = >>> eli.getPartialList(1,100); >>> >>> Will return 100 values, but if I add this: >>> >>> >>> findOptions.setResultSetType(EntityFindOptions.TYPE_SCROLL_INSENSITIVE); >>> // need this for partial list reads >>> >>> I get timeouts/out of memory errors. Which means that I can't >>> actually run through my result set - except for the first 100 >>> returned records. >>> >>> Ruth >>> >>> >>> >>> BJ Freeman wrote: >>> >>>> couple of things to help on the db side >>>> http://secure.linuxports.com/pgsql-novice/2002-04/msg00182.php >>>> On Tue, 2006-08-01 at 14:51 +0000, Karl O. Pinc wrote: >>>> >>>> > > Whatever happened to the rpm packaging of the 7.x libpq? >>>> > > (Which enabled programs linked against the old libraries >>>> > > to be used with a 8.x. postgresql.) >>>> >>>> http://developer.postgresql.org/~devrim/rpms/compat/ >>>> >>>> >>>> >>>> Ruth Hoffman sent the following on 8/1/2006 9:12 AM: >>>> >>>>> Hi David: >>>>> Ok, thats fair. >>>>> Thanks for your help. >>>>> Ruth >>>>> >>>>> David E Jones wrote: >>>>> >>>>>> >>>>>> Sounds like you have some digging to do then. >>>>>> >>>>>> Please understand this is most likely a database configuration >>>>>> problem, not a problem that can be fixed with code or other >>>>>> changes in OFBiz. >>>>>> >>>>>> The best thing to do is review the JDBC driver documentation to >>>>>> see if special parameters are needed, or in general do some >>>>>> debugging to see exactly what is going on. >>>>>> >>>>>> Unless someone else has any tips from recent experience for you, >>>>>> I'm not sure how much help this mailing list can be... >>>>>> >>>>>> Best of luck, >>>>>> -David >>>>>> >>>>>> >>>>>> On Aug 1, 2006, at 9:20 AM, Ruth Hoffman wrote: >>>>>> >>>>>>> Thanks David: >>>>>>> Now I have the same client version as the server version (here's >>>>>>> the log file): >>>>>>> >>>>>>> 2006-08-01 10:14:06,553 (main) [ >>>>>>> DatabaseUtil.java:765:INFO ] >>>>>>> Database Product Name is PostgreSQL >>>>>>> 2006-08-01 10:14:06,555 (main) [ >>>>>>> DatabaseUtil.java:766:INFO ] >>>>>>> Database Product Version is 8.1.4 >>>>>>> 2006-08-01 10:14:06,557 (main) [ >>>>>>> DatabaseUtil.java:774:INFO ] >>>>>>> Database Driver Name is PostgreSQL Native Driver >>>>>>> 2006-08-01 10:14:06,561 (main) [ >>>>>>> DatabaseUtil.java:775:INFO ] >>>>>>> Database Driver Version is PostgreSQL 8.1 JDBC3 with SSL >>>>>>> (build 407) >>>>>>> >>>>>>> And I just verified that: Webtools/Entity Reference; >>>>>>> Webtools/XML export & programmatic EntityListIterator...all >>>>>>> cause either an out of memory error or, a transaction time out >>>>>>> error. >>>>>>> >>>>>>> TIA >>>>>>> Ruth >>>>>>> >>>>>>> >>>>>>> >>>>>>> David E Jones wrote: >>>>>>> >>>>>>>> >>>>>>>> On Aug 1, 2006, at 8:50 AM, Ruth Hoffman wrote: >>>>>>>> >>>>>>>>> David: >>>>>>>>> Please clarify, if you could - are you saying that PostgresSQL >>>>>>>>> v. 8.1.4 (as a client) is not compatible with PostgresSQL >>>>>>>>> 8.0? (as installed on my remote db) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> I'm saying look at the pgsql web site because there are issues >>>>>>>> with using the wrong version of the JDBC driver for a given >>>>>>>> version of the database. >>>>>>>> >>>>>>>> In other words, there are two different things: the database, >>>>>>>> and the JDBC driver. Each has its own version. If the JDBC >>>>>>>> driver version does not work with the version of the database >>>>>>>> it is talking to, things will not work. >>>>>>>> >>>>>>>>> To your knowledge, has anyone ever successfully used the >>>>>>>>> Entity Engine with remote (not on the same server) database >>>>>>>>> tables of over 1.5Milliion records? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Yes, there are many OFBiz installations with single tables >>>>>>>> even larger than this and they work fine. >>>>>>>> >>>>>>>> -David >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >> > |
Re: java.lang.OutOfMemoryError - still, please some advice?
|
|
Hi Ruth:
just to let you know I have not forgot. I finally got down to something I understand which is results sets in the JDBC 2.0 API http://java.sun.com/j2se/1.4/docs/api/java/sql/ResultSet.html and the low level interface that ofbiz uses for cursor work, being framework/entity/src/org/ofbiz/entity/jdbc class CursorResultSet extends AbstractCursorHandler I figure by next week some time I will have worked back up to the webtools code. so far don't know enough to make a opinion. or see anything to fix. BJ Freeman sent the following on 8/1/2006 11:24 AM: > I am walking down the ofbiz code till I get to the actual routine that > structures the call to the DB. > one thing I notice is > findOptions.useCache(false);//default is true. > I have not gone far enough to be certain, but I am guessing this means > cache in memory, or use the cache which is memory based. > See the other email about excucutefind. > > Note: till I get all the way thru, I really won't understand this enough > go be a novice.. LOL. > > Ruth Hoffman sent the following on 8/1/2006 11:03 AM: >> Hi BJ: >> Yes, I tested all with the same drivers/same target database. I tested >> the Webtools stuff on both versions 7759 & 427128. I tested the >> findListIteratorByCondition only on version 7759. >> >> I'd be happy to help...just tell me where to start looking.. >> Ruth >> >> BJ Freeman wrote: >> >>> if all those were using the same driver, and DB then it most definitely >>> would point to ofbiz. >>> however if those use different drivers, then that add another variable >>> to the equation. >>> >>> You have gone deeper into the code than I have. >>> it will take me while to figure out exactly what you have tested and >>> what the results should be. >>> >>> will get back to you shortly. >>> >>> >>> Ruth Hoffman sent the following on 8/1/2006 9:44 AM: >>> >>>> Thanks BJ - >>>> It works fine using psql (and with mixed versions) - I can do select >>>> *'s to my heart's content. It also works with pgAdmin - albiet >>>> slowwwwwwly. And, it works with PHP - we have some PHP code that >>>> already does these queries. So, I'm guessing its not a Postgres >>>> problem, per sa. But then, I'm not a DB expert. >>>> >>>> I am also able to do this: >>>> >>>> EntityFindOptions findOptions = new EntityFindOptions(); >>>> findOptions.setDistinct(false); // this >>>> doesnt seem to work when used with a non-primary key field >>>> findOptions.setFetchSize(100); // need to set >>>> fetch size or it will never return >>>> // >>>> findOptions.setResultSetType(EntityFindOptions.TYPE_SCROLL_INSENSITIVE); >>>> // need this for partial list reads >>>> // >>>> findOptions.setResultSetType(EntityFindOptions.CONCUR_READ_ONLY); >>>> boolean beganTransaction = false; >>>> try { >>>> GenericValue thisCdr = null; >>>> String calling_number = null; >>>> eli = >>>> delegator.findListIteratorByCondition("cdrdata", null, null, null, >>>> null, findOptions); List searchSet = >>>> eli.getPartialList(1,100); >>>> >>>> Will return 100 values, but if I add this: >>>> >>>> >>>> findOptions.setResultSetType(EntityFindOptions.TYPE_SCROLL_INSENSITIVE); >>>> // need this for partial list reads >>>> >>>> I get timeouts/out of memory errors. Which means that I can't >>>> actually run through my result set - except for the first 100 >>>> returned records. >>>> >>>> Ruth >>>> >>>> >>>> >>>> BJ Freeman wrote: >>>> >>>>> couple of things to help on the db side >>>>> http://secure.linuxports.com/pgsql-novice/2002-04/msg00182.php >>>>> On Tue, 2006-08-01 at 14:51 +0000, Karl O. Pinc wrote: >>>>> >>>>> > > Whatever happened to the rpm packaging of the 7.x libpq? >>>>> > > (Which enabled programs linked against the old libraries >>>>> > > to be used with a 8.x. postgresql.) >>>>> >>>>> http://developer.postgresql.org/~devrim/rpms/compat/ >>>>> >>>>> >>>>> >>>>> Ruth Hoffman sent the following on 8/1/2006 9:12 AM: >>>>> >>>>>> Hi David: >>>>>> Ok, thats fair. >>>>>> Thanks for your help. >>>>>> Ruth >>>>>> >>>>>> David E Jones wrote: >>>>>> >>>>>>> >>>>>>> Sounds like you have some digging to do then. >>>>>>> >>>>>>> Please understand this is most likely a database configuration >>>>>>> problem, not a problem that can be fixed with code or other >>>>>>> changes in OFBiz. >>>>>>> >>>>>>> The best thing to do is review the JDBC driver documentation to >>>>>>> see if special parameters are needed, or in general do some >>>>>>> debugging to see exactly what is going on. >>>>>>> >>>>>>> Unless someone else has any tips from recent experience for you, >>>>>>> I'm not sure how much help this mailing list can be... >>>>>>> >>>>>>> Best of luck, >>>>>>> -David >>>>>>> >>>>>>> >>>>>>> On Aug 1, 2006, at 9:20 AM, Ruth Hoffman wrote: >>>>>>> >>>>>>>> Thanks David: >>>>>>>> Now I have the same client version as the server version >>>>>>>> (here's the log file): >>>>>>>> >>>>>>>> 2006-08-01 10:14:06,553 (main) [ >>>>>>>> DatabaseUtil.java:765:INFO ] >>>>>>>> Database Product Name is PostgreSQL >>>>>>>> 2006-08-01 10:14:06,555 (main) [ >>>>>>>> DatabaseUtil.java:766:INFO ] >>>>>>>> Database Product Version is 8.1.4 >>>>>>>> 2006-08-01 10:14:06,557 (main) [ >>>>>>>> DatabaseUtil.java:774:INFO ] >>>>>>>> Database Driver Name is PostgreSQL Native Driver >>>>>>>> 2006-08-01 10:14:06,561 (main) [ >>>>>>>> DatabaseUtil.java:775:INFO ] >>>>>>>> Database Driver Version is PostgreSQL 8.1 JDBC3 with SSL >>>>>>>> (build 407) >>>>>>>> >>>>>>>> And I just verified that: Webtools/Entity Reference; >>>>>>>> Webtools/XML export & programmatic EntityListIterator...all >>>>>>>> cause either an out of memory error or, a transaction time out >>>>>>>> error. >>>>>>>> >>>>>>>> TIA >>>>>>>> Ruth >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> David E Jones wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> On Aug 1, 2006, at 8:50 AM, Ruth Hoffman wrote: >>>>>>>>> >>>>>>>>>> David: >>>>>>>>>> Please clarify, if you could - are you saying that >>>>>>>>>> PostgresSQL v. 8.1.4 (as a client) is not compatible with >>>>>>>>>> PostgresSQL 8.0? (as installed on my remote db) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> I'm saying look at the pgsql web site because there are issues >>>>>>>>> with using the wrong version of the JDBC driver for a given >>>>>>>>> version of the database. >>>>>>>>> >>>>>>>>> In other words, there are two different things: the database, >>>>>>>>> and the JDBC driver. Each has its own version. If the JDBC >>>>>>>>> driver version does not work with the version of the database >>>>>>>>> it is talking to, things will not work. >>>>>>>>> >>>>>>>>>> To your knowledge, has anyone ever successfully used the >>>>>>>>>> Entity Engine with remote (not on the same server) database >>>>>>>>>> tables of over 1.5Milliion records? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Yes, there are many OFBiz installations with single tables >>>>>>>>> even larger than this and they work fine. >>>>>>>>> >>>>>>>>> -David >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>> >> > |
Re: java.lang.OutOfMemoryError - still, please some advice?
|
|
Hi BJ:
Wow! Thanks for the effort. I've not had time to do more than my initial trials. For now, I've turned my attention to another part of the project I'm working on - if you find out anything, for example some tests I can run against this database, please let me know. The database in question is behind a firewall and not accessible on the open Internet - so I'll need to make arrangements to work with it. Thanks again. Ruth BJ Freeman wrote: > Hi Ruth: > just to let you know I have not forgot. > I finally got down to something I understand which is results sets in > the JDBC 2.0 API > http://java.sun.com/j2se/1.4/docs/api/java/sql/ResultSet.html > and the low level interface that ofbiz uses for cursor work, being > framework/entity/src/org/ofbiz/entity/jdbc > class CursorResultSet extends AbstractCursorHandler > > I figure by next week some time I will have worked back up to the > webtools code. > > so far don't know enough to make a opinion. or see anything to fix. > > > > > BJ Freeman sent the following on 8/1/2006 11:24 AM: > >> I am walking down the ofbiz code till I get to the actual routine >> that structures the call to the DB. >> one thing I notice is >> findOptions.useCache(false);//default is true. >> I have not gone far enough to be certain, but I am guessing this >> means cache in memory, or use the cache which is memory based. >> See the other email about excucutefind. >> >> Note: till I get all the way thru, I really won't understand this >> enough go be a novice.. LOL. >> >> Ruth Hoffman sent the following on 8/1/2006 11:03 AM: >> >>> Hi BJ: >>> Yes, I tested all with the same drivers/same target database. I >>> tested the Webtools stuff on both versions 7759 & 427128. I tested >>> the findListIteratorByCondition only on version 7759. >>> >>> I'd be happy to help...just tell me where to start looking.. >>> Ruth >>> >>> BJ Freeman wrote: >>> >>>> if all those were using the same driver, and DB then it most >>>> definitely >>>> would point to ofbiz. >>>> however if those use different drivers, then that add another variable >>>> to the equation. >>>> >>>> You have gone deeper into the code than I have. >>>> it will take me while to figure out exactly what you have tested >>>> and what the results should be. >>>> >>>> will get back to you shortly. >>>> >>>> >>>> Ruth Hoffman sent the following on 8/1/2006 9:44 AM: >>>> >>>>> Thanks BJ - >>>>> It works fine using psql (and with mixed versions) - I can do >>>>> select *'s to my heart's content. It also works with pgAdmin - >>>>> albiet slowwwwwwly. And, it works with PHP - we have some PHP code >>>>> that already does these queries. So, I'm guessing its not a >>>>> Postgres problem, per sa. But then, I'm not a DB expert. >>>>> >>>>> I am also able to do this: >>>>> >>>>> EntityFindOptions findOptions = new EntityFindOptions(); >>>>> findOptions.setDistinct(false); // this >>>>> doesnt seem to work when used with a non-primary key field >>>>> findOptions.setFetchSize(100); // need to set >>>>> fetch size or it will never return >>>>> // >>>>> findOptions.setResultSetType(EntityFindOptions.TYPE_SCROLL_INSENSITIVE); >>>>> // need this for partial list reads >>>>> // >>>>> findOptions.setResultSetType(EntityFindOptions.CONCUR_READ_ONLY); >>>>> boolean beganTransaction = false; >>>>> try { >>>>> GenericValue thisCdr = null; >>>>> String calling_number = null; >>>>> eli = >>>>> delegator.findListIteratorByCondition("cdrdata", null, null, null, >>>>> null, findOptions); List searchSet = >>>>> eli.getPartialList(1,100); >>>>> >>>>> Will return 100 values, but if I add this: >>>>> >>>>> >>>>> findOptions.setResultSetType(EntityFindOptions.TYPE_SCROLL_INSENSITIVE); >>>>> // need this for partial list reads >>>>> >>>>> I get timeouts/out of memory errors. Which means that I can't >>>>> actually run through my result set - except for the first 100 >>>>> returned records. >>>>> >>>>> Ruth >>>>> >>>>> >>>>> >>>>> BJ Freeman wrote: >>>>> >>>>>> couple of things to help on the db side >>>>>> http://secure.linuxports.com/pgsql-novice/2002-04/msg00182.php >>>>>> On Tue, 2006-08-01 at 14:51 +0000, Karl O. Pinc wrote: >>>>>> >>>>>> > > Whatever happened to the rpm packaging of the 7.x libpq? >>>>>> > > (Which enabled programs linked against the old libraries >>>>>> > > to be used with a 8.x. postgresql.) >>>>>> >>>>>> http://developer.postgresql.org/~devrim/rpms/compat/ >>>>>> >>>>>> >>>>>> >>>>>> Ruth Hoffman sent the following on 8/1/2006 9:12 AM: >>>>>> >>>>>>> Hi David: >>>>>>> Ok, thats fair. >>>>>>> Thanks for your help. >>>>>>> Ruth >>>>>>> >>>>>>> David E Jones wrote: >>>>>>> >>>>>>>> >>>>>>>> Sounds like you have some digging to do then. >>>>>>>> >>>>>>>> Please understand this is most likely a database configuration >>>>>>>> problem, not a problem that can be fixed with code or other >>>>>>>> changes in OFBiz. >>>>>>>> >>>>>>>> The best thing to do is review the JDBC driver documentation to >>>>>>>> see if special parameters are needed, or in general do some >>>>>>>> debugging to see exactly what is going on. >>>>>>>> >>>>>>>> Unless someone else has any tips from recent experience for >>>>>>>> you, I'm not sure how much help this mailing list can be... >>>>>>>> >>>>>>>> Best of luck, >>>>>>>> -David >>>>>>>> >>>>>>>> >>>>>>>> On Aug 1, 2006, at 9:20 AM, Ruth Hoffman wrote: >>>>>>>> >>>>>>>>> Thanks David: >>>>>>>>> Now I have the same client version as the server version >>>>>>>>> (here's the log file): >>>>>>>>> >>>>>>>>> 2006-08-01 10:14:06,553 (main) [ >>>>>>>>> DatabaseUtil.java:765:INFO ] >>>>>>>>> Database Product Name is PostgreSQL >>>>>>>>> 2006-08-01 10:14:06,555 (main) [ >>>>>>>>> DatabaseUtil.java:766:INFO ] >>>>>>>>> Database Product Version is 8.1.4 >>>>>>>>> 2006-08-01 10:14:06,557 (main) [ >>>>>>>>> DatabaseUtil.java:774:INFO ] >>>>>>>>> Database Driver Name is PostgreSQL Native Driver >>>>>>>>> 2006-08-01 10:14:06,561 (main) [ >>>>>>>>> DatabaseUtil.java:775:INFO ] >>>>>>>>> Database Driver Version is PostgreSQL 8.1 JDBC3 with SSL >>>>>>>>> (build 407) >>>>>>>>> >>>>>>>>> And I just verified that: Webtools/Entity Reference; >>>>>>>>> Webtools/XML export & programmatic EntityListIterator...all >>>>>>>>> cause either an out of memory error or, a transaction time >>>>>>>>> out error. >>>>>>>>> >>>>>>>>> TIA >>>>>>>>> Ruth >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> David E Jones wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Aug 1, 2006, at 8:50 AM, Ruth Hoffman wrote: >>>>>>>>>> >>>>>>>>>>> David: >>>>>>>>>>> Please clarify, if you could - are you saying that >>>>>>>>>>> PostgresSQL v. 8.1.4 (as a client) is not compatible with >>>>>>>>>>> PostgresSQL 8.0? (as installed on my remote db) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I'm saying look at the pgsql web site because there are >>>>>>>>>> issues with using the wrong version of the JDBC driver for >>>>>>>>>> a given version of the database. >>>>>>>>>> >>>>>>>>>> In other words, there are two different things: the database, >>>>>>>>>> and the JDBC driver. Each has its own version. If the JDBC >>>>>>>>>> driver version does not work with the version of the >>>>>>>>>> database it is talking to, things will not work. >>>>>>>>>> >>>>>>>>>>> To your knowledge, has anyone ever successfully used the >>>>>>>>>>> Entity Engine with remote (not on the same server) >>>>>>>>>>> database tables of over 1.5Milliion records? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Yes, there are many OFBiz installations with single tables >>>>>>>>>> even larger than this and they work fine. >>>>>>>>>> >>>>>>>>>> -David >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> >>> >> > |
«
Return to OFBiz - User
|
1 view|%1 views
| Free forum by Nabble | Edit this page |