Demos: disk full

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Demos: disk full

Jacques Le Roux
Administrator
Hi,

This night I tried to create a log from the cron I run everyday at 3 AM to see if demos need update (based on svn)
But for a reason I got a disk full (the log was 47GB on 51GB available on the VM). I got rid of the file but I can't recover the VM

I will ask infra, in the meantime the demos will be unavailable

Jacques

Reply | Threaded
Open this post in threaded view
|

Re: Demos: disk full

Jacques Le Roux
Administrator
Trunk demo is back, the others will recover soon.
We have an issue (maybe 2) with stable ecommerce demodata loading, I will look at it...

Jacques

Le 03/07/2014 08:57, Jacques Le Roux a écrit :

> Hi,
>
> This night I tried to create a log from the cron I run everyday at 3 AM to see if demos need update (based on svn)
> But for a reason I got a disk full (the log was 47GB on 51GB available on the VM). I got rid of the file but I can't recover the VM
>
> I will ask infra, in the meantime the demos will be unavailable
>
> Jacques
>
>

--
Reply | Threaded
Open this post in threaded view
|

Re: Demos: disk full

Jacques Le Roux
Administrator
We got this error on R12.04 which saturated the disk again

2014-07-03 10:22:59,589 (org.ofbiz.service.job.JobPoller@2078b6ef) [   GenericDelegator.java:1296:ERROR]
---- exception report ----------------------------------------------------------
Failure in storeByCondition operation for entity [JobSandbox]: org.ofbiz.entity.GenericDataSourceException: Generic Entity Exception occured in
updateByCondition (SQL Exception while executing the following:UPDATE OFBIZ.JOB_SANDBOX SET RUN_BY_INSTANCE_ID = ?, STATUS_ID = ? WHERE ((RUN_TIME <=
? AND START_DATE_TIME IS NULL AND CANCEL_DATE_TIME IS NULL AND RUN_BY_INSTANCE_ID IS NULL) AND (POOL_ID IS NULL OR POOL_ID = ?)) (Container was opened
in read-only mode.)). Rolling back transaction.
Exception: org.ofbiz.entity.GenericDataSourceException
Message: Generic Entity Exception occured in updateByCondition (SQL Exception while executing the following:UPDATE OFBIZ.JOB_SANDBOX SET
RUN_BY_INSTANCE_ID = ?, STATUS_ID = ? WHERE ((RUN_TIME <= ? AND START_DATE_TIME IS NULL AND CANCEL_DATE_TIME IS NULL AND RUN_BY_INSTANCE_ID IS NULL)
AND (POOL_ID IS NULL OR POOL_ID = ?)) (Container was opened in read-only mode.))
---- cause ---------------------------------------------------------------------
Exception: org.ofbiz.entity.GenericDataSourceException
Message: SQL Exception while executing the following:UPDATE OFBIZ.JOB_SANDBOX SET RUN_BY_INSTANCE_ID = ?, STATUS_ID = ? WHERE ((RUN_TIME <= ? AND
START_DATE_TIME IS NULL AND CANCEL_DATE_TIME IS NULL AND RUN_BY_INSTANCE_ID IS NULL) AND (POOL_ID IS NULL OR POOL_ID = ?)) (Container was opened in
read-only mode.)
---- cause ---------------------------------------------------------------------
Exception: java.sql.SQLTransactionRollbackException
Message: Container was opened in read-only mode.
---- cause ---------------------------------------------------------------------
Exception: org.apache.derby.impl.jdbc.EmbedSQLException
Message: Container was opened in read-only mode.
---- cause ---------------------------------------------------------------------
Exception: org.apache.derby.iapi.error.StandardException
Message: Container was opened in read-only mode.
---- stack trace ---------------------------------------------------------------
ERROR 40XD1: Container was opened in read-only mode.

It normally relates to https://issues.apache.org/jira/browse/DERBY-541 but we use derby-10.8.2.2.jar in R12.04

Let's see if this happens again... All is restarting...

Jacques

Le 03/07/2014 11:05, Jacques Le Roux a écrit :

> Trunk demo is back, the others will recover soon.
> We have an issue (maybe 2) with stable ecommerce demodata loading, I will look at it...
>
> Jacques
>
> Le 03/07/2014 08:57, Jacques Le Roux a écrit :
>> Hi,
>>
>> This night I tried to create a log from the cron I run everyday at 3 AM to see if demos need update (based on svn)
>> But for a reason I got a disk full (the log was 47GB on 51GB available on the VM). I got rid of the file but I can't recover the VM
>>
>> I will ask infra, in the meantime the demos will be unavailable
>>
>> Jacques
>>
>>
>

--
Reply | Threaded
Open this post in threaded view
|

Re: Demos: disk full

Jacques Le Roux
Administrator
Seems more a permission issue, I will check those...

Jacques

Le 03/07/2014 17:50, Jacques Le Roux a écrit :

> We got this error on R12.04 which saturated the disk again
>
> 2014-07-03 10:22:59,589 (org.ofbiz.service.job.JobPoller@2078b6ef) [   GenericDelegator.java:1296:ERROR]
> ---- exception report ----------------------------------------------------------
> Failure in storeByCondition operation for entity [JobSandbox]: org.ofbiz.entity.GenericDataSourceException: Generic Entity Exception occured in
> updateByCondition (SQL Exception while executing the following:UPDATE OFBIZ.JOB_SANDBOX SET RUN_BY_INSTANCE_ID = ?, STATUS_ID = ? WHERE ((RUN_TIME
> <= ? AND START_DATE_TIME IS NULL AND CANCEL_DATE_TIME IS NULL AND RUN_BY_INSTANCE_ID IS NULL) AND (POOL_ID IS NULL OR POOL_ID = ?)) (Container was
> opened in read-only mode.)). Rolling back transaction.
> Exception: org.ofbiz.entity.GenericDataSourceException
> Message: Generic Entity Exception occured in updateByCondition (SQL Exception while executing the following:UPDATE OFBIZ.JOB_SANDBOX SET
> RUN_BY_INSTANCE_ID = ?, STATUS_ID = ? WHERE ((RUN_TIME <= ? AND START_DATE_TIME IS NULL AND CANCEL_DATE_TIME IS NULL AND RUN_BY_INSTANCE_ID IS NULL)
> AND (POOL_ID IS NULL OR POOL_ID = ?)) (Container was opened in read-only mode.))
> ---- cause ---------------------------------------------------------------------
> Exception: org.ofbiz.entity.GenericDataSourceException
> Message: SQL Exception while executing the following:UPDATE OFBIZ.JOB_SANDBOX SET RUN_BY_INSTANCE_ID = ?, STATUS_ID = ? WHERE ((RUN_TIME <= ? AND
> START_DATE_TIME IS NULL AND CANCEL_DATE_TIME IS NULL AND RUN_BY_INSTANCE_ID IS NULL) AND (POOL_ID IS NULL OR POOL_ID = ?)) (Container was opened in
> read-only mode.)
> ---- cause ---------------------------------------------------------------------
> Exception: java.sql.SQLTransactionRollbackException
> Message: Container was opened in read-only mode.
> ---- cause ---------------------------------------------------------------------
> Exception: org.apache.derby.impl.jdbc.EmbedSQLException
> Message: Container was opened in read-only mode.
> ---- cause ---------------------------------------------------------------------
> Exception: org.apache.derby.iapi.error.StandardException
> Message: Container was opened in read-only mode.
> ---- stack trace ---------------------------------------------------------------
> ERROR 40XD1: Container was opened in read-only mode.
>
> It normally relates to https://issues.apache.org/jira/browse/DERBY-541 but we use derby-10.8.2.2.jar in R12.04
>
> Let's see if this happens again... All is restarting...
>
> Jacques
>
> Le 03/07/2014 11:05, Jacques Le Roux a écrit :
>> Trunk demo is back, the others will recover soon.
>> We have an issue (maybe 2) with stable ecommerce demodata loading, I will look at it...
>>
>> Jacques
>>
>> Le 03/07/2014 08:57, Jacques Le Roux a écrit :
>>> Hi,
>>>
>>> This night I tried to create a log from the cron I run everyday at 3 AM to see if demos need update (based on svn)
>>> But for a reason I got a disk full (the log was 47GB on 51GB available on the VM). I got rid of the file but I can't recover the VM
>>>
>>> I will ask infra, in the meantime the demos will be unavailable
>>>
>>> Jacques
>>>
>>>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Demos: disk full

Pierre Smits
Jacques,

It seems to me that 51 GB would be sufficient as a capacity for log storage
and data for many days before reaching limits.

But, is extensive logging necessary for the demo environment(s)? Shouldn't
clean-up be executed with each daily restart?

Regards,

Pierre Smits

*ORRTIZ.COM <http://www.orrtiz.com>*
Services & Solutions for Cloud-
Based Manufacturing, Professional
Services and Retail & Trade
http://www.orrtiz.com
Reply | Threaded
Open this post in threaded view
|

Re: Demos: disk full

Jacques Le Roux
Administrator
See my last message, seems that Derby ran amok, not sure why yet, the write permissions are correct
Let's see if it reproduces...

Jacques

Le 03/07/2014 18:04, Pierre Smits a écrit :

> Jacques,
>
> It seems to me that 51 GB would be sufficient as a capacity for log storage
> and data for many days before reaching limits.
>
> But, is extensive logging necessary for the demo environment(s)? Shouldn't
> clean-up be executed with each daily restart?
>
> Regards,
>
> Pierre Smits
>
> *ORRTIZ.COM <http://www.orrtiz.com>*
> Services & Solutions for Cloud-
> Based Manufacturing, Professional
> Services and Retail & Trade
> http://www.orrtiz.com
>

--
Reply | Threaded
Open this post in threaded view
|

Re: Demos: disk full

Jacques Le Roux
Administrator
In reply to this post by Jacques Le Roux

Le 03/07/2014 11:05, Jacques Le Roux a écrit :
> We have an issue (maybe 2) with stable ecommerce demodata loading, I will look at it...
>

It was an incompatibility in Lucene component, I did not load it in stable (R13.07) for now...

[java] 2014-07-02 20:24:39,766 (main) [ ServiceDispatcher.java:512:INFO ] [[Sync service failed...- total:0.0,since last(Begin):0.0]] -
'entity-default / indexProduct'
      [java] 2014-07-02 20:24:39,768 (main) [ TransactionUtil.java:378:WARN ]
      [java] ---- exception report ----------------------------------------------------------
      [java] [TransactionUtil.setRollbackOnly] Calling transaction setRollbackOnly; this stack trace shows where this is happening:
      [java] Exception: java.lang.Exception
      [java] Message: Service [indexProduct] threw an unexpected exception/error
      [java] ---- stack trace ---------------------------------------------------------------
      [java] java.lang.Exception: Service [indexProduct] threw an unexpected exception/error
      [java] org.ofbiz.entity.transaction.TransactionUtil.setRollbackOnly(TransactionUtil.java:378)
      [java] org.ofbiz.entity.transaction.TransactionUtil.rollback(TransactionUtil.java:319)
      [java] org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:517)
      [java] org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:232)
      [java] org.ofbiz.service.GenericDispatcherFactory$GenericDispatcher.runSync(GenericDispatcherFactory.java:83)
      [java] org.ofbiz.entityext.eca.EntityEcaAction.runAction(EntityEcaAction.java:92)
      [java] org.ofbiz.entityext.eca.EntityEcaRule.eval(EntityEcaRule.java:142)
      [java] org.ofbiz.entityext.eca.DelegatorEcaHandler.evalRules(DelegatorEcaHandler.java:83)
      [java] org.ofbiz.entity.GenericDelegator$EntityEcaRuleRunner.evalRules(GenericDelegator.java:2276)
      [java] org.ofbiz.entity.GenericDelegator.create(GenericDelegator.java:871)
      [java] org.ofbiz.entity.GenericDelegator.storeAll(GenericDelegator.java:1355)
      [java] org.ofbiz.entity.util.EntitySaxReader.writeValues(EntitySaxReader.java:326)
      [java] org.ofbiz.entity.util.EntitySaxReader.parse(EntitySaxReader.java:296)
      [java] org.ofbiz.entity.util.EntitySaxReader.parse(EntitySaxReader.java:245)
      [java] org.ofbiz.entity.util.EntityDataLoader.loadData(EntityDataLoader.java:257)
      [java] org.ofbiz.entityext.data.EntityDataLoadContainer.loadContainer(EntityDataLoadContainer.java:492)
      [java] org.ofbiz.entityext.data.EntityDataLoadContainer.start(EntityDataLoadContainer.java:238)
      [java] org.ofbiz.base.container.ContainerLoader.start(ContainerLoader.java:238)
      [java] org.ofbiz.base.start.Start.startStartLoaders(Start.java:340)
      [java] org.ofbiz.base.start.Start.start(Start.java:382)
      [java] org.ofbiz.base.start.Start.main(Start.java:122)
      [java] --------------------------------------------------------------------------------
      [java]
      [java] 2014-07-02 20:24:39,769 (main) [ EntityEcaAction.java:114:ERROR]
      [java] ---- exception report ----------------------------------------------------------
      [java] Error running Entity ECA action service
      [java] Exception: org.ofbiz.service.GenericServiceException
      [java] Message: Service [indexProduct] target threw an unexpected exception (org/apache/lucene/index/CorruptIndexException : Unsupported
major.minor version 51.0)

Jacques
Reply | Threaded
Open this post in threaded view
|

Re: Demos: disk full

Jacques Le Roux
Administrator
OK this is resolved, all demos are now running under Java 7, so Lucene works also with R13.07 (with svn-external from trunk)

A demo is possibly reset at 3AM GTM if a commit has been done since last check. So trunk will be reset more often than releases branches.

Jacques


Le 03/07/2014 20:02, Jacques Le Roux a écrit :

>
> Le 03/07/2014 11:05, Jacques Le Roux a écrit :
>> We have an issue (maybe 2) with stable ecommerce demodata loading, I will look at it...
>>
>
> It was an incompatibility in Lucene component, I did not load it in stable (R13.07) for now...
>
> [java] 2014-07-02 20:24:39,766 (main) [ ServiceDispatcher.java:512:INFO ] [[Sync service failed...- total:0.0,since last(Begin):0.0]] -
> 'entity-default / indexProduct'
>      [java] 2014-07-02 20:24:39,768 (main) [ TransactionUtil.java:378:WARN ]
>      [java] ---- exception report ----------------------------------------------------------
>      [java] [TransactionUtil.setRollbackOnly] Calling transaction setRollbackOnly; this stack trace shows where this is happening:
>      [java] Exception: java.lang.Exception
>      [java] Message: Service [indexProduct] threw an unexpected exception/error
>      [java] ---- stack trace ---------------------------------------------------------------
>      [java] java.lang.Exception: Service [indexProduct] threw an unexpected exception/error
>      [java] org.ofbiz.entity.transaction.TransactionUtil.setRollbackOnly(TransactionUtil.java:378)
>      [java] org.ofbiz.entity.transaction.TransactionUtil.rollback(TransactionUtil.java:319)
>      [java] org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:517)
>      [java] org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:232)
>      [java] org.ofbiz.service.GenericDispatcherFactory$GenericDispatcher.runSync(GenericDispatcherFactory.java:83)
>      [java] org.ofbiz.entityext.eca.EntityEcaAction.runAction(EntityEcaAction.java:92)
>      [java] org.ofbiz.entityext.eca.EntityEcaRule.eval(EntityEcaRule.java:142)
>      [java] org.ofbiz.entityext.eca.DelegatorEcaHandler.evalRules(DelegatorEcaHandler.java:83)
>      [java] org.ofbiz.entity.GenericDelegator$EntityEcaRuleRunner.evalRules(GenericDelegator.java:2276)
>      [java] org.ofbiz.entity.GenericDelegator.create(GenericDelegator.java:871)
>      [java] org.ofbiz.entity.GenericDelegator.storeAll(GenericDelegator.java:1355)
>      [java] org.ofbiz.entity.util.EntitySaxReader.writeValues(EntitySaxReader.java:326)
>      [java] org.ofbiz.entity.util.EntitySaxReader.parse(EntitySaxReader.java:296)
>      [java] org.ofbiz.entity.util.EntitySaxReader.parse(EntitySaxReader.java:245)
>      [java] org.ofbiz.entity.util.EntityDataLoader.loadData(EntityDataLoader.java:257)
>      [java] org.ofbiz.entityext.data.EntityDataLoadContainer.loadContainer(EntityDataLoadContainer.java:492)
>      [java] org.ofbiz.entityext.data.EntityDataLoadContainer.start(EntityDataLoadContainer.java:238)
>      [java] org.ofbiz.base.container.ContainerLoader.start(ContainerLoader.java:238)
>      [java] org.ofbiz.base.start.Start.startStartLoaders(Start.java:340)
>      [java] org.ofbiz.base.start.Start.start(Start.java:382)
>      [java] org.ofbiz.base.start.Start.main(Start.java:122)
>      [java] --------------------------------------------------------------------------------
>      [java]
>      [java] 2014-07-02 20:24:39,769 (main) [ EntityEcaAction.java:114:ERROR]
>      [java] ---- exception report ----------------------------------------------------------
>      [java] Error running Entity ECA action service
>      [java] Exception: org.ofbiz.service.GenericServiceException
>      [java] Message: Service [indexProduct] target threw an unexpected exception (org/apache/lucene/index/CorruptIndexException : Unsupported
> major.minor version 51.0)
>
> Jacques
>

--
Reply | Threaded
Open this post in threaded view
|

Re: Demos: disk full

JulienNicolas
Well done.

Thanks !

Julien.

Le 04/07/2014 08:48, Jacques Le Roux a écrit :

> OK this is resolved, all demos are now running under Java 7, so Lucene
> works also with R13.07 (with svn-external from trunk)
>
> A demo is possibly reset at 3AM GTM if a commit has been done since
> last check. So trunk will be reset more often than releases branches.
>
> Jacques
>
>
> Le 03/07/2014 20:02, Jacques Le Roux a écrit :
>>
>> Le 03/07/2014 11:05, Jacques Le Roux a écrit :
>>> We have an issue (maybe 2) with stable ecommerce demodata loading, I
>>> will look at it...
>>>
>>
>> It was an incompatibility in Lucene component, I did not load it in
>> stable (R13.07) for now...
>>
>> [java] 2014-07-02 20:24:39,766 (main) [
>> ServiceDispatcher.java:512:INFO ] [[Sync service failed...-
>> total:0.0,since last(Begin):0.0]] - 'entity-default / indexProduct'
>>      [java] 2014-07-02 20:24:39,768 (main) [
>> TransactionUtil.java:378:WARN ]
>>      [java] ---- exception report
>> ----------------------------------------------------------
>>      [java] [TransactionUtil.setRollbackOnly] Calling transaction
>> setRollbackOnly; this stack trace shows where this is happening:
>>      [java] Exception: java.lang.Exception
>>      [java] Message: Service [indexProduct] threw an unexpected
>> exception/error
>>      [java] ---- stack trace
>> ---------------------------------------------------------------
>>      [java] java.lang.Exception: Service [indexProduct] threw an
>> unexpected exception/error
>>      [java]
>> org.ofbiz.entity.transaction.TransactionUtil.setRollbackOnly(TransactionUtil.java:378)
>>      [java]
>> org.ofbiz.entity.transaction.TransactionUtil.rollback(TransactionUtil.java:319)
>>      [java]
>> org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:517)
>>      [java]
>> org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:232)
>>      [java]
>> org.ofbiz.service.GenericDispatcherFactory$GenericDispatcher.runSync(GenericDispatcherFactory.java:83)
>>      [java]
>> org.ofbiz.entityext.eca.EntityEcaAction.runAction(EntityEcaAction.java:92)
>>      [java]
>> org.ofbiz.entityext.eca.EntityEcaRule.eval(EntityEcaRule.java:142)
>>      [java]
>> org.ofbiz.entityext.eca.DelegatorEcaHandler.evalRules(DelegatorEcaHandler.java:83)
>>      [java]
>> org.ofbiz.entity.GenericDelegator$EntityEcaRuleRunner.evalRules(GenericDelegator.java:2276)
>>      [java]
>> org.ofbiz.entity.GenericDelegator.create(GenericDelegator.java:871)
>>      [java]
>> org.ofbiz.entity.GenericDelegator.storeAll(GenericDelegator.java:1355)
>>      [java]
>> org.ofbiz.entity.util.EntitySaxReader.writeValues(EntitySaxReader.java:326)
>>      [java]
>> org.ofbiz.entity.util.EntitySaxReader.parse(EntitySaxReader.java:296)
>>      [java]
>> org.ofbiz.entity.util.EntitySaxReader.parse(EntitySaxReader.java:245)
>>      [java]
>> org.ofbiz.entity.util.EntityDataLoader.loadData(EntityDataLoader.java:257)
>>      [java]
>> org.ofbiz.entityext.data.EntityDataLoadContainer.loadContainer(EntityDataLoadContainer.java:492)
>>      [java]
>> org.ofbiz.entityext.data.EntityDataLoadContainer.start(EntityDataLoadContainer.java:238)
>>      [java]
>> org.ofbiz.base.container.ContainerLoader.start(ContainerLoader.java:238)
>>      [java] org.ofbiz.base.start.Start.startStartLoaders(Start.java:340)
>>      [java] org.ofbiz.base.start.Start.start(Start.java:382)
>>      [java] org.ofbiz.base.start.Start.main(Start.java:122)
>>      [java]
>> --------------------------------------------------------------------------------
>>      [java]
>>      [java] 2014-07-02 20:24:39,769 (main) [
>> EntityEcaAction.java:114:ERROR]
>>      [java] ---- exception report
>> ----------------------------------------------------------
>>      [java] Error running Entity ECA action service
>>      [java] Exception: org.ofbiz.service.GenericServiceException
>>      [java] Message: Service [indexProduct] target threw an
>> unexpected exception (org/apache/lucene/index/CorruptIndexException :
>> Unsupported major.minor version 51.0)
>>
>> Jacques
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Demos: disk full

Pierre Smits
Jacques,

Is resetting of demo based on trunk a manual job, or automated?

Regards,

Pierre Smits

*ORRTIZ.COM <http://www.orrtiz.com>*
Services & Solutions for Cloud-
Based Manufacturing, Professional
Services and Retail & Trade
http://www.orrtiz.com
Reply | Threaded
Open this post in threaded view
|

Re: Demos: disk full

Jacques Le Roux
Administrator
it's a cron at 3AM GMT, why?

Jacques

Le 04/07/2014 09:01, Pierre Smits a écrit :

> Jacques,
>
> Is resetting of demo based on trunk a manual job, or automated?
>
> Regards,
>
> Pierre Smits
>
> *ORRTIZ.COM <http://www.orrtiz.com>*
> Services & Solutions for Cloud-
> Based Manufacturing, Professional
> Services and Retail & Trade
> http://www.orrtiz.com
>

--