large entity field (trackingDigest) is truncated

[Mike Voytovich-2]

Hi,

I’m attempting to use the UPS shipment code in OFBiz to generate shipping labels.  Note that we’re still using the default Derby database.

OFBiz is able to send the confirm/accept requests and receive the responses, but an error is returned from UPS in handleUpsShipmentAcceptResponse - “The XML document is not well formed”.

It turns out that we’re properly receiving a 20,000 byte “ShipmentDigest” from UPS in the handleUpsShipmentConfirmResponse method; but it looks like this is being truncated to exactly 8K when written out to the ShipmentRouteSegment entity.  i.e., the ShipmentDigest data is correct in / ofbiz/work/ofbiz/upscert/UpsShipmentConfirmResponse log file; but is 8K (with the remainder padded with "�") in the UpsShipmentAcceptRequest log file.

The trackingDigest field-type is “very-long” with a SQL type of “CLOB”, so I would expect it to save ~20K without issue.

Has anyone else run into this type of problem?  Is there something that needs to be configured in Derby to allow large entity fields?  I did find some documentation on Derby for setting the CLOB size, but I don’t see anywhere in OFBiz where this is done.

thanks,
-mike

[BJ Freeman]

derby should not be used for production system.
you might try the Derby docs on apache
http://db.apache.org/derby/docs/

=========================
BJ Freeman
http://bjfreeman.elance.com
Strategic Power Office with Suppilier Automation <http://www.businessesnetwork.com/automation/viewforum.php?f=93>
Specialtymarket.com <http://www.specialtymarket.com/>

Systems Integrator-- Glad to Assist

Chat  Y! messenger: bjfr33man
Linkedin
<http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk=tab_pro>


Mike Voytovich sent the following on 3/1/2010 9:43 PM:

> Hi,
>
> I’m attempting to use the UPS shipment code in OFBiz to generate shipping labels.  Note that we’re still using the default Derby database.
>
> OFBiz is able to send the confirm/accept requests and receive the responses, but an error is returned from UPS in handleUpsShipmentAcceptResponse - “The XML document is not well formed”.
>
> It turns out that we’re properly receiving a 20,000 byte “ShipmentDigest” from UPS in the handleUpsShipmentConfirmResponse method; but it looks like this is being truncated to exactly 8K when written out to the ShipmentRouteSegment entity.  i.e., the ShipmentDigest data is correct in / ofbiz/work/ofbiz/upscert/UpsShipmentConfirmResponse log file; but is 8K (with the remainder padded with "&#0;") in the UpsShipmentAcceptRequest log file.
>
> The trackingDigest field-type is “very-long” with a SQL type of “CLOB”, so I would expect it to save ~20K without issue.
>
> Has anyone else run into this type of problem?  Is there something that needs to be configured in Derby to allow large entity fields?  I did find some documentation on Derby for setting the CLOB size, but I don’t see anywhere in OFBiz where this is done.
>
> thanks,
> -mike
>

[Mike Voytovich-2]

Hi BJ,

We're still in development, so I was hoping to continue to use Derby for the
time being.  

And yes, I did check the Derby docs - but as I mentioned below, I can't find
anyplace where OFBiz changes the default size for CLOB.

I'll keep digging...

thanks,
-mike



On 3/2/10 12:06 AM, "BJ Freeman" <[hidden email]> wrote:

> derby should not be used for production system.
> you might try the Derby docs on apache
> http://db.apache.org/derby/docs/
>
> =========================
> BJ Freeman
> http://bjfreeman.elance.com
> Strategic Power Office with Suppilier Automation
> <http://www.businessesnetwork.com/automation/viewforum.php?f=93>
> Specialtymarket.com <http://www.specialtymarket.com/>
>
> Systems Integrator-- Glad to Assist
>
> Chat  Y! messenger: bjfr33man
> Linkedin
> <http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk=tab
> _pro>
>
>
> Mike Voytovich sent the following on 3/1/2010 9:43 PM:
>> Hi,
>>
>> I¹m attempting to use the UPS shipment code in OFBiz to generate shipping
>> labels.  Note that we¹re still using the default Derby database.
>>
>> OFBiz is able to send the confirm/accept requests and receive the responses,
>> but an error is returned from UPS in handleUpsShipmentAcceptResponse - ³The
>> XML document is not well formed².
>>
>> It turns out that we¹re properly receiving a 20,000 byte ³ShipmentDigest²
>> from UPS in the handleUpsShipmentConfirmResponse method; but it looks like
>> this is being truncated to exactly 8K when written out to the
>> ShipmentRouteSegment entity.  i.e., the ShipmentDigest data is correct in /
>> ofbiz/work/ofbiz/upscert/UpsShipmentConfirmResponse log file; but is 8K (with
>> the remainder padded with "&#0;") in the UpsShipmentAcceptRequest log file.
>>
>> The trackingDigest field-type is ³very-long² with a SQL type of ³CLOB², so I
>> would expect it to save ~20K without issue.
>>
>> Has anyone else run into this type of problem?  Is there something that needs
>> to be configured in Derby to allow large entity fields?  I did find some
>> documentation on Derby for setting the CLOB size, but I don¹t see anywhere in
>> OFBiz where this is done.
>>
>> thanks,
>> -mike
>>
>
>

[BJ Freeman]

if I understand you
framework\entity\fieldtype\fieldtypederby.xml
shows
<field-type-def type="very-long" sql-type="CLOB"
java-type="String"></field-type-def>

this is as far as ofbiz goes.


=========================
BJ Freeman
http://bjfreeman.elance.com
Strategic Power Office with Suppilier Automation <http://www.businessesnetwork.com/automation/viewforum.php?f=93>
Specialtymarket.com <http://www.specialtymarket.com/>

Systems Integrator-- Glad to Assist

Chat  Y! messenger: bjfr33man
Linkedin
<http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk=tab_pro>


Mike Voytovich sent the following on 3/2/2010 7:11 AM:

> Hi BJ,
>
> We're still in development, so I was hoping to continue to use Derby for the
> time being.  
>
> And yes, I did check the Derby docs - but as I mentioned below, I can't find
> anyplace where OFBiz changes the default size for CLOB.
>
> I'll keep digging...
>
> thanks,
> -mike
>
>
>
> On 3/2/10 12:06 AM, "BJ Freeman" <[hidden email]> wrote:
>
>> derby should not be used for production system.
>> you might try the Derby docs on apache
>> http://db.apache.org/derby/docs/
>>
>> =========================
>> BJ Freeman
>> http://bjfreeman.elance.com
>> Strategic Power Office with Suppilier Automation
>> <http://www.businessesnetwork.com/automation/viewforum.php?f=93>
>> Specialtymarket.com <http://www.specialtymarket.com/>
>>
>> Systems Integrator-- Glad to Assist
>>
>> Chat  Y! messenger: bjfr33man
>> Linkedin
>> <http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk=tab
>> _pro>
>>
>>
>> Mike Voytovich sent the following on 3/1/2010 9:43 PM:
>>> Hi,
>>>
>>> I¹m attempting to use the UPS shipment code in OFBiz to generate shipping
>>> labels.  Note that we¹re still using the default Derby database.
>>>
>>> OFBiz is able to send the confirm/accept requests and receive the responses,
>>> but an error is returned from UPS in handleUpsShipmentAcceptResponse - ³The
>>> XML document is not well formed².
>>>
>>> It turns out that we¹re properly receiving a 20,000 byte ³ShipmentDigest²
>>> from UPS in the handleUpsShipmentConfirmResponse method; but it looks like
>>> this is being truncated to exactly 8K when written out to the
>>> ShipmentRouteSegment entity.  i.e., the ShipmentDigest data is correct in /
>>> ofbiz/work/ofbiz/upscert/UpsShipmentConfirmResponse log file; but is 8K (with
>>> the remainder padded with "&#0;") in the UpsShipmentAcceptRequest log file.
>>>
>>> The trackingDigest field-type is ³very-long² with a SQL type of ³CLOB², so I
>>> would expect it to save ~20K without issue.
>>>
>>> Has anyone else run into this type of problem?  Is there something that needs
>>> to be configured in Derby to allow large entity fields?  I did find some
>>> documentation on Derby for setting the CLOB size, but I don¹t see anywhere in
>>> OFBiz where this is done.
>>>
>>> thanks,
>>> -mike
>>>
>>
>
>

[Mike Voytovich-2]

The problem was in fact with Derby.  I switched over to MySQL, and now I see
the correct data read out of the ShipmentRouteSegment entity.

I think it is indeed an OFBiz bug, since Derby is the default database out
of the box - and it will surely bite another new user who is either trying
to use the UPS shipping support, or any entity that requires a field > 8K.

I've filed Jira issue OFBIZ-3529 regarding this issue:

https://issues.apache.org/jira/browse/OFBIZ-3529

thanks,
-mike



On 3/2/10 10:00 AM, "BJ Freeman" <[hidden email]> wrote:

> if I understand you
> framework\entity\fieldtype\fieldtypederby.xml
> shows
> <field-type-def type="very-long" sql-type="CLOB"
> java-type="String"></field-type-def>
>
> this is as far as ofbiz goes.
>
>
> =========================
> BJ Freeman
> http://bjfreeman.elance.com
> Strategic Power Office with Suppilier Automation
> <http://www.businessesnetwork.com/automation/viewforum.php?f=93>
> Specialtymarket.com <http://www.specialtymarket.com/>
>
> Systems Integrator-- Glad to Assist
>
> Chat  Y! messenger: bjfr33man
> Linkedin
> <http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk=tab
> _pro>
>
>
> Mike Voytovich sent the following on 3/2/2010 7:11 AM:
>> Hi BJ,
>>
>> We're still in development, so I was hoping to continue to use Derby for the
>> time being.  
>>
>> And yes, I did check the Derby docs - but as I mentioned below, I can't find
>> anyplace where OFBiz changes the default size for CLOB.
>>
>> I'll keep digging...
>>
>> thanks,
>> -mike
>>
>>
>>
>> On 3/2/10 12:06 AM, "BJ Freeman" <[hidden email]> wrote:
>>
>>> derby should not be used for production system.
>>> you might try the Derby docs on apache
>>> http://db.apache.org/derby/docs/
>>>
>>> =========================
>>> BJ Freeman
>>> http://bjfreeman.elance.com
>>> Strategic Power Office with Suppilier Automation
>>> <http://www.businessesnetwork.com/automation/viewforum.php?f=93>
>>> Specialtymarket.com <http://www.specialtymarket.com/>
>>>
>>> Systems Integrator-- Glad to Assist
>>>
>>> Chat  Y! messenger: bjfr33man
>>> Linkedin
>>> <http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk=t
>>> ab
>>> _pro>
>>>
>>>
>>> Mike Voytovich sent the following on 3/1/2010 9:43 PM:
>>>> Hi,
>>>>
>>>> I¹m attempting to use the UPS shipment code in OFBiz to generate shipping
>>>> labels.  Note that we¹re still using the default Derby database.
>>>>
>>>> OFBiz is able to send the confirm/accept requests and receive the
>>>> responses,
>>>> but an error is returned from UPS in handleUpsShipmentAcceptResponse - ³The
>>>> XML document is not well formed².
>>>>
>>>> It turns out that we¹re properly receiving a 20,000 byte ³ShipmentDigest²
>>>> from UPS in the handleUpsShipmentConfirmResponse method; but it looks like
>>>> this is being truncated to exactly 8K when written out to the
>>>> ShipmentRouteSegment entity.  i.e., the ShipmentDigest data is correct in /
>>>> ofbiz/work/ofbiz/upscert/UpsShipmentConfirmResponse log file; but is 8K
>>>> (with
>>>> the remainder padded with "&#0;") in the UpsShipmentAcceptRequest log file.
>>>>
>>>> The trackingDigest field-type is ³very-long² with a SQL type of ³CLOB², so
>>>> I
>>>> would expect it to save ~20K without issue.
>>>>
>>>> Has anyone else run into this type of problem?  Is there something that
>>>> needs
>>>> to be configured in Derby to allow large entity fields?  I did find some
>>>> documentation on Derby for setting the CLOB size, but I don¹t see anywhere
>>>> in
>>>> OFBiz where this is done.
>>>>
>>>> thanks,
>>>> -mike
>>>>
>>>
>>
>>
>
>

[Scott Gray-2]

Hi Mike,

I have no idea if this will help but it will at least allow us to rule one thing out, could you please try changing:
sql-type="CLOB"
to:
sql-type="CLOB(2M)"

Thanks
Scott

HotWax Media
http://www.hotwaxmedia.com

On 2/03/2010, at 5:06 PM, Mike Voytovich wrote:

> The problem was in fact with Derby.  I switched over to MySQL, and now I see
> the correct data read out of the ShipmentRouteSegment entity.
>
> I think it is indeed an OFBiz bug, since Derby is the default database out
> of the box - and it will surely bite another new user who is either trying
> to use the UPS shipping support, or any entity that requires a field > 8K.
>
> I've filed Jira issue OFBIZ-3529 regarding this issue:
>
> https://issues.apache.org/jira/browse/OFBIZ-3529
>
> thanks,
> -mike
>
>
>
> On 3/2/10 10:00 AM, "BJ Freeman" <[hidden email]> wrote:
>
>> if I understand you
>> framework\entity\fieldtype\fieldtypederby.xml
>> shows
>> <field-type-def type="very-long" sql-type="CLOB"
>> java-type="String"></field-type-def>
>>
>> this is as far as ofbiz goes.
>>
>>
>> =========================
>> BJ Freeman
>> http://bjfreeman.elance.com
>> Strategic Power Office with Suppilier Automation
>> <http://www.businessesnetwork.com/automation/viewforum.php?f=93>
>> Specialtymarket.com <http://www.specialtymarket.com/>
>>
>> Systems Integrator-- Glad to Assist
>>
>> Chat  Y! messenger: bjfr33man
>> Linkedin
>> <http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk=tab
>> _pro>
>>
>>
>> Mike Voytovich sent the following on 3/2/2010 7:11 AM:
>>> Hi BJ,
>>>
>>> We're still in development, so I was hoping to continue to use Derby for the
>>> time being.  
>>>
>>> And yes, I did check the Derby docs - but as I mentioned below, I can't find
>>> anyplace where OFBiz changes the default size for CLOB.
>>>
>>> I'll keep digging...
>>>
>>> thanks,
>>> -mike
>>>
>>>
>>>
>>> On 3/2/10 12:06 AM, "BJ Freeman" <[hidden email]> wrote:
>>>
>>>> derby should not be used for production system.
>>>> you might try the Derby docs on apache
>>>> http://db.apache.org/derby/docs/
>>>>
>>>> =========================
>>>> BJ Freeman
>>>> http://bjfreeman.elance.com
>>>> Strategic Power Office with Suppilier Automation
>>>> <http://www.businessesnetwork.com/automation/viewforum.php?f=93>
>>>> Specialtymarket.com <http://www.specialtymarket.com/>
>>>>
>>>> Systems Integrator-- Glad to Assist
>>>>
>>>> Chat  Y! messenger: bjfr33man
>>>> Linkedin
>>>> <http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk=t
>>>> ab
>>>> _pro>
>>>>
>>>>
>>>> Mike Voytovich sent the following on 3/1/2010 9:43 PM:
>>>>> Hi,
>>>>>
>>>>> I’m attempting to use the UPS shipment code in OFBiz to generate shipping
>>>>> labels.  Note that we’re still using the default Derby database.
>>>>>
>>>>> OFBiz is able to send the confirm/accept requests and receive the
>>>>> responses,
>>>>> but an error is returned from UPS in handleUpsShipmentAcceptResponse - “The
>>>>> XML document is not well formed”.
>>>>>
>>>>> It turns out that we’re properly receiving a 20,000 byte “ShipmentDigest”
>>>>> from UPS in the handleUpsShipmentConfirmResponse method; but it looks like
>>>>> this is being truncated to exactly 8K when written out to the
>>>>> ShipmentRouteSegment entity.  i.e., the ShipmentDigest data is correct in /
>>>>> ofbiz/work/ofbiz/upscert/UpsShipmentConfirmResponse log file; but is 8K
>>>>> (with
>>>>> the remainder padded with "&#0;") in the UpsShipmentAcceptRequest log file.
>>>>>
>>>>> The trackingDigest field-type is “very-long” with a SQL type of “CLOB”, so
>>>>> I
>>>>> would expect it to save ~20K without issue.
>>>>>
>>>>> Has anyone else run into this type of problem?  Is there something that
>>>>> needs
>>>>> to be configured in Derby to allow large entity fields?  I did find some
>>>>> documentation on Derby for setting the CLOB size, but I don’t see anywhere
>>>>> in
>>>>> OFBiz where this is done.
>>>>>
>>>>> thanks,
>>>>> -mike
>>>>>
>>>>
>>>
>>>
>>
>>
>

[Mike Voytovich-2]

Hi Scott,

Thanks for the suggestion.  I made the diff below and performed a clean-all
and rebuilt - and it did NOT fix the problem...

regards,
-mike

--- framework/entity/fieldtype/fieldtypederby.xml    (revision 140)
+++ framework/entity/fieldtype/fieldtypederby.xml    (working copy)
@@ -42,7 +42,7 @@
     <field-type-def type="very-short" sql-type="VARCHAR(10)"
java-type="String"></field-type-def>
     <field-type-def type="short-varchar" sql-type="VARCHAR(60)"
java-type="String"></field-type-def>
     <field-type-def type="long-varchar" sql-type="VARCHAR(255)"
java-type="String"></field-type-def>
-    <field-type-def type="very-long" sql-type="CLOB"
java-type="String"></field-type-def>
+    <field-type-def type="very-long" sql-type="CLOB(2M)"
java-type="String"></field-type-def>
 
     <field-type-def type="comment" sql-type="VARCHAR(255)"
java-type="String"></field-type-def>
     <field-type-def type="description" sql-type="VARCHAR(255)"
java-type="String"></field-type-def>



On 3/3/10 6:31 AM, "Scott Gray" <[hidden email]> wrote:

> Hi Mike,
>
> I have no idea if this will help but it will at least allow us to rule one
> thing out, could you please try changing:
> sql-type="CLOB"
> to:
> sql-type="CLOB(2M)"
>
> Thanks
> Scott
>
> HotWax Media
> http://www.hotwaxmedia.com
>
> On 2/03/2010, at 5:06 PM, Mike Voytovich wrote:
>
>> The problem was in fact with Derby.  I switched over to MySQL, and now I see
>> the correct data read out of the ShipmentRouteSegment entity.
>>
>> I think it is indeed an OFBiz bug, since Derby is the default database out
>> of the box - and it will surely bite another new user who is either trying
>> to use the UPS shipping support, or any entity that requires a field > 8K.
>>
>> I've filed Jira issue OFBIZ-3529 regarding this issue:
>>
>> https://issues.apache.org/jira/browse/OFBIZ-3529
>>
>> thanks,
>> -mike
>>
>>
>>
>> On 3/2/10 10:00 AM, "BJ Freeman" <[hidden email]> wrote:
>>
>>> if I understand you
>>> framework\entity\fieldtype\fieldtypederby.xml
>>> shows
>>> <field-type-def type="very-long" sql-type="CLOB"
>>> java-type="String"></field-type-def>
>>>
>>> this is as far as ofbiz goes.
>>>
>>>
>>> =========================
>>> BJ Freeman
>>> http://bjfreeman.elance.com
>>> Strategic Power Office with Suppilier Automation
>>> <http://www.businessesnetwork.com/automation/viewforum.php?f=93>
>>> Specialtymarket.com <http://www.specialtymarket.com/>
>>>
>>> Systems Integrator-- Glad to Assist
>>>
>>> Chat  Y! messenger: bjfr33man
>>> Linkedin
>>> <http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk=t
>>> ab
>>> _pro>
>>>
>>>
>>> Mike Voytovich sent the following on 3/2/2010 7:11 AM:
>>>> Hi BJ,
>>>>
>>>> We're still in development, so I was hoping to continue to use Derby for
>>>> the
>>>> time being.  
>>>>
>>>> And yes, I did check the Derby docs - but as I mentioned below, I can't
>>>> find
>>>> anyplace where OFBiz changes the default size for CLOB.
>>>>
>>>> I'll keep digging...
>>>>
>>>> thanks,
>>>> -mike
>>>>
>>>>
>>>>
>>>> On 3/2/10 12:06 AM, "BJ Freeman" <[hidden email]> wrote:
>>>>
>>>>> derby should not be used for production system.
>>>>> you might try the Derby docs on apache
>>>>> http://db.apache.org/derby/docs/
>>>>>
>>>>> =========================
>>>>> BJ Freeman
>>>>> http://bjfreeman.elance.com
>>>>> Strategic Power Office with Suppilier Automation
>>>>> <http://www.businessesnetwork.com/automation/viewforum.php?f=93>
>>>>> Specialtymarket.com <http://www.specialtymarket.com/>
>>>>>
>>>>> Systems Integrator-- Glad to Assist
>>>>>
>>>>> Chat  Y! messenger: bjfr33man
>>>>> Linkedin
>>>>> <http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk
>>>>> =t
>>>>> ab
>>>>> _pro>
>>>>>
>>>>>
>>>>> Mike Voytovich sent the following on 3/1/2010 9:43 PM:
>>>>>> Hi,
>>>>>>
>>>>>> I¹m attempting to use the UPS shipment code in OFBiz to generate shipping
>>>>>> labels.  Note that we¹re still using the default Derby database.
>>>>>>
>>>>>> OFBiz is able to send the confirm/accept requests and receive the
>>>>>> responses,
>>>>>> but an error is returned from UPS in handleUpsShipmentAcceptResponse -
>>>>>> ³The
>>>>>> XML document is not well formed².
>>>>>>
>>>>>> It turns out that we¹re properly receiving a 20,000 byte ³ShipmentDigest²
>>>>>> from UPS in the handleUpsShipmentConfirmResponse method; but it looks
>>>>>> like
>>>>>> this is being truncated to exactly 8K when written out to the
>>>>>> ShipmentRouteSegment entity.  i.e., the ShipmentDigest data is correct in
>>>>>> /
>>>>>> ofbiz/work/ofbiz/upscert/UpsShipmentConfirmResponse log file; but is 8K
>>>>>> (with
>>>>>> the remainder padded with "&#0;") in the UpsShipmentAcceptRequest log
>>>>>> file.
>>>>>>
>>>>>> The trackingDigest field-type is ³very-long² with a SQL type of ³CLOB²,
>>>>>> so
>>>>>> I
>>>>>> would expect it to save ~20K without issue.
>>>>>>
>>>>>> Has anyone else run into this type of problem?  Is there something that
>>>>>> needs
>>>>>> to be configured in Derby to allow large entity fields?  I did find some
>>>>>> documentation on Derby for setting the CLOB size, but I don¹t see
>>>>>> anywhere
>>>>>> in
>>>>>> OFBiz where this is done.
>>>>>>
>>>>>> thanks,
>>>>>> -mike
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>

[Jeroen van der Wal]

We're experiencing the same issue after merging with the latest trunk.
We still have to do a root cause analysis. Perhaps something with the
new conversion framework?

--
Jeroen van der Wal
Stromboli b.v.
[phone]