data import - product images

[Jesse Thomas]

Is there is a way of using a data reader to load a product image and
have it treated as though its uploaded as an "Original Image"
(generating the other sizes needed)? Or whats the best way to load
images during data migration?

The images are available in the file system and via http, several
thousand of them. Thanks in advance for any help or advice!

Thanks,

     Jesse

[Rishi Solanki]

Fix the number of images you would like to use for order. Fix there size
and pattern you like to put in the file system. Put that path in the
product data and push all images in file system separately.

You may like to add script to do that which will be bit customization.
Please refer catalog > products > edit product > edit product contents for
details.


HTH!

--
Rishi Solanki



On 24 Jan 2018 1:00 pm, "Jesse Thomas" <[hidden email]> wrote:

Is there is a way of using a data reader to load a product image and have
it treated as though its uploaded as an "Original Image" (generating the
other sizes needed)? Or whats the best way to load images during data
migration?

The images are available in the file system and via http, several thousand
of them. Thanks in advance for any help or advice!

Thanks,

    Jesse

[Shi Jinghai-3]

On generating the other sizes needed, it's already in the source code, see
https://github.com/apache/ofbiz-framework/blob/trunk/applications/product/src/main/java/org/apache/ofbiz/product/image/ScaleImage.java

-----邮件原件-----
发件人: Jesse Thomas [mailto:[hidden email]]
发送时间: 2018年1月24日 15:30
收件人: [hidden email]
主题: data import - product images

Is there is a way of using a data reader to load a product image and have it treated as though its uploaded as an "Original Image"
(generating the other sizes needed)? Or whats the best way to load images during data migration?

The images are available in the file system and via http, several thousand of them. Thanks in advance for any help or advice!

Thanks,

     Jesse

[Jesse Thomas]

Thanks Shi and Rishi for the responses.

I was hoping to do this the hard way and l learn some new tricks in the
process. More specifically I was hoping to use a ftl template (like in
./specialpurpose/ecommerce/data/DemoTree.xml) to call a service or
function directly. Making the solution 100% OFB.

If you know of any examples or clues please let me know.

Thanks again!


On 1/24/2018 11:27 PM, Shi Jinghai wrote:

> On generating the other sizes needed, it's already in the source code, see
> https://github.com/apache/ofbiz-framework/blob/trunk/applications/product/src/main/java/org/apache/ofbiz/product/image/ScaleImage.java
>
> -----邮件原件-----
> 发件人: Jesse Thomas [mailto:[hidden email]]
> 发送时间: 2018年1月24日 15:30
> 收件人: [hidden email]
> 主题: data import - product images
>
> Is there is a way of using a data reader to load a product image and have it treated as though its uploaded as an "Original Image"
> (generating the other sizes needed)? Or whats the best way to load images during data migration?
>
> The images are available in the file system and via http, several thousand of them. Thanks in advance for any help or advice!
>
> Thanks,
>
>       Jesse
>

[Mike Z]

A well written perl or shell script can take "original.jpg" in directory
"x" and create, multiple larger/smaller images with the same "root" name.
You can use image magic, an image processor package, to do this.  Basic
stuff for anyone who knows shell scripting.

You can then import these images using a single XML data_reader file, and
import it into OFB.  Example:

<?xml version="1.0" encoding="UTF-8"?>
<entity-engine-xml>
<Product
  productId="798936836182"
  smallImageUrl="/images/products/10000/1001MINIGOLF/small.jpg"
  mediumImageUrl="/images/products/10000/1001MINIGOLF/medium.jpg"
  largeImageUrl="/images/products/10000/1001MINIGOLF/large.jpg"
  originalImageUrl="/images/products/10000/1001MINIGOLF/original.jpg"
  detailImageUrl="/images/products/10000/1001MINIGOLF/detail.jpg"
/>

[plus 1000s more].....

</entity-engine-xml>

A single script can generate the above XML file (with 1000s of products),
resize 1000s of "original.jpg" into multiple images, and import into OFB,
all in one pass.  Standard stuff for anyone competent with perl/shell
scripting.

What you don't want is make java do busy, CPU intensive work and have it
render different sizes "on the fly", if that is what you are suggesting.

On Wed, Jan 24, 2018 at 11:56 PM, Jesse Thomas <[hidden email]>
wrote:

> Thanks Shi and Rishi for the responses.
>
> I was hoping to do this the hard way and l learn some new tricks in the
> process. More specifically I was hoping to use a ftl template (like in
> ./specialpurpose/ecommerce/data/DemoTree.xml) to call a service or
> function directly. Making the solution 100% OFB.
>
> If you know of any examples or clues please let me know.
>
> Thanks again!
>
>
>
> On 1/24/2018 11:27 PM, Shi Jinghai wrote:
>
>> On generating the other sizes needed, it's already in the source code, see
>> https://github.com/apache/ofbiz-framework/blob/trunk/applica
>> tions/product/src/main/java/org/apache/ofbiz/product/image
>> /ScaleImage.java
>>
>> -----邮件原件-----
>> 发件人: Jesse Thomas [mailto:[hidden email]]
>> 发送时间: 2018年1月24日 15:30
>> 收件人: [hidden email]
>> 主题: data import - product images
>>
>> Is there is a way of using a data reader to load a product image and have
>> it treated as though its uploaded as an "Original Image"
>> (generating the other sizes needed)? Or whats the best way to load images
>> during data migration?
>>
>> The images are available in the file system and via http, several
>> thousand of them. Thanks in advance for any help or advice!
>>
>> Thanks,
>>
>>       Jesse
>>
>>
>

[Shi Jinghai-3]

Thanks Mike, I agree with you if such image processing happens regularly, perl is a good choice but system may be complicated.

The image processing speed of JDK 7 is dramatically slower than JDK 6, and sometimes wrong, as Kodak and others withdrew their image algorithm properties when Java transferred from Sun to Oracle.

Java will struggle on ImageIO for a while, hope JDK 9 have a big improvement. [1]

Kind Regards,

Shi Jinghai


[1] https://bugs.openjdk.java.net/browse/JDK-8041125



发件人: Mike [mailto:[hidden email]]
发送时间: 2018年1月26日 0:26
收件人: user
抄送: Shi Jinghai
主题: Re: data import - product images

A well written perl or shell script can take "original.jpg" in directory "x" and create, multiple larger/smaller images with the same "root" name. You can use image magic, an image processor package, to do this.  Basic stuff for anyone who knows shell scripting.

You can then import these images using a single XML data_reader file, and import it into OFB.  Example:

<?xml version="1.0" encoding="UTF-8"?>
<entity-engine-xml>
<Product
  productId="798936836182"
  smallImageUrl="/images/products/10000/1001MINIGOLF/small.jpg"
  mediumImageUrl="/images/products/10000/1001MINIGOLF/medium.jpg"
  largeImageUrl="/images/products/10000/1001MINIGOLF/large.jpg"
  originalImageUrl="/images/products/10000/1001MINIGOLF/original.jpg"
  detailImageUrl="/images/products/10000/1001MINIGOLF/detail.jpg"
/>

[plus 1000s more].....

</entity-engine-xml>

A single script can generate the above XML file (with 1000s of products), resize 1000s of "original.jpg" into multiple images, and import into OFB, all in one pass.  Standard stuff for anyone competent with perl/shell scripting.

What you don't want is make java do busy, CPU intensive work and have it render different sizes "on the fly", if that is what you are suggesting.

On Wed, Jan 24, 2018 at 11:56 PM, Jesse Thomas <[hidden email]<mailto:[hidden email]>> wrote:
Thanks Shi and Rishi for the responses.

I was hoping to do this the hard way and l learn some new tricks in the process. More specifically I was hoping to use a ftl template (like in ./specialpurpose/ecommerce/data/DemoTree.xml) to call a service or function directly. Making the solution 100% OFB.

If you know of any examples or clues please let me know.

Thanks again!



On 1/24/2018 11:27 PM, Shi Jinghai wrote:
On generating the other sizes needed, it's already in the source code, see
https://github.com/apache/ofbiz-framework/blob/trunk/applications/product/src/main/java/org/apache/ofbiz/product/image/ScaleImage.java

-----邮件原件-----
发件人: Jesse Thomas [mailto:[hidden email]<mailto:[hidden email]>]
发送时间: 2018年1月24日 15:30
收件人: [hidden email]<mailto:[hidden email]>
主题: data import - product images

Is there is a way of using a data reader to load a product image and have it treated as though its uploaded as an "Original Image"
(generating the other sizes needed)? Or whats the best way to load images during data migration?

The images are available in the file system and via http, several thousand of them. Thanks in advance for any help or advice!

Thanks,

      Jesse

[Jacques Le Roux]

Thanks Jinghai,

That's an interesting information!

Jacques


Le 26/01/2018 à 02:57, Shi Jinghai a écrit :

> Thanks Mike, I agree with you if such image processing happens regularly, perl is a good choice but system may be complicated.
>
> The image processing speed of JDK 7 is dramatically slower than JDK 6, and sometimes wrong, as Kodak and others withdrew their image algorithm properties when Java transferred from Sun to Oracle.
>
> Java will struggle on ImageIO for a while, hope JDK 9 have a big improvement. [1]
>
> Kind Regards,
>
> Shi Jinghai
>
>
> [1] https://bugs.openjdk.java.net/browse/JDK-8041125
>
>
>
> 发件人: Mike [mailto:[hidden email]]
> 发送时间: 2018年1月26日 0:26
> 收件人: user
> 抄送: Shi Jinghai
> 主题: Re: data import - product images
>
> A well written perl or shell script can take "original.jpg" in directory "x" and create, multiple larger/smaller images with the same "root" name. You can use image magic, an image processor package, to do this.  Basic stuff for anyone who knows shell scripting.
>
> You can then import these images using a single XML data_reader file, and import it into OFB.  Example:
>
> <?xml version="1.0" encoding="UTF-8"?>
> <entity-engine-xml>
> <Product
>    productId="798936836182"
>    smallImageUrl="/images/products/10000/1001MINIGOLF/small.jpg"
>    mediumImageUrl="/images/products/10000/1001MINIGOLF/medium.jpg"
>    largeImageUrl="/images/products/10000/1001MINIGOLF/large.jpg"
>    originalImageUrl="/images/products/10000/1001MINIGOLF/original.jpg"
>    detailImageUrl="/images/products/10000/1001MINIGOLF/detail.jpg"
> />
>
> [plus 1000s more].....
>
> </entity-engine-xml>
>
> A single script can generate the above XML file (with 1000s of products), resize 1000s of "original.jpg" into multiple images, and import into OFB, all in one pass.  Standard stuff for anyone competent with perl/shell scripting.
>
> What you don't want is make java do busy, CPU intensive work and have it render different sizes "on the fly", if that is what you are suggesting.
>
> On Wed, Jan 24, 2018 at 11:56 PM, Jesse Thomas <[hidden email]<mailto:[hidden email]>> wrote:
> Thanks Shi and Rishi for the responses.
>
> I was hoping to do this the hard way and l learn some new tricks in the process. More specifically I was hoping to use a ftl template (like in ./specialpurpose/ecommerce/data/DemoTree.xml) to call a service or function directly. Making the solution 100% OFB.
>
> If you know of any examples or clues please let me know.
>
> Thanks again!
>
>
>
> On 1/24/2018 11:27 PM, Shi Jinghai wrote:
> On generating the other sizes needed, it's already in the source code, see
> https://github.com/apache/ofbiz-framework/blob/trunk/applications/product/src/main/java/org/apache/ofbiz/product/image/ScaleImage.java
>
> -----邮件原件-----
> 发件人: Jesse Thomas [mailto:[hidden email]<mailto:[hidden email]>]
> 发送时间: 2018年1月24日 15:30
> 收件人: [hidden email]<mailto:[hidden email]>
> 主题: data import - product images
>
> Is there is a way of using a data reader to load a product image and have it treated as though its uploaded as an "Original Image"
> (generating the other sizes needed)? Or whats the best way to load images during data migration?
>
> The images are available in the file system and via http, several thousand of them. Thanks in advance for any help or advice!
>
> Thanks,
>
>        Jesse
>
>

[Jesse Thomas]

I have written a service that checks a folder for image files, resizes
and updates the Product in OFBiz based on the file name equaling the
productId. It works for a few hundred images, but when I process a few
thousand images at the end it hits a transaction timeout. All the images
are resized and saved. But the product records don't get updated. All
I'm doing in the database is updating the Product image fields, is there
a way to force the commit on each product instead at the end of the
service call?

Thanks,
     Jesse

On 1/26/2018 12:48 AM, Jacques Le Roux wrote:

> Thanks Jinghai,
>
> That's an interesting information!
>
> Jacques
>
>
> Le 26/01/2018 à 02:57, Shi Jinghai a écrit :
>> Thanks Mike, I agree with you if such image processing happens
>> regularly, perl is a good choice but system may be complicated.
>>
>> The image processing speed of JDK 7 is dramatically slower than JDK
>> 6, and sometimes wrong, as Kodak and others withdrew their image
>> algorithm properties when Java transferred from Sun to Oracle.
>>
>> Java will struggle on ImageIO for a while, hope JDK 9 have a big
>> improvement. [1]
>>
>> Kind Regards,
>>
>> Shi Jinghai
>>
>>
>> [1] https://bugs.openjdk.java.net/browse/JDK-8041125
>>
>>
>>
>> 发件人: Mike [mailto:[hidden email]]
>> 发送时间: 2018年1月26日 0:26
>> 收件人: user
>> 抄送: Shi Jinghai
>> 主题: Re: data import - product images
>>
>> A well written perl or shell script can take "original.jpg" in
>> directory "x" and create, multiple larger/smaller images with the
>> same "root" name. You can use image magic, an image processor
>> package, to do this.  Basic stuff for anyone who knows shell scripting.
>>
>> You can then import these images using a single XML data_reader file,
>> and import it into OFB.  Example:
>>
>> <?xml version="1.0" encoding="UTF-8"?>
>> <entity-engine-xml>
>> <Product
>>    productId="798936836182"
>>    smallImageUrl="/images/products/10000/1001MINIGOLF/small.jpg"
>> mediumImageUrl="/images/products/10000/1001MINIGOLF/medium.jpg"
>>    largeImageUrl="/images/products/10000/1001MINIGOLF/large.jpg"
>> originalImageUrl="/images/products/10000/1001MINIGOLF/original.jpg"
>> detailImageUrl="/images/products/10000/1001MINIGOLF/detail.jpg"
>> />
>>
>> [plus 1000s more].....
>>
>> </entity-engine-xml>
>>
>> A single script can generate the above XML file (with 1000s of
>> products), resize 1000s of "original.jpg" into multiple images, and
>> import into OFB, all in one pass.  Standard stuff for anyone
>> competent with perl/shell scripting.
>>
>> What you don't want is make java do busy, CPU intensive work and have
>> it render different sizes "on the fly", if that is what you are
>> suggesting.
>>
>> On Wed, Jan 24, 2018 at 11:56 PM, Jesse Thomas
>> <[hidden email]<mailto:[hidden email]>> wrote:
>> Thanks Shi and Rishi for the responses.
>>
>> I was hoping to do this the hard way and l learn some new tricks in
>> the process. More specifically I was hoping to use a ftl template
>> (like in ./specialpurpose/ecommerce/data/DemoTree.xml) to call a
>> service or function directly. Making the solution 100% OFB.
>>
>> If you know of any examples or clues please let me know.
>>
>> Thanks again!
>>
>>
>>
>> On 1/24/2018 11:27 PM, Shi Jinghai wrote:
>> On generating the other sizes needed, it's already in the source
>> code, see
>> https://github.com/apache/ofbiz-framework/blob/trunk/applications/product/src/main/java/org/apache/ofbiz/product/image/ScaleImage.java 
>>
>>
>> -----邮件原件-----
>> 发件人: Jesse Thomas
>> [mailto:[hidden email]<mailto:[hidden email]>]
>> 发送时间: 2018年1月24日 15:30
>> 收件人: [hidden email]<mailto:[hidden email]>
>> 主题: data import - product images
>>
>> Is there is a way of using a data reader to load a product image and
>> have it treated as though its uploaded as an "Original Image"
>> (generating the other sizes needed)? Or whats the best way to load
>> images during data migration?
>>
>> The images are available in the file system and via http, several
>> thousand of them. Thanks in advance for any help or advice!
>>
>> Thanks,
>>
>>        Jesse
>>
>>
>

[Swapnil Mane]

Hello Jesse,

Solution to avoid transaction timeout is to schedule your job (service)
https://demo-trunk.ofbiz.apache.org/webtools/control/scheduleJob


- Best  Regards,
Swapnil M Mane


On Fri, Feb 23, 2018 at 1:59 PM, Jesse Thomas <[hidden email]>
wrote:

> I have written a service that checks a folder for image files, resizes and
> updates the Product in OFBiz based on the file name equaling the productId.
> It works for a few hundred images, but when I process a few thousand images
> at the end it hits a transaction timeout. All the images are resized and
> saved. But the product records don't get updated. All I'm doing in the
> database is updating the Product image fields, is there a way to force the
> commit on each product instead at the end of the service call?
>
> Thanks,
>     Jesse
>
>
> On 1/26/2018 12:48 AM, Jacques Le Roux wrote:
>
>> Thanks Jinghai,
>>
>> That's an interesting information!
>>
>> Jacques
>>
>>
>> Le 26/01/2018 à 02:57, Shi Jinghai a écrit :
>>
>>> Thanks Mike, I agree with you if such image processing happens
>>> regularly, perl is a good choice but system may be complicated.
>>>
>>> The image processing speed of JDK 7 is dramatically slower than JDK 6,
>>> and sometimes wrong, as Kodak and others withdrew their image algorithm
>>> properties when Java transferred from Sun to Oracle.
>>>
>>> Java will struggle on ImageIO for a while, hope JDK 9 have a big
>>> improvement. [1]
>>>
>>> Kind Regards,
>>>
>>> Shi Jinghai
>>>
>>>
>>> [1] https://bugs.openjdk.java.net/browse/JDK-8041125
>>>
>>>
>>>
>>> 发件人: Mike [mailto:[hidden email]]
>>> 发送时间: 2018年1月26日 0:26
>>> 收件人: user
>>> 抄送: Shi Jinghai
>>> 主题: Re: data import - product images
>>>
>>> A well written perl or shell script can take "original.jpg" in directory
>>> "x" and create, multiple larger/smaller images with the same "root" name.
>>> You can use image magic, an image processor package, to do this.  Basic
>>> stuff for anyone who knows shell scripting.
>>>
>>> You can then import these images using a single XML data_reader file,
>>> and import it into OFB.  Example:
>>>
>>> <?xml version="1.0" encoding="UTF-8"?>
>>> <entity-engine-xml>
>>> <Product
>>>    productId="798936836182"
>>>    smallImageUrl="/images/products/10000/1001MINIGOLF/small.jpg"
>>> mediumImageUrl="/images/products/10000/1001MINIGOLF/medium.jpg"
>>>    largeImageUrl="/images/products/10000/1001MINIGOLF/large.jpg"
>>> originalImageUrl="/images/products/10000/1001MINIGOLF/original.jpg"
>>> detailImageUrl="/images/products/10000/1001MINIGOLF/detail.jpg"
>>> />
>>>
>>> [plus 1000s more].....
>>>
>>> </entity-engine-xml>
>>>
>>> A single script can generate the above XML file (with 1000s of
>>> products), resize 1000s of "original.jpg" into multiple images, and import
>>> into OFB, all in one pass.  Standard stuff for anyone competent with
>>> perl/shell scripting.
>>>
>>> What you don't want is make java do busy, CPU intensive work and have it
>>> render different sizes "on the fly", if that is what you are suggesting.
>>>
>>> On Wed, Jan 24, 2018 at 11:56 PM, Jesse Thomas <[hidden email]
>>> <mailto:[hidden email]>> wrote:
>>> Thanks Shi and Rishi for the responses.
>>>
>>> I was hoping to do this the hard way and l learn some new tricks in the
>>> process. More specifically I was hoping to use a ftl template (like in
>>> ./specialpurpose/ecommerce/data/DemoTree.xml) to call a service or
>>> function directly. Making the solution 100% OFB.
>>>
>>> If you know of any examples or clues please let me know.
>>>
>>> Thanks again!
>>>
>>>
>>>
>>> On 1/24/2018 11:27 PM, Shi Jinghai wrote:
>>> On generating the other sizes needed, it's already in the source code,
>>> see
>>> https://github.com/apache/ofbiz-framework/blob/trunk/applica
>>> tions/product/src/main/java/org/apache/ofbiz/product/image/
>>> ScaleImage.java
>>>
>>> -----邮件原件-----
>>> 发件人: Jesse Thomas [mailto:[hidden email]<mailto:
>>> [hidden email]>]
>>> 发送时间: 2018年1月24日 15:30
>>> 收件人: [hidden email]<mailto:[hidden email]>
>>> 主题: data import - product images
>>>
>>> Is there is a way of using a data reader to load a product image and
>>> have it treated as though its uploaded as an "Original Image"
>>> (generating the other sizes needed)? Or whats the best way to load
>>> images during data migration?
>>>
>>> The images are available in the file system and via http, several
>>> thousand of them. Thanks in advance for any help or advice!
>>>
>>> Thanks,
>>>
>>>        Jesse
>>>
>>>
>>>
>>
>

[Gil Portenseigne]

Hello Jesse,

You could manually manage your transactions within your service using :

actualTransaction = TransactionUtil.suspend(); //suspend the current
transaction
TransactionUtil.begin(600); //Start a new one
[...]
if (ServiceUtil.isError(result)) {
     TransactionUtil.rollback(); //Rollback the new one
} else {
     TransactionUtil.commit(); // Or Commit it
}
TransactionUtil.resume(actualTransaction); //resume the suspended
transaction.

With this solution you keep your process sync, and you can manage errors.

HTH

Gil


On 23/02/2018 09:29, Jesse Thomas wrote:

> I have written a service that checks a folder for image files, resizes
> and updates the Product in OFBiz based on the file name equaling the
> productId. It works for a few hundred images, but when I process a few
> thousand images at the end it hits a transaction timeout. All the
> images are resized and saved. But the product records don't get
> updated. All I'm doing in the database is updating the Product image
> fields, is there a way to force the commit on each product instead at
> the end of the service call?
>
> Thanks,
>     Jesse
>
> On 1/26/2018 12:48 AM, Jacques Le Roux wrote:
>> Thanks Jinghai,
>>
>> That's an interesting information!
>>
>> Jacques
>>
>>
>> Le 26/01/2018 à 02:57, Shi Jinghai a écrit :
>>> Thanks Mike, I agree with you if such image processing happens
>>> regularly, perl is a good choice but system may be complicated.
>>>
>>> The image processing speed of JDK 7 is dramatically slower than JDK
>>> 6, and sometimes wrong, as Kodak and others withdrew their image
>>> algorithm properties when Java transferred from Sun to Oracle.
>>>
>>> Java will struggle on ImageIO for a while, hope JDK 9 have a big
>>> improvement. [1]
>>>
>>> Kind Regards,
>>>
>>> Shi Jinghai
>>>
>>>
>>> [1] https://bugs.openjdk.java.net/browse/JDK-8041125
>>>
>>>
>>>
>>> 发件人: Mike [mailto:[hidden email]]
>>> 发送时间: 2018年1月26日 0:26
>>> 收件人: user
>>> 抄送: Shi Jinghai
>>> 主题: Re: data import - product images
>>>
>>> A well written perl or shell script can take "original.jpg" in
>>> directory "x" and create, multiple larger/smaller images with the
>>> same "root" name. You can use image magic, an image processor
>>> package, to do this.  Basic stuff for anyone who knows shell scripting.
>>>
>>> You can then import these images using a single XML data_reader
>>> file, and import it into OFB.  Example:
>>>
>>> <?xml version="1.0" encoding="UTF-8"?>
>>> <entity-engine-xml>
>>> <Product
>>>    productId="798936836182"
>>> smallImageUrl="/images/products/10000/1001MINIGOLF/small.jpg"
>>> mediumImageUrl="/images/products/10000/1001MINIGOLF/medium.jpg"
>>> largeImageUrl="/images/products/10000/1001MINIGOLF/large.jpg"
>>> originalImageUrl="/images/products/10000/1001MINIGOLF/original.jpg"
>>> detailImageUrl="/images/products/10000/1001MINIGOLF/detail.jpg"
>>> />
>>>
>>> [plus 1000s more].....
>>>
>>> </entity-engine-xml>
>>>
>>> A single script can generate the above XML file (with 1000s of
>>> products), resize 1000s of "original.jpg" into multiple images, and
>>> import into OFB, all in one pass.  Standard stuff for anyone
>>> competent with perl/shell scripting.
>>>
>>> What you don't want is make java do busy, CPU intensive work and
>>> have it render different sizes "on the fly", if that is what you are
>>> suggesting.
>>>
>>> On Wed, Jan 24, 2018 at 11:56 PM, Jesse Thomas
>>> <[hidden email]<mailto:[hidden email]>> wrote:
>>> Thanks Shi and Rishi for the responses.
>>>
>>> I was hoping to do this the hard way and l learn some new tricks in
>>> the process. More specifically I was hoping to use a ftl template
>>> (like in ./specialpurpose/ecommerce/data/DemoTree.xml) to call a
>>> service or function directly. Making the solution 100% OFB.
>>>
>>> If you know of any examples or clues please let me know.
>>>
>>> Thanks again!
>>>
>>>
>>>
>>> On 1/24/2018 11:27 PM, Shi Jinghai wrote:
>>> On generating the other sizes needed, it's already in the source
>>> code, see
>>> https://github.com/apache/ofbiz-framework/blob/trunk/applications/product/src/main/java/org/apache/ofbiz/product/image/ScaleImage.java 
>>>
>>>
>>> -----邮件原件-----
>>> 发件人: Jesse Thomas
>>> [mailto:[hidden email]<mailto:[hidden email]>]
>>> 发送时间: 2018年1月24日 15:30
>>> 收件人: [hidden email]<mailto:[hidden email]>
>>> 主题: data import - product images
>>>
>>> Is there is a way of using a data reader to load a product image and
>>> have it treated as though its uploaded as an "Original Image"
>>> (generating the other sizes needed)? Or whats the best way to load
>>> images during data migration?
>>>
>>> The images are available in the file system and via http, several
>>> thousand of them. Thanks in advance for any help or advice!
>>>
>>> Thanks,
>>>
>>>        Jesse
>>>
>>>
>>
>

[Rishi Solanki]

Jesse,
Assuming that you have a wrapper service which call resize images and data
store service for each product. Lets say wrapperService() is your main
service and inlineProductUpdateSerive() is the service which invoke in each
iteration in wrapperService(). The inlineProductUpdateSerive() cakk in the
wrapperService should look like this;

dispatcher.runSync("inlineProductUpdateSerive", 300, true, serviceInCtx);

OR

dispatcher.runAsync("inlineProductUpdateSerive", true, serviceInCtx);

In first option above, if service will return error then simply log it so
that service will continue run for other products. In second case true
value passed for persist that means it will re try 3 times on failure.

Bottom line is no need to handle the transaction manually OFBiz service
engine facilitate you to do that by just passing some flags. Finally
increase the transaction time in the service definition of wrapperService()
and call the inlineProductUpdateSerive() as mentioned above.

HTH!



--

Rishi Solanki
Sr Manager, Enterprise Software Development
HotWax Systems Pvt. Ltd.
Direct: [phone]
http://www.hotwaxsystems.com
www.hotwax.co

On Fri, Feb 23, 2018 at 2:25 PM, gil portenseigne <
[hidden email]> wrote:

> Hello Jesse,
>
> You could manually manage your transactions within your service using :
>
> actualTransaction = TransactionUtil.suspend(); //suspend the current
> transaction
> TransactionUtil.begin(600); //Start a new one
> [...]
> if (ServiceUtil.isError(result)) {
>     TransactionUtil.rollback(); //Rollback the new one
> } else {
>     TransactionUtil.commit(); // Or Commit it
> }
> TransactionUtil.resume(actualTransaction); //resume the suspended
> transaction.
>
> With this solution you keep your process sync, and you can manage errors.
>
> HTH
>
> Gil
>
>
>
> On 23/02/2018 09:29, Jesse Thomas wrote:
>
>> I have written a service that checks a folder for image files, resizes
>> and updates the Product in OFBiz based on the file name equaling the
>> productId. It works for a few hundred images, but when I process a few
>> thousand images at the end it hits a transaction timeout. All the images
>> are resized and saved. But the product records don't get updated. All I'm
>> doing in the database is updating the Product image fields, is there a way
>> to force the commit on each product instead at the end of the service call?
>>
>> Thanks,
>>     Jesse
>>
>> On 1/26/2018 12:48 AM, Jacques Le Roux wrote:
>>
>>> Thanks Jinghai,
>>>
>>> That's an interesting information!
>>>
>>> Jacques
>>>
>>>
>>> Le 26/01/2018 à 02:57, Shi Jinghai a écrit :
>>>
>>>> Thanks Mike, I agree with you if such image processing happens
>>>> regularly, perl is a good choice but system may be complicated.
>>>>
>>>> The image processing speed of JDK 7 is dramatically slower than JDK 6,
>>>> and sometimes wrong, as Kodak and others withdrew their image algorithm
>>>> properties when Java transferred from Sun to Oracle.
>>>>
>>>> Java will struggle on ImageIO for a while, hope JDK 9 have a big
>>>> improvement. [1]
>>>>
>>>> Kind Regards,
>>>>
>>>> Shi Jinghai
>>>>
>>>>
>>>> [1] https://bugs.openjdk.java.net/browse/JDK-8041125
>>>>
>>>>
>>>>
>>>> 发件人: Mike [mailto:[hidden email]]
>>>> 发送时间: 2018年1月26日 0:26
>>>> 收件人: user
>>>> 抄送: Shi Jinghai
>>>> 主题: Re: data import - product images
>>>>
>>>> A well written perl or shell script can take "original.jpg" in
>>>> directory "x" and create, multiple larger/smaller images with the same
>>>> "root" name. You can use image magic, an image processor package, to do
>>>> this.  Basic stuff for anyone who knows shell scripting.
>>>>
>>>> You can then import these images using a single XML data_reader file,
>>>> and import it into OFB.  Example:
>>>>
>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>> <entity-engine-xml>
>>>> <Product
>>>>    productId="798936836182"
>>>> smallImageUrl="/images/products/10000/1001MINIGOLF/small.jpg"
>>>> mediumImageUrl="/images/products/10000/1001MINIGOLF/medium.jpg"
>>>> largeImageUrl="/images/products/10000/1001MINIGOLF/large.jpg"
>>>> originalImageUrl="/images/products/10000/1001MINIGOLF/original.jpg"
>>>> detailImageUrl="/images/products/10000/1001MINIGOLF/detail.jpg"
>>>> />
>>>>
>>>> [plus 1000s more].....
>>>>
>>>> </entity-engine-xml>
>>>>
>>>> A single script can generate the above XML file (with 1000s of
>>>> products), resize 1000s of "original.jpg" into multiple images, and import
>>>> into OFB, all in one pass.  Standard stuff for anyone competent with
>>>> perl/shell scripting.
>>>>
>>>> What you don't want is make java do busy, CPU intensive work and have
>>>> it render different sizes "on the fly", if that is what you are suggesting.
>>>>
>>>> On Wed, Jan 24, 2018 at 11:56 PM, Jesse Thomas <
>>>> [hidden email]<mailto:[hidden email]>> wrote:
>>>> Thanks Shi and Rishi for the responses.
>>>>
>>>> I was hoping to do this the hard way and l learn some new tricks in the
>>>> process. More specifically I was hoping to use a ftl template (like in
>>>> ./specialpurpose/ecommerce/data/DemoTree.xml) to call a service or
>>>> function directly. Making the solution 100% OFB.
>>>>
>>>> If you know of any examples or clues please let me know.
>>>>
>>>> Thanks again!
>>>>
>>>>
>>>>
>>>> On 1/24/2018 11:27 PM, Shi Jinghai wrote:
>>>> On generating the other sizes needed, it's already in the source code,
>>>> see
>>>> https://github.com/apache/ofbiz-framework/blob/trunk/applica
>>>> tions/product/src/main/java/org/apache/ofbiz/product/image/
>>>> ScaleImage.java
>>>>
>>>> -----邮件原件-----
>>>> 发件人: Jesse Thomas [mailto:[hidden email]<mailto:
>>>> [hidden email]>]
>>>> 发送时间: 2018年1月24日 15:30
>>>> 收件人: [hidden email]<mailto:[hidden email]>
>>>> 主题: data import - product images
>>>>
>>>> Is there is a way of using a data reader to load a product image and
>>>> have it treated as though its uploaded as an "Original Image"
>>>> (generating the other sizes needed)? Or whats the best way to load
>>>> images during data migration?
>>>>
>>>> The images are available in the file system and via http, several
>>>> thousand of them. Thanks in advance for any help or advice!
>>>>
>>>> Thanks,
>>>>
>>>>        Jesse
>>>>
>>>>
>>>>
>>>
>>
>

[Jesse Thomas]

Gil, Thanks for the tip, managing transactions like that was exactly
what I was looking for and has gotten me farther down the road. Now all
the product changes are updated/committed as they are processed and not
rolled back if there is an error at the end of the service.

Rishi, I'll try the method you suggest too. I have all my code in one
service/function right now and it would be good to break it up.

I'm still getting an error and series of warnings, sometimes. If I
schedule the service to run (/webtools/control/scheduleJob) I get no
errors. If I run it from /webtools/control/runService I get this series
of errors and warnings...

|TransactionUtil               |W| In getSetRollbackOnlyCause no stack
placeholder was in place, here is the current location: (stack trace...)
|JavaEventHandler              |E| null
org.apache.ofbiz.entity.transaction.GenericTransactionException: Roll
back error (with no rollbackOnly cause found), could not commit
transaction, was rolled back instead: java.lang.Exception: Transaction
has timed out (Transaction has timed out)(stack trace...)
|TransactionUtil               |W| In setTransactionBeginStack a stack
placeholder was already in place, here is where the transaction began:
(stack trace...)
|TransactionUtil               |W| In setTransactionBeginStack a stack
placeholder was already in place, here is the current location: (stack
trace...)

I'm going to clean up my code and try a few more things before asking
for more help. This has been a good learning experience and I have all
my images loaded. Thanks everyone for all your help!

On 2/23/2018 10:43 PM, Rishi Solanki wrote:

> Jesse,
> Assuming that you have a wrapper service which call resize images and data
> store service for each product. Lets say wrapperService() is your main
> service and inlineProductUpdateSerive() is the service which invoke in each
> iteration in wrapperService(). The inlineProductUpdateSerive() cakk in the
> wrapperService should look like this;
>
> dispatcher.runSync("inlineProductUpdateSerive", 300, true, serviceInCtx);
>
> OR
>
> dispatcher.runAsync("inlineProductUpdateSerive", true, serviceInCtx);
>
> In first option above, if service will return error then simply log it so
> that service will continue run for other products. In second case true
> value passed for persist that means it will re try 3 times on failure.
>
> Bottom line is no need to handle the transaction manually OFBiz service
> engine facilitate you to do that by just passing some flags. Finally
> increase the transaction time in the service definition of wrapperService()
> and call the inlineProductUpdateSerive() as mentioned above.
>
> HTH!
>
>
>
> --
>
> Rishi Solanki
> Sr Manager, Enterprise Software Development
> HotWax Systems Pvt. Ltd.
> Direct: [phone]
> http://www.hotwaxsystems.com
> www.hotwax.co
>
> On Fri, Feb 23, 2018 at 2:25 PM, gil portenseigne <
> [hidden email]> wrote:
>
>> Hello Jesse,
>>
>> You could manually manage your transactions within your service using :
>>
>> actualTransaction = TransactionUtil.suspend(); //suspend the current
>> transaction
>> TransactionUtil.begin(600); //Start a new one
>> [...]
>> if (ServiceUtil.isError(result)) {
>>      TransactionUtil.rollback(); //Rollback the new one
>> } else {
>>      TransactionUtil.commit(); // Or Commit it
>> }
>> TransactionUtil.resume(actualTransaction); //resume the suspended
>> transaction.
>>
>> With this solution you keep your process sync, and you can manage errors.
>>
>> HTH
>>
>> Gil
>>
>>
>>
>> On 23/02/2018 09:29, Jesse Thomas wrote:
>>
>>> I have written a service that checks a folder for image files, resizes
>>> and updates the Product in OFBiz based on the file name equaling the
>>> productId. It works for a few hundred images, but when I process a few
>>> thousand images at the end it hits a transaction timeout. All the images
>>> are resized and saved. But the product records don't get updated. All I'm
>>> doing in the database is updating the Product image fields, is there a way
>>> to force the commit on each product instead at the end of the service call?
>>>
>>> Thanks,
>>>      Jesse
>>>
>>> On 1/26/2018 12:48 AM, Jacques Le Roux wrote:
>>>
>>>> Thanks Jinghai,
>>>>
>>>> That's an interesting information!
>>>>
>>>> Jacques
>>>>
>>>>
>>>> Le 26/01/2018 à 02:57, Shi Jinghai a écrit :
>>>>
>>>>> Thanks Mike, I agree with you if such image processing happens
>>>>> regularly, perl is a good choice but system may be complicated.
>>>>>
>>>>> The image processing speed of JDK 7 is dramatically slower than JDK 6,
>>>>> and sometimes wrong, as Kodak and others withdrew their image algorithm
>>>>> properties when Java transferred from Sun to Oracle.
>>>>>
>>>>> Java will struggle on ImageIO for a while, hope JDK 9 have a big
>>>>> improvement. [1]
>>>>>
>>>>> Kind Regards,
>>>>>
>>>>> Shi Jinghai
>>>>>
>>>>>
>>>>> [1] https://bugs.openjdk.java.net/browse/JDK-8041125
>>>>>
>>>>>
>>>>>
>>>>> 发件人: Mike [mailto:[hidden email]]
>>>>> 发送时间: 2018年1月26日 0:26
>>>>> 收件人: user
>>>>> 抄送: Shi Jinghai
>>>>> 主题: Re: data import - product images
>>>>>
>>>>> A well written perl or shell script can take "original.jpg" in
>>>>> directory "x" and create, multiple larger/smaller images with the same
>>>>> "root" name. You can use image magic, an image processor package, to do
>>>>> this.  Basic stuff for anyone who knows shell scripting.
>>>>>
>>>>> You can then import these images using a single XML data_reader file,
>>>>> and import it into OFB.  Example:
>>>>>
>>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>>> <entity-engine-xml>
>>>>> <Product
>>>>>     productId="798936836182"
>>>>> smallImageUrl="/images/products/10000/1001MINIGOLF/small.jpg"
>>>>> mediumImageUrl="/images/products/10000/1001MINIGOLF/medium.jpg"
>>>>> largeImageUrl="/images/products/10000/1001MINIGOLF/large.jpg"
>>>>> originalImageUrl="/images/products/10000/1001MINIGOLF/original.jpg"
>>>>> detailImageUrl="/images/products/10000/1001MINIGOLF/detail.jpg"
>>>>> />
>>>>>
>>>>> [plus 1000s more].....
>>>>>
>>>>> </entity-engine-xml>
>>>>>
>>>>> A single script can generate the above XML file (with 1000s of
>>>>> products), resize 1000s of "original.jpg" into multiple images, and import
>>>>> into OFB, all in one pass.  Standard stuff for anyone competent with
>>>>> perl/shell scripting.
>>>>>
>>>>> What you don't want is make java do busy, CPU intensive work and have
>>>>> it render different sizes "on the fly", if that is what you are suggesting.
>>>>>
>>>>> On Wed, Jan 24, 2018 at 11:56 PM, Jesse Thomas <
>>>>> [hidden email]<mailto:[hidden email]>> wrote:
>>>>> Thanks Shi and Rishi for the responses.
>>>>>
>>>>> I was hoping to do this the hard way and l learn some new tricks in the
>>>>> process. More specifically I was hoping to use a ftl template (like in
>>>>> ./specialpurpose/ecommerce/data/DemoTree.xml) to call a service or
>>>>> function directly. Making the solution 100% OFB.
>>>>>
>>>>> If you know of any examples or clues please let me know.
>>>>>
>>>>> Thanks again!
>>>>>
>>>>>
>>>>>
>>>>> On 1/24/2018 11:27 PM, Shi Jinghai wrote:
>>>>> On generating the other sizes needed, it's already in the source code,
>>>>> see
>>>>> https://github.com/apache/ofbiz-framework/blob/trunk/applica
>>>>> tions/product/src/main/java/org/apache/ofbiz/product/image/
>>>>> ScaleImage.java
>>>>>
>>>>> -----邮件原件-----
>>>>> 发件人: Jesse Thomas [mailto:[hidden email]<mailto:
>>>>> [hidden email]>]
>>>>> 发送时间: 2018年1月24日 15:30
>>>>> 收件人: [hidden email]<mailto:[hidden email]>
>>>>> 主题: data import - product images
>>>>>
>>>>> Is there is a way of using a data reader to load a product image and
>>>>> have it treated as though its uploaded as an "Original Image"
>>>>> (generating the other sizes needed)? Or whats the best way to load
>>>>> images during data migration?
>>>>>
>>>>> The images are available in the file system and via http, several
>>>>> thousand of them. Thanks in advance for any help or advice!
>>>>>
>>>>> Thanks,
>>>>>
>>>>>         Jesse
>>>>>
>>>>>
>>>>>