What is a ETL tool in OFBiz?
Locked
What is a ETL tool in OFBiz?
 
|
Hi community,
Now I'm working on OFBiz and need to integrate it to several systems. I have to get data from several sources to store in OFBiz data model. I need to use ETL tool for help this process but I don't know what is a ETL tool that OFBiz use. Can anyone tell me what is a ETL tool in OFBiz? Regards, Chatree Srichart |
|
|
it depends on the data form you using.
for files https://cwiki.apache.org/confluence/display/OFBIZ/Handling+of+External+data if you have a db you can connect to ofbiz then you can create the entities and write minilang scripts to pull from the old one, map then store in ofbiz. https://demo-trunk.ofbiz.apache.org/webtools/control/view/ModelInduceFromDb your need to define a datasource in the entityengine.xml for your db, to use this tool. Chatree Srichart sent the following on 7/12/2010 12:19 PM: > Hi community, > > Now I'm working on OFBiz and need to integrate it to several systems. I have > to get data from several sources to store in OFBiz data model. > I need to use ETL tool for help this process but I don't know what is a ETL > tool that OFBiz use. > > Can anyone tell me what is a ETL tool in OFBiz? > > Regards, > Chatree Srichart > |
Re: What is a ETL tool in OFBiz?
|
|
Hi, BJ Freeman
Thanks for your quick answer. Regards, Chatree Srichart On Tue, Jul 13, 2010 at 2:53 AM, BJ Freeman <[hidden email]> wrote: > it depends on the data form you using. > for files > https://cwiki.apache.org/confluence/display/OFBIZ/Handling+of+External+data > > if you have a db you can connect to ofbiz then you can > create the entities and write minilang scripts to pull from the old one, > map then store in ofbiz. > https://demo-trunk.ofbiz.apache.org/webtools/control/view/ModelInduceFromDb > your need to define a datasource in the entityengine.xml for your db, to > use this tool. > > > > > Chatree Srichart sent the following on 7/12/2010 12:19 PM: > > Hi community, >> >> Now I'm working on OFBiz and need to integrate it to several systems. I >> have >> to get data from several sources to store in OFBiz data model. >> I need to use ETL tool for help this process but I don't know what is a >> ETL >> tool that OFBiz use. >> >> Can anyone tell me what is a ETL tool in OFBiz? >> >> Regards, >> Chatree Srichart >> >> |
|
|
your welcome
in addition if you got the DB route you can write scripts to map ofbiz to the other db and use SECAS. so if say a createcustomers service is run you can have your SECA call your script and load the other DB. from you legacy db you can have scheduled services that pull from it to ofbiz. Chatree Srichart sent the following on 7/12/2010 1:05 PM: > Hi, BJ Freeman > Thanks for your quick answer. > > Regards, > Chatree Srichart > > On Tue, Jul 13, 2010 at 2:53 AM, BJ Freeman<[hidden email]> wrote: > >> it depends on the data form you using. >> for files >> https://cwiki.apache.org/confluence/display/OFBIZ/Handling+of+External+data >> >> if you have a db you can connect to ofbiz then you can >> create the entities and write minilang scripts to pull from the old one, >> map then store in ofbiz. >> https://demo-trunk.ofbiz.apache.org/webtools/control/view/ModelInduceFromDb >> your need to define a datasource in the entityengine.xml for your db, to >> use this tool. >> >> >> >> >> Chatree Srichart sent the following on 7/12/2010 12:19 PM: >> >> Hi community, >>> >>> Now I'm working on OFBiz and need to integrate it to several systems. I >>> have >>> to get data from several sources to store in OFBiz data model. >>> I need to use ETL tool for help this process but I don't know what is a >>> ETL >>> tool that OFBiz use. >>> >>> Can anyone tell me what is a ETL tool in OFBiz? >>> >>> Regards, >>> Chatree Srichart >>> >>> > |
Re: What is a ETL tool in OFBiz?
|
|
For reference:
After evaluating few ETL tools - Pentaho Kettle, Jitterbit, Talend, we found out that the most easier way to integrate data in Ofbiz is using Talend. The process: - Connect Talend to the external DB - For each row, prepare the data (doing transformations, joins with other tables, call web services to get ISO codes) - Call an ofbiz service to create the data in Ofbiz DB (like create product) Ofbiz services can be called as web service (real pain with the way parameters are encoded in maps) or through XML-RPC The process is really easy, since Talend can connect to many data sources, and importing the data using Ofbiz services ensure that the data is properly created. Bilgin On Mon, Jul 12, 2010 at 10:07 PM, BJ Freeman <[hidden email]> wrote: > your welcome > in addition if you got the DB route you can write scripts to map ofbiz to > the other db and use SECAS. > so if say a createcustomers service is run you can have your SECA call your > script and load the other DB. > from you legacy db you can have scheduled services that pull from it to > ofbiz. > > Chatree Srichart sent the following on 7/12/2010 1:05 PM: > > Hi, BJ Freeman >> Thanks for your quick answer. >> >> Regards, >> Chatree Srichart >> >> On Tue, Jul 13, 2010 at 2:53 AM, BJ Freeman<[hidden email]> wrote: >> >> it depends on the data form you using. >>> for files >>> >>> https://cwiki.apache.org/confluence/display/OFBIZ/Handling+of+External+data >>> >>> if you have a db you can connect to ofbiz then you can >>> create the entities and write minilang scripts to pull from the old one, >>> map then store in ofbiz. >>> >>> https://demo-trunk.ofbiz.apache.org/webtools/control/view/ModelInduceFromDb >>> your need to define a datasource in the entityengine.xml for your db, to >>> use this tool. >>> >>> >>> >>> >>> Chatree Srichart sent the following on 7/12/2010 12:19 PM: >>> >>> Hi community, >>> >>>> >>>> Now I'm working on OFBiz and need to integrate it to several systems. I >>>> have >>>> to get data from several sources to store in OFBiz data model. >>>> I need to use ETL tool for help this process but I don't know what is a >>>> ETL >>>> tool that OFBiz use. >>>> >>>> Can anyone tell me what is a ETL tool in OFBiz? >>>> >>>> Regards, >>>> Chatree Srichart >>>> >>>> >>>> >> |
Re: What is a ETL tool in OFBiz?
Administrator
|
Thanks BJ, Bilgin,
I will put it in FAQ, except if someone takes the time to dot it... Jacques From: "Bilgin Ibryam" <[hidden email]> > For reference: > > After evaluating few ETL tools - Pentaho Kettle, Jitterbit, Talend, we found > out that the most easier way to integrate data in Ofbiz is using Talend. > > The process: > > - Connect Talend to the external DB > - For each row, prepare the data (doing transformations, joins with other > tables, call web services to get ISO codes) > - Call an ofbiz service to create the data in Ofbiz DB (like create product) > > Ofbiz services can be called as web service (real pain with the way > parameters are encoded in maps) or through XML-RPC > > The process is really easy, since Talend can connect to many data sources, > and importing the data using Ofbiz services ensure that the data is properly > created. > > Bilgin > > On Mon, Jul 12, 2010 at 10:07 PM, BJ Freeman <[hidden email]> wrote: > >> your welcome >> in addition if you got the DB route you can write scripts to map ofbiz to >> the other db and use SECAS. >> so if say a createcustomers service is run you can have your SECA call your >> script and load the other DB. >> from you legacy db you can have scheduled services that pull from it to >> ofbiz. >> >> Chatree Srichart sent the following on 7/12/2010 1:05 PM: >> >> Hi, BJ Freeman >>> Thanks for your quick answer. >>> >>> Regards, >>> Chatree Srichart >>> >>> On Tue, Jul 13, 2010 at 2:53 AM, BJ Freeman<[hidden email]> wrote: >>> >>> it depends on the data form you using. >>>> for files >>>> >>>> https://cwiki.apache.org/confluence/display/OFBIZ/Handling+of+External+data >>>> >>>> if you have a db you can connect to ofbiz then you can >>>> create the entities and write minilang scripts to pull from the old one, >>>> map then store in ofbiz. >>>> >>>> https://demo-trunk.ofbiz.apache.org/webtools/control/view/ModelInduceFromDb >>>> your need to define a datasource in the entityengine.xml for your db, to >>>> use this tool. >>>> >>>> >>>> >>>> >>>> Chatree Srichart sent the following on 7/12/2010 12:19 PM: >>>> >>>> Hi community, >>>> >>>>> >>>>> Now I'm working on OFBiz and need to integrate it to several systems. I >>>>> have >>>>> to get data from several sources to store in OFBiz data model. >>>>> I need to use ETL tool for help this process but I don't know what is a >>>>> ETL >>>>> tool that OFBiz use. >>>>> >>>>> Can anyone tell me what is a ETL tool in OFBiz? >>>>> >>>>> Regards, >>>>> Chatree Srichart >>>>> >>>>> >>>>> >>> > |
Re: What is a ETL tool in OFBiz?
|
Administrator
|
Hi Bilgin,
Before putting this into the FAQ, I was wondering why you choose to use Talend over OOTB OFBiz datafiles? I guess there are advantages, could you please name some? Thanks Jacques From: "Jacques Le Roux" <[hidden email]> > Thanks BJ, Bilgin, > > I will put it in FAQ, except if someone takes the time to dot it... > > Jacques > > From: "Bilgin Ibryam" <[hidden email]> >> For reference: >> >> After evaluating few ETL tools - Pentaho Kettle, Jitterbit, Talend, we found >> out that the most easier way to integrate data in Ofbiz is using Talend. >> >> The process: >> >> - Connect Talend to the external DB >> - For each row, prepare the data (doing transformations, joins with other >> tables, call web services to get ISO codes) >> - Call an ofbiz service to create the data in Ofbiz DB (like create product) >> >> Ofbiz services can be called as web service (real pain with the way >> parameters are encoded in maps) or through XML-RPC >> >> The process is really easy, since Talend can connect to many data sources, >> and importing the data using Ofbiz services ensure that the data is properly >> created. >> >> Bilgin >> >> On Mon, Jul 12, 2010 at 10:07 PM, BJ Freeman <[hidden email]> wrote: >> >>> your welcome >>> in addition if you got the DB route you can write scripts to map ofbiz to >>> the other db and use SECAS. >>> so if say a createcustomers service is run you can have your SECA call your >>> script and load the other DB. >>> from you legacy db you can have scheduled services that pull from it to >>> ofbiz. >>> >>> Chatree Srichart sent the following on 7/12/2010 1:05 PM: >>> >>> Hi, BJ Freeman >>>> Thanks for your quick answer. >>>> >>>> Regards, >>>> Chatree Srichart >>>> >>>> On Tue, Jul 13, 2010 at 2:53 AM, BJ Freeman<[hidden email]> wrote: >>>> >>>> it depends on the data form you using. >>>>> for files >>>>> >>>>> https://cwiki.apache.org/confluence/display/OFBIZ/Handling+of+External+data >>>>> >>>>> if you have a db you can connect to ofbiz then you can >>>>> create the entities and write minilang scripts to pull from the old one, >>>>> map then store in ofbiz. >>>>> >>>>> https://demo-trunk.ofbiz.apache.org/webtools/control/view/ModelInduceFromDb >>>>> your need to define a datasource in the entityengine.xml for your db, to >>>>> use this tool. >>>>> >>>>> >>>>> >>>>> >>>>> Chatree Srichart sent the following on 7/12/2010 12:19 PM: >>>>> >>>>> Hi community, >>>>> >>>>>> >>>>>> Now I'm working on OFBiz and need to integrate it to several systems. I >>>>>> have >>>>>> to get data from several sources to store in OFBiz data model. >>>>>> I need to use ETL tool for help this process but I don't know what is a >>>>>> ETL >>>>>> tool that OFBiz use. >>>>>> >>>>>> Can anyone tell me what is a ETL tool in OFBiz? >>>>>> >>>>>> Regards, >>>>>> Chatree Srichart >>>>>> >>>>>> >>>>>> >>>> >> > |
Re: What is a ETL tool in OFBiz?
|
|
Sure. The main advance of using Talend was that it is able to extract data
from many datasources and then invoke ofbiz service thru xml-rpc (or soap). Importing with OFBiz datafiles (or with xml files) requires handling of all the low-level tasks (primary keys, relatations, types, timestamps) to be done manually. Consider that you want to import the postal address for a person. You have to create 5-6 entities and manage the relations, types correctly... On the other hand, with Talend (or similar tool) you have to extract the needed data from legacy data source and call createPartyPostalAddress service. No need to worry if you missed to populate some of the entities. Bilgin On Sat, Sep 25, 2010 at 1:14 PM, Jacques Le Roux < [hidden email]> wrote: > Hi Bilgin, > > Before putting this into the FAQ, I was wondering why you choose to use > Talend over OOTB OFBiz datafiles? > I guess there are advantages, could you please name some? > > Thanks > > Jacques > > From: "Jacques Le Roux" <[hidden email]> > > Thanks BJ, Bilgin, >> >> I will put it in FAQ, except if someone takes the time to dot it... >> >> Jacques >> >> From: "Bilgin Ibryam" <[hidden email]> >> >>> For reference: >>> >>> After evaluating few ETL tools - Pentaho Kettle, Jitterbit, Talend, we >>> found >>> out that the most easier way to integrate data in Ofbiz is using Talend. >>> >>> The process: >>> >>> - Connect Talend to the external DB >>> - For each row, prepare the data (doing transformations, joins with other >>> tables, call web services to get ISO codes) >>> - Call an ofbiz service to create the data in Ofbiz DB (like create >>> product) >>> >>> Ofbiz services can be called as web service (real pain with the way >>> parameters are encoded in maps) or through XML-RPC >>> >>> The process is really easy, since Talend can connect to many data >>> sources, >>> and importing the data using Ofbiz services ensure that the data is >>> properly >>> created. >>> >>> Bilgin >>> >>> On Mon, Jul 12, 2010 at 10:07 PM, BJ Freeman <[hidden email]> >>> wrote: >>> >>> your welcome >>>> in addition if you got the DB route you can write scripts to map ofbiz >>>> to >>>> the other db and use SECAS. >>>> so if say a createcustomers service is run you can have your SECA call >>>> your >>>> script and load the other DB. >>>> from you legacy db you can have scheduled services that pull from it to >>>> ofbiz. >>>> >>>> Chatree Srichart sent the following on 7/12/2010 1:05 PM: >>>> >>>> Hi, BJ Freeman >>>> >>>>> Thanks for your quick answer. >>>>> >>>>> Regards, >>>>> Chatree Srichart >>>>> >>>>> On Tue, Jul 13, 2010 at 2:53 AM, BJ Freeman<[hidden email]> >>>>> wrote: >>>>> >>>>> it depends on the data form you using. >>>>> >>>>>> for files >>>>>> >>>>>> >>>>>> https://cwiki.apache.org/confluence/display/OFBIZ/Handling+of+External+data >>>>>> >>>>>> if you have a db you can connect to ofbiz then you can >>>>>> create the entities and write minilang scripts to pull from the old >>>>>> one, >>>>>> map then store in ofbiz. >>>>>> >>>>>> >>>>>> https://demo-trunk.ofbiz.apache.org/webtools/control/view/ModelInduceFromDb >>>>>> your need to define a datasource in the entityengine.xml for your db, >>>>>> to >>>>>> use this tool. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Chatree Srichart sent the following on 7/12/2010 12:19 PM: >>>>>> >>>>>> Hi community, >>>>>> >>>>>> >>>>>>> Now I'm working on OFBiz and need to integrate it to several systems. >>>>>>> I >>>>>>> have >>>>>>> to get data from several sources to store in OFBiz data model. >>>>>>> I need to use ETL tool for help this process but I don't know what is >>>>>>> a >>>>>>> ETL >>>>>>> tool that OFBiz use. >>>>>>> >>>>>>> Can anyone tell me what is a ETL tool in OFBiz? >>>>>>> >>>>>>> Regards, >>>>>>> Chatree Srichart >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>> >>> >> > |
Re: What is a ETL tool in OFBiz?
|
Administrator
|
Thanks Bilgin,
This will help to feed the FAQ. Notably the example helps much. Jacques From: "Bilgin Ibryam" <[hidden email]> > Sure. The main advance of using Talend was that it is able to extract data > from many datasources and then invoke ofbiz service thru xml-rpc (or soap). > > Importing with OFBiz datafiles (or with xml files) requires handling of all > the low-level tasks (primary keys, relatations, types, timestamps) to be > done manually. > Consider that you want to import the postal address for a person. You have > to create 5-6 entities and manage the relations, types correctly... On the > other hand, with Talend (or similar tool) you have to extract the needed > data from legacy data source and call createPartyPostalAddress service. No > need to worry if you missed to populate some of the entities. > > Bilgin > > On Sat, Sep 25, 2010 at 1:14 PM, Jacques Le Roux < > [hidden email]> wrote: > >> Hi Bilgin, >> >> Before putting this into the FAQ, I was wondering why you choose to use >> Talend over OOTB OFBiz datafiles? >> I guess there are advantages, could you please name some? >> >> Thanks >> >> Jacques >> >> From: "Jacques Le Roux" <[hidden email]> >> >> Thanks BJ, Bilgin, >>> >>> I will put it in FAQ, except if someone takes the time to dot it... >>> >>> Jacques >>> >>> From: "Bilgin Ibryam" <[hidden email]> >>> >>>> For reference: >>>> >>>> After evaluating few ETL tools - Pentaho Kettle, Jitterbit, Talend, we >>>> found >>>> out that the most easier way to integrate data in Ofbiz is using Talend. >>>> >>>> The process: >>>> >>>> - Connect Talend to the external DB >>>> - For each row, prepare the data (doing transformations, joins with other >>>> tables, call web services to get ISO codes) >>>> - Call an ofbiz service to create the data in Ofbiz DB (like create >>>> product) >>>> >>>> Ofbiz services can be called as web service (real pain with the way >>>> parameters are encoded in maps) or through XML-RPC >>>> >>>> The process is really easy, since Talend can connect to many data >>>> sources, >>>> and importing the data using Ofbiz services ensure that the data is >>>> properly >>>> created. >>>> >>>> Bilgin >>>> >>>> On Mon, Jul 12, 2010 at 10:07 PM, BJ Freeman <[hidden email]> >>>> wrote: >>>> >>>> your welcome >>>>> in addition if you got the DB route you can write scripts to map ofbiz >>>>> to >>>>> the other db and use SECAS. >>>>> so if say a createcustomers service is run you can have your SECA call >>>>> your >>>>> script and load the other DB. >>>>> from you legacy db you can have scheduled services that pull from it to >>>>> ofbiz. >>>>> >>>>> Chatree Srichart sent the following on 7/12/2010 1:05 PM: >>>>> >>>>> Hi, BJ Freeman >>>>> >>>>>> Thanks for your quick answer. >>>>>> >>>>>> Regards, >>>>>> Chatree Srichart >>>>>> >>>>>> On Tue, Jul 13, 2010 at 2:53 AM, BJ Freeman<[hidden email]> >>>>>> wrote: >>>>>> >>>>>> it depends on the data form you using. >>>>>> >>>>>>> for files >>>>>>> >>>>>>> >>>>>>> https://cwiki.apache.org/confluence/display/OFBIZ/Handling+of+External+data >>>>>>> >>>>>>> if you have a db you can connect to ofbiz then you can >>>>>>> create the entities and write minilang scripts to pull from the old >>>>>>> one, >>>>>>> map then store in ofbiz. >>>>>>> >>>>>>> >>>>>>> https://demo-trunk.ofbiz.apache.org/webtools/control/view/ModelInduceFromDb >>>>>>> your need to define a datasource in the entityengine.xml for your db, >>>>>>> to >>>>>>> use this tool. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Chatree Srichart sent the following on 7/12/2010 12:19 PM: >>>>>>> >>>>>>> Hi community, >>>>>>> >>>>>>> >>>>>>>> Now I'm working on OFBiz and need to integrate it to several systems. >>>>>>>> I >>>>>>>> have >>>>>>>> to get data from several sources to store in OFBiz data model. >>>>>>>> I need to use ETL tool for help this process but I don't know what is >>>>>>>> a >>>>>>>> ETL >>>>>>>> tool that OFBiz use. >>>>>>>> >>>>>>>> Can anyone tell me what is a ETL tool in OFBiz? >>>>>>>> >>>>>>>> Regards, >>>>>>>> Chatree Srichart >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>> >>>> >>> >> > |
|
|
In reply to this post by Bilgin Ibryam-2
I would think if you have two database that do not match that you will
have to do the process at one end or the other of matching the data.\ as well as making sure you have all the requirements of the service your calling. where if you connect to the db through ofbiz and setup the entities, you marshal the data from the remote db and call the service the same way. but you cut out any other layers. Bilgin Ibryam sent the following on 9/25/2010 11:54 AM: > Sure. The main advance of using Talend was that it is able to extract data > from many datasources and then invoke ofbiz service thru xml-rpc (or soap). > > Importing with OFBiz datafiles (or with xml files) requires handling of all > the low-level tasks (primary keys, relatations, types, timestamps) to be > done manually. > Consider that you want to import the postal address for a person. You have > to create 5-6 entities and manage the relations, types correctly... On the > other hand, with Talend (or similar tool) you have to extract the needed > data from legacy data source and call createPartyPostalAddress service. No > need to worry if you missed to populate some of the entities. > > Bilgin > > On Sat, Sep 25, 2010 at 1:14 PM, Jacques Le Roux< > [hidden email]> wrote: > >> Hi Bilgin, >> >> Before putting this into the FAQ, I was wondering why you choose to use >> Talend over OOTB OFBiz datafiles? >> I guess there are advantages, could you please name some? >> >> Thanks >> >> Jacques >> >> From: "Jacques Le Roux"<[hidden email]> >> >> Thanks BJ, Bilgin, >>> >>> I will put it in FAQ, except if someone takes the time to dot it... >>> >>> Jacques >>> >>> From: "Bilgin Ibryam"<[hidden email]> >>> >>>> For reference: >>>> >>>> After evaluating few ETL tools - Pentaho Kettle, Jitterbit, Talend, we >>>> found >>>> out that the most easier way to integrate data in Ofbiz is using Talend. >>>> >>>> The process: >>>> >>>> - Connect Talend to the external DB >>>> - For each row, prepare the data (doing transformations, joins with other >>>> tables, call web services to get ISO codes) >>>> - Call an ofbiz service to create the data in Ofbiz DB (like create >>>> product) >>>> >>>> Ofbiz services can be called as web service (real pain with the way >>>> parameters are encoded in maps) or through XML-RPC >>>> >>>> The process is really easy, since Talend can connect to many data >>>> sources, >>>> and importing the data using Ofbiz services ensure that the data is >>>> properly >>>> created. >>>> >>>> Bilgin >>>> >>>> On Mon, Jul 12, 2010 at 10:07 PM, BJ Freeman<[hidden email]> >>>> wrote: >>>> >>>> your welcome >>>>> in addition if you got the DB route you can write scripts to map ofbiz >>>>> to >>>>> the other db and use SECAS. >>>>> so if say a createcustomers service is run you can have your SECA call >>>>> your >>>>> script and load the other DB. >>>>> from you legacy db you can have scheduled services that pull from it to >>>>> ofbiz. >>>>> >>>>> Chatree Srichart sent the following on 7/12/2010 1:05 PM: >>>>> >>>>> Hi, BJ Freeman >>>>> >>>>>> Thanks for your quick answer. >>>>>> >>>>>> Regards, >>>>>> Chatree Srichart >>>>>> >>>>>> On Tue, Jul 13, 2010 at 2:53 AM, BJ Freeman<[hidden email]> >>>>>> wrote: >>>>>> >>>>>> it depends on the data form you using. >>>>>> >>>>>>> for files >>>>>>> >>>>>>> >>>>>>> https://cwiki.apache.org/confluence/display/OFBIZ/Handling+of+External+data >>>>>>> >>>>>>> if you have a db you can connect to ofbiz then you can >>>>>>> create the entities and write minilang scripts to pull from the old >>>>>>> one, >>>>>>> map then store in ofbiz. >>>>>>> >>>>>>> >>>>>>> https://demo-trunk.ofbiz.apache.org/webtools/control/view/ModelInduceFromDb >>>>>>> your need to define a datasource in the entityengine.xml for your db, >>>>>>> to >>>>>>> use this tool. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Chatree Srichart sent the following on 7/12/2010 12:19 PM: >>>>>>> >>>>>>> Hi community, >>>>>>> >>>>>>> >>>>>>>> Now I'm working on OFBiz and need to integrate it to several systems. >>>>>>>> I >>>>>>>> have >>>>>>>> to get data from several sources to store in OFBiz data model. >>>>>>>> I need to use ETL tool for help this process but I don't know what is >>>>>>>> a >>>>>>>> ETL >>>>>>>> tool that OFBiz use. >>>>>>>> >>>>>>>> Can anyone tell me what is a ETL tool in OFBiz? >>>>>>>> >>>>>>>> Regards, >>>>>>>> Chatree Srichart >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>> >>>> >>> >> > |
Re: What is a ETL tool in OFBiz?
|
Administrator
|
BJ, Bilgin,
I understand that using the OFBiz datafiles tools is good when you don't have a legacy or external DB at hand (ie you have only flat files or such) I remember you already spoke about this method when relationnal DB(s) is available, and it makes sense indeed (simpler you only have to specify the Entity). What would be the drawbacks compared to what Bilgin suggest (fields types, is Talend able to handle triggers or embedded functions effects?, etc.) Thanks Jacques From: "BJ Freeman" <[hidden email]> >I would think if you have two database that do not match that you will have to do the process at one end or the other of matching >the data.\ > as well as making sure you have all the requirements of the service your calling. > where if you connect to the db through ofbiz and setup the entities, > you marshal the data from the remote db and call the service the same way. > but you cut out any other layers. > > Bilgin Ibryam sent the following on 9/25/2010 11:54 AM: >> Sure. The main advance of using Talend was that it is able to extract data >> from many datasources and then invoke ofbiz service thru xml-rpc (or soap). >> >> Importing with OFBiz datafiles (or with xml files) requires handling of all >> the low-level tasks (primary keys, relatations, types, timestamps) to be >> done manually. >> Consider that you want to import the postal address for a person. You have >> to create 5-6 entities and manage the relations, types correctly... On the >> other hand, with Talend (or similar tool) you have to extract the needed >> data from legacy data source and call createPartyPostalAddress service. No >> need to worry if you missed to populate some of the entities. >> >> Bilgin >> >> On Sat, Sep 25, 2010 at 1:14 PM, Jacques Le Roux< >> [hidden email]> wrote: >> >>> Hi Bilgin, >>> >>> Before putting this into the FAQ, I was wondering why you choose to use >>> Talend over OOTB OFBiz datafiles? >>> I guess there are advantages, could you please name some? >>> >>> Thanks >>> >>> Jacques >>> >>> From: "Jacques Le Roux"<[hidden email]> >>> >>> Thanks BJ, Bilgin, >>>> >>>> I will put it in FAQ, except if someone takes the time to dot it... >>>> >>>> Jacques >>>> >>>> From: "Bilgin Ibryam"<[hidden email]> >>>> >>>>> For reference: >>>>> >>>>> After evaluating few ETL tools - Pentaho Kettle, Jitterbit, Talend, we >>>>> found >>>>> out that the most easier way to integrate data in Ofbiz is using Talend. >>>>> >>>>> The process: >>>>> >>>>> - Connect Talend to the external DB >>>>> - For each row, prepare the data (doing transformations, joins with other >>>>> tables, call web services to get ISO codes) >>>>> - Call an ofbiz service to create the data in Ofbiz DB (like create >>>>> product) >>>>> >>>>> Ofbiz services can be called as web service (real pain with the way >>>>> parameters are encoded in maps) or through XML-RPC >>>>> >>>>> The process is really easy, since Talend can connect to many data >>>>> sources, >>>>> and importing the data using Ofbiz services ensure that the data is >>>>> properly >>>>> created. >>>>> >>>>> Bilgin >>>>> >>>>> On Mon, Jul 12, 2010 at 10:07 PM, BJ Freeman<[hidden email]> >>>>> wrote: >>>>> >>>>> your welcome >>>>>> in addition if you got the DB route you can write scripts to map ofbiz >>>>>> to >>>>>> the other db and use SECAS. >>>>>> so if say a createcustomers service is run you can have your SECA call >>>>>> your >>>>>> script and load the other DB. >>>>>> from you legacy db you can have scheduled services that pull from it to >>>>>> ofbiz. >>>>>> >>>>>> Chatree Srichart sent the following on 7/12/2010 1:05 PM: >>>>>> >>>>>> Hi, BJ Freeman >>>>>> >>>>>>> Thanks for your quick answer. >>>>>>> >>>>>>> Regards, >>>>>>> Chatree Srichart >>>>>>> >>>>>>> On Tue, Jul 13, 2010 at 2:53 AM, BJ Freeman<[hidden email]> >>>>>>> wrote: >>>>>>> >>>>>>> it depends on the data form you using. >>>>>>> >>>>>>>> for files >>>>>>>> >>>>>>>> >>>>>>>> https://cwiki.apache.org/confluence/display/OFBIZ/Handling+of+External+data >>>>>>>> >>>>>>>> if you have a db you can connect to ofbiz then you can >>>>>>>> create the entities and write minilang scripts to pull from the old >>>>>>>> one, >>>>>>>> map then store in ofbiz. >>>>>>>> >>>>>>>> >>>>>>>> https://demo-trunk.ofbiz.apache.org/webtools/control/view/ModelInduceFromDb >>>>>>>> your need to define a datasource in the entityengine.xml for your db, >>>>>>>> to >>>>>>>> use this tool. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Chatree Srichart sent the following on 7/12/2010 12:19 PM: >>>>>>>> >>>>>>>> Hi community, >>>>>>>> >>>>>>>> >>>>>>>>> Now I'm working on OFBiz and need to integrate it to several systems. >>>>>>>>> I >>>>>>>>> have >>>>>>>>> to get data from several sources to store in OFBiz data model. >>>>>>>>> I need to use ETL tool for help this process but I don't know what is >>>>>>>>> a >>>>>>>>> ETL >>>>>>>>> tool that OFBiz use. >>>>>>>>> >>>>>>>>> Can anyone tell me what is a ETL tool in OFBiz? >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Chatree Srichart >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >>>> >>> >> > |
|
|
actually was not talking about data flat files as a I was about using a
datasource to connect to a remote DB and inducing the entities for it, then writing mininlanq to pull in data from the remote db then mapping to the service and calling the service. seems to me a lot simpler. only draw back is if the remote db is behind a fire wall. Jacques Le Roux sent the following on 9/25/2010 3:15 PM: > BJ, Bilgin, > > I understand that using the OFBiz datafiles tools is good when you don't > have a legacy or external DB at hand (ie you have only flat files or such) > I remember you already spoke about this method when relationnal DB(s) is > available, and it makes sense indeed (simpler you only have to specify > the Entity). What would be the drawbacks compared to what Bilgin suggest > (fields types, is Talend able to handle triggers or embedded functions > effects?, etc.) > > Thanks > > Jacques > > From: "BJ Freeman" <[hidden email]> >> I would think if you have two database that do not match that you will >> have to do the process at one end or the other of matching the data.\ >> as well as making sure you have all the requirements of the service >> your calling. >> where if you connect to the db through ofbiz and setup the entities, >> you marshal the data from the remote db and call the service the same >> way. >> but you cut out any other layers. >> >> Bilgin Ibryam sent the following on 9/25/2010 11:54 AM: >>> Sure. The main advance of using Talend was that it is able to extract >>> data >>> from many datasources and then invoke ofbiz service thru xml-rpc (or >>> soap). >>> >>> Importing with OFBiz datafiles (or with xml files) requires handling >>> of all >>> the low-level tasks (primary keys, relatations, types, timestamps) to be >>> done manually. >>> Consider that you want to import the postal address for a person. You >>> have >>> to create 5-6 entities and manage the relations, types correctly... >>> On the >>> other hand, with Talend (or similar tool) you have to extract the needed >>> data from legacy data source and call createPartyPostalAddress >>> service. No >>> need to worry if you missed to populate some of the entities. >>> >>> Bilgin >>> >>> On Sat, Sep 25, 2010 at 1:14 PM, Jacques Le Roux< >>> [hidden email]> wrote: >>> >>>> Hi Bilgin, >>>> >>>> Before putting this into the FAQ, I was wondering why you choose to use >>>> Talend over OOTB OFBiz datafiles? >>>> I guess there are advantages, could you please name some? >>>> >>>> Thanks >>>> >>>> Jacques >>>> >>>> From: "Jacques Le Roux"<[hidden email]> >>>> >>>> Thanks BJ, Bilgin, >>>>> >>>>> I will put it in FAQ, except if someone takes the time to dot it... >>>>> >>>>> Jacques >>>>> >>>>> From: "Bilgin Ibryam"<[hidden email]> >>>>> >>>>>> For reference: >>>>>> >>>>>> After evaluating few ETL tools - Pentaho Kettle, Jitterbit, >>>>>> Talend, we >>>>>> found >>>>>> out that the most easier way to integrate data in Ofbiz is using >>>>>> Talend. >>>>>> >>>>>> The process: >>>>>> >>>>>> - Connect Talend to the external DB >>>>>> - For each row, prepare the data (doing transformations, joins >>>>>> with other >>>>>> tables, call web services to get ISO codes) >>>>>> - Call an ofbiz service to create the data in Ofbiz DB (like create >>>>>> product) >>>>>> >>>>>> Ofbiz services can be called as web service (real pain with the way >>>>>> parameters are encoded in maps) or through XML-RPC >>>>>> >>>>>> The process is really easy, since Talend can connect to many data >>>>>> sources, >>>>>> and importing the data using Ofbiz services ensure that the data is >>>>>> properly >>>>>> created. >>>>>> >>>>>> Bilgin >>>>>> >>>>>> On Mon, Jul 12, 2010 at 10:07 PM, BJ Freeman<[hidden email]> >>>>>> wrote: >>>>>> >>>>>> your welcome >>>>>>> in addition if you got the DB route you can write scripts to map >>>>>>> ofbiz >>>>>>> to >>>>>>> the other db and use SECAS. >>>>>>> so if say a createcustomers service is run you can have your SECA >>>>>>> call >>>>>>> your >>>>>>> script and load the other DB. >>>>>>> from you legacy db you can have scheduled services that pull from >>>>>>> it to >>>>>>> ofbiz. >>>>>>> >>>>>>> Chatree Srichart sent the following on 7/12/2010 1:05 PM: >>>>>>> >>>>>>> Hi, BJ Freeman >>>>>>> >>>>>>>> Thanks for your quick answer. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Chatree Srichart >>>>>>>> >>>>>>>> On Tue, Jul 13, 2010 at 2:53 AM, BJ Freeman<[hidden email]> >>>>>>>> wrote: >>>>>>>> >>>>>>>> it depends on the data form you using. >>>>>>>> >>>>>>>>> for files >>>>>>>>> >>>>>>>>> >>>>>>>>> https://cwiki.apache.org/confluence/display/OFBIZ/Handling+of+External+data >>>>>>>>> >>>>>>>>> >>>>>>>>> if you have a db you can connect to ofbiz then you can >>>>>>>>> create the entities and write minilang scripts to pull from the >>>>>>>>> old >>>>>>>>> one, >>>>>>>>> map then store in ofbiz. >>>>>>>>> >>>>>>>>> >>>>>>>>> https://demo-trunk.ofbiz.apache.org/webtools/control/view/ModelInduceFromDb >>>>>>>>> >>>>>>>>> your need to define a datasource in the entityengine.xml for >>>>>>>>> your db, >>>>>>>>> to >>>>>>>>> use this tool. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Chatree Srichart sent the following on 7/12/2010 12:19 PM: >>>>>>>>> >>>>>>>>> Hi community, >>>>>>>>> >>>>>>>>> >>>>>>>>>> Now I'm working on OFBiz and need to integrate it to several >>>>>>>>>> systems. >>>>>>>>>> I >>>>>>>>>> have >>>>>>>>>> to get data from several sources to store in OFBiz data model. >>>>>>>>>> I need to use ETL tool for help this process but I don't know >>>>>>>>>> what is >>>>>>>>>> a >>>>>>>>>> ETL >>>>>>>>>> tool that OFBiz use. >>>>>>>>>> >>>>>>>>>> Can anyone tell me what is a ETL tool in OFBiz? >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Chatree Srichart >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>> >>>>> >>>> >>> >> > > > |
Re: What is a ETL tool in OFBiz?
|
Administrator
|
I did actually the same. I should have separated my sentences in sections.
2 st lines were about flat files ("when you don't have a legacy or external DB at hand"). Then about the direct DB access ("when relationnal DB(s) is available"). It seems to me also to be the simpler method... I thought about possible issues with legacy field types. But it's anyway easy to create specific fieldtypes in OFBiz if ever needed. Jacques From: "BJ Freeman" <[hidden email]> > actually was not talking about data flat files as a I was about using a datasource to connect to a remote DB and inducing the > entities for it, then writing mininlanq to pull in data from the remote db then mapping to the service and calling the service. > seems to me a lot simpler. > only draw back is if the remote db is behind a fire wall. > > > Jacques Le Roux sent the following on 9/25/2010 3:15 PM: >> BJ, Bilgin, >> >> I understand that using the OFBiz datafiles tools is good when you don't >> have a legacy or external DB at hand (ie you have only flat files or such) >> I remember you already spoke about this method when relationnal DB(s) is >> available, and it makes sense indeed (simpler you only have to specify >> the Entity). What would be the drawbacks compared to what Bilgin suggest >> (fields types, is Talend able to handle triggers or embedded functions >> effects?, etc.) >> >> Thanks >> >> Jacques >> >> From: "BJ Freeman" <[hidden email]> >>> I would think if you have two database that do not match that you will >>> have to do the process at one end or the other of matching the data.\ >>> as well as making sure you have all the requirements of the service >>> your calling. >>> where if you connect to the db through ofbiz and setup the entities, >>> you marshal the data from the remote db and call the service the same >>> way. >>> but you cut out any other layers. >>> >>> Bilgin Ibryam sent the following on 9/25/2010 11:54 AM: >>>> Sure. The main advance of using Talend was that it is able to extract >>>> data >>>> from many datasources and then invoke ofbiz service thru xml-rpc (or >>>> soap). >>>> >>>> Importing with OFBiz datafiles (or with xml files) requires handling >>>> of all >>>> the low-level tasks (primary keys, relatations, types, timestamps) to be >>>> done manually. >>>> Consider that you want to import the postal address for a person. You >>>> have >>>> to create 5-6 entities and manage the relations, types correctly... >>>> On the >>>> other hand, with Talend (or similar tool) you have to extract the needed >>>> data from legacy data source and call createPartyPostalAddress >>>> service. No >>>> need to worry if you missed to populate some of the entities. >>>> >>>> Bilgin >>>> >>>> On Sat, Sep 25, 2010 at 1:14 PM, Jacques Le Roux< >>>> [hidden email]> wrote: >>>> >>>>> Hi Bilgin, >>>>> >>>>> Before putting this into the FAQ, I was wondering why you choose to use >>>>> Talend over OOTB OFBiz datafiles? >>>>> I guess there are advantages, could you please name some? >>>>> >>>>> Thanks >>>>> >>>>> Jacques >>>>> >>>>> From: "Jacques Le Roux"<[hidden email]> >>>>> >>>>> Thanks BJ, Bilgin, >>>>>> >>>>>> I will put it in FAQ, except if someone takes the time to dot it... >>>>>> >>>>>> Jacques >>>>>> >>>>>> From: "Bilgin Ibryam"<[hidden email]> >>>>>> >>>>>>> For reference: >>>>>>> >>>>>>> After evaluating few ETL tools - Pentaho Kettle, Jitterbit, >>>>>>> Talend, we >>>>>>> found >>>>>>> out that the most easier way to integrate data in Ofbiz is using >>>>>>> Talend. >>>>>>> >>>>>>> The process: >>>>>>> >>>>>>> - Connect Talend to the external DB >>>>>>> - For each row, prepare the data (doing transformations, joins >>>>>>> with other >>>>>>> tables, call web services to get ISO codes) >>>>>>> - Call an ofbiz service to create the data in Ofbiz DB (like create >>>>>>> product) >>>>>>> >>>>>>> Ofbiz services can be called as web service (real pain with the way >>>>>>> parameters are encoded in maps) or through XML-RPC >>>>>>> >>>>>>> The process is really easy, since Talend can connect to many data >>>>>>> sources, >>>>>>> and importing the data using Ofbiz services ensure that the data is >>>>>>> properly >>>>>>> created. >>>>>>> >>>>>>> Bilgin >>>>>>> >>>>>>> On Mon, Jul 12, 2010 at 10:07 PM, BJ Freeman<[hidden email]> >>>>>>> wrote: >>>>>>> >>>>>>> your welcome >>>>>>>> in addition if you got the DB route you can write scripts to map >>>>>>>> ofbiz >>>>>>>> to >>>>>>>> the other db and use SECAS. >>>>>>>> so if say a createcustomers service is run you can have your SECA >>>>>>>> call >>>>>>>> your >>>>>>>> script and load the other DB. >>>>>>>> from you legacy db you can have scheduled services that pull from >>>>>>>> it to >>>>>>>> ofbiz. >>>>>>>> >>>>>>>> Chatree Srichart sent the following on 7/12/2010 1:05 PM: >>>>>>>> >>>>>>>> Hi, BJ Freeman >>>>>>>> >>>>>>>>> Thanks for your quick answer. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Chatree Srichart >>>>>>>>> >>>>>>>>> On Tue, Jul 13, 2010 at 2:53 AM, BJ Freeman<[hidden email]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> it depends on the data form you using. >>>>>>>>> >>>>>>>>>> for files >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> https://cwiki.apache.org/confluence/display/OFBIZ/Handling+of+External+data >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> if you have a db you can connect to ofbiz then you can >>>>>>>>>> create the entities and write minilang scripts to pull from the >>>>>>>>>> old >>>>>>>>>> one, >>>>>>>>>> map then store in ofbiz. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> https://demo-trunk.ofbiz.apache.org/webtools/control/view/ModelInduceFromDb >>>>>>>>>> >>>>>>>>>> your need to define a datasource in the entityengine.xml for >>>>>>>>>> your db, >>>>>>>>>> to >>>>>>>>>> use this tool. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Chatree Srichart sent the following on 7/12/2010 12:19 PM: >>>>>>>>>> >>>>>>>>>> Hi community, >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Now I'm working on OFBiz and need to integrate it to several >>>>>>>>>>> systems. >>>>>>>>>>> I >>>>>>>>>>> have >>>>>>>>>>> to get data from several sources to store in OFBiz data model. >>>>>>>>>>> I need to use ETL tool for help this process but I don't know >>>>>>>>>>> what is >>>>>>>>>>> a >>>>>>>>>>> ETL >>>>>>>>>>> tool that OFBiz use. >>>>>>>>>>> >>>>>>>>>>> Can anyone tell me what is a ETL tool in OFBiz? >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Chatree Srichart >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >> >> > |
Re: What is a ETL tool in OFBiz?
|
|
In reply to this post by BJ Freeman
On Sun, Sep 26, 2010 at 2:11 AM, BJ Freeman <[hidden email]> wrote:
> actually was not talking about data flat files as a I was about using a > datasource to connect to a remote DB and inducing the entities for it, then > writing mininlanq to pull in data from the remote db then mapping to the > service and calling the service. > seems to me a lot simpler. > In my case I didn't have access to the remote DB and also didn't know the structure of the data. So it was easier for the client to use Talend for extracting data (as he alrady knows the data structure of DB) and send these data to OFBiz with xml-rpc Bilgin |
Re: What is a ETL tool in OFBiz?
|
Administrator
|
Thanks Bilgin,
I have enough now to write a really usefull FAQ Jacques From: "Bilgin Ibryam" <[hidden email]> > On Sun, Sep 26, 2010 at 2:11 AM, BJ Freeman <[hidden email]> wrote: > >> actually was not talking about data flat files as a I was about using a >> datasource to connect to a remote DB and inducing the entities for it, then >> writing mininlanq to pull in data from the remote db then mapping to the >> service and calling the service. >> seems to me a lot simpler. >> > > In my case I didn't have access to the remote DB and also didn't know the > structure of the data. So it was easier for the client to use Talend for > extracting data (as he alrady knows the data structure of DB) and send these > data to OFBiz with xml-rpc > > Bilgin > |
Re: What is a ETL tool in OFBiz?
|
Administrator
|
Done at https://cwiki.apache.org/confluence/display/OFBIZ/Handling+of+External+data
I have refactored and updated you work BJ. BTW, I recommend Scite and its variation NotePad++ as simple text editor http://notepad-plus-plus.org/ NotePad++ is stuffed with really cool tools (see TextFx menu options) Thanks Jacques From: "Jacques Le Roux" <[hidden email]> > Thanks Bilgin, > > I have enough now to write a really usefull FAQ > > Jacques > > From: "Bilgin Ibryam" <[hidden email]> >> On Sun, Sep 26, 2010 at 2:11 AM, BJ Freeman <[hidden email]> wrote: >> >>> actually was not talking about data flat files as a I was about using a >>> datasource to connect to a remote DB and inducing the entities for it, then >>> writing mininlanq to pull in data from the remote db then mapping to the >>> service and calling the service. >>> seems to me a lot simpler. >>> >> >> In my case I didn't have access to the remote DB and also didn't know the >> structure of the data. So it was easier for the client to use Talend for >> extracting data (as he alrady knows the data structure of DB) and send these >> data to OFBiz with xml-rpc >> >> Bilgin >> > |
|
|
thanks
Jacques Le Roux sent the following on 10/2/2010 2:47 AM: > Done at > https://cwiki.apache.org/confluence/display/OFBIZ/Handling+of+External+data > I have refactored and updated you work BJ. > BTW, I recommend Scite and its variation NotePad++ as simple text editor > http://notepad-plus-plus.org/ NotePad++ is stuffed with really cool > tools (see TextFx menu options) > > Thanks > > Jacques > > From: "Jacques Le Roux" <[hidden email]> >> Thanks Bilgin, >> >> I have enough now to write a really usefull FAQ >> >> Jacques >> >> From: "Bilgin Ibryam" <[hidden email]> >>> On Sun, Sep 26, 2010 at 2:11 AM, BJ Freeman <[hidden email]> wrote: >>> >>>> actually was not talking about data flat files as a I was about using a >>>> datasource to connect to a remote DB and inducing the entities for >>>> it, then >>>> writing mininlanq to pull in data from the remote db then mapping to >>>> the >>>> service and calling the service. >>>> seems to me a lot simpler. >>>> >>> >>> In my case I didn't have access to the remote DB and also didn't know >>> the >>> structure of the data. So it was easier for the client to use Talend for >>> extracting data (as he alrady knows the data structure of DB) and >>> send these >>> data to OFBiz with xml-rpc >>> >>> Bilgin >>> >> > > |
«
Return to OFBiz - User
|
1 view|%1 views
| Free forum by Nabble | Edit this page |