Re: importing data into ofbiz
Posted by David E Jones-2 on Jul 07, 2006; 8:03pm
URL: http://ofbiz.116.s1.nabble.com/importing-data-into-ofbiz-tp169514p169515.html
Stepping back a bit and looking at this conceptually:
What you are proposing would be an import format that would work well for a certain piece of software that the data is coming from.
In fact, the normalized data model of OFBiz is far easier to transform data to than a more normalized form would be.
Consider Software A and Software B that both have very functionality specific, redundant, and generally non-normalized data models. The cardinality of fields relative to a product may be different, and incompatible, and they may treat similar information in very different ways. Because the OFBiz model is more flexible it may be possible to migrate A->OFBiz and B->OFBiz but it may not be possible to migrate A->B or B->A.
So, yes, something is needed to transform incoming data but trying to create something that is less normalized will just make it easier for some systems (those that are close to it) and far more difficult for other systems (that have more conceptual mismatches with it).
This is basically the unavoidable law of de-normalization.
As far as tools and such go, FreeMarker is actually a pretty good XML transformation tool. The Entity Engine import stuff does already support using FTL files along with an incoming XML file to pre-transform the data. There is an example in the ofbiz/applications/ecommerce/data/DemoContent.xml file. Notice the root element is "entity-engine-transform-xml" instead of "entity-engine-xml".
-David
Si Chen wrote:
> Hi everybody. I've been thinking about importing data into ofbiz. One
> of the issues is that we have a highly normalized data model which would
> require extensive mapping from external systems. Nevertheless, I think
> it is possible to create some standard "template" entities for importing
> data and thus create a standard practice and, eventually, standard tools
> for doing it. Here's what I'm thinking, given in the context of
> importing product data:
>
> 1. Create a standard non-normalized "holding" entity which contains
> productId, quantity on hand, available to promise, average cost, reorder
> point. Note that none of these are linked to any entity in ofbiz.
> 2. Add a timestamp fields to identify when the data was loaded into the
> "holding" entity and when it was imported into ofbiz.
> 3. Write a standard datafile xml import template which corresponds to
> this "holding" entity.
> 4. Write some processing script or service which takes new data from
> the holding entity and insert it into ofbiz. For example, in this case
> it would be to create InventoryItem, ProductAverageCost, and maybe some
> AcctgTrans (maybe . . .)
>
> It may be most efficient to write these holding entities corresponding
> to popular software programs where people might want to import data
> from, so they can mirror those programs' data layout more easily.
>
> Si
URL: http://ofbiz.116.s1.nabble.com/importing-data-into-ofbiz-tp169514p169515.html
Stepping back a bit and looking at this conceptually:
What you are proposing would be an import format that would work well for a certain piece of software that the data is coming from.
In fact, the normalized data model of OFBiz is far easier to transform data to than a more normalized form would be.
Consider Software A and Software B that both have very functionality specific, redundant, and generally non-normalized data models. The cardinality of fields relative to a product may be different, and incompatible, and they may treat similar information in very different ways. Because the OFBiz model is more flexible it may be possible to migrate A->OFBiz and B->OFBiz but it may not be possible to migrate A->B or B->A.
So, yes, something is needed to transform incoming data but trying to create something that is less normalized will just make it easier for some systems (those that are close to it) and far more difficult for other systems (that have more conceptual mismatches with it).
This is basically the unavoidable law of de-normalization.
As far as tools and such go, FreeMarker is actually a pretty good XML transformation tool. The Entity Engine import stuff does already support using FTL files along with an incoming XML file to pre-transform the data. There is an example in the ofbiz/applications/ecommerce/data/DemoContent.xml file. Notice the root element is "entity-engine-transform-xml" instead of "entity-engine-xml".
-David
Si Chen wrote:
> Hi everybody. I've been thinking about importing data into ofbiz. One
> of the issues is that we have a highly normalized data model which would
> require extensive mapping from external systems. Nevertheless, I think
> it is possible to create some standard "template" entities for importing
> data and thus create a standard practice and, eventually, standard tools
> for doing it. Here's what I'm thinking, given in the context of
> importing product data:
>
> 1. Create a standard non-normalized "holding" entity which contains
> productId, quantity on hand, available to promise, average cost, reorder
> point. Note that none of these are linked to any entity in ofbiz.
> 2. Add a timestamp fields to identify when the data was loaded into the
> "holding" entity and when it was imported into ofbiz.
> 3. Write a standard datafile xml import template which corresponds to
> this "holding" entity.
> 4. Write some processing script or service which takes new data from
> the holding entity and insert it into ofbiz. For example, in this case
> it would be to create InventoryItem, ProductAverageCost, and maybe some
> AcctgTrans (maybe . . .)
>
> It may be most efficient to write these holding entities corresponding
> to popular software programs where people might want to import data
> from, so they can mirror those programs' data layout more easily.
>
> Si
smime.p7s (4K) | Free forum by Nabble | Edit this page |