Re: importing data into ofbiz

Posted by David E Jones-2 on Jul 10, 2006; 6:27pm
URL: http://ofbiz.116.s1.nabble.com/importing-data-into-ofbiz-tp169514p169521.html


The point of transaction management is to have a set of operations succeed or fail all together.

Would it really be easier with a large or small file to have all of the data go in except for the failures and then try to pick out all of the failures and fix and get those in individually in another file or by slimming down the original?

Another problem with not failing the entire file is if there are dependencies between the data or problems might be caused by having partial data in the system, then you'll have cascading failures....

So how would a partial failure be handled, or what sort of a process would it fit into?

-David


Si Chen wrote:

> Still, one issue I have is "exceptions handling": it seems that entity
> engine xml import would just rollback everything if one of the records
> to be imported is bad.  Isn't better (or standard) to have a system
> where the data could be imported and then each exception handled
> individually?
>
> Si
>
>
> On Jul 7, 2006, at 1:27 PM, David E. Jones wrote:
>
>>
>> Yes, this looks like a good description of the general idea.
>>
>> -David
>>
>>
>> Si Chen wrote:
>>> David,
>>> Yes, the idea was to create something that would be better for some
>>> applications (namely, the one it's built for) than others.
>>> But I'm very curious about this entity-engine-transform-xml.  So is
>>> this idea to paste a big block of XML into
>>> <entity-engine-transform-xml> file and then write the .FTL to do the
>>> transformation?  And then run xml import on that file?
>>> And what about the freemarker template?  Does it fall under their
>>> "Declarative XML Processing" documentation here:
>>> http://freemarker.org/docs/xgui_declarative.html  I looked at it
>>> briefly and recognized some of syntax.
>>> Si
>>> On Jul 7, 2006, at 12:03 PM, David E. Jones wrote:
>>>>
>>>> Stepping back a bit and looking at this conceptually:
>>>>
>>>> What you are proposing would be an import format that would work
>>>> well for a certain piece of software that the data is coming from.
>>>>
>>>> In fact, the normalized data model of OFBiz is far easier to
>>>> transform data to than a more normalized form would be.
>>>>
>>>> Consider Software A and Software B that both have very functionality
>>>> specific, redundant, and generally non-normalized data models. The
>>>> cardinality of fields relative to a product may be different, and
>>>> incompatible, and they may treat similar information in very
>>>> different ways. Because the OFBiz model is more flexible it may be
>>>> possible to migrate A->OFBiz and B->OFBiz but it may not be possible
>>>> to migrate A->B or B->A.
>>>>
>>>> So, yes, something is needed to transform incoming data but trying
>>>> to create something that is less normalized will just make it easier
>>>> for some systems (those that are close to it) and far more difficult
>>>> for other systems (that have more conceptual mismatches with it).
>>>>
>>>> This is basically the unavoidable law of de-normalization.
>>>>
>>>> As far as tools and such go, FreeMarker is actually a pretty good
>>>> XML transformation tool. The Entity Engine import stuff does already
>>>> support using FTL files along with an incoming XML file to
>>>> pre-transform the data. There is an example in the
>>>> ofbiz/applications/ecommerce/data/DemoContent.xml file. Notice the
>>>> root element is "entity-engine-transform-xml" instead of
>>>> "entity-engine-xml".
>>>>
>>>> -David
>>>>
>>>>
>>>> Si Chen wrote:
>>>>> Hi everybody.  I've been thinking about importing data into ofbiz.  
>>>>> One of the issues is that we have a highly normalized data model
>>>>> which would require extensive mapping from external systems.  
>>>>> Nevertheless, I think it is possible to create some standard
>>>>> "template" entities for importing data and thus create a standard
>>>>> practice and, eventually, standard tools for doing it.  Here's what
>>>>> I'm thinking, given in the context of importing product data:
>>>>> 1.  Create a standard non-normalized "holding" entity which
>>>>> contains productId, quantity on hand, available to promise, average
>>>>> cost, reorder point.  Note that none of these are linked to any
>>>>> entity in ofbiz.
>>>>> 2.  Add a timestamp fields to identify when the data was loaded
>>>>> into the "holding" entity and when it was imported into ofbiz.
>>>>> 3.  Write a standard datafile xml import template which corresponds
>>>>> to this "holding" entity.
>>>>> 4.  Write some processing script or service which takes new data
>>>>> from the holding entity and insert it into ofbiz.  For example, in
>>>>> this case it would be to create InventoryItem, ProductAverageCost,
>>>>> and maybe some AcctgTrans (maybe . . .)
>>>>> It may be most efficient to write these holding entities
>>>>> corresponding to popular software programs where people might want
>>>>> to import data from, so they can mirror those programs' data layout
>>>>> more easily.
>>>>> Si
>