OFBiz › OFBiz - Dev

Dev - OFBiz Datawarehouse Component

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

10 messages Options

Oleg Kozyrev Jr.

Dev - OFBiz Datawarehouse Component

Hello all,

At the moment I'm working on implementing Data Warehouse functionality into OFBiz (http://jira.undersunconsulting.com/browse/OFBIZ-719). It would be nice to hear your ideas/suggessions on several points below.

In general, there are 3 major parts in this component:
1. Building Data Warehouse (ETL, Extraction Transformation Loading).
Now this part is based on Octopus (http://octopus.objectweb.org). At the moment this part doesn't use any OFBiz (Entity Engine) functionality, but it can extract/transform/load data from different data sources (DB, Excel, ...) and put them into db (OFBiz DB, for example). There is a special engine (structure is close to minilang engine structure) for doing this work. Next version will be more close to Entity Engine terms (I mean that now everything is defined in terms of db tables, but should be defined in terms of OFBiz entities).

2. Building OLAP cube and executing queries. This is based on Mondrian (http://mondrian.sourceforge.net). Right now you can define cube in terms of db tables. Similar to this: http://mondrian.sourceforge.net/schema.html#Cubes_and_dimensions
At the moment there is one low-level OFBiz Java service, which executes MDX queries.

3. User Interface. Currently I'm working on implementing GUI. I think it will be based on JPivot (http://jpivot.sourceforge.net/). The problem is that JPivot provides JSP tag library and it is closely connected with Mondrian. So, I'm thinking about full rework of this tag library into FTL tags (like <@ofbizUrl>, <@ofbizContentUrl>, etc.). It will look like here: http://jpivot.sourceforge.net/temp-N1018C.html

I appreciate any comments on this points. Especially about third one.

Thanks,

Oleg.

_______________________________________________
Dev mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/dev

Michael Irving

Re: Dev - OFBiz Datawarehouse Component

Might want to simply implement one of these:

1) http://www.opensourcestrategies.com/ofbiz/ofbiz-birt.pdf
2) http://www.pentaho.org

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On
Behalf Of Oleg Kozyrev Jr.
Sent: Friday, April 21, 2006 1:53 AM
To: [hidden email]
Subject: [OFBiz] Dev - OFBiz Datawarehouse Component

Hello all,

At the moment I'm working on implementing Data Warehouse functionality into
OFBiz (http://jira.undersunconsulting.com/browse/OFBIZ-719). It would be
nice to hear your ideas/suggessions on several points below.

In general, there are 3 major parts in this component:
1. Building Data Warehouse (ETL, Extraction Transformation Loading).
Now this part is based on Octopus (http://octopus.objectweb.org). At the
moment this part doesn't use any OFBiz (Entity Engine) functionality, but it
can extract/transform/load data from different data sources (DB, Excel, ...)
and put them into db (OFBiz DB, for example). There is a special engine
(structure is close to minilang engine structure) for doing this work. Next
version will be more close to Entity Engine terms (I mean that now
everything is defined in terms of db tables, but should be defined in terms
of OFBiz entities).

2. Building OLAP cube and executing queries. This is based on Mondrian
(http://mondrian.sourceforge.net). Right now you can define cube in terms of
db tables. Similar to this:
http://mondrian.sourceforge.net/schema.html#Cubes_and_dimensions
At the moment there is one low-level OFBiz Java service, which executes MDX
queries.

3. User Interface. Currently I'm working on implementing GUI. I think it
will be based on JPivot (http://jpivot.sourceforge.net/). The problem is
that JPivot provides JSP tag library and it is closely connected with
Mondrian. So, I'm thinking about full rework of this tag library into FTL
tags (like <@ofbizUrl>, <@ofbizContentUrl>, etc.). It will look like here:
http://jpivot.sourceforge.net/temp-N1018C.html

I appreciate any comments on this points. Especially about third one.

Thanks,

Oleg.

_______________________________________________
Dev mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/dev

_______________________________________________
Dev mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/dev

Oleg Kozyrev Jr.

Re: Dev - OFBiz Datawarehouse Component

In reply to this post by Oleg Kozyrev Jr.

Michael,

Thanks for your answer.

I've heard about BIRT. As far as I understand it can help in building reports over an already prepared data warehouse. But, the main purpose of component I'm working on is to build this data warehouse and to provide easy access to it. Certainly without GUI (reports and so on) it's useless. Also, BIRT can't help in analysing data in real-time. I mean, when you are studing sales you can't view any unprepared dependencies, because report should be prepared in advance.

I've also heard about Pentaho... As far as I'm a student I 'must' do all this stuff, because this is my term paper. :)
Still Ofbiz component have much in common with Pentaho.

Thank you one more time.

Oleg.

-----Original Message-----
From: "Michael Irving" <[hidden email]>
To: "'OFBiz Project Development Discussion'" <[hidden email]>
Date: Fri, 21 Apr 2006 08:13:21 -0400
Subject: Re: [OFBiz] Dev - OFBiz Datawarehouse Component

>
> Might want to simply implement one of these:
>
> 1) http://www.opensourcestrategies.com/ofbiz/ofbiz-birt.pdf
> 2) http://www.pentaho.org
>
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On
> Behalf Of Oleg Kozyrev Jr.
> Sent: Friday, April 21, 2006 1:53 AM
> To: [hidden email]
> Subject: [OFBiz] Dev - OFBiz Datawarehouse Component
>
> Hello all,
>
> At the moment I'm working on implementing Data Warehouse functionality into
> OFBiz (http://jira.undersunconsulting.com/browse/OFBIZ-719). It would be
> nice to hear your ideas/suggessions on several points below.
>
> In general, there are 3 major parts in this component:
> 1. Building Data Warehouse (ETL, Extraction Transformation Loading).
> Now this part is based on Octopus (http://octopus.objectweb.org). At the
> moment this part doesn't use any OFBiz (Entity Engine) functionality, but it
> can extract/transform/load data from different data sources (DB, Excel, ...)
> and put them into db (OFBiz DB, for example). There is a special engine
> (structure is close to minilang engine structure) for doing this work. Next
> version will be more close to Entity Engine terms (I mean that now
> everything is defined in terms of db tables, but should be defined in terms
> of OFBiz entities).
>
> 2. Building OLAP cube and executing queries. This is based on Mondrian
> (http://mondrian.sourceforge.net). Right now you can define cube in terms of
> db tables. Similar to this:
> http://mondrian.sourceforge.net/schema.html#Cubes_and_dimensions
> At the moment there is one low-level OFBiz Java service, which executes MDX
> queries.
>
> 3. User Interface. Currently I'm working on implementing GUI. I think it
> will be based on JPivot (http://jpivot.sourceforge.net/). The problem is
> that JPivot provides JSP tag library and it is closely connected with
> Mondrian. So, I'm thinking about full rework of this tag library into FTL
> tags (like <@ofbizUrl>, <@ofbizContentUrl>, etc.). It will look like here:
> http://jpivot.sourceforge.net/temp-N1018C.html
>
> I appreciate any comments on this points. Especially about third one.
>
> Thanks,
>
> Oleg.
>
> _______________________________________________
> Dev mailing list
> [hidden email]
> http://lists.ofbiz.org/mailman/listinfo/dev
>
>
> _______________________________________________
> Dev mailing list
> [hidden email]
> http://lists.ofbiz.org/mailman/listinfo/dev
>

Как познакомиться с десятками людей за один день? Мы знаем один простой способ http://r.mail.ru/cln2844/agent.mail.ru/talk.html

_______________________________________________
Dev mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/dev

David E. Jones

Re: Dev - OFBiz Datawarehouse Component

In reply to this post by Oleg Kozyrev Jr.

Oleg,

A couple of thoughts:

1. the libraries for these packages could be included with OFBiz, but check licensing (GPL and LGPL are no-nos)

2. for #3 you might want to instead look into the FreeMarker support of JSP tag libraries; note that to support these in OFBiz right now it requires a code change to load the JSP taglibs in the FreeMarkerWorker.java class; that isn't a problem if the libraries go into the project, it can stay there permanently

-David

Oleg Kozyrev Jr. wrote:

> Hello all,
>
> At the moment I'm working on implementing Data Warehouse functionality into OFBiz (http://jira.undersunconsulting.com/browse/OFBIZ-719). It would be nice to hear your ideas/suggessions on several points below.
>
> In general, there are 3 major parts in this component:
> 1. Building Data Warehouse (ETL, Extraction Transformation Loading).
> Now this part is based on Octopus (http://octopus.objectweb.org). At the moment this part doesn't use any OFBiz (Entity Engine) functionality, but it can extract/transform/load data from different data sources (DB, Excel, ...) and put them into db (OFBiz DB, for example). There is a special engine (structure is close to minilang engine structure) for doing this work. Next version will be more close to Entity Engine terms (I mean that now everything is defined in terms of db tables, but should be defined in terms of OFBiz entities).
>
> 2. Building OLAP cube and executing queries. This is based on Mondrian (http://mondrian.sourceforge.net). Right now you can define cube in terms of db tables. Similar to this: http://mondrian.sourceforge.net/schema.html#Cubes_and_dimensions
> At the moment there is one low-level OFBiz Java service, which executes MDX queries.
>
> 3. User Interface. Currently I'm working on implementing GUI. I think it will be based on JPivot (http://jpivot.sourceforge.net/). The problem is that JPivot provides JSP tag library and it is closely connected with Mondrian. So, I'm thinking about full rework of this tag library into FTL tags (like <@ofbizUrl>, <@ofbizContentUrl>, etc.). It will look like here: http://jpivot.sourceforge.net/temp-N1018C.html
>
> I appreciate any comments on this points. Especially about third one.
>
> Thanks,
>
> Oleg.
>
> _______________________________________________
> Dev mailing list
> [hidden email]
> http://lists.ofbiz.org/mailman/listinfo/dev

_______________________________________________
Dev mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/dev

Oleg Kozyrev Jr.

Re: Dev - OFBiz Datawarehouse Component

David,

Mondrian and JPivot are CPL. Is it ok?
Octopus is GPL. Probably it would be possible to include it into specialized/libraries like Shark or something?

Some comments on your second point. Yes, I was thinking about using FreeMarker support for JSP. There are several problems:
* if one want to use this option he has to add special listener to web-application (web.xml)
* OFBiz is going away from using JSPs (regions, etc)
* Anyway these tags are using Mondrian functionality inside themselfs, but they should work with OFBiz Service Engine. That's why they should be reworked.

Am I right that you think that using FreeMarker instead of JSP is better in some way?

Oleg.

-----Original Message-----
From: David E Jones <[hidden email]>
To: OFBiz Project Development Discussion <[hidden email]>
Date: Fri, 21 Apr 2006 12:43:56 -0600
Subject: Re: [OFBiz] Dev - OFBiz Datawarehouse Component

>
>
> Oleg,
>
> A couple of thoughts:
>
> 1. the libraries for these packages could be included with OFBiz, but check licensing (GPL and LGPL are no-nos)
>
> 2. for #3 you might want to instead look into the FreeMarker support of JSP tag libraries; note that to support these in OFBiz right now it requires a code change to load the JSP taglibs in the FreeMarkerWorker.java class; that isn't a problem if the libraries go into the project, it can stay there permanently
>
> -David
>
>
> Oleg Kozyrev Jr. wrote:
> > Hello all,
> >
> > At the moment I'm working on implementing Data Warehouse functionality into OFBiz (http://jira.undersunconsulting.com/browse/OFBIZ-719). It would be nice to hear your ideas/suggessions on several points below.
> >
> > In general, there are 3 major parts in this component:
> > 1. Building Data Warehouse (ETL, Extraction Transformation Loading).
> > Now this part is based on Octopus (http://octopus.objectweb.org). At the moment this part doesn't use any OFBiz (Entity Engine) functionality, but it can extract/transform/load data from different data sources (DB, Excel, ...) and put them into db (OFBiz DB, for example). There is a special engine (structure is close to minilang engine structure) for doing this work. Next version will be more close to Entity Engine terms (I mean that now everything is defined in terms of db tables, but should be defined in terms of OFBiz entities).
> >
> > 2. Building OLAP cube and executing queries. This is based on Mondrian (http://mondrian.sourceforge.net). Right now you can define cube in terms of db tables. Similar to this: http://mondrian.sourceforge.net/schema.html#Cubes_and_dimensions
> > At the moment there is one low-level OFBiz Java service, which executes MDX queries.
> >
> > 3. User Interface. Currently I'm working on implementing GUI. I think it will be based on JPivot (http://jpivot.sourceforge.net/). The problem is that JPivot provides JSP tag library and it is closely connected with Mondrian. So, I'm thinking about full rework of this tag library into FTL tags (like <@ofbizUrl>, <@ofbizContentUrl>, etc.). It will look like here: http://jpivot.sourceforge.net/temp-N1018C.html
> >
> > I appreciate any comments on this points. Especially about third one.
> >
> > Thanks,
> >
> > Oleg.
> >
> > _______________________________________________
> > Dev mailing list
> > [hidden email]
> > http://lists.ofbiz.org/mailman/listinfo/dev
>
> _______________________________________________
> Dev mailing list
> [hidden email]
> http://lists.ofbiz.org/mailman/listinfo/dev
>

Жуки@Mail.ru - маленькие герои в мире большого спорта!
http://r.mail.ru/cln3055/zhuki.mail.ru/

_______________________________________________
Dev mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/dev

Oleg Kozyrev Jr.

Re: Dev - OFBiz Datawarehouse Component

In reply to this post by David E. Jones

David,

Could you, please, look at Jira issue for FreeMarkerWorker.java (http://jira.undersunconsulting.com/browse/OFBIZ-848)

If it's ok, could you commit it?

Thanks,

Oleg.

-----Original Message-----
From: David E Jones <[hidden email]>
To: OFBiz Project Development Discussion <[hidden email]>
Date: Fri, 21 Apr 2006 12:43:56 -0600
Subject: Re: [OFBiz] Dev - OFBiz Datawarehouse Component

David E. Jones

Re: Dev - OFBiz Datawarehouse Component

In reply to this post by Oleg Kozyrev Jr.

Oleg,

Yes, the Common Public License is just fine and won't be a problem.

The GPL license, on the other hand is a very big problem. In fact, it's such a big problem that we can't even include code that requires a GPL library. If it is LGPL we can include the code, but not that actual library (which is what the specialized/libraries component has, ie just LGPL, no GPL).

On the other issue... perhaps there is some confusion. I am saying use FreeMarker but use the built in support for JSP tag libraries in FreeMarker. This is different from transforms, so that other issue you mentioned wouldn't help this (which consequently puts in lower on my already un-manage-ably long list of issues to work on). For info on this see the FreeMarker docs:

http://www.freemarker.org/docs/pgui_misc_servlet.html

A little way down there is a section titled "Using JSP custom tags in FTL".

-David

Oleg Kozyrev Jr. wrote:

> David,
>
> Mondrian and JPivot are CPL. Is it ok?
> Octopus is GPL. Probably it would be possible to include it into specialized/libraries like Shark or something?
>
> Some comments on your second point. Yes, I was thinking about using FreeMarker support for JSP. There are several problems:
> * if one want to use this option he has to add special listener to web-application (web.xml)
> * OFBiz is going away from using JSPs (regions, etc)
> * Anyway these tags are using Mondrian functionality inside themselfs, but they should work with OFBiz Service Engine. That's why they should be reworked.
>
> Am I right that you think that using FreeMarker instead of JSP is better in some way?
>
> Oleg.
>
> -----Original Message-----
> From: David E Jones <[hidden email]>
> To: OFBiz Project Development Discussion <[hidden email]>
> Date: Fri, 21 Apr 2006 12:43:56 -0600
> Subject: Re: [OFBiz] Dev - OFBiz Datawarehouse Component
>
>>
>> Oleg,
>>
>> A couple of thoughts:
>>
>> 1. the libraries for these packages could be included with OFBiz, but check licensing (GPL and LGPL are no-nos)
>>
>> 2. for #3 you might want to instead look into the FreeMarker support of JSP tag libraries; note that to support these in OFBiz right now it requires a code change to load the JSP taglibs in the FreeMarkerWorker.java class; that isn't a problem if the libraries go into the project, it can stay there permanently
>>
>> -David
>>
>>
>> Oleg Kozyrev Jr. wrote:
>>> Hello all,
>>>
>>> At the moment I'm working on implementing Data Warehouse functionality into OFBiz (http://jira.undersunconsulting.com/browse/OFBIZ-719). It would be nice to hear your ideas/suggessions on several points below.
>>>
>>> In general, there are 3 major parts in this component:
>>> 1. Building Data Warehouse (ETL, Extraction Transformation Loading).
>>> Now this part is based on Octopus (http://octopus.objectweb.org). At the moment this part doesn't use any OFBiz (Entity Engine) functionality, but it can extract/transform/load data from different data sources (DB, Excel, ...) and put them into db (OFBiz DB, for example). There is a special engine (structure is close to minilang engine structure) for doing this work. Next version will be more close to Entity Engine terms (I mean that now everything is defined in terms of db tables, but should be defined in terms of OFBiz entities).
>>>
>>> 2. Building OLAP cube and executing queries. This is based on Mondrian (http://mondrian.sourceforge.net). Right now you can define cube in terms of db tables. Similar to this: http://mondrian.sourceforge.net/schema.html#Cubes_and_dimensions
>>> At the moment there is one low-level OFBiz Java service, which executes MDX queries.
>>>
>>> 3. User Interface. Currently I'm working on implementing GUI. I think it will be based on JPivot (http://jpivot.sourceforge.net/). The problem is that JPivot provides JSP tag library and it is closely connected with Mondrian. So, I'm thinking about full rework of this tag library into FTL tags (like <@ofbizUrl>, <@ofbizContentUrl>, etc.). It will look like here: http://jpivot.sourceforge.net/temp-N1018C.html
>>>
>>> I appreciate any comments on this points. Especially about third one.
>>>
>>> Thanks,
>>>
>>> Oleg.
>>>
>>> _______________________________________________
>>> Dev mailing list
>>> [hidden email]
>>> http://lists.ofbiz.org/mailman/listinfo/dev
>>
>> _______________________________________________
>> Dev mailing list
>> [hidden email]
>> http://lists.ofbiz.org/mailman/listinfo/dev
>>
>
>
> Жуки@Mail.ru - маленькие герои в мире большого спорта!
> http://r.mail.ru/cln3055/zhuki.mail.ru/
>
> _______________________________________________
> Dev mailing list
> [hidden email]
> http://lists.ofbiz.org/mailman/listinfo/dev

_______________________________________________
Dev mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/dev

davidnwelton

Re: Dev - OFBiz Datawarehouse Component

> The GPL license, on the other hand is a very big problem. In fact, it's such a big problem that we can't even include code that requires a GPL library. If it is LGPL we can include the code, but not that actual library (which is what the specialized/libraries component has, ie just LGPL, no GPL).

It's always a possibility to create something like "ofbiz-extras" on
sourceforge and put some GPL components there, as long as the main
codebase doesn't depend on them.

--
David N. Welton
- http://www.dedasys.com/davidw/

Linux, Open Source Consulting
- http://www.dedasys.com/

_______________________________________________
Dev mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/dev

David E. Jones

Re: Dev - OFBiz Datawarehouse Component

Yes, we have considered doing something like this. Basically a GPL add-on to OFBiz that can use other GPL licensed stuff. In a way it's a nice idea, though I do have concerns about too much work being done on GPL derivatives that are forced into a GPL license (all of the code in this would _have_ to be GPL licensed) rather than using a license like Apache 2.0 that makes the resulting software much more flexible and less worrisome to use.

-David J.

David Welton wrote:

>> The GPL license, on the other hand is a very big problem. In fact, it's such a big problem that we can't even include code that requires a GPL library. If it is LGPL we can include the code, but not that actual library (which is what the specialized/libraries component has, ie just LGPL, no GPL).
>
> It's always a possibility to create something like "ofbiz-extras" on
> sourceforge and put some GPL components there, as long as the main
> codebase doesn't depend on them.
>
> --
> David N. Welton
> - http://www.dedasys.com/davidw/
>
> Linux, Open Source Consulting
> - http://www.dedasys.com/
>
> _______________________________________________
> Dev mailing list
> [hidden email]
> http://lists.ofbiz.org/mailman/listinfo/dev

_______________________________________________
Dev mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/dev

David E. Jones

Re: Dev - OFBiz Datawarehouse Component

Maybe I should comment on this a little more...

For now the plan is to keep the specialized directly on the Undersun SVN server where everything else is right now. We plan to keep that in operation long after the main code base has moved over to the hoped-for official Apache SVN server.

The point of this is to facilitate OFBiz-related efforts. Of course, anyone can create a SourceForge project or use other resources to host their own OFBiz derivative, and maybe at some point we'll encourage that anyway... I guess it depends on how many people are interested and how much collaboration ends up happening that needs tools to help along. So far I haven't seen a whole lot of that among the OFBiz derivatives I'm aware of, so I don't know how this will work itself out... ;)

-David

David E Jones wrote:

> Yes, we have considered doing something like this. Basically a GPL add-on to OFBiz that can use other GPL licensed stuff. In a way it's a nice idea, though I do have concerns about too much work being done on GPL derivatives that are forced into a GPL license (all of the code in this would _have_ to be GPL licensed) rather than using a license like Apache 2.0 that makes the resulting software much more flexible and less worrisome to use.
>
> -David J.
>
>
> David Welton wrote:
>>> The GPL license, on the other hand is a very big problem. In fact, it's such a big problem that we can't even include code that requires a GPL library. If it is LGPL we can include the code, but not that actual library (which is what the specialized/libraries component has, ie just LGPL, no GPL).
>> It's always a possibility to create something like "ofbiz-extras" on
>> sourceforge and put some GPL components there, as long as the main
>> codebase doesn't depend on them.
>>
>> --
>> David N. Welton
>> - http://www.dedasys.com/davidw/
>>
>> Linux, Open Source Consulting
>> - http://www.dedasys.com/
>>
>> _______________________________________________
>> Dev mailing list
>> [hidden email]
>> http://lists.ofbiz.org/mailman/listinfo/dev
>
> _______________________________________________
> Dev mailing list
> [hidden email]
> http://lists.ofbiz.org/mailman/listinfo/dev

_______________________________________________
Dev mailing list
[hidden email]
http://lists.ofbiz.org/mailman/listinfo/dev