Discussion: Mashups

[Adrian Crum]

I need to add mashup capability to my local OFBiz copy. My immediate
need is a web page scraper that will gather copy counts from our copiers
and store them as meter readings.

I would like to implement this in a generic way that might be fed back
into the project. I don't want to reinvent the wheel, however. So, I'm
opening up the topic for discussion.

Is there any interest in having something like a "mashup engine" or
"mashup services" in the framework? If yes, what are some strategies we
can use? Are there any open source libraries we can leverage in an
implementation?

-Adrian

[Jacques Le Roux]

Sorry Adrian,

What would be a "mashup engine" or "mashup services" ? Actually what is a mashup capability ?

Jacques

From: "Adrian Crum" <[hidden email]>

>I need to add mashup capability to my local OFBiz copy. My immediate
> need is a web page scraper that will gather copy counts from our copiers
> and store them as meter readings.
>
> I would like to implement this in a generic way that might be fed back
> into the project. I don't want to reinvent the wheel, however. So, I'm
> opening up the topic for discussion.
>
> Is there any interest in having something like a "mashup engine" or
> "mashup services" in the framework? If yes, what are some strategies we
> can use? Are there any open source libraries we can leverage in an
> implementation?
>
> -Adrian
>

[Adrian Crum]

Here's a good tutorial -

http://java.sun.com/developer/technicalArticles/J2EE/mashup_1/

-Adrian

Jacques Le Roux wrote:

> Sorry Adrian,
>
> What would be a "mashup engine" or "mashup services" ? Actually what is
> a mashup capability ?
>
> Jacques
>
> From: "Adrian Crum" <[hidden email]>
>> I need to add mashup capability to my local OFBiz copy. My immediate
>> need is a web page scraper that will gather copy counts from our
>> copiers and store them as meter readings.
>>
>> I would like to implement this in a generic way that might be fed back
>> into the project. I don't want to reinvent the wheel, however. So, I'm
>> opening up the topic for discussion.
>>
>> Is there any interest in having something like a "mashup engine" or
>> "mashup services" in the framework? If yes, what are some strategies
>> we can use? Are there any open source libraries we can leverage in an
>> implementation?
>>
>> -Adrian
>>
>

[Jacques Le Roux]

From: "Adrian Crum" <[hidden email]>
> Here's a good tutorial -
>
> http://java.sun.com/developer/technicalArticles/J2EE/mashup_1/

Interesting, thanks
Jacques

> -Adrian
>
> Jacques Le Roux wrote:
>> Sorry Adrian,
>>
>> What would be a "mashup engine" or "mashup services" ? Actually what is
>> a mashup capability ?
>>
>> Jacques
>>
>> From: "Adrian Crum" <[hidden email]>
>>> I need to add mashup capability to my local OFBiz copy. My immediate
>>> need is a web page scraper that will gather copy counts from our
>>> copiers and store them as meter readings.
>>>
>>> I would like to implement this in a generic way that might be fed back
>>> into the project. I don't want to reinvent the wheel, however. So, I'm
>>> opening up the topic for discussion.
>>>
>>> Is there any interest in having something like a "mashup engine" or
>>> "mashup services" in the framework? If yes, what are some strategies
>>> we can use? Are there any open source libraries we can leverage in an
>>> implementation?
>>>
>>> -Adrian
>>>
>>
>

[Peter Ford-5]

Adrian,

Looking forward to have progress on it.



On Wed, Jul 9, 2008 at 4:48 PM, Jacques Le Roux <
[hidden email]> wrote:

> From: "Adrian Crum" <[hidden email]>
>
>> Here's a good tutorial -
>>
>> http://java.sun.com/developer/technicalArticles/J2EE/mashup_1/
>>
>
> Interesting, thanks
> Jacques
>
>
>  -Adrian
>>
>> Jacques Le Roux wrote:
>>
>>> Sorry Adrian,
>>>
>>> What would be a "mashup engine" or "mashup services" ? Actually what is a
>>> mashup capability ?
>>>
>>> Jacques
>>>
>>> From: "Adrian Crum" <[hidden email]>
>>>
>>>> I need to add mashup capability to my local OFBiz copy. My immediate
>>>> need is a web page scraper that will gather copy counts from our copiers and
>>>> store them as meter readings.
>>>>
>>>> I would like to implement this in a generic way that might be fed back
>>>> into the project. I don't want to reinvent the wheel, however. So, I'm
>>>> opening up the topic for discussion.
>>>>
>>>> Is there any interest in having something like a "mashup engine" or
>>>> "mashup services" in the framework? If yes, what are some strategies we can
>>>> use? Are there any open source libraries we can leverage in an
>>>> implementation?
>>>>
>>>> -Adrian
>>>>
>>>>
>>>
>>

--
Peter Ford
USA

[Adrian Crum-2]

I created a basic service to retrieve external web pages and I also created a sample scraper - a form extraction service. It all seems to work well.

I implemented these as services so they can be used inside screen widget <actions> elements and in minilang.

I want to wait a little while before committing so I can get additional feedback from the community.

The sendExternalRequest service takes the following arguments:

extRequestUrl - the external URL to invoke
extRequestType - get, post, or empty (for REST type URLs)
extRequestParams - request parameters Map

It returns extResponse - the web page as a String.

The getFormParametersFromExtResponse service takes the following arguments:

extResponse - The response String
formId - The id attribute to search for
formName - The form name to search for

It returns:

extResponse - The original response String
formAction - The form's action attribute
formMethod - The form's method attribute
formFields - The form's input fields Map

The idea being you could modify the contents of the formFields Map and then use the returned values in another call to sendExternalRequest.

I picture the Mashup services working like a chain - you invoke the initial request, then take the response and pass it through various scrapers and/or XML element extractors.

Comments are welcome.

-Adrian

--- On Thu, 7/10/08, Peter Ford <[hidden email]> wrote:

> From: Peter Ford <[hidden email]>
> Subject: Re: Discussion: Mashups
> To: [hidden email]
> Date: Thursday, July 10, 2008, 1:23 AM
> Adrian,
>
> Looking forward to have progress on it.
>
>
>
> On Wed, Jul 9, 2008 at 4:48 PM, Jacques Le Roux <
> [hidden email]> wrote:
>
> > From: "Adrian Crum"
> <[hidden email]>
> >
> >> Here's a good tutorial -
> >>
> >>
> http://java.sun.com/developer/technicalArticles/J2EE/mashup_1/
> >>
> >
> > Interesting, thanks
> > Jacques
> >
> >
> >  -Adrian
> >>
> >> Jacques Le Roux wrote:
> >>
> >>> Sorry Adrian,
> >>>
> >>> What would be a "mashup engine" or
> "mashup services" ? Actually what is a
> >>> mashup capability ?
> >>>
> >>> Jacques
> >>>
> >>> From: "Adrian Crum"
> <[hidden email]>
> >>>
> >>>> I need to add mashup capability to my
> local OFBiz copy. My immediate
> >>>> need is a web page scraper that will
> gather copy counts from our copiers and
> >>>> store them as meter readings.
> >>>>
> >>>> I would like to implement this in a
> generic way that might be fed back
> >>>> into the project. I don't want to
> reinvent the wheel, however. So, I'm
> >>>> opening up the topic for discussion.
> >>>>
> >>>> Is there any interest in having something
> like a "mashup engine" or
> >>>> "mashup services" in the
> framework? If yes, what are some strategies we can
> >>>> use? Are there any open source libraries
> we can leverage in an
> >>>> implementation?
> >>>>
> >>>> -Adrian
> >>>>
> >>>>
> >>>
> >>
>
>
> --
> Peter Ford
> USA

     

[Jacques Le Roux]

Adrian,

Is there a reason why you did not create a Jira issue and attach a patch ?

Thanks

Jacques

From: "Adrian Crum" <[hidden email]>

>I created a basic service to retrieve external web pages and I also created a sample scraper - a form extraction service. It all
>seems to work well.
>
> I implemented these as services so they can be used inside screen widget <actions> elements and in minilang.
>
> I want to wait a little while before committing so I can get additional feedback from the community.
>
> The sendExternalRequest service takes the following arguments:
>
> extRequestUrl - the external URL to invoke
> extRequestType - get, post, or empty (for REST type URLs)
> extRequestParams - request parameters Map
>
> It returns extResponse - the web page as a String.
>
> The getFormParametersFromExtResponse service takes the following arguments:
>
> extResponse - The response String
> formId - The id attribute to search for
> formName - The form name to search for
>
> It returns:
>
> extResponse - The original response String
> formAction - The form's action attribute
> formMethod - The form's method attribute
> formFields - The form's input fields Map
>
> The idea being you could modify the contents of the formFields Map and then use the returned values in another call to
> sendExternalRequest.
>
> I picture the Mashup services working like a chain - you invoke the initial request, then take the response and pass it through
> various scrapers and/or XML element extractors.
>
> Comments are welcome.
>
> -Adrian
>
> --- On Thu, 7/10/08, Peter Ford <[hidden email]> wrote:
>
>> From: Peter Ford <[hidden email]>
>> Subject: Re: Discussion: Mashups
>> To: [hidden email]
>> Date: Thursday, July 10, 2008, 1:23 AM
>> Adrian,
>>
>> Looking forward to have progress on it.
>>
>>
>>
>> On Wed, Jul 9, 2008 at 4:48 PM, Jacques Le Roux <
>> [hidden email]> wrote:
>>
>> > From: "Adrian Crum"
>> <[hidden email]>
>> >
>> >> Here's a good tutorial -
>> >>
>> >>
>> http://java.sun.com/developer/technicalArticles/J2EE/mashup_1/
>> >>
>> >
>> > Interesting, thanks
>> > Jacques
>> >
>> >
>> >  -Adrian
>> >>
>> >> Jacques Le Roux wrote:
>> >>
>> >>> Sorry Adrian,
>> >>>
>> >>> What would be a "mashup engine" or
>> "mashup services" ? Actually what is a
>> >>> mashup capability ?
>> >>>
>> >>> Jacques
>> >>>
>> >>> From: "Adrian Crum"
>> <[hidden email]>
>> >>>
>> >>>> I need to add mashup capability to my
>> local OFBiz copy. My immediate
>> >>>> need is a web page scraper that will
>> gather copy counts from our copiers and
>> >>>> store them as meter readings.
>> >>>>
>> >>>> I would like to implement this in a
>> generic way that might be fed back
>> >>>> into the project. I don't want to
>> reinvent the wheel, however. So, I'm
>> >>>> opening up the topic for discussion.
>> >>>>
>> >>>> Is there any interest in having something
>> like a "mashup engine" or
>> >>>> "mashup services" in the
>> framework? If yes, what are some strategies we can
>> >>>> use? Are there any open source libraries
>> we can leverage in an
>> >>>> implementation?
>> >>>>
>> >>>> -Adrian
>> >>>>
>> >>>>
>> >>>
>> >>
>>
>>
>> --
>> Peter Ford
>> USA
>
>
>
>

[Adrian Crum-2]

I didn't implement it in the trunk code, it is in my custom application. Let me find a place to put it in the trunk, then I'll create a patch and upload it to Jira.

-Adrian


--- On Sat, 7/12/08, Jacques Le Roux <[hidden email]> wrote:

> From: Jacques Le Roux <[hidden email]>
> Subject: Re: Discussion: Mashups
> To: [hidden email], [hidden email]
> Date: Saturday, July 12, 2008, 11:26 AM
> Adrian,
>
> Is there a reason why you did not create a Jira issue and
> attach a patch ?
>
> Thanks
>
> Jacques
>
> From: "Adrian Crum" <[hidden email]>
> >I created a basic service to retrieve external web
> pages and I also created a sample scraper - a form
> extraction service. It all
> >seems to work well.
> >
> > I implemented these as services so they can be used
> inside screen widget <actions> elements and in
> minilang.
> >
> > I want to wait a little while before committing so I
> can get additional feedback from the community.
> >
> > The sendExternalRequest service takes the following
> arguments:
> >
> > extRequestUrl - the external URL to invoke
> > extRequestType - get, post, or empty (for REST type
> URLs)
> > extRequestParams - request parameters Map
> >
> > It returns extResponse - the web page as a String.
> >
> > The getFormParametersFromExtResponse service takes the
> following arguments:
> >
> > extResponse - The response String
> > formId - The id attribute to search for
> > formName - The form name to search for
> >
> > It returns:
> >
> > extResponse - The original response String
> > formAction - The form's action attribute
> > formMethod - The form's method attribute
> > formFields - The form's input fields Map
> >
> > The idea being you could modify the contents of the
> formFields Map and then use the returned values in another
> call to
> > sendExternalRequest.
> >
> > I picture the Mashup services working like a chain -
> you invoke the initial request, then take the response and
> pass it through
> > various scrapers and/or XML element extractors.
> >
> > Comments are welcome.
> >
> > -Adrian
> >
> > --- On Thu, 7/10/08, Peter Ford
> <[hidden email]> wrote:
> >
> >> From: Peter Ford <[hidden email]>
> >> Subject: Re: Discussion: Mashups
> >> To: [hidden email]
> >> Date: Thursday, July 10, 2008, 1:23 AM
> >> Adrian,
> >>
> >> Looking forward to have progress on it.
> >>
> >>
> >>
> >> On Wed, Jul 9, 2008 at 4:48 PM, Jacques Le Roux
> <
> >> [hidden email]> wrote:
> >>
> >> > From: "Adrian Crum"
> >> <[hidden email]>
> >> >
> >> >> Here's a good tutorial -
> >> >>
> >> >>
> >>
> http://java.sun.com/developer/technicalArticles/J2EE/mashup_1/
> >> >>
> >> >
> >> > Interesting, thanks
> >> > Jacques
> >> >
> >> >
> >> >  -Adrian
> >> >>
> >> >> Jacques Le Roux wrote:
> >> >>
> >> >>> Sorry Adrian,
> >> >>>
> >> >>> What would be a "mashup
> engine" or
> >> "mashup services" ? Actually what is a
> >> >>> mashup capability ?
> >> >>>
> >> >>> Jacques
> >> >>>
> >> >>> From: "Adrian Crum"
> >> <[hidden email]>
> >> >>>
> >> >>>> I need to add mashup capability
> to my
> >> local OFBiz copy. My immediate
> >> >>>> need is a web page scraper that
> will
> >> gather copy counts from our copiers and
> >> >>>> store them as meter readings.
> >> >>>>
> >> >>>> I would like to implement this in
> a
> >> generic way that might be fed back
> >> >>>> into the project. I don't
> want to
> >> reinvent the wheel, however. So, I'm
> >> >>>> opening up the topic for
> discussion.
> >> >>>>
> >> >>>> Is there any interest in having
> something
> >> like a "mashup engine" or
> >> >>>> "mashup services" in
> the
> >> framework? If yes, what are some strategies we can
> >> >>>> use? Are there any open source
> libraries
> >> we can leverage in an
> >> >>>> implementation?
> >> >>>>
> >> >>>> -Adrian
> >> >>>>
> >> >>>>
> >> >>>
> >> >>
> >>
> >>
> >> --
> >> Peter Ford
> >> USA
> >
> >
> >
> >