[TYPO3-english] Sponsorship for mnogosearch + restricted content

[Georg Ringer-2]

Hi everybody,

I am searching for some people and companies who are interested in a
great feature for the extension mnogosearch [1]: Indexing restricted
content.

--- Skip if you know the extension already -------
This is a great search engine which is very fast and simple to use (but
needs to be compiled on the server)

Examples:
[2] Blog of Dmitry
[3] Project I did with 15.000 pages, several thousand other records, ...
--------------------------------------------------

Missing Feature is being able to index also restricted content and show
only the results which fit the access of the current user. This is not a
very easy task and therefore Dmitry himself is the perfect guy doing it.

There is just one thing missing: YOU and your money!

Seriously - not everything can be done for free or just for fun!

So if you need this feature, want to have it or just wanna do something
good please step forward and say so! It is not yet the time to speak
about the sum which is necessary but to get all the people together who
are interested in it!

Thanks a lot for reading and helping
Georg
- inspiring people to share -

---
[1] http://typo3.org/extensions/repository/view/mnogosearch/current/
[2] http://dmitry-dulepov.com/
[3] http://www.wandern.com/suche.html


_______________________________________________
TYPO3-english mailing list
TYPO3-english@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-english

[Peter Russ]

--- Original Nachricht ---
Absender:   Georg Ringer
Datum:       11.09.2009 13:06:

> Hi everybody,
>
> I am searching for some people and companies who are interested in a
> great feature for the extension mnogosearch [1]: Indexing restricted
> content.
>
> --- Skip if you know the extension already -------
> This is a great search engine which is very fast and simple to use (but
> needs to be compiled on the server)
>
> Examples:
> [2] Blog of Dmitry
> [3] Project I did with 15.000 pages, several thousand other records, ...
> --------------------------------------------------
>
> Missing Feature is being able to index also restricted content and show
> only the results which fit the access of the current user. This is not a
> very easy task and therefore Dmitry himself is the perfect guy doing it.
>
> There is just one thing missing: YOU and your money!
>
> Seriously - not everything can be done for free or just for fun!
>
> So if you need this feature, want to have it or just wanna do something
> good please step forward and say so! It is not yet the time to speak
> about the sum which is necessary but to get all the people together who
> are interested in it!
>
> Thanks a lot for reading and helping
> Georg
> - inspiring people to share -
>
> ---
> [1] http://typo3.org/extensions/repository/view/mnogosearch/current/
> [2] http://dmitry-dulepov.com/
> [3] http://www.wandern.com/suche.html
>
>

Hallo Georg,

I disagree, that's "not time to speak about sum"!
If it takes e.g. 1k€ it's easier to get 10 people to sponsor than with
1M€ ;-)

As Dimitry might be the only one how can give an estimation on the
required "ressources", I would higly appreciate his input on this matter.

Regs. Peter.

--
Fiat lux! Docendo discimus.
_____________________________
uon GbR

http://www.uon.li
http://www.xing.com/profile/Peter_Russ
_______________________________________________
TYPO3-english mailing list
TYPO3-english@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-english

[Dmitry Dulepov]

Hi!

Georg Ringer wrote:
> I am searching for some people and companies who are interested in a
> great feature for the extension mnogosearch [1]: Indexing restricted
> content.

This feature is technically difficult to implement. I can have a look at it but it will take significant time even for the research. I am sure that the solution exists but it needs to be found. Therefore we have two separate processes:
- finding a solution
- developing it

The cost of the second can be estimated only after the first is completed. I can estimate the cost of the first if there is many enough people who want this feature.

So who wants this feature? Please, speak up.

--
Dmitry Dulepov
Facebook: http://www.facebook.com/dmitryd
Twitter: http://twitter.com/dmitryd
Skype: liels_bugs
_______________________________________________
TYPO3-english mailing list
TYPO3-english@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-english

[Wiel, J.A.M. van de]

Hi Dmitry,
Personally I still want this for our website.

..sadly I'm not the one with the budget around here so being able to give "the suits" some sort of rough ballpark figure would be very helpful to my efforts of selling this idea internally.

Bas
________________________________________
From: typo3-english-bounces@... [typo3-english-bounces@...] On Behalf Of Dmitry Dulepov [dmitry.dulepov@...]
Sent: Friday, September 11, 2009 6:41 PM
To: typo3-english@...
Subject: Re: [TYPO3-english] Sponsorship for mnogosearch + restricted content

Hi!

Georg Ringer wrote:
> I am searching for some people and companies who are interested in a
> great feature for the extension mnogosearch [1]: Indexing restricted
> content.

This feature is technically difficult to implement. I can have a look at it but it will take significant time even for the research. I am sure that the solution exists but it needs to be found. Therefore we have two separate processes:
- finding a solution
- developing it

The cost of the second can be estimated only after the first is completed. I can estimate the cost of the first if there is many enough people who want this feature.

So who wants this feature? Please, speak up.

--
Dmitry Dulepov
Facebook: http://www.facebook.com/dmitryd
Twitter: http://twitter.com/dmitryd
Skype: liels_bugs
_______________________________________________
TYPO3-english mailing list
TYPO3-english@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-english
_______________________________________________
TYPO3-english mailing list
TYPO3-english@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-english

[Olivier Dobberkau]

On 11.09.09 13:06, Georg Ringer wrote:
>
> I am searching for some people and companies who are interested in a
> great feature for the extension mnogosearch [1]: Indexing restricted
> content.

hi.

just a silly question? how does indexed search handle this?
cant you hook into this? or how does mongosearch create the index document?

olivier
_______________________________________________
TYPO3-english mailing list
TYPO3-english@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-english

[Dmitry Dulepov]

Hi!

Olivier Dobberkau wrote:
> just a silly question? how does indexed search handle this?

Indexed search works inside TYPO3 while external search engines work outside. Therefore indexed search is able to fake FE user group while none of external search engines can do it and have the same URL in the search results.

> cant you hook into this? or how does mongosearch create the index document?

Mnogosearch has two ways to index documents: using HTML (indexes content only, skips menu and the rest of repeating parts) and records (like tt_news or FAQ).

--
Dmitry Dulepov
Facebook: http://www.facebook.com/dmitryd
Twitter: http://twitter.com/dmitryd
Skype: liels_bugs
_______________________________________________
TYPO3-english mailing list
TYPO3-english@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-english

[Ralf Hettinger]

Thanks Georg, for pushing this further.

I'm interested - looking forward to Dmitry's estimate of part I.

Regards
Ralf

Am 11.09.2009 13:06 schrieb Georg Ringer:

> Hi everybody,
>
> I am searching for some people and companies who are interested in a
> great feature for the extension mnogosearch [1]: Indexing restricted
> content.
>
> --- Skip if you know the extension already -------
> This is a great search engine which is very fast and simple to use (but
> needs to be compiled on the server)
>
> Examples:
> [2] Blog of Dmitry
> [3] Project I did with 15.000 pages, several thousand other records, ...
> --------------------------------------------------
>
> Missing Feature is being able to index also restricted content and show
> only the results which fit the access of the current user. This is not a
> very easy task and therefore Dmitry himself is the perfect guy doing it.
>
> There is just one thing missing: YOU and your money!
>
> Seriously - not everything can be done for free or just for fun!
>
> So if you need this feature, want to have it or just wanna do something
> good please step forward and say so! It is not yet the time to speak
> about the sum which is necessary but to get all the people together who
> are interested in it!
>
> Thanks a lot for reading and helping
> Georg
> - inspiring people to share -
>
> ---
> [1] http://typo3.org/extensions/repository/view/mnogosearch/current/
> [2] http://dmitry-dulepov.com/
> [3] http://www.wandern.com/suche.html
>
>

_______________________________________________
TYPO3-english mailing list
TYPO3-english@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-english

[Dmitry Dulepov]

Hi!

Ralf Hettinger wrote:
> I'm interested - looking forward to Dmitry's estimate of part I.

I think it should take some thinking that I can do on the background after the 15th of October. After thinking I will need to look deeper into some issues and that will take 300-400€. I must tell that it does not guarantee the result. But if it gives a solution, it can be also a solution for other alternatives, such as Lucene, Sphynx or Solr. I will think first and only if I have ideas, we will need to get the money for it. No ideas, no need for money. I think it is fair enough.

This all is about phase 1 only.

--
Dmitry Dulepov
Facebook: http://www.facebook.com/dmitryd
Twitter: http://twitter.com/dmitryd
Skype: liels_bugs
_______________________________________________
TYPO3-english mailing list
TYPO3-english@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-english

[Ralf Hettinger]

Sounds more than fair enough to me.

Thank you - and thanks for giving us a date :)
Looking forward to middle of October then.

Regards
Ralf

Am 16.09.2009 21:16 schrieb Dmitry Dulepov:
> Hi!
>
> Ralf Hettinger wrote:
>> I'm interested - looking forward to Dmitry's estimate of part I.
>
> I think it should take some thinking that I can do on the background after the 15th of October. After thinking I will need to look deeper into some issues and that will take 300-400€. I must tell that it does not guarantee the result. But if it gives a solution, it can be also a solution for other alternatives, such as Lucene, Sphynx or Solr. I will think first and only if I have ideas, we will need to get the money for it. No ideas, no need for money. I think it is fair enough.
>
> This all is about phase 1 only.
>
_______________________________________________
TYPO3-english mailing list
TYPO3-english@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-english

[Jigal van Hemert-3]

Hi,

Dmitry Dulepov wrote:
> guarantee the result. But if it gives a solution, it can be also a
> solution for other alternatives, such as Lucene, Sphynx or Solr. I will

d.k.d. seems to have a solution for both indexing and searching restricted
content according to user privileges in their new solr based search
extension (extbase extension) since they list it as one of the features.

Maybe this can somehow be used for Mnogosearch too?

Regards,
--
Jigal van Hemert.

_______________________________________________
TYPO3-english mailing list
TYPO3-english@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-english

[Dmitry Dulepov]

Hi!

Jigal van Hemert wrote:
> d.k.d. seems to have a solution for both indexing and searching restricted
> content according to user privileges in their new solr based search
> extension (extbase extension) since they list it as one of the features.
>
> Maybe this can somehow be used for Mnogosearch too?

Do you mean the information from the podcast? I heard they ~plan~ to do it but I do not know if they did. There are no docs and no ChangeLog. I had a quick look to the extension, there are many classes but I do not fancy digging into them right now. I think I have ideas already how to make user-based restrictions. I will need to prove them in October.

I must tell certain things about mnoGoSearch that are currently different from Solr:
- it stays compatible with 4.2 (not Extbase-d)
- it does not require Java enterprise server
- it is fully documented and free (there is no "private" version)
- anyone can use it by simply following the manual, which describes everything in details

In general, I think it is good that we have indexed search, mnogosearch, lucene and now solr. The more the better. Everyone can choose his preferred solution, which is great! Who knows, may be I have used Lucene if I did not find mnoGoSearch first :)

--
Dmitry Dulepov
Facebook: http://www.facebook.com/dmitryd
Twitter: http://twitter.com/dmitryd
Skype: liels_bugs
_______________________________________________
TYPO3-english mailing list
TYPO3-english@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-english

[Olivier Dobberkau]

On 17.09.09 16:40, Dmitry Dulepov wrote:

> it is fully documented and free (there is no "private" version)

hello.

just to make this clear. the private version pays the community edition.

documentation will come. and the solr usecase is cetainly different.

for those interested in code:

http://forge.typo3.org/repositories/entry/extension-solr/trunk/classes/class.tx_solr_indexer.php

search for the access part.

many greetings,

olivier
_______________________________________________
TYPO3-english mailing list
TYPO3-english@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-english

[Feodor Rykhtik]

Hi,

It will be interesting to see the solution of Dmitry. I thought about
something like this in putting additional "security" meta marker in
each page, which lists the groups, and to use this part of meta data
during search request.

About mnoGoSearch VS Solr : it's different categories. The biggest
positive stuff about mng : in Debian Lenny (I think in similar distros
also) - it's completely easy to install the search engine, and You
don't need to have standalone VM/server for that. Solr is different :
there is more features, more cool stuff, it's very cool what d.k.d
made the community version.

Best Regards,
Fedir
_______________________________________________
TYPO3-english mailing list
TYPO3-english@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-english

[Ingo Renner-2]

Dmitry Dulepov wrote:

Hi to all interested in search engines (and especially Solr!)


> Do you mean the information from the podcast? I heard they ~plan~ to do it but I do not know if they did. There are no docs and no ChangeLog. I had a quick look to the extension, there are many classes but I do not fancy digging into them right now.

I think I have to clarify some things here:

* We actually _did_ implement searching for access restricted content,
and it's not as complicated as you make people think it is. :)

* documentation will follow, maybe this weekend, otherwise next week.
It's surely not complicated to set up Solr, there're even some scripts
provided with the extension that help you do that.

Here's a quick guide though:

1. Set up a J2EE container (Jetty is easy, but Tomcat works too of course)
2. Set up Solr (1. and 2. are covered by a script in
EXT:solr/resources/shell)
3. Copy schema.xml and solrconfig.xml to the Solr configuration directory
4. Install EXT:solr
5. Set the Solr connection in plugin.tx_solr.solr
6. Don't forget about config.index_enable
7. That's it, nothing more to do :)

* The many classes may be a good indication for structured and modular
code ;)

> I must tell certain things about mnoGoSearch that are currently different from Solr:
> - it stays compatible with 4.2 (not Extbase-d)

* EXT:solr is _not_ based on EXT:extbase
* However, the published version works with TYPO3 4.3 only

> - it does not require Java enterprise server
> - it is fully documented and free (there is no "private" version)

* EXT:solr is free, so is our so called "private" version.
The only difference is that the released version does not offer all the
features we have in our "private" version. These features are missing
either because they're not completely ready yet or because they didn't
get enough sponsorship to be released yet.

However, we've set ourselves a kind of "community contract": Whenever
the community provides a bigger contribution to the released version,
we'll in return contribute another feature on top of that.

Missing features in the community version:

* Auto Suggest
* More Like This
* Compatibility with TYPO3 versions 4.1 - 4.3

It's also easy to get access to the "private" version:
Simply contribute to the development of EXT:solr, you'll then get early
access to all features, setup support, and influence on what get's added
to the extension with which priority that will later be available in the
community version, too. It's just a kind of sponsorship model to ensure
the development of the extension.


I hope this clears some things up

all the best
Ingo

--
Ingo Renner
TYPO3 Core Developer, Release Manager TYPO3 4.2

_______________________________________________
TYPO3-english mailing list
TYPO3-english@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-english

[Dmitry Dulepov]

Hi!

Ingo Renner wrote:
> * We actually _did_ implement searching for access restricted content,
> and it's not as complicated as you make people think it is. :)

Basically you accuse me of lying to people, right?

--
Dmitry Dulepov
Facebook: http://www.facebook.com/dmitryd
Twitter: http://twitter.com/dmitryd
Skype: liels_bugs
_______________________________________________
TYPO3-english mailing list
TYPO3-english@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-english

[Jigal van Hemert-3]

Ingo Renner wrote:
> * We actually _did_ implement searching for access restricted content,
> and it's not as complicated as you make people think it is. :)

Two things:

1. As far as I could see in the source code of the 'Apache solr search'
extension only access restricted pages are implemented. There was a
remark with TODO check restrictions for content elements. Correct?

2. Indexing with solr is done from the TYPO3 extension. It converts
TYPO3 pages to solr documents and sends them to the solr http interface.
Indexing with mnogosearch is done from the mnogosearch application. This
makes it a lot more complicated to let it access content which can
normally only be seen by logged in users. The PHP module for mnogosearch
only contains functions to search information.

> * documentation will follow, maybe this weekend, otherwise next week.
Documentation is really a must have for any extension... :-)

Both extensions do need to implement some 'must have' features in order
to become the search engines which todays clients require. Thanks to big
search applications such as Google people are much more demanding when
it comes to a search box on their web site.

Oh, and please don't fight! Share :-)

Regards,
--
Jigal van Hemert.
_______________________________________________
TYPO3-english mailing list
TYPO3-english@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-english

[Dmitry Dulepov]

Hi!

Jigal van Hemert wrote:
> Both extensions do need to implement some 'must have' features in order
> to become the search engines which todays clients require.

This is too abstract to cause any action from my side. Unless you have something like a list of wanted features, I cannot do anything :)

> Oh, and please don't fight! Share :-)

I do not see a reason or need for fighting. I do not do business with mnogosearch and I see no need to compete with anyone or market mnogosearch as the best solution ever. I use it personally and this is what matters for me. If others find it useful, cool for me but not more than that.

Regarding DKD and Solr. DKD is a great company, they did a lot of cool sites and they want to do business with Solr. Why not? They are there to earn money and they chose Solr for reasons only they know and Ingo did a good job on Solr TYPO3 extension. There is no reason for fighting, I think DKD should get their reward for a good work through new and more customers :) I am sure they can support their clients' needs very well with Solr. It is good for everybody because it works for TYPO3's good too!

Regarding me and mnogosearch. Originally I implemented mnogosearch for my own projects because 5 minutes of waiting for a single search with indexed search was totally inadequate. Mnogosearch gave me 0.003 sec on the same data (nearly 30,000 records that time and over 1,500,000 now with nearly the same time). Mnogosearch works perfectly for me: sections, synonyms, word grammar forms, spelling check, speed — this is what I required at that project and got easily. It has far more features that I use or implemented for TYPO3. These features are easily accessible for anyone who can read official mnogosearch docs at mnogosearch.org. The rest of existing extension features (like record indexing or file indexing) was implemented by request from TYPO3 users. Since I have everything that I need personally, I can only add features that people ask specifically (like restricted content search). I cannot implement the feature named "must have" because it is not really a feature descript
ion :) Give me a list and I will see if I have time and wish to do it. It is that simple.

--
Dmitry Dulepov
Facebook: http://www.facebook.com/dmitryd
Twitter: http://twitter.com/dmitryd
Skype: liels_bugs
_______________________________________________
TYPO3-english mailing list
TYPO3-english@...
http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-english

[Ralf Hettinger]

Thank you for this update. I'll stay tuned :)

Cheers
Ralf

Am 28.10.2009 12:20 schrieb Dmitry Dulepov:

> Hi!
>
> Georg Ringer wrote:
>> I am searching for some people and companies who are interested in a
>> great feature for the extension mnogosearch [1]: Indexing restricted
>> content.
>
> To give you an update... Currently I am investigating this but it is a very difficult problem :( It is more difficult than I thought. Here is a little background about problems we have.
>
> Mnogosearch indexes URLs. It is exactly like Google: one URL equals to one page. TYPO3 can have many different pages with the same URL due to user groups. This does not fir to mnogosearch at all. Therefore mnogosearch has to be cheated in some way. For example, it can have different URLs for the same page depending on the user group. Then, at the search time, URLs have to be normalized to their typical look. This is possible and not a big problem.
>
> There are two other problems that make the solution almost impossible.
>
> The first problem is that content elements can belong to various groups. Let's assume that there is anonymous content, content that belong to group #1 and content that belong to #2. It means that we have to index the same page four times (anonymous, anonymous + #1, anonymous + #2, anonymous + #1 + #2). That is ok. Now how do we tell mnogosearch what kind of result do we need to fetch? We could use tags or categories. This would work but not reliabely. And there is another problem: duplicate search results.
>
> This final problem is the worse one. Suppose we search for a word that appears in the anonymous content from the example above. If user is in groups #1 and #2, it will give us four results: anonymous page, anonymous + #1, anonymous + #2, anonymous + #1 + #2. These are all different pages to mnogosearch. They have to be filtered out to produce proper results. And this task is not solvable at the moment.
>
> I afraid having a proper solution for handling permissions at content element level is not possible at the moment.
>
> Now I am going to look to access control only at page level. Hopefully this will be easier. Stay tuned.
>

_______________________________________________
TYPO3-english mailing list
TYPO3-english@...
http://lists.typo3.org/cgi-bin/mailman/listinfo/typo3-english