I think, a good architecture for Otter would be a modular one with local servers to which commands can be sent, similar to X on Unix systems. As far as I understand it, X usually runs as a local server on a Unix workstation and programs written to use X for graphical output use it to display their user interface. As I understand Otter, it uses engines that can draw formatted web pages from HTML files and displays the result in a prominent area of its program window. Since the decorations, that Otter places around the naked rendering area, might not be ideal for every environment, it would be nice to separate the control elements from the canvas. From what I've read, this is already useful for full-screen mode. It could also be useful for different user interfaces, for instance one optimized for touch screens.
Now, to protect against accusations of being bloated, how about presenting Otter as simply a lean interface to control those specialized engines, that people would like to be part of their browser? Ideally, this lean interface could be freely configured in its visual characteristcs by the users themselves. The control interface should send messages to those components, that provide the actual functionality, via some communication facility. Sockets seem to be a standard abstraction that components can use to connect to each other and then communicate via some protocol. Network sockets in particular would allow to send commands from anywhere. It would allow, for instance, to have a control interface on your smartphone and the rendering canvas on your monitor. I don't know, if people would use this to watch web videos from their couch, but it could also be used for other things like adding torrents to your Otter download component running on your computer at home from some other place where you happen to be. Also, Otter would not be bound to one user interface. Adapting Otter to the holographic 3D interfaces, that might be in fashion ten years from now, would be much simpler, if the components, that Otter's current user interface controls, could be controlled through messages rather than called through hard-coded mappings of buttons to procedures.
P.S.: Thinking about it, X on Unix systems might not have been the most representative thing to illustrate the concept. I hope I could make clear what I meant with the subsequent explanations.
I don't think there are binary packages just yet. All I've seen is are packages for Qt 4.x. I have yet to attempt a full build on my computer, but it looks as if building everything from scratch might be the most fail-safe option. Do you know of any memory requirements to build Qt and other dependencies of Otter? I read somewhere some time ago, that it is difficult to built Firefox on some machines, because it requires outrageous amounts of RAM.
It would be nice, if you could prepare a fail-safe make script, ideally somewhat configurable to test several settings, that builds Otter and all its dependencies from scratch. I would like to build it on an unused web server. This would probably be far from fast, but since the server could built Otter all the time, it would allow daily builds, unless building takes more than 24 hours. Users who want to test daily builds could download them from there. For that, I would have to investigate how to automate the process.
OK, I ran the debugger gdb and it said there are no debug symbols in Otter and Qt Creator. Backtrace was short, for Otter it was:
Program received signal SIGSEGV, Segmentation fault. #0 0xb137a445 in XGetXCBConnection () from /usr/local/lib/libX11-xcb.so.1 #1 0x00000000 in ?? ()
For Qt Creator, it was almost the same, except that there was only line #0 and no line #1 and the first number (memory address?) in line #0 was different. Both apparently failed in XGetXCBConnection () from /usr/local/lib/libX11-xcb.so.1.
I guess I need to find some other setup. While I was able to install Qt Creator from the provided installer and compiling Otter (after installing Xorg and gst-plugins-base and libxslt from Tiny Core Linux's package manager), trying to start either of Qt Creator or Otter fails immediately with the notice "Segmentation fault". Is there any way to get a more detailed report about the problem than that?
Note also, that the instructions in Otter's INSTALL file did not work for me as described there. "qmake" could not be found, so I looked for it in the folder that the Qt Creator installer created and found there a qmake file in a gcc/bin/ folder, which I then used with its full path instead. "make" didn't work either. So I installed GNU Make from Tiny Core Linux's package manager and entered the command again. This lead to "g++" not found. So I installed the "compiletc" package from Tiny Core Linux's package manager, which installs a couple of packages for a build environment, like gcc, make, patch and somesuch. After doing that, I tried again and compilation started. Building Otter took about ten minutes on my computer, which uses an Intel Atom D525 (http://ark.intel.com/products/49490) as its processor.
I'm not sure, whether the Qt Creator installer also installed its own compiler that I could have used, and if so, how? Would I have to pass some information about its existence to the "make" program? Or do I have to define some environment variable? How would that be done? Sorry, if these appear to be trivial questions, I'm not that familiar with how to do advanced things on Linux. Windows makes it easier to create environment variables and configure other things, since graphic tools exist, where you can click on what looks like it could be what you want.
The compilation environment I installed from Tiny Core Linux's package manager might have come with different configurations than Qt Creator's own compiler, if one was included. I think, for instance, that Tiny Core Linux uses eglibc by default as its C library. I'm not sure, if Qt uses any C libraries and assumes, perhaps, that it can call gclibc. Could this be a cause for Otter and Qt Creator immediately quitting with "Segmentation fault"? Note, that Qt Creator's graphical installer program itself ran fine.
Anyway, I see the ten minutes it took to compile Otter as a confirmation to my wish to find an already compiled package of Otter for download. Casual users might otherwise find installation too lengthy, and if for some reason it doesn't work the first time, the software might be considered broken, causing people not to come back.
I have now downloaded the latest Qt Creator and installed. I'll soon try to build Otter. Since there are weekly releases now, is there some place where to find the weekly release's source code? I didn't see a file on the sourceforge download page. Oh, and in case you intend to try to install Qt Creator on Tiny Core Linux, I found that some libgl or so file was missing. Installing the Xorg package made the error message go away.
OK, I've now looked into the AdBlock filter documentation. The rules presented there seem to make writing a filter rule file easy enough. Will Otter implement its own interpreter for these rules? How and when will URLs be checked against the rules in a filter file? I wonder how element hiding will work. Does Otter fetch the HTML file from the server, does some work on it, and then feeds the processed character stream to the rendering engine, or does WebKit do more work on its own?
Since such a thing as element hiding already exists in the AdBlock rules, I wonder whether this could be extended to some degree. Looking at HTTrack's filtering options (http://www.httrack.com/html/filters.html), I can see how, for instance, file-size based filtering or hiding could be added somehow to the AdBlock rules. However, AdBlock and HTTrack have different purposes. Filters based on file size might not be that relevant for an ad blocker.
However, what currently serves only content filtering could be of more general utility, if each URL has to be checked anyway. The rule matching could also be used to trigger special handling of certain URLs. For instance, users could specify that certain addresses must only be called via HTTPS instead of HTTP. I know that I have to be careful with my web mailer, lest I log in over the unencrypted interface. Since both encryted and unencrypted login sites are provided under the same URL, this would free me from having to manually type the protocol prefix, since it would default to HTTP, if I didn't.
Going further this path, it should also be possible to specify the secure protocol and cyphers to be used for individual sites. Currently, this cannot be done on a per-site basis in Opera, leading to the situation that some pages don't load, if you disable certain cyphers or protocol versions, while at the same time, some sites select cyphers in the order in which they appear in Opera's list, although the servers would actually support more recent cyphers, if the browser reports the older ones as unsupported to the server.
So I wonder where the best place would be to configure such things in the browser. It could be in the site specific settings, but having some way to specify rule based triggers seems to be attractive nonetheless.
I guess, it could be made selectable, which interpreter (and thus, which rule syntax) to use. That way, only one list would have to be kept in memory. I agree that it is not necessary to have multiple engines active at the same time right from the beginning. A modular approach for engine integration would be useful, so that people could plug in their own interpreters, if they want to experiment.
The warning-raising rule block was just a quick idea. Having some notice about blocked content on the site currently loaded would be useful (maybe show it for a few seconds in some corner). Still, it would be nice, if certain blocked content could be made "more visible" than other content, so there should be some way to mark those rules that you want to be more visible. URLs of blocked content marked for presentation in such a "more visible" way could perhaps be printed in bold or somesuch, when shown in an error console.
I agree that it would often be useful to have tabs with modified or disabled content blocker. I have certain sites that I usually block, but which I occasionally want to visit nonetheless. For that, I suggest named filter sets, i.e., have a way to give names to your filter rule sets. Which rule set to apply for a given tab could than be selected from a popup menu. If you wanted to maintain only one list in memory, however, you would have to think about extension rule sets, that would be concatenated to the list in memory and used for the tabs that use them. I think, the approach I presented in my previous post would already be good for that scheme, since pattern blocks would be checked in their order of appearance. Say, you normally want to block "example.com" and therefore have this rule in your standard rule set:
[exclude] *example.com*
Let's say, you now want to allow "example.com" in some tab but keep all your other rules intact. You could have this in an extension rule set:
[include] *example.com*
As you can see, if you concatenate this to your standard rule set and evaluate the rules in their order of appearance, "example.com" gets first excluded only to be finally included again. Not very efficient, but it is only for your current tab where you want to have that exception. The alternative to extension rule sets would be to have completely independent rule sets. In the end, there are probably use cases for both. So you should ideally be able to see and select in the popup menu for your tab both the independent rule sets and the extension rule sets and be able to load exactly one independent rule set and an arbitrary number of extension rule sets per tab to have full flexibility. To distingish the sets, their names could be shown in different colours or with some prefix prepended. So, in the popup menu, you could find, for instance (where "--" be a comment):
Standard -- your standard rule set Fancy -- some alternative rule set, that you have defined [E] allow foo.com -- some extension rule set, where you explicitly allow "foo.com"; "[E]" denotes "extension rule set" [E] allow blablub -- some other extension rule set, where you allow or deny some other patterns
The standard rule set would be selected by default for each tab. If you unselect the "Standard" entry in the popup menu, content blocking would be disabled entirely for the affected tab.
I don't know the AdBlock format but if several interpreters are possible, I'd like to see at least one simple thing like Opera's urlfilter.ini. I opened that file from Opera's profile folder and it looks like this:
The patterns are optionally followed by "=UUID:crypticsequence", which is concatenated to each pattern after you edit the rules from within Opera itself and save them. I don't know what the UUID is used for. If omitted, the filter still works. If you replace the "prioritize excludelist=1" with "prioritize excludelist=0", every URL that matches any pattern from the "[include]" block is allowed, regardless if the pattern also matches any pattern from the "[exclude]" block, and at the same time, patterns not included in the "[include]" block count as filtered out. This is rather useless, since all you can do with this scheme is to either allow everything or allow only certain patterns (whitelisting, since everything else is blocked). In Opera itself, you cannot change the "[include]" patterns at all, and the predefined pattern is "*". You can only add patterns that you want to exclude.
I suggest, that the useless "[prefs]" block be omitted and that the "[include]" and and "[exclude]" blocks are evaluated in order of appearance and that a sequence of "[include]" and "[exclude]" blocks be allowed. For instance, Opera itself does not care if an additional "[include]" block follows the "[exclude]" block, leading to the problem that you cannot specify exceptions to your "[exclude]" patterns. In the proposed scheme, an entered URL would be checked against the patterns in the corresponding blocks in the order of their appearance and treated accordingly. If a URL matches an "[exlude]" pattern, the URL is filtered and the engine can jump to the next "[include]" block to check whether an explicit inclusion rule overrides the previous exclusion rule and so on. That way, it would be possible to exclude "someurl.net" but to include "proxy.net/someurl.net" by specifying this properly. Note, that the example in my previous post was not very well chosen, since translated to the scheme suggested here, it would read:
[include] ; first rule block * [exclude] ; second rule block *foo.com* [include] ; third rule block (may include URLs previously excluded) *yourproxy.com*
The last block would actually also allow, for instance, "http://foo.com/somefoo/yourproxy.com", which is an unintended edge case, although "foo.com" would still be blocked in most cases (those, that don't contain "yourproxy.com"; you get the idea). You'd need to be careful with your rule specifications to cover all edge cases. That is why I said that it would ultimately also be desirable to specify filters for structural URL parts, where the filter would only target, for instance, the domain name, not the whole text string. This would be a convenience to avoid having to craft very complicated to decypher regular expressions. I don't think that a simple filter engine should actually implement full regular expression parsing, since that seems to be overkill for most cases. If at all, such an engine could still be implemented later.
As an addition, further block types could be introduced, like "[warn]", which could mean that the user shall be warned if some URL matches any pattern (and show which one) specified therein. This could be nice for debugging, if a user is uncertain whether some of his rules lead to collateral damage like blocking URLs that he didn't intend to block. Or possibly for some other purposes that you might come up with. These extra block types would, however, be merely "nice to have" and could be implemented later.
In the suggested scheme, whitelisting would be implemented as follows:
[exclude] ; everything is excluded, * [include] ; except the URLs that you explicitly specify pattern-1 pattern-2 .. pattern-n
@Somebody, uhm... Static build sounds nice, but it would be harder to accomplish, it would require statically compiled Qt which would take many hours to compile (and then the same amount for 32 bit build) due to enormous size of WebKit itself. For beta I would go for shipping .so files from SDK, static build could be reconsidered later.
OK, maybe break it into individual component blocks, such that large components like WebKit and Qt are included alongside Otter itself inside an archive. Otter could then be linked dynamically to the large dependencies, that you don't want to build each time, but the libraries necessary to run Otter would be delivered together with Otter, just like (I assume) it is done for the Windows version.
Opera allowed to block addresses by specifying a list of patterns to block. The annoying thing about that was, that it wasn't possible to specify exceptions from the patterns. Thus, some rules simply could not be specified. Also, each pattern was treated as a complete address string. You could not specify, for instance, that only the server name should be checked for the pattern. Example: You want to block everything from server "foo.com" but not block "yourproxy.com/foo.com". You couldn't specify this in Opera.
I hope that these limitations won't be in Otter. As a first easy solution, it should be possible to specify both inclusion and exclusion rules, which are applied in the order of appearance, just like filters are specified in httrack (http://www.httrack.com/). Thus, if "-" means "exclude" and "+" means "include", it should be possible to specify something like:
+"*" -- include everything -"*foo.com*" -- but exclude "foo.com" +"*yourproxy.com/*" -- but include, for instance, "yourproxy.com/foo.com"
Exact syntax is debatable, but in the above filter example, you might have guessed that "*" stands for "arbitrary length wildcard" and "--" for comments. However, the simple fact, that the rules are checked in order of appearance, means, that the filter would already be much more powerful than Opera's.
Implementing some way to specify rules that only apply to certain structural elements of the URL string, like domain or host names, could be delayed to the future.
@Somebody, yes, basic installer doesn't give any real advantages over basic ZIP file. But apparently some users like to have it anyway, for unknown reasons. ;-)
I could try to provide such packages (32 and 64 bit) for Linux, shipping only needed SO files, but I have no idea if it will work fine with all of the most prominent distributions...
I think, there is only one thing to decide for installation, and that is whether Otter's profile should be managed for each system user individually (i.e., a profile directory in User folder on Windows or in home directory on Unix) or centrally for all users (i.e., portably in some specified folder; I think the Windows Opera installer had that option for a portable installation).
When building packages for Linux, you should test them with a minimal distribution like Tiny Core Linux (http://tinycorelinux.net/), since it should work everywhere if it works there. All dependencies should be statically linked, lest there's any chance that some component is not available in some distribution. You should be able to use Tiny Core Linux without hassle alonside your normal installation, since it runs completely in RAM. You just need an empty USB stick, to which you can write their 72 MB CorePlus image with dd. To get a minimal build environment in a running Tiny Core Linux, the "compiletc" package can be used from their graphical "Apps" package manager.
@Luxor, I thought about this idea earlier but dismissed it since it could lead to malicious comments about otter that cannot swim. :-D
I can't imagine that happening and wouldn't be concerned about such a hypothetical misinterpretation. It would be a contradiction in itself. Otters are known to be good swimmers. A lifebelt would rather symbolize rescue from the sinking ship that is Opera. The otter would bring people that lifebelt.
Why is an installer needed? Is "installation" more complicated than unpacking an archive to some place?
Although this topic is about Windows, while at it, let me express that at least for the first beta version I hope for a ready-built executable binary for Linux, which does not seem to be provided for any alpha version. Is there a reason for that? Opera used to be available for Linux as one single file that could be run. Otter seems to suggest that users first load some several hundred MB large Qt-foo and spend an indefinite time to compile Otter from source for each release. This is not good.
