From german.mb at gmail.com Thu Apr 2 00:03:15 2015 From: german.mb at gmail.com (=?UTF-8?Q?Germ=C3=A1n_M=2E_Bravo?=) Date: Wed, 1 Apr 2015 17:03:15 -0600 Subject: [Xapian-devel] Remote protocol abstracted Message-ID: I have been working in the abstraction of the remote protocol to separate it from the server and the connection. The purpose of this is I need to use the Xapian remote protocol in a server we are working on (named Xapiand: https://github.com/Kronuz/Xapiand) which is similar to some extent to the `xapian-tcpsrv` in that it also serves databases remotely through TCP, using the binary protocol. What is different about it is that it can serve any number of databases using an event driven mechanism for receiving and attending clients. Additionally, it also serves clients using a custom HTTP protocol which is JSON based and RESTful. I have the following commits in my Xapian fork at https://github.com/Kronuz/xapian/commits/master: 1. Remote protocol with support for selection of database directory ------------------------------------------------------------------- If the user requests to open a remote database server, it can optionally require the remote backend to tell the server what database it wants to open, if the remote server is configured as a server without a default directory (xapian-tcpsrv remains to always have a default directory at the moment). The binary protocol was changed in the way that a new command can be sent to the server (MSG_SELECT) in case the server tells the client no default directory is selected, the server then reacts to this command by opening and selecting the requested database as the active database. 2. Split msg_query and msg_getmset ---------------------------------- Originally, `msg_query()` handled two commands by the client: `MSG_QUERY` and also `MSG_GETMSET` (which was marked as "used during a conversation"). This changes this and makes it so `msg_query()` returns even when the client hasn't yet sent a `MSG_GETMSET`, but instead only keeping the current satate of the query in the `MatchState` struct and setting the expected `required_type` for the next command. The purpose of this is that an event based remote protocol server might still not yet have the `MSG_GETMSET` message from the client as a response, and it'd be bad practice (or catastrophic) to block the thread while the message arrives. This blocking (if any) should be handled further up, by the event loop. 3. Method run() split in two: run() and run_one() ------------------------------------------------- The method run() calls run_one() in a loop, but we don't always want this behavior. The reason is, again, event based messaging might decide to start attending other clients on the same thread without blocking, thus an infinite while for these type of servers is not desired. 4. Abstract RemoteProtocol class added -------------------------------------- RemoteServer extends from abstract RemoteProtocol, and only implements a few members needed by the protocol. This effectively separates the protocol from the connection and the server. RemoteProtocol with virtual abstract methods for: * get_message - Receives a new message (waiting when needed). * send_message - Sends a message to the client. * get_db - Reserves a database when the class needs to use it. * release_db - Releases the database when the class is not using it. * select_db - It's called when the protocol needs to select a new database to work on. * shutdown - It's called when the protocol requires to shutdown. Each command in the RemoteProtocol now do a two additional things: 1. Use get_db() to reserve and get a read-only or writable database. 2. Relese the reserved database using release_db() Implements RemoteProtocol on the RemoteServer: * get_db - Always returns the same database object (db or wdb) * release_db - Does nothing in this server (database is never released). * get_message and send_message - Call RemoteConnection methods. 5. RemoteProtocol implementation moved to remoteprotocol.cc ----------------------------------------------------------- Added `remoteprotocol.cc` with the implementation of `RemoteProtocol` and header file remoteprotocol.h with added `RemoteProtocol` abstract class definition moved to `include/xapian/remoteprotocol.h` The reason for this is other projects might want to use the `RemoteProtocol` class for implementing a custom remote server. Xapiand is one of these projects. (https://github.com/Kronuz/Xapiand) Please, any comments and feedback would be appreciated. German M. Bravo (Kronuz) -------------- next part -------------- An HTML attachment was scrubbed... URL: From german.mb at gmail.com Thu Apr 2 00:03:15 2015 From: german.mb at gmail.com (=?UTF-8?Q?Germ=C3=A1n_M=2E_Bravo?=) Date: Wed, 1 Apr 2015 17:03:15 -0600 Subject: [Xapian-devel] Remote protocol abstracted Message-ID: I have been working in the abstraction of the remote protocol to separate it from the server and the connection. The purpose of this is I need to use the Xapian remote protocol in a server we are working on (named Xapiand: https://github.com/Kronuz/Xapiand) which is similar to some extent to the `xapian-tcpsrv` in that it also serves databases remotely through TCP, using the binary protocol. What is different about it is that it can serve any number of databases using an event driven mechanism for receiving and attending clients. Additionally, it also serves clients using a custom HTTP protocol which is JSON based and RESTful. I have the following commits in my Xapian fork at https://github.com/Kronuz/xapian/commits/master: 1. Remote protocol with support for selection of database directory ------------------------------------------------------------------- If the user requests to open a remote database server, it can optionally require the remote backend to tell the server what database it wants to open, if the remote server is configured as a server without a default directory (xapian-tcpsrv remains to always have a default directory at the moment). The binary protocol was changed in the way that a new command can be sent to the server (MSG_SELECT) in case the server tells the client no default directory is selected, the server then reacts to this command by opening and selecting the requested database as the active database. 2. Split msg_query and msg_getmset ---------------------------------- Originally, `msg_query()` handled two commands by the client: `MSG_QUERY` and also `MSG_GETMSET` (which was marked as "used during a conversation"). This changes this and makes it so `msg_query()` returns even when the client hasn't yet sent a `MSG_GETMSET`, but instead only keeping the current satate of the query in the `MatchState` struct and setting the expected `required_type` for the next command. The purpose of this is that an event based remote protocol server might still not yet have the `MSG_GETMSET` message from the client as a response, and it'd be bad practice (or catastrophic) to block the thread while the message arrives. This blocking (if any) should be handled further up, by the event loop. 3. Method run() split in two: run() and run_one() ------------------------------------------------- The method run() calls run_one() in a loop, but we don't always want this behavior. The reason is, again, event based messaging might decide to start attending other clients on the same thread without blocking, thus an infinite while for these type of servers is not desired. 4. Abstract RemoteProtocol class added -------------------------------------- RemoteServer extends from abstract RemoteProtocol, and only implements a few members needed by the protocol. This effectively separates the protocol from the connection and the server. RemoteProtocol with virtual abstract methods for: * get_message - Receives a new message (waiting when needed). * send_message - Sends a message to the client. * get_db - Reserves a database when the class needs to use it. * release_db - Releases the database when the class is not using it. * select_db - It's called when the protocol needs to select a new database to work on. * shutdown - It's called when the protocol requires to shutdown. Each command in the RemoteProtocol now do a two additional things: 1. Use get_db() to reserve and get a read-only or writable database. 2. Relese the reserved database using release_db() Implements RemoteProtocol on the RemoteServer: * get_db - Always returns the same database object (db or wdb) * release_db - Does nothing in this server (database is never released). * get_message and send_message - Call RemoteConnection methods. 5. RemoteProtocol implementation moved to remoteprotocol.cc ----------------------------------------------------------- Added `remoteprotocol.cc` with the implementation of `RemoteProtocol` and header file remoteprotocol.h with added `RemoteProtocol` abstract class definition moved to `include/xapian/remoteprotocol.h` The reason for this is other projects might want to use the `RemoteProtocol` class for implementing a custom remote server. Xapiand is one of these projects. (https://github.com/Kronuz/Xapiand) Please, any comments and feedback would be appreciated. German M. Bravo (Kronuz) -------------- next part -------------- An HTML attachment was scrubbed... URL: