[Libwebsockets] v2.0 coming very soon

Thomas Spitz thomas.spitz at hestia-france.com
Thu Jun 2 10:49:26 CEST 2016


Hello Andy,

I have seen that you released v2.0.2 (on
https://libwebsockets.org/git/libwebsockets/) but I am surprised there is
no LWSMPRO_CALLBACK in /lib/libwebsockets.h given that you mention it in
https://github.com/warmcat/libwebsockets/blob/master/README.coding.md

Best regards,
Thomas Spitz


2016-05-26 22:31 GMT+02:00 Andy Green <andy at warmcat.com>:

>
>
> On May 26, 2016 11:19:05 PM GMT+08:00, Thomas Spitz <
> thomas.spitz at hestia-france.com> wrote:
> >>
> >>     Maybe you should better prototype it with the high end stuff and
> >>
> >>     when it works, look at reducing that to code, including the
> >plugin
> >>     content direct in your app etc if you still want to.
> >> This is what I have done. First lwsws, then test-server-v2.0 then
> >> test-server then my program
> >> This leads me to use lws in the old manner because my main ws
> >protocol
> >> (which is part of my program) cannot be launched as plugin (too much
> >> work to make it work as separate program and also too hard to debug
> >then)
> >> I don't really understand your situation, but it is not a separate
> >> program.  As I mentioned the one plugin can service both ws and http
> >in the
> >> one callback function.  Maybe I misunderstand.
> >>
> >> My problem might be sum up to the following : I cannot easily
> >separate
> >test-server program into 2 programs compiled separately
> >(test-server-v2.0
> >on one side and protocol_my_protocol on other side). In fact my program
> >share data with each client (mutex, * client_wsi, program_data,
> >client_data, client pthread, ...) AND my program is also able to
> >start/stop
> >the server (run into a thread) in order to reconfigure it. I'm
>
> I dunno if this is the point, but in the info struct there is this
>
>  * @user:       CONTEXT: optional user pointer that can be recovered via
> the context
>  *              pointer using lws_context_user
>
> So you can pass in here a pointer to your shared data struct between the
> main app and lws service thread.  Everywhere in your code that has access
> to lws_context, eg, every protocol plugin callback, has access to that
> struct; you can share the struct declaration by #include the same header in
> the plugin as the main app.
>
> That means eg a protocol plugin running in its service thread can directly
> share and change any data with the main app's thread.
>
> Although a plugin
>
>  - has its code separated
>
>  - may be built separately from the main app
>
> It is NOT a separate thread or process or has private virtual memory
> restriction like it was fork()ed.  When the plugin is loaded, it becomes
> part of the process using lws, almost exactly like it was all compiled
> together originally.
>
> It doesn't have access to main app global vars directly, that's true, but
> if you put the related globals in a (global) struct and set info.user to
> point to it when you create the context, all plugins can have access to it
> easily giving the same result.
>
> My guess is every objection to plugins has a similar solution, although
> maybe some of your problems need new apis once I understand them.
>
> >I will unfortunately have to implement low-level code for static (GET
> >>
> >> file like leaf.png with/without authentification) and dynamic (HTTP
> >> GET/POST command) HTTP serving.
> >> Can you tell a little more about why you "have" to do the low level
> >> serving code for static files?
> >>
> >> -> some files have to be retrieved from other locations than where
> >"test.html" is (for instance from a given folder in a SD card)
> >
> >> If it's auth, the generic-sessions plugin is aimed at providing that.
> >
> >In order not to enable anybody retrieve those files on SD card, I need
> >to
> >protect them by login/password (In fact the only users enabled are the
> >user
> >that manage to connect through our ws protocol using our own auth
> >policy).
> >I could have used your future generic-sessions plugin (except I am
> >afraid
> >sqlite is redundant with my actual 400 users login/pwd struct) but as I
> >cannot use plugins due to separate compilation and shared compenents..
>
> 400 users sounds to me like you really need sqlite actually.
>
> If you look into that I think you will end up with needing a "session".
> Then the amount of duplicated work will be very big.
>
> sqlite also has a handy feature you can do sql from the commandline on the
> db, eg to init your 400 users if you won't let them create their own
> accounts over the web ui.
>
> >I agree you'll have to do your own dynamic HTTP serving code, but this
> >can
> >> coexist in the same plugin - the same source file - as the ws serving
> >part,
> >> if that helps.
> >
> >Without plugins, I think of serving dynamic HTTP using general
> >callback_http
>
> Yes, for the dynamic html, you are into that one way or another.
>
> But architecturally, you should be able to do it from a plugin.
>
> >Hope you will keep test-server always up to date
> >>
> >>
> >> One minor question : Is it possible to do port redirection with
> >> test-server (http 80 -> redirect to https 7681). By the way one
> >drawback
> >> Sure... you make a vhost at the port you want to redirect, and tell
> >it the
> >> new URL in the origin  (This one redirects SSL on port 7681, where I
> >used
> >> to run the test server, to SSL on port 443 and the URL where it now
> >lives)
> >>     {
> >>         # the old test server ran this on :7681, put a redirect
> >>         # there to take us to the new location
> >>     "name": "libwebsockets.org",
> >>     "port": "7681",
> >>      "host-ssl-key":  "/etc/pki/tls/private/libwebsockets.org.key",
> >>      "host-ssl-cert": "/etc/pki/tls/certs/libwebsockets.org.crt",
> >>      "host-ssl-ca":   "/etc/pki/tls/certs/libwebsockets.org.cer",
> >>      "mounts": [{
> >>        "mountpoint": "/",
> >>        "origin": ">https://libwebsockets.org/testserver/"
> >>      }]
> >>   }
> >> You should be able to give an origin like
> >> "origin": ">https://libwebsockets.org:7681/testserver/"
> >>
> >> vhost need plugin support if I understood correctly. If yes, then I
> >don't
> >see how to make it work with vhost
>
> No: vhosts are baked into the main lib and are unrelated to plugins.
>
> You can deliver an array of protocols to them when you create them, and
> you mention protocol names in the info.pvo linked list to enable +
> configure them on the vhost.
>
> If plugins are enabled, lws will bind any protocol names to the plugin
> that matches, at init time; it makes a new protocols array if so with your
> array and then any dynamically found protocols enabled by mentions in pvo
> appended.  But it's completely optional, you can do everything just with
> the static array of protocols you fed the vhost and no plugins, since
> vhosts only talks about protocols using their name, it doesn't care where
> it found them originally.
>
> -Andy
>
> >
> >
> >2016-05-26 16:13 GMT+02:00 Andy Green <andy at warmcat.com>:
> >
> >>
> >>
> >> On 05/26/2016 08:40 PM, Thomas Spitz wrote:
> >>
> >>>     Maybe you should better prototype it with the high end stuff and
> >>>     when it works, look at reducing that to code, including the
> >plugin
> >>>     content direct in your app etc if you still want to.
> >>>
> >>> This is what I have done. First lwsws, then test-server-v2.0 then
> >>> test-server then my program
> >>>
> >>> This leads me to use lws in the old manner because my main ws
> >protocol
> >>> (which is part of my program) cannot be launched as plugin (too much
> >>> work to make it work as separate program and also too hard to debug
> >then)
> >>>
> >>
> >> I don't really understand your situation, but it is not a separate
> >> program.  As I mentioned the one plugin can service both ws and http
> >in the
> >> one callback function.  Maybe I misunderstand.
> >>
> >> I will unfortunately have to implement low-level code for static (GET
> >>> file like leaf.png with/without authentification) and dynamic (HTTP
> >>> GET/POST command) HTTP serving.
> >>>
> >>
> >> Can you tell a little more about why you "have" to do the low level
> >> serving code for static files?  If it's auth, the generic-sessions
> >plugin
> >> is aimed at providing that.  If there's something special about your
> >auth
> >> needs, I am very curious to hear about that atm.  Generally I am very
> >much
> >> trying to provide static file serving with session auth "out of the
> >box".
> >>
> >> I agree you'll have to do your own dynamic HTTP serving code, but
> >this can
> >> coexist in the same plugin - the same source file - as the ws serving
> >part,
> >> if that helps.
> >>
> >> Hope you will keep test-server always up to date
> >>>
> >>> One minor question : Is it possible to do port redirection with
> >>> test-server (http 80 -> redirect to https 7681). By the way one
> >drawback
> >>>
> >>
> >> Sure... you make a vhost at the port you want to redirect, and tell
> >it the
> >> new URL in the origin  (This one redirects SSL on port 7681, where I
> >used
> >> to run the test server, to SSL on port 443 and the URL where it now
> >lives)
> >>
> >>     {
> >>         # the old test server ran this on :7681, put a redirect
> >>         # there to take us to the new location
> >>     "name": "libwebsockets.org",
> >>     "port": "7681",
> >>      "host-ssl-key":  "/etc/pki/tls/private/libwebsockets.org.key",
> >>      "host-ssl-cert": "/etc/pki/tls/certs/libwebsockets.org.crt",
> >>      "host-ssl-ca":   "/etc/pki/tls/certs/libwebsockets.org.cer",
> >>      "mounts": [{
> >>        "mountpoint": "/",
> >>        "origin": ">https://libwebsockets.org/testserver/"
> >>      }]
> >>   }
> >>
> >> You should be able to give an origin like
> >>
> >> "origin": ">https://libwebsockets.org:7681/testserver/"
> >>
> >> with redirection using lwsws is that we have to specify a URL when
> >doing
> >>> port redirection. For instance if we have server at 192.168.1.3
> >exposed
> >>> to the web as websockets.org <http://websockets.org> then we need
> >two
> >>> redirections in order to make it work on both LAN and WAN
> >>>
> >>
> >> Hm you can configure your local DNS to tell you 192.168.1.3 is the
> >address
> >> of your server, to local clients.  That's what I do here to access my
> >own
> >> internet-facing services.
> >>
> >> If it's a laptop and you take it outside, he gets external DNS server
> >and
> >> uses the external address automatically.
> >>
> >> That's not really an lws problem, if I understood it.
> >>
> >> -Andy
> >>
> >> Thanks for the support.
> >>>
> >>> Best regards,
> >>> Thomas
> >>>
> >>> 2016-05-26 11:48 GMT+02:00 Andy Green <andy at warmcat.com
> >>> <mailto:andy at warmcat.com>>:
> >>>
> >>>
> >>>
> >>>
> >>>     On 05/26/2016 05:15 PM, Thomas Spitz wrote:
> >>>
> >>>         Hello Andy,
> >>>
> >>>         On one hand, I have tried to use the new plugin approach in
> >my
> >>>         program
> >>>         but it is very hard in my case because lws server and my
> >main
> >>>         plugin (ws
> >>>         connection) are very tied together (they share many
> >variables
> >>>         that would
> >>>         need to be exposed to my application through shm or other
> >>>         sharing options).
> >>>         On the other hand, in order to get advantage of better HTTP
> >file
> >>>         handling, auth management and further improvements, I would
> >like
> >>>         not to
> >>>         have to write my own callback_http.
> >>>
> >>>
> >>>     If you are going to serve dynamic http, you'll need to have your
> >own
> >>>     http handling.  But that can be in the same plugin - the same
> >>>     protocol callback - as the ws parts, so there is no problem
> >about
> >>>     "sharing variables".
> >>>
> >>>         In other word, is it possible to declare my own protocol
> >>>         definitions and
> >>>         user callback contents like in first test-server but without
> >>>         declaring
> >>>         http-callback. That is to say
> >>>
> >>>
> >>>     If you need it, because you're not serving files but generated
> >html,
> >>>     you need it.  If you're serving files, like leaf.jpg, you can
> >use
> >>>     mounts.
> >>>
> >>>              /* list of supported protocols and callbacks */
> >>>              static struct lws_protocols protocols[] =
> >>>              {
> >>>              { "MY_PROTOCOL", callback_my_protocol, sizeof(struct
> >>>              per_session_data__my_protocole), 4096},
> >>>              { NULL, NULL, 0, 0 } /* terminator */
> >>>              };
> >>>
> >>>     ...
> >>>
> >>>         it is not clear for me wether libuv (and so plugins) are
> >>>         required or nor.
> >>>
> >>>
> >>>     Lws is pretty flexible, it does not itself need libuv or
> >plugins.
> >>>     But you are into writing a lot of code and structs if you go
> >that way.
> >>>
> >>>     One of the knock-ons is then you must provide an explicit
> >http-only
> >>>     protocols[0] yourself.
> >>>
> >>>     If you use the plugins stuff, actually you can avoid it -->
> >>>
> >>>         I also do not understand how test-server-v2.0 manage to
> >serve
> >>>         leaf.png
> >>>         and any other file in any other folder (what test-server
> >isn't
> >>>         able to
> >>>         do) using HTTP
> >>>
> >>>
> >>>     test-server-v2.0.c is even more remarkable if you notice he
> >doesn't
> >>>     define any protocols at all.  Lws defines a dummy "protocols[0]"
> >one
> >>>     then and assembles the protocol structs dynamically from the
> >plugins
> >>>     enabled in the config (or info.pvo if you do it by hand with
> >structs).
> >>>
> >>>     The reason it can serve everything is it declares a mount:
> >>>
> >>>     static const struct lws_http_mount mount = {
> >>>              (struct lws_http_mount *)&mount_post,           /*
> >>>     linked-list pointer to next*/
> >>>              "/",            /* mountpoint in URL namespace on this
> >vhost
> >>> */
> >>>              LOCAL_RESOURCE_PATH, /* where to go on the filesystem
> >for
> >>>     that */
> >>>              "test.html",    /* default filename if none given */
> >>>              NULL,
> >>>              NULL,
> >>>              NULL,
> >>>              NULL,
> >>>              0,
> >>>              0,
> >>>              0,
> >>>              0,
> >>>              0,
> >>>              0,
> >>>              LWSMPRO_FILE,   /* mount type is a directory in a
> >filesystem
> >>> */
> >>>              1,              /* strlen("/"), ie length of the
> >mountpoint
> >>> */
> >>>     };
> >>>
> >>>     binding the URL namespace from / to LOCAL_RESOURCE_PATH, which
> >is
> >>>     /usr/share/libwebsockets-test-server or wherever you installed
> >lws.
> >>>
> >>>     Once you set it up, unless it finds another mount that's a
> >better
> >>>     match to the URL, lws will automatically serve files from that
> >dir
> >>>     when asked for things in /.  leaf.jpg is in that directory.
> >>>
> >>>     The equivalent bit of lwsws conf is this
> >>>
> >>>             {
> >>>             "mountpoint": "/",
> >>>             "origin": "file:///usr/share/libwebsockets-test-server",
> >>>             "default": "test.html",
> >>>             }
> >>>
> >>>     Lws lets you also declare mounts that are bound to a protocol
> >name.
> >>>
> >>>             {
> >>>              "mountpoint": "/dynamic",
> >>>              "origin": "callback://my-protocol-name"
> >>>             }
> >>>
> >>>     HTTP GET or POST down /dynamic/* will occur on the callback for
> >>>     "my-protocol-name". (The name /dynamic is not meaningful, it
> >could
> >>>     be anything).
> >>>
> >>>     So this way you can automate serving static content so it is no
> >>>     effort, and isolate serving dynamic content for a region of the
> >URL
> >>>     space in a plugin, by name.
> >>>
> >>>         As you can see I am quite confusing with all the new
> >>>         possibilities. If
> >>>         my questions are not clear, I can send you part of my
> >code...
> >>>
> >>>
> >>>     It's difficult to answer in a more straightforward way because
> >the
> >>>     best solution is pushing to make it as abstract as possible
> >(lwsws,
> >>>     libuv, protocol plugins) but you want to implement it all in
> >code.
> >>>     So while explaining you should use plugins you're also asking
> >how to
> >>>     do it without them.
> >>>
> >>>     Maybe you should better prototype it with the high end stuff and
> >>>     when it works, look at reducing that to code, including the
> >plugin
> >>>     content direct in your app etc if you still want to.
> >>>
> >>>     -Andy
> >>>
> >>>         Best regards,
> >>>         Thomas
> >>>
> >>>
> >>>
> >>>         2016-05-18 9:06 GMT+02:00 Andy Green <andy at warmcat.com
> >>>         <mailto:andy at warmcat.com>
> >>>         <mailto:andy at warmcat.com <mailto:andy at warmcat.com>>>:
> >>>
> >>>
> >>>
> >>>              On May 18, 2016 2:46:13 PM GMT+08:00, Thomas Spitz
> >>>              <thomas.spitz at hestia-france.com
> >>>         <mailto:thomas.spitz at hestia-france.com>
> >>>              <mailto:thomas.spitz at hestia-france.com
> >>>
> >>>         <mailto:thomas.spitz at hestia-france.com>>> wrote:
> >>>              >Hello Andy,
> >>>              >
> >>>              >I think I will use plugins however regarding lwsws I
> >will
> >>>         rather not
> >>>              >use it
> >>>
> >>>              That'll get you most of the benefits.
> >>>
> >>>              >because my application need to computably
> >computationally
> >>>         (not
> >>>              >manually)
> >>>              >configure ports, redirect and protocols. Thus, I think
> >>>         going through
> >>>              >JSON
> >>>              >files will certainly be more complex than dealing with
> >>>         native C
> >>>              >structures... I might be wrong...
> >>>
> >>>              You know your application better than I do... if it's
> >>>         easier to wire
> >>>              up with structs, no problem.
> >>>
> >>>              One observation though... JSON is easy to
> >machine-generate
> >>>         if need be.
> >>>
> >>>              -Andy
> >>>
> >>>               >Best regards,
> >>>               >Thomas
> >>>               >
> >>>               >
> >>>               >2016-05-17 20:48 GMT+02:00 Andy Green
> ><andy at warmcat.com
> >>>         <mailto:andy at warmcat.com>
> >>>              <mailto:andy at warmcat.com <mailto:andy at warmcat.com>>>:
> >>>               >
> >>>               >>
> >>>               >>
> >>>               >> On May 18, 2016 1:43:04 AM GMT+08:00, Thomas Spitz
> ><
> >>>               >> thomas.spitz at hestia-france.com
> >>>         <mailto:thomas.spitz at hestia-france.com>
> >>>              <mailto:thomas.spitz at hestia-france.com
> >>>
> >>>         <mailto:thomas.spitz at hestia-france.com>>> wrote:
> >>>               >> >I think I will have to go for option 1 as you
> >explained
> >>> in
> >>>               >>
> >>>
> >>>           >>
> >>>
> >https://libwebsockets.org/pipermail/libwebsockets/2016-April/002303.html
> >>>               >> >in
> >>>               >> >order to get an easy and maintainable solution for
> >port
> >>>               >redirections,
> >>>               >> >http
> >>>               >> >processing, ...:
> >>>               >>
> >>>               >> It's workable, but compared to just having a custom
> >>>         JSON config
> >>>              and a
> >>>               >> plugin, with everything else off the shelf, it's
> >not
> >>>         the easiest or
> >>>               >most
> >>>               >> maintainable way.
> >>>               >>
> >>>               >> >1) Keep your existing code and attach your own
> >mounts
> >>>         at vhost
> >>>               >creation
> >>>               >> >time
> >>>               >> >
> >>>               >> >
> >>>               >>
> >>>
> >>>           >
> >>>
> >
> https://github.com/warmcat/libwebsockets/blob/master/README.coding.md#using-lws-v2-vhosts
> >>>               >> >
> >>>               >> >Lws will serve the mounted contents automatically
> >from
> >>>         where it's
> >>>               >> >mounted
> >>>               >> >in the server namespace
> >>>               >> >
> >>>               >> >Could you confirm that this solution requires
> >libuv
> >>>         and or libev? I
> >>>               >> >hope
> >>>               >> >this will not stress too much my emdedded system
> >;-)
> >>>               >>
> >>>               >> Plugins are what implies libuv (not libev).
> >However I
> >>>         think you
> >>>              find
> >>>               >for
> >>>               >> Arm v7 running Linux, the idea libuv will 'strain'
> >it
> >>>         is probably
> >>>               >just
> >>>               >> prejudice.  Libuv is cheap as a crossplatform
> >wrapper
> >>>         around socket
> >>>               >events
> >>>               >> or timers, and we're otherwise using it for
> >>>         crossplatform dirent and
> >>>               >plugin
> >>>               >> operations at init.
> >>>               >>
> >>>               >> Until recently when it got updated to a Rpi 3,
> >>>         libwebsockets.org <http://libwebsockets.org>
> >>>              <http://libwebsockets.org> was
> >>>
> >>>               >a
> >>>               >> 1GHz Armv7, before that a series of weaker Arm
> >devices
> >>>         like CA8,
> >>>              even
> >>>               >the
> >>>               >> weakest ones with 256MB would have had no trouble
> >with
> >>>         libuv in the
> >>>               >mix.
> >>>               >>
> >>>               >> >I think this would have been nice to have cgitest,
> >>>         server-status in
> >>>               >> >libwebsockets-test-server-v2.0 and
> >>>         libwebsockets-test-server to
> >>>               >compare
> >>>               >> >the
> >>>               >> >three solutions (the third solution is lwsws).
> >>>               >>
> >>>               >> The old test server code is bloated with adding
> >support
> >>>         for the
> >>>               >kitchen
> >>>               >> sink in there, making it difficult to use as the
> >>>         example it's
> >>>               >intended to
> >>>               >> be; the advantage of the plugins + lwsws for
> >>>         maintainability is
> >>>               >clear.
> >>>               >>
> >>>               >> You can add everything in the plugins + lwsws JSON
> >into
> >>>               >test-server-v2.0.c
> >>>               >> as structs + protocols + api calls, but then it
> >will
> >>> become
> >>>               >inflexible and
> >>>               >> huge... that's a big step backwards.
> >>>               >>
> >>>               >> So the official test-server-v2.0.c should stay like
> >it
> >>>         is, just
> >>>               >having
> >>>               >> enough 'config by struct' to demonstrate the
> >concept
> >>>         and leveraging
> >>>               >the
> >>>               >> plugins, where a standalone server app is possible
> >>>         lwsws is the way
> >>>               >to 'do
> >>>               >> everything' flexibly and maintainably.
> >>>               >>
> >>>               >> >libwebsockets-test-server is
> >>>               >> >probably not necessary as I assume it will be a
> >>>         depreciated
> >>>              solution
> >>>               >in
> >>>               >> >the
> >>>               >> >future? I will certainly try to complete
> >>>               >libwebsockets-test-server-v2.0
> >>>               >> >tomorrow.
> >>>               >>
> >>>               >> Yes the old all-in-one test server will continue to
> >be
> >>>         supported
> >>>               >(indeed
> >>>               >> the default, since everything else needs enabling
> >at
> >>>         cmake).
> >>>              But new
> >>>               >stuff
> >>>               >> - I plan to discuss a completely new, highly
> >abstracted
> >>>         protocol
> >>>               >plugin
> >>>               >> project soon, and suitable third-party plugins
> >could
> >>>         become provided
> >>>               >as
> >>>               >> part of lws - will be provided as 'mix and
> >matchable'
> >>>         standalone
> >>>               >plugins
> >>>               >> only.
> >>>               >>
> >>>               >> -Andy
> >>>               >>
> >>>               >> >Best regards,
> >>>               >> >Thomas
> >>>               >>
> >>>               >>
> >>>
> >>>
> >>>
> >>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://libwebsockets.org/pipermail/libwebsockets/attachments/20160602/2df6d8b8/attachment.html>


More information about the Libwebsockets mailing list