[Libwebsockets] lws + libuv ( mac/freebsd ), explicit_vhosts

Andy Green andy at warmcat.com
Wed Nov 30 00:31:08 CET 2016


On Tue, 2016-11-29 at 18:41 +0200, Aleksey Zhukovskiy wrote:
> Hi 
> I proposed patch - solution my problem. Added private function int
> lws_uv_initvhost(struct lws_vhost* vh, struct lws* wsi) to libuv.c
> that makes initialization of vhost (from lws_uv_initloow - while( vh
> ) {...}) when libuv loop already started only. It called for every
> vhost creation and for each created vhost in lws_uv_initloop. When
> libuv loop is not started does nothing. Check if initialization on
> vhost already done.
> NO changes for user requires. No any additional calls to _loop(), we
> can just add vhosts at any time. Hope I have not broke anything.

Thanks.

I modified it a bit, protocols that are enabled on a vhost are supposed
to get callbacks for their creation one time, per-vhost.  So I added
this into the create vhost api if it sees the server has already
started.

It doesn't break lwsws or the libuv test server, maybe you can check if
still OK for your usecase, it's pushed on master.

I also added a note in the commitlog about privileges... if you add a
vhost late, after the server starts, then root privs will have been
dropped.  If your new vhost enables a protocol that wants to make use
of the initially elevated privs (eg, open a file in a restricted part
of the filesystem, but use it after privs dropped so you can do nothing
else with it), it will break.

> If you wrote this code, may be you can watch it again. server.c:155
> vhost->context->pt[m].wsi_listening = wsi;
> 
> ++  lws_uv_initvhost( vhost, wsi ); ++ // This is my addition
> 
> if (insert_wsi_socket_into_fds(vhost->context, wsi))
>    goto bail;
> 
> vhost->context->count_wsi_allocated++;
> vhost->lserv_wsi = wsi;
> 
> I feel something wrong with it but I didnt get all. Is first
> assignment OK if insert_wsi_socket will fail? And last two lines.. 
> count_wsi_allocated increases only if insert succeded. Maybe its not
> quite correct?

It's possible something wrong, but I didn't see what.

If we can't track the wsi using the fds stuff (this handles fd-> wsi
mapping for poll()), we fail to create the vhost and the server will
exit... so only need to free() things.  Normally this won't happen but
it is possible with "late" vhost that the server is so busy there is no
space for a new listen fd for the vhost.

If we can, then we go on, the count_wsi_allocated is purely for stats,
ie the server-status protocol displays it

https://libwebsockets.org/server-status/

-Andy

> 
> 
> 2016-11-29 13:27 GMT+02:00 Andy Green <andy at warmcat.com>:
> > On Tue, 2016-11-29 at 13:12 +0200, Aleksey Zhukovskiy wrote:
> > > In this way I cant add vhost after calling
> > lws_uv_initloop(context,
> > > NULL, 0) right?
> > 
> > This "libuvification" of the vhost listen sockets is a thing
> > peculiar
> > to libuv.
> > 
> > It's not that it seals off the vhost list, IIRC the reason for it
> > is it
> > cannot instantiate the listen socket if the libuv loop was not
> > already
> > running; in many cases that happens only after vhost initialization
> > so
> > it has to be deferred until we know we will truly instantiate the
> > real
> > listen socket and start the loop.
> > 
> > > I'll try to initialize new vhost after start. When new instance
> > of my
> > > daemon starts it checks for another one by connection to PORT,
> > and if
> > > success sends BYE. Another instance stops listening (when
> > receives
> > > BYE), and only then new daemon starts to listen this PORT. During
> > > this time first instance does some initializing job using
> > > libwebsockets too. Another instance keeps working while any
> > > connections are alive. It gives grace restarting of service with
> > no
> > > some pain for customers. I want to try to keep this functional
> > > because now I know the reason of crashing ))
> > > btw I do not kill vhosts while running. Just create
> > 
> > A proper "reload" type functionality would also need to deal with
> > killing or modifying vhosts too.  But anyway the bottom line is if
> > you
> > add a vhost later - much later - with libuv, you must do what
> > lws_uv_initloop() does to vhost listen sockets to the new guy's
> > vhost
> > listen socket.  That's the only limitation I think.
> > 
> > You can probably modify lws_uv_initloop() to be safe to call again
> > on
> > it.  Eg, 
> > 
> > a) add a :1 member in the pt struct to say if the global pt init
> > part
> > has already been done, if so, skip it
> > 
> > b) where at the moment we just blindly do
> > 
> > vh->lserv_wsi->w_read.context = context;  ...etc...
> > 
> > because we only expect to run one time, you can check vh-
> > >lserv_wsi-
> > >w_read.context != NULL and if so, just skip because it's already
> > initialized.  That way it'll init new guys when run again.
> > 
> > Patches welcome.
> > 
> > -Andy
> > 
> > >
> > > 2016-11-29 13:00 GMT+02:00 Andy Green <andy at warmcat.com>:
> > > > On Tue, 2016-11-29 at 12:57 +0200, Aleksey Zhukovskiy wrote:
> > > > > Seems to initialization of all vhosts takes place in
> > > > > lws_uv_initloop() (module libuv.c) at while( vh ) { ... }
> > > > > So if I create vhost later, then I have empty field w_read
> > which
> > > > > needs for libuv operations.
> > > > > no callback when creating wsi of new vhost? 
> > > >
> > > > Using my powers of ESP I just managed to answer this before you
> > > > asked
> > > > it.
> > > >
> > > > As the example code does, you should call that after you have
> > > > defined
> > > > all your vhosts.  If you're not doing that atm, ie, not
> > following
> > > > what
> > > > the example code does for libuv case, then that mystery is
> > solved.
> > > >
> > > > -Andy
> > > >
> > > > > 2016-11-29 12:44 GMT+02:00 Aleksey Zhukovskiy <beetle at gambler
> > .ru>
> > > > :
> > > > > > I just want to say that on FreeBSD, libdl is part of libc,
> > so
> > > > if
> > > > > > you add libdl for unix (CMakeLists.txt)
> > > > > >         if (UNIX AND LWS_WITH_PLUGINS)
> > > > > >                 set(CMAKE_C_FLAGS "-fPIC ${CMAKE_C_FLAGS}")
> > > > > >                target_link_libraries(websockets dl)
> > > > > >         endif()
> > > > > > when linking C executable bin/libwebsockets-test-fraggle we
> > > > face
> > > > > > error "cannot find -ldl" and can't finish make process.
> > This
> > > > may be
> > > > > > a unnecessary disapointment for some freebsd users.
> > > > > > maybe better to patch contition to
> > > > > >         if (UNIX AND LWS_WITH_PLUGINS AND NOT(
> > > > ${CMAKE_SYSTEM_NAME}
> > > > > > MATCHES "FreeBSD" ))
> > > > > > because Im not sure about entire BSD-family
> > > > > >
> > > > > > I built debug versions of libuv and libwebsockets and still
> > > > debug
> > > > > > it. Hope to send you some interesting a little bit later.
> > > > > > Know exactly that segfault occurs in libuv because
> > loop==NULL
> > > > in
> > > > > > handle. It occurs because wsi->w_read is not initialized
> > (fully
> > > > > > zeroed). I didn't find initialization block already.
> > > > > >
> > > > > > 2016-11-29 12:01 GMT+02:00 Andy Green <andy at warmcat.com>:
> > > > > > > On Tue, 2016-11-29 at 11:31 +0200, Aleksey Zhukovskiy
> > wrote:
> > > > > > > > First thing I found when trying lwsws on freebsd -
> > error at
> > > > > > > linking.
> > > > > > > > Freebsd does not have libdl (included in std),
> > > > CMakeLists.txt
> > > > > > > doesn't
> > > > > > >
> > > > > > > Mmm the plugins are handled by libuv.  There's an old dl
> > > > > > > implementation
> > > > > > > lurking around to keep Travis happy, which has ancient
> > libuv
> > > > by
> > > > > > > default
> > > > > > > that has no dl support.  It's just there to make it build
> > if
> > > > the
> > > > > > > libuv
> > > > > > > is too old.
> > > > > > >
> > > > > > > Somehow you are building this junk libdl implementation
> > not
> > > > the
> > > > > > > libuv-
> > > > > > > specific one.
> > > > > > >
> > > > > > > > have to add target_link_libraries(websockets dl). This
> > > > occured
> > > > > > > when I
> > > > > > > > activated lwsws and it follows LWS_WITH_PLUGINS. Simply
> > > > > > > commenting
> > > > > > > > this line solved the problem.
> > > > > > >
> > > > > > > I don't think it solved the problem... killing plugins
> > hid
> > > > the
> > > > > > > symptom.
> > > > > > >
> > > > > > > The old dl stuff lives in ./lib/lws-plat-unix, and is
> > > > protected
> > > > > > > like
> > > > > > > this
> > > > > > >
> > > > > > > #ifdef LWS_WITH_PLUGINS
> > > > > > >
> > > > > > > #if defined(LWS_USE_LIBUV) && UV_VERSION_MAJOR > 0
> > > > > > >
> > > > > > > /* libuv.c implements these in a cross-platform way */
> > > > > > >
> > > > > > > #else
> > > > > > >
> > > > > > > ... the old, dummy implementation
> > > > > > >
> > > > > > > #endif
> > > > > > >
> > > > > > > So something is wrong either with LWS_USE_LIBUV or
> > > > > > > UV_VERSION_MAJOR on
> > > > > > > your system.  On Travis with the ancient libuv,
> > > > UV_VERSION_MAJOR
> > > > > > > is 0,
> > > > > > > so it uses this placeholder libdl code.  But on a system
> > with
> > > > > > > recent
> > > > > > > libuv, that should never happen and there is no direct
> > > > dependency
> > > > > > > on
> > > > > > > libdl (on Linux, libuv wants libdl.  I dunno what libuv
> > does
> > > > on
> > > > > > > BSD for
> > > > > > > dynamic linking type operations but assuming BSD has that
> > > > > > > concept, it
> > > > > > > should have a suitable implementation using the generic
> > libuv
> > > > > > > api).
> > > > > > >
> > > > > > > > After some time my testing pointed to core of problem:
> > I
> > > > can
> > > > > > > not add
> > > > > > > > vhost just after initializing libuv loop. If I add
> > vhost
> > > > before
> > > > > > > it
> > > > > > > > all is ok. Howewer its important to me  to have this
> > > > > > > possibility. I
> > > > > > > > can start listen for some ports when only another
> > instance
> > > > of
> > > > > > > my
> > > > > > > > daemon frees it. So maybe is there some way to
> > initialize
> > > > but
> > > > > > > "sleep"
> > > > > > > > some vhost? Then I can create it before main cycle. Or
> > > > maybe
> > > > > > > its
> > > > > > > > better to provide feature of adding vhost on working
> > loop.
> > > > > > >
> > > > > > > I don't see why anything cares.  Lws should not care how
> > late
> > > > it
> > > > > > > is to
> > > > > > > the party after the loop starts.  In fact it supports
> > > > "foreign
> > > > > > > loops"
> > > > > > > that exist completely outside lws, and can have done
> > whatever
> > > > > > > before
> > > > > > > lws even starts up.
> > > > > > >
> > > > > > > Actually your backtrace is saying that **libuv** cares,
> > it
> > > > blows
> > > > > > > up
> > > > > > > inside libuv when handling what looks to be a socket fd. 
> > You
> > > > > > > will need
> > > > > > > to add debuginfo or whatever the bsd equivalent is to
> > libuv,
> > > > so
> > > > > > > gdb can
> > > > > > > show us exactly where he chokes in libuv and we can get a
> > > > clue
> > > > > > > about
> > > > > > > why.
> > > > > > >
> > > > > > > > How do you do hot reload of configuration of lwsws
> > without
> > > > > > > restart if
> > > > > > > > added new listening ports in config?
> > > > > > >
> > > > > > > That concept doesn't exist in lws... vhosts are created
> > one
> > > > time
> > > > > > > after
> > > > > > > the contexts are created.  For my usecases, that has
> > always
> > > > been
> > > > > > > fine,
> > > > > > > I just restart the service.
> > > > > > >
> > > > > > > Patches are always welcome though... it should be easy to
> > add
> > > > > > > vhosts
> > > > > > > randomly but I am not sure how cleanly you can kill a
> > vhost
> > > > is
> > > > > > > it's
> > > > > > > happening with the rest of the system staying up.
> > > > > > >
> > > > > > > -Andy
> > > > > > >
> > > > > > > >  
> > > > > > > >
> > > > > > > > 2016-11-28 23:31 GMT+02:00 Andy Green <andy at warmcat.com
> > >:
> > > > > > > > >
> > > > > > > > > On November 29, 2016 4:42:02 AM GMT+08:00, Aleksey
> > > > Zhukovskiy
> > > > > > > <beet
> > > > > > > > > le at gambler.ru> wrote:
> > > > > > > > > >Andy,
> > > > > > > > > >thanx for answers, thats what I thought about libuv.
> > My
> > > > > > > version of
> > > > > > > > > it
> > > > > > > > > >is
> > > > > > > > > >1.10 (seems ok?)
> > > > > > > > >
> > > > > > > > > Yes.
> > > > > > > > >
> > > > > > > > > If you run the libuv test server instead of your
> > code, is
> > > > it
> > > > > > > still
> > > > > > > > > broken?
> > > > > > > > >
> > > > > > > > > https://github.com/warmcat/libwebsockets/blob/master/
> > test
> > > > -ser
> > > > > > > ver/te
> > > > > > > > > st-server-libuv.c
> > > > > > > > >
> > > > > > > > > How about if you run lwsws?
> > > > > > > > >
> > > > > > > > > https://github.com/warmcat/libwebsockets/blob/master/
> > READ
> > > > ME.l
> > > > > > > wsws.m
> > > > > > > > > d
> > > > > > > > >
> > > > > > > > > Both of these implement lws + libuv and don't crash
> > > > here...
> > > > > > > if they
> > > > > > > > > don't crash for you either it looks like you might
> > not be
> > > > > > > initing
> > > > > > > > > the libuv loop correctly in you code.  If they do
> > crash,
> > > > can
> > > > > > > you
> > > > > > > > > install debuginfo for libuv so we can see more
> > detail?
> > > > > > > > >
> > > > > > > > > If all you want is a generic server with your own
> > > > protocols
> > > > > > > you can
> > > > > > > > > use lwsws out of the box with no code changes,
> > > > configuring it
> > > > > > > with
> > > > > > > > > JSON (eg, multiple vhosts) and making your protocol a
> > > > > > > standalone
> > > > > > > > > plugin
> > > > > > > > >
> > > > > > > > > https://github.com/warmcat/libwebsockets/tree/master/
> > plug
> > > > in-s
> > > > > > > tandal
> > > > > > > > > one
> > > > > > > > >
> > > > > > > > > This model is serving https://libwebsockets.org +
> > > > > > > > > https://warmcat.com , eg the test ws server is done
> > > > entirely
> > > > > > > with
> > > > > > > > > protocol plugins... this is much cleaner, eg
> > > > > > > > >
> > > > > > > > > https://github.com/warmcat/libwebsockets/blob/master/
> > plug
> > > > ins/
> > > > > > > protoc
> > > > > > > > > ol_dumb_increment.c
> > > > > > > > >
> > > > > > > > > -Andy
> > > > > > > > >
> > > > > > > > > >I made new tests. I've switched off SSL, excess
> > > > protocols,
> > > > > > > all
> > > > > > > > > >extensions.
> > > > > > > > > >I do not make some connects now, dont create vhosts.
> > > > I've
> > > > > > > > > simplified
> > > > > > > > > >all
> > > > > > > > > >names and parameters. I've built libws with
> > switching
> > > > off
> > > > > > > some
> > > > > > > > > >extra-parameters.
> > > > > > > > > >And I am still crashing.  At first call to
> > > > > > > > > >lws_create_vhost( lwscontext, &info )  - gdb
> > backtrace
> > > > is
> > > > > > > the same
> > > > > > > > > I
> > > > > > > > > >sent
> > > > > > > > > >This is dump of libwebsockets (freebsd, Apple is the
> > > > same)
> > > > > > > > > >
> > > > > > > > > >[2016/11/28 22:28:27:2159] NOTICE: Initial logging
> > level
> > > > 7
> > > > > > > > > >[2016/11/28 22:28:27:2159] NOTICE: Libwebsockets
> > > > version:
> > > > > > > 2.1.0
> > > > > > > > > >beetle at B10.1-v2.0.0-188-ge67dfe1
> > > > > > > > > >[2016/11/28 22:28:27:2159] NOTICE: IPV6 not compiled
> > in
> > > > > > > > > >[2016/11/28 22:28:27:2159] NOTICE: libev support not
> > > > > > > compiled in
> > > > > > > > > >[2016/11/28 22:28:27:2159] NOTICE: libuv support
> > > > compiled in
> > > > > > > and
> > > > > > > > > >enabled
> > > > > > > > > >[2016/11/28 22:28:27:2160] NOTICE:  Threads: 1 each
> > > > 13950
> > > > > > > fds
> > > > > > > > > >[2016/11/28 22:28:27:2161] NOTICE:  mem: platform fd
> > > > map:
> > > > > > > 111600
> > > > > > > > > bytes
> > > > > > > > > >[2016/11/28 22:28:27:2161] NOTICE:  Compiled with
> > > > OpenSSL
> > > > > > > support
> > > > > > > > > >[2016/11/28 22:28:27:2161] NOTICE:  SSL disabled: no
> > > > > > > > > >LWS_SERVER_OPTION_DO_SSL_GLOBAL_INIT
> > > > > > > > > >[2016/11/28 22:28:27:2161] NOTICE:  mem: per-conn: 
> >    
> > > >    
> > > > > > > 688
> > > > > > > > > bytes +
> > > > > > > > > >protocol rx buf
> > > > > > > > > >[2016/11/28 22:28:27:2161] NOTICE: 
> > canonical_hostname =
> > > > > > > B10.1
> > > > > > > > > >[2016/11/28 22:28:27:2162] NOTICE: Creating Vhost
> > > > > > > 'localhost' port
> > > > > > > > > >60000, 1
> > > > > > > > > >protocols, IPv6 off
> > > > > > > > > >
> > > > > > > > > >After clearing LWS_SERVER_OPTION_EXPLICIT_VHOSTS
> > added
> > > > two
> > > > > > > lines
> > > > > > > > > >[2016/11/28 22:29:40:3383] NOTICE: Creating Vhost
> > > > 'default'
> > > > > > > port
> > > > > > > > > 0, 1
> > > > > > > > > >protocols, IPv6 off
> > > > > > > > > >[2016/11/28 22:29:40:3400] NOTICE:  Listening on
> > port
> > > > 60815
> > > > > > > > > >
> > > > > > > > > >but finally segfault catched us. Seems to be my
> > vhost
> > > > has
> > > > > > > some
> > > > > > > > > problem.
> > > > > > > > > >"Segfaulty" problem.
> > > > > > > > > >at this moment I have no more time for experiments,
> > > > tomorrow
> > > > > > > will
> > > > > > > > > try
> > > > > > > > > >to
> > > > > > > > > >continue
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >2016-11-27 23:45 GMT+02:00 Andy Green <andy at warmcat.
> > com>
> > > > :
> > > > > > > > > >
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> On November 28, 2016 4:22:52 AM GMT+08:00, Andy
> > Green
> > > > > > > > > ><andy at warmcat.com>
> > > > > > > > > >> wrote:
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> >On November 28, 2016 1:26:18 AM GMT+08:00,
> > Aleksey
> > > > > > > Zhukovskiy
> > > > > > > > > >> ><beetle at gambler.ru> wrote:
> > > > > > > > > >>
> > > > > > > > > >> >>1. It there a reason to use libuv/ev for better
> > > > > > > perfomance at
> > > > > > > > > all?
> > > > > > > > > >> >
> > > > > > > > > >> >Yes, it will use epoll() rather than poll().  If
> > you
> > > > have
> > > > > > > many
> > > > > > > > > >> >connections, this is faster.
> > > > > > > > > >>
> > > > > > > > > >> Well, on bsd, kqueue.  Basically libuv has
> > backends
> > > > for
> > > > > > > the best
> > > > > > > > > >native
> > > > > > > > > >> event wait on the platform and converts to generic
> > > > libuv
> > > > > > > events.
> > > > > > > > > >>
> > > > > > > > > >> -Andy
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> > 
> 
> 



More information about the Libwebsockets mailing list