The Design and Implementation of the FreeBSD Operating System, Second Edition
Now available: The Design and Implementation of the FreeBSD Operating System (Second Edition)


[ source navigation ] [ diff markup ] [ identifier search ] [ freetext search ] [ file search ] [ list types ] [ track identifier ]

FreeBSD/Linux Kernel Cross Reference
sys/kern/uipc_socket.c

Version: -  FREEBSD  -  FREEBSD-13-STABLE  -  FREEBSD-13-0  -  FREEBSD-12-STABLE  -  FREEBSD-12-0  -  FREEBSD-11-STABLE  -  FREEBSD-11-0  -  FREEBSD-10-STABLE  -  FREEBSD-10-0  -  FREEBSD-9-STABLE  -  FREEBSD-9-0  -  FREEBSD-8-STABLE  -  FREEBSD-8-0  -  FREEBSD-7-STABLE  -  FREEBSD-7-0  -  FREEBSD-6-STABLE  -  FREEBSD-6-0  -  FREEBSD-5-STABLE  -  FREEBSD-5-0  -  FREEBSD-4-STABLE  -  FREEBSD-3-STABLE  -  FREEBSD22  -  l41  -  OPENBSD  -  linux-2.6  -  MK84  -  PLAN9  -  xnu-8792 
SearchContext: -  none  -  3  -  10 

    1 /*-
    2  * Copyright (c) 1982, 1986, 1988, 1990, 1993
    3  *      The Regents of the University of California.
    4  * Copyright (c) 2004 The FreeBSD Foundation
    5  * Copyright (c) 2004-2008 Robert N. M. Watson
    6  * All rights reserved.
    7  *
    8  * Redistribution and use in source and binary forms, with or without
    9  * modification, are permitted provided that the following conditions
   10  * are met:
   11  * 1. Redistributions of source code must retain the above copyright
   12  *    notice, this list of conditions and the following disclaimer.
   13  * 2. Redistributions in binary form must reproduce the above copyright
   14  *    notice, this list of conditions and the following disclaimer in the
   15  *    documentation and/or other materials provided with the distribution.
   16  * 4. Neither the name of the University nor the names of its contributors
   17  *    may be used to endorse or promote products derived from this software
   18  *    without specific prior written permission.
   19  *
   20  * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
   21  * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
   22  * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
   23  * ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
   24  * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
   25  * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
   26  * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
   27  * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
   28  * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
   29  * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
   30  * SUCH DAMAGE.
   31  *
   32  *      @(#)uipc_socket.c       8.3 (Berkeley) 4/15/94
   33  */
   34 
   35 /*
   36  * Comments on the socket life cycle:
   37  *
   38  * soalloc() sets of socket layer state for a socket, called only by
   39  * socreate() and sonewconn().  Socket layer private.
   40  *
   41  * sodealloc() tears down socket layer state for a socket, called only by
   42  * sofree() and sonewconn().  Socket layer private.
   43  *
   44  * pru_attach() associates protocol layer state with an allocated socket;
   45  * called only once, may fail, aborting socket allocation.  This is called
   46  * from socreate() and sonewconn().  Socket layer private.
   47  *
   48  * pru_detach() disassociates protocol layer state from an attached socket,
   49  * and will be called exactly once for sockets in which pru_attach() has
   50  * been successfully called.  If pru_attach() returned an error,
   51  * pru_detach() will not be called.  Socket layer private.
   52  *
   53  * pru_abort() and pru_close() notify the protocol layer that the last
   54  * consumer of a socket is starting to tear down the socket, and that the
   55  * protocol should terminate the connection.  Historically, pru_abort() also
   56  * detached protocol state from the socket state, but this is no longer the
   57  * case.
   58  *
   59  * socreate() creates a socket and attaches protocol state.  This is a public
   60  * interface that may be used by socket layer consumers to create new
   61  * sockets.
   62  *
   63  * sonewconn() creates a socket and attaches protocol state.  This is a
   64  * public interface  that may be used by protocols to create new sockets when
   65  * a new connection is received and will be available for accept() on a
   66  * listen socket.
   67  *
   68  * soclose() destroys a socket after possibly waiting for it to disconnect.
   69  * This is a public interface that socket consumers should use to close and
   70  * release a socket when done with it.
   71  *
   72  * soabort() destroys a socket without waiting for it to disconnect (used
   73  * only for incoming connections that are already partially or fully
   74  * connected).  This is used internally by the socket layer when clearing
   75  * listen socket queues (due to overflow or close on the listen socket), but
   76  * is also a public interface protocols may use to abort connections in
   77  * their incomplete listen queues should they no longer be required.  Sockets
   78  * placed in completed connection listen queues should not be aborted for
   79  * reasons described in the comment above the soclose() implementation.  This
   80  * is not a general purpose close routine, and except in the specific
   81  * circumstances described here, should not be used.
   82  *
   83  * sofree() will free a socket and its protocol state if all references on
   84  * the socket have been released, and is the public interface to attempt to
   85  * free a socket when a reference is removed.  This is a socket layer private
   86  * interface.
   87  *
   88  * NOTE: In addition to socreate() and soclose(), which provide a single
   89  * socket reference to the consumer to be managed as required, there are two
   90  * calls to explicitly manage socket references, soref(), and sorele().
   91  * Currently, these are generally required only when transitioning a socket
   92  * from a listen queue to a file descriptor, in order to prevent garbage
   93  * collection of the socket at an untimely moment.  For a number of reasons,
   94  * these interfaces are not preferred, and should be avoided.
   95  *
   96  * NOTE: With regard to VNETs the general rule is that callers do not set
   97  * curvnet. Exceptions to this rule include soabort(), sodisconnect(),
   98  * sofree() (and with that sorele(), sotryfree()), as well as sonewconn()
   99  * and sorflush(), which are usually called from a pre-set VNET context.
  100  * sopoll() currently does not need a VNET context to be set.
  101  */
  102 
  103 #include <sys/cdefs.h>
  104 __FBSDID("$FreeBSD: releng/11.2/sys/kern/uipc_socket.c 333511 2018-05-12 01:55:24Z jtl $");
  105 
  106 #include "opt_inet.h"
  107 #include "opt_inet6.h"
  108 #include "opt_compat.h"
  109 
  110 #include <sys/param.h>
  111 #include <sys/systm.h>
  112 #include <sys/fcntl.h>
  113 #include <sys/limits.h>
  114 #include <sys/lock.h>
  115 #include <sys/mac.h>
  116 #include <sys/malloc.h>
  117 #include <sys/mbuf.h>
  118 #include <sys/mutex.h>
  119 #include <sys/domain.h>
  120 #include <sys/file.h>                   /* for struct knote */
  121 #include <sys/hhook.h>
  122 #include <sys/kernel.h>
  123 #include <sys/khelp.h>
  124 #include <sys/event.h>
  125 #include <sys/eventhandler.h>
  126 #include <sys/poll.h>
  127 #include <sys/proc.h>
  128 #include <sys/protosw.h>
  129 #include <sys/socket.h>
  130 #include <sys/socketvar.h>
  131 #include <sys/resourcevar.h>
  132 #include <net/route.h>
  133 #include <sys/signalvar.h>
  134 #include <sys/stat.h>
  135 #include <sys/sx.h>
  136 #include <sys/sysctl.h>
  137 #include <sys/taskqueue.h>
  138 #include <sys/uio.h>
  139 #include <sys/jail.h>
  140 #include <sys/syslog.h>
  141 #include <netinet/in.h>
  142 
  143 #include <net/vnet.h>
  144 
  145 #include <security/mac/mac_framework.h>
  146 
  147 #include <vm/uma.h>
  148 
  149 #ifdef COMPAT_FREEBSD32
  150 #include <sys/mount.h>
  151 #include <sys/sysent.h>
  152 #include <compat/freebsd32/freebsd32.h>
  153 #endif
  154 
  155 static int      soreceive_rcvoob(struct socket *so, struct uio *uio,
  156                     int flags);
  157 
  158 static void     filt_sordetach(struct knote *kn);
  159 static int      filt_soread(struct knote *kn, long hint);
  160 static void     filt_sowdetach(struct knote *kn);
  161 static int      filt_sowrite(struct knote *kn, long hint);
  162 static int      filt_solisten(struct knote *kn, long hint);
  163 static int inline hhook_run_socket(struct socket *so, void *hctx, int32_t h_id);
  164 fo_kqfilter_t   soo_kqfilter;
  165 
  166 static struct filterops solisten_filtops = {
  167         .f_isfd = 1,
  168         .f_detach = filt_sordetach,
  169         .f_event = filt_solisten,
  170 };
  171 static struct filterops soread_filtops = {
  172         .f_isfd = 1,
  173         .f_detach = filt_sordetach,
  174         .f_event = filt_soread,
  175 };
  176 static struct filterops sowrite_filtops = {
  177         .f_isfd = 1,
  178         .f_detach = filt_sowdetach,
  179         .f_event = filt_sowrite,
  180 };
  181 
  182 so_gen_t        so_gencnt;      /* generation count for sockets */
  183 
  184 MALLOC_DEFINE(M_SONAME, "soname", "socket name");
  185 MALLOC_DEFINE(M_PCB, "pcb", "protocol control block");
  186 
  187 #define VNET_SO_ASSERT(so)                                              \
  188         VNET_ASSERT(curvnet != NULL,                                    \
  189             ("%s:%d curvnet is NULL, so=%p", __func__, __LINE__, (so)));
  190 
  191 VNET_DEFINE(struct hhook_head *, socket_hhh[HHOOK_SOCKET_LAST + 1]);
  192 #define V_socket_hhh            VNET(socket_hhh)
  193 
  194 /*
  195  * Limit on the number of connections in the listen queue waiting
  196  * for accept(2).
  197  * NB: The original sysctl somaxconn is still available but hidden
  198  * to prevent confusion about the actual purpose of this number.
  199  */
  200 static u_int somaxconn = SOMAXCONN;
  201 
  202 static int
  203 sysctl_somaxconn(SYSCTL_HANDLER_ARGS)
  204 {
  205         int error;
  206         int val;
  207 
  208         val = somaxconn;
  209         error = sysctl_handle_int(oidp, &val, 0, req);
  210         if (error || !req->newptr )
  211                 return (error);
  212 
  213         /*
  214          * The purpose of the UINT_MAX / 3 limit, is so that the formula
  215          *   3 * so_qlimit / 2
  216          * below, will not overflow.
  217          */
  218 
  219         if (val < 1 || val > UINT_MAX / 3)
  220                 return (EINVAL);
  221 
  222         somaxconn = val;
  223         return (0);
  224 }
  225 SYSCTL_PROC(_kern_ipc, OID_AUTO, soacceptqueue, CTLTYPE_UINT | CTLFLAG_RW,
  226     0, sizeof(int), sysctl_somaxconn, "I",
  227     "Maximum listen socket pending connection accept queue size");
  228 SYSCTL_PROC(_kern_ipc, KIPC_SOMAXCONN, somaxconn,
  229     CTLTYPE_UINT | CTLFLAG_RW | CTLFLAG_SKIP,
  230     0, sizeof(int), sysctl_somaxconn, "I",
  231     "Maximum listen socket pending connection accept queue size (compat)");
  232 
  233 static int numopensockets;
  234 SYSCTL_INT(_kern_ipc, OID_AUTO, numopensockets, CTLFLAG_RD,
  235     &numopensockets, 0, "Number of open sockets");
  236 
  237 /*
  238  * accept_mtx locks down per-socket fields relating to accept queues.  See
  239  * socketvar.h for an annotation of the protected fields of struct socket.
  240  */
  241 struct mtx accept_mtx;
  242 MTX_SYSINIT(accept_mtx, &accept_mtx, "accept", MTX_DEF);
  243 
  244 /*
  245  * so_global_mtx protects so_gencnt, numopensockets, and the per-socket
  246  * so_gencnt field.
  247  */
  248 static struct mtx so_global_mtx;
  249 MTX_SYSINIT(so_global_mtx, &so_global_mtx, "so_glabel", MTX_DEF);
  250 
  251 /*
  252  * General IPC sysctl name space, used by sockets and a variety of other IPC
  253  * types.
  254  */
  255 SYSCTL_NODE(_kern, KERN_IPC, ipc, CTLFLAG_RW, 0, "IPC");
  256 
  257 /*
  258  * Initialize the socket subsystem and set up the socket
  259  * memory allocator.
  260  */
  261 static uma_zone_t socket_zone;
  262 int     maxsockets;
  263 
  264 static void
  265 socket_zone_change(void *tag)
  266 {
  267 
  268         maxsockets = uma_zone_set_max(socket_zone, maxsockets);
  269 }
  270 
  271 static void
  272 socket_hhook_register(int subtype)
  273 {
  274         
  275         if (hhook_head_register(HHOOK_TYPE_SOCKET, subtype,
  276             &V_socket_hhh[subtype],
  277             HHOOK_NOWAIT|HHOOK_HEADISINVNET) != 0)
  278                 printf("%s: WARNING: unable to register hook\n", __func__);
  279 }
  280 
  281 static void
  282 socket_hhook_deregister(int subtype)
  283 {
  284         
  285         if (hhook_head_deregister(V_socket_hhh[subtype]) != 0)
  286                 printf("%s: WARNING: unable to deregister hook\n", __func__);
  287 }
  288 
  289 static void
  290 socket_init(void *tag)
  291 {
  292 
  293         socket_zone = uma_zcreate("socket", sizeof(struct socket), NULL, NULL,
  294             NULL, NULL, UMA_ALIGN_PTR, UMA_ZONE_NOFREE);
  295         maxsockets = uma_zone_set_max(socket_zone, maxsockets);
  296         uma_zone_set_warning(socket_zone, "kern.ipc.maxsockets limit reached");
  297         EVENTHANDLER_REGISTER(maxsockets_change, socket_zone_change, NULL,
  298             EVENTHANDLER_PRI_FIRST);
  299 }
  300 SYSINIT(socket, SI_SUB_PROTO_DOMAININIT, SI_ORDER_ANY, socket_init, NULL);
  301 
  302 static void
  303 socket_vnet_init(const void *unused __unused)
  304 {
  305         int i;
  306 
  307         /* We expect a contiguous range */
  308         for (i = 0; i <= HHOOK_SOCKET_LAST; i++)
  309                 socket_hhook_register(i);
  310 }
  311 VNET_SYSINIT(socket_vnet_init, SI_SUB_PROTO_DOMAININIT, SI_ORDER_ANY,
  312     socket_vnet_init, NULL);
  313 
  314 static void
  315 socket_vnet_uninit(const void *unused __unused)
  316 {
  317         int i;
  318 
  319         for (i = 0; i <= HHOOK_SOCKET_LAST; i++)
  320                 socket_hhook_deregister(i);
  321 }
  322 VNET_SYSUNINIT(socket_vnet_uninit, SI_SUB_PROTO_DOMAININIT, SI_ORDER_ANY,
  323     socket_vnet_uninit, NULL);
  324 
  325 /*
  326  * Initialise maxsockets.  This SYSINIT must be run after
  327  * tunable_mbinit().
  328  */
  329 static void
  330 init_maxsockets(void *ignored)
  331 {
  332 
  333         TUNABLE_INT_FETCH("kern.ipc.maxsockets", &maxsockets);
  334         maxsockets = imax(maxsockets, maxfiles);
  335 }
  336 SYSINIT(param, SI_SUB_TUNABLES, SI_ORDER_ANY, init_maxsockets, NULL);
  337 
  338 /*
  339  * Sysctl to get and set the maximum global sockets limit.  Notify protocols
  340  * of the change so that they can update their dependent limits as required.
  341  */
  342 static int
  343 sysctl_maxsockets(SYSCTL_HANDLER_ARGS)
  344 {
  345         int error, newmaxsockets;
  346 
  347         newmaxsockets = maxsockets;
  348         error = sysctl_handle_int(oidp, &newmaxsockets, 0, req);
  349         if (error == 0 && req->newptr) {
  350                 if (newmaxsockets > maxsockets &&
  351                     newmaxsockets <= maxfiles) {
  352                         maxsockets = newmaxsockets;
  353                         EVENTHANDLER_INVOKE(maxsockets_change);
  354                 } else
  355                         error = EINVAL;
  356         }
  357         return (error);
  358 }
  359 SYSCTL_PROC(_kern_ipc, OID_AUTO, maxsockets, CTLTYPE_INT|CTLFLAG_RW,
  360     &maxsockets, 0, sysctl_maxsockets, "IU",
  361     "Maximum number of sockets available");
  362 
  363 /*
  364  * Socket operation routines.  These routines are called by the routines in
  365  * sys_socket.c or from a system process, and implement the semantics of
  366  * socket operations by switching out to the protocol specific routines.
  367  */
  368 
  369 /*
  370  * Get a socket structure from our zone, and initialize it.  Note that it
  371  * would probably be better to allocate socket and PCB at the same time, but
  372  * I'm not convinced that all the protocols can be easily modified to do
  373  * this.
  374  *
  375  * soalloc() returns a socket with a ref count of 0.
  376  */
  377 static struct socket *
  378 soalloc(struct vnet *vnet)
  379 {
  380         struct socket *so;
  381 
  382         so = uma_zalloc(socket_zone, M_NOWAIT | M_ZERO);
  383         if (so == NULL)
  384                 return (NULL);
  385 #ifdef MAC
  386         if (mac_socket_init(so, M_NOWAIT) != 0) {
  387                 uma_zfree(socket_zone, so);
  388                 return (NULL);
  389         }
  390 #endif
  391         if (khelp_init_osd(HELPER_CLASS_SOCKET, &so->osd)) {
  392                 uma_zfree(socket_zone, so);
  393                 return (NULL);
  394         }
  395 
  396         SOCKBUF_LOCK_INIT(&so->so_snd, "so_snd");
  397         SOCKBUF_LOCK_INIT(&so->so_rcv, "so_rcv");
  398         sx_init(&so->so_snd.sb_sx, "so_snd_sx");
  399         sx_init(&so->so_rcv.sb_sx, "so_rcv_sx");
  400         TAILQ_INIT(&so->so_snd.sb_aiojobq);
  401         TAILQ_INIT(&so->so_rcv.sb_aiojobq);
  402         TASK_INIT(&so->so_snd.sb_aiotask, 0, soaio_snd, so);
  403         TASK_INIT(&so->so_rcv.sb_aiotask, 0, soaio_rcv, so);
  404 #ifdef VIMAGE
  405         VNET_ASSERT(vnet != NULL, ("%s:%d vnet is NULL, so=%p",
  406             __func__, __LINE__, so));
  407         so->so_vnet = vnet;
  408 #endif
  409         /* We shouldn't need the so_global_mtx */
  410         if (hhook_run_socket(so, NULL, HHOOK_SOCKET_CREATE)) {
  411                 /* Do we need more comprehensive error returns? */
  412                 uma_zfree(socket_zone, so);
  413                 return (NULL);
  414         }
  415         mtx_lock(&so_global_mtx);
  416         so->so_gencnt = ++so_gencnt;
  417         ++numopensockets;
  418 #ifdef VIMAGE
  419         vnet->vnet_sockcnt++;
  420 #endif
  421         mtx_unlock(&so_global_mtx);
  422 
  423         return (so);
  424 }
  425 
  426 /*
  427  * Free the storage associated with a socket at the socket layer, tear down
  428  * locks, labels, etc.  All protocol state is assumed already to have been
  429  * torn down (and possibly never set up) by the caller.
  430  */
  431 static void
  432 sodealloc(struct socket *so)
  433 {
  434 
  435         KASSERT(so->so_count == 0, ("sodealloc(): so_count %d", so->so_count));
  436         KASSERT(so->so_pcb == NULL, ("sodealloc(): so_pcb != NULL"));
  437 
  438         mtx_lock(&so_global_mtx);
  439         so->so_gencnt = ++so_gencnt;
  440         --numopensockets;       /* Could be below, but faster here. */
  441 #ifdef VIMAGE
  442         VNET_ASSERT(so->so_vnet != NULL, ("%s:%d so_vnet is NULL, so=%p",
  443             __func__, __LINE__, so));
  444         so->so_vnet->vnet_sockcnt--;
  445 #endif
  446         mtx_unlock(&so_global_mtx);
  447         if (so->so_rcv.sb_hiwat)
  448                 (void)chgsbsize(so->so_cred->cr_uidinfo,
  449                     &so->so_rcv.sb_hiwat, 0, RLIM_INFINITY);
  450         if (so->so_snd.sb_hiwat)
  451                 (void)chgsbsize(so->so_cred->cr_uidinfo,
  452                     &so->so_snd.sb_hiwat, 0, RLIM_INFINITY);
  453         /* remove accept filter if one is present. */
  454         if (so->so_accf != NULL)
  455                 do_setopt_accept_filter(so, NULL);
  456 #ifdef MAC
  457         mac_socket_destroy(so);
  458 #endif
  459         hhook_run_socket(so, NULL, HHOOK_SOCKET_CLOSE);
  460 
  461         crfree(so->so_cred);
  462         khelp_destroy_osd(&so->osd);
  463         sx_destroy(&so->so_snd.sb_sx);
  464         sx_destroy(&so->so_rcv.sb_sx);
  465         SOCKBUF_LOCK_DESTROY(&so->so_snd);
  466         SOCKBUF_LOCK_DESTROY(&so->so_rcv);
  467         uma_zfree(socket_zone, so);
  468 }
  469 
  470 /*
  471  * socreate returns a socket with a ref count of 1.  The socket should be
  472  * closed with soclose().
  473  */
  474 int
  475 socreate(int dom, struct socket **aso, int type, int proto,
  476     struct ucred *cred, struct thread *td)
  477 {
  478         struct protosw *prp;
  479         struct socket *so;
  480         int error;
  481 
  482         if (proto)
  483                 prp = pffindproto(dom, proto, type);
  484         else
  485                 prp = pffindtype(dom, type);
  486 
  487         if (prp == NULL) {
  488                 /* No support for domain. */
  489                 if (pffinddomain(dom) == NULL)
  490                         return (EAFNOSUPPORT);
  491                 /* No support for socket type. */
  492                 if (proto == 0 && type != 0)
  493                         return (EPROTOTYPE);
  494                 return (EPROTONOSUPPORT);
  495         }
  496         if (prp->pr_usrreqs->pru_attach == NULL ||
  497             prp->pr_usrreqs->pru_attach == pru_attach_notsupp)
  498                 return (EPROTONOSUPPORT);
  499 
  500         if (prison_check_af(cred, prp->pr_domain->dom_family) != 0)
  501                 return (EPROTONOSUPPORT);
  502 
  503         if (prp->pr_type != type)
  504                 return (EPROTOTYPE);
  505         so = soalloc(CRED_TO_VNET(cred));
  506         if (so == NULL)
  507                 return (ENOBUFS);
  508 
  509         TAILQ_INIT(&so->so_incomp);
  510         TAILQ_INIT(&so->so_comp);
  511         so->so_type = type;
  512         so->so_cred = crhold(cred);
  513         if ((prp->pr_domain->dom_family == PF_INET) ||
  514             (prp->pr_domain->dom_family == PF_INET6) ||
  515             (prp->pr_domain->dom_family == PF_ROUTE))
  516                 so->so_fibnum = td->td_proc->p_fibnum;
  517         else
  518                 so->so_fibnum = 0;
  519         so->so_proto = prp;
  520 #ifdef MAC
  521         mac_socket_create(cred, so);
  522 #endif
  523         knlist_init_mtx(&so->so_rcv.sb_sel.si_note, SOCKBUF_MTX(&so->so_rcv));
  524         knlist_init_mtx(&so->so_snd.sb_sel.si_note, SOCKBUF_MTX(&so->so_snd));
  525         so->so_count = 1;
  526         /*
  527          * Auto-sizing of socket buffers is managed by the protocols and
  528          * the appropriate flags must be set in the pru_attach function.
  529          */
  530         CURVNET_SET(so->so_vnet);
  531         error = (*prp->pr_usrreqs->pru_attach)(so, proto, td);
  532         CURVNET_RESTORE();
  533         if (error) {
  534                 KASSERT(so->so_count == 1, ("socreate: so_count %d",
  535                     so->so_count));
  536                 so->so_count = 0;
  537                 sodealloc(so);
  538                 return (error);
  539         }
  540         *aso = so;
  541         return (0);
  542 }
  543 
  544 #ifdef REGRESSION
  545 static int regression_sonewconn_earlytest = 1;
  546 SYSCTL_INT(_regression, OID_AUTO, sonewconn_earlytest, CTLFLAG_RW,
  547     &regression_sonewconn_earlytest, 0, "Perform early sonewconn limit test");
  548 #endif
  549 
  550 /*
  551  * When an attempt at a new connection is noted on a socket which accepts
  552  * connections, sonewconn is called.  If the connection is possible (subject
  553  * to space constraints, etc.) then we allocate a new structure, properly
  554  * linked into the data structure of the original socket, and return this.
  555  * Connstatus may be 0, or SS_ISCONFIRMING, or SS_ISCONNECTED.
  556  *
  557  * Note: the ref count on the socket is 0 on return.
  558  */
  559 struct socket *
  560 sonewconn(struct socket *head, int connstatus)
  561 {
  562         static struct timeval lastover;
  563         static struct timeval overinterval = { 60, 0 };
  564         static int overcount;
  565 
  566         struct socket *so;
  567         int over;
  568 
  569         ACCEPT_LOCK();
  570         over = (head->so_qlen > 3 * head->so_qlimit / 2);
  571         ACCEPT_UNLOCK();
  572 #ifdef REGRESSION
  573         if (regression_sonewconn_earlytest && over) {
  574 #else
  575         if (over) {
  576 #endif
  577                 overcount++;
  578 
  579                 if (ratecheck(&lastover, &overinterval)) {
  580                         log(LOG_DEBUG, "%s: pcb %p: Listen queue overflow: "
  581                             "%i already in queue awaiting acceptance "
  582                             "(%d occurrences)\n",
  583                             __func__, head->so_pcb, head->so_qlen, overcount);
  584 
  585                         overcount = 0;
  586                 }
  587 
  588                 return (NULL);
  589         }
  590         VNET_ASSERT(head->so_vnet != NULL, ("%s:%d so_vnet is NULL, head=%p",
  591             __func__, __LINE__, head));
  592         so = soalloc(head->so_vnet);
  593         if (so == NULL) {
  594                 log(LOG_DEBUG, "%s: pcb %p: New socket allocation failure: "
  595                     "limit reached or out of memory\n",
  596                     __func__, head->so_pcb);
  597                 return (NULL);
  598         }
  599         if ((head->so_options & SO_ACCEPTFILTER) != 0)
  600                 connstatus = 0;
  601         so->so_head = head;
  602         so->so_type = head->so_type;
  603         so->so_options = head->so_options &~ SO_ACCEPTCONN;
  604         so->so_linger = head->so_linger;
  605         so->so_state = head->so_state | SS_NOFDREF;
  606         so->so_fibnum = head->so_fibnum;
  607         so->so_proto = head->so_proto;
  608         so->so_cred = crhold(head->so_cred);
  609 #ifdef MAC
  610         mac_socket_newconn(head, so);
  611 #endif
  612         knlist_init_mtx(&so->so_rcv.sb_sel.si_note, SOCKBUF_MTX(&so->so_rcv));
  613         knlist_init_mtx(&so->so_snd.sb_sel.si_note, SOCKBUF_MTX(&so->so_snd));
  614         VNET_SO_ASSERT(head);
  615         if (soreserve(so, head->so_snd.sb_hiwat, head->so_rcv.sb_hiwat)) {
  616                 sodealloc(so);
  617                 log(LOG_DEBUG, "%s: pcb %p: soreserve() failed\n",
  618                     __func__, head->so_pcb);
  619                 return (NULL);
  620         }
  621         if ((*so->so_proto->pr_usrreqs->pru_attach)(so, 0, NULL)) {
  622                 sodealloc(so);
  623                 log(LOG_DEBUG, "%s: pcb %p: pru_attach() failed\n",
  624                     __func__, head->so_pcb);
  625                 return (NULL);
  626         }
  627         so->so_rcv.sb_lowat = head->so_rcv.sb_lowat;
  628         so->so_snd.sb_lowat = head->so_snd.sb_lowat;
  629         so->so_rcv.sb_timeo = head->so_rcv.sb_timeo;
  630         so->so_snd.sb_timeo = head->so_snd.sb_timeo;
  631         so->so_rcv.sb_flags |= head->so_rcv.sb_flags & SB_AUTOSIZE;
  632         so->so_snd.sb_flags |= head->so_snd.sb_flags & SB_AUTOSIZE;
  633         so->so_state |= connstatus;
  634         ACCEPT_LOCK();
  635         /*
  636          * The accept socket may be tearing down but we just
  637          * won a race on the ACCEPT_LOCK.
  638          * However, if sctp_peeloff() is called on a 1-to-many
  639          * style socket, the SO_ACCEPTCONN doesn't need to be set.
  640          */
  641         if (!(head->so_options & SO_ACCEPTCONN) &&
  642             ((head->so_proto->pr_protocol != IPPROTO_SCTP) ||
  643              (head->so_type != SOCK_SEQPACKET))) {
  644                 SOCK_LOCK(so);
  645                 so->so_head = NULL;
  646                 sofree(so);             /* NB: returns ACCEPT_UNLOCK'ed. */
  647                 return (NULL);
  648         }
  649         if (connstatus) {
  650                 TAILQ_INSERT_TAIL(&head->so_comp, so, so_list);
  651                 so->so_qstate |= SQ_COMP;
  652                 head->so_qlen++;
  653         } else {
  654                 /*
  655                  * Keep removing sockets from the head until there's room for
  656                  * us to insert on the tail.  In pre-locking revisions, this
  657                  * was a simple if(), but as we could be racing with other
  658                  * threads and soabort() requires dropping locks, we must
  659                  * loop waiting for the condition to be true.
  660                  */
  661                 while (head->so_incqlen > head->so_qlimit) {
  662                         struct socket *sp;
  663                         sp = TAILQ_FIRST(&head->so_incomp);
  664                         TAILQ_REMOVE(&head->so_incomp, sp, so_list);
  665                         head->so_incqlen--;
  666                         sp->so_qstate &= ~SQ_INCOMP;
  667                         sp->so_head = NULL;
  668                         ACCEPT_UNLOCK();
  669                         soabort(sp);
  670                         ACCEPT_LOCK();
  671                 }
  672                 TAILQ_INSERT_TAIL(&head->so_incomp, so, so_list);
  673                 so->so_qstate |= SQ_INCOMP;
  674                 head->so_incqlen++;
  675         }
  676         ACCEPT_UNLOCK();
  677         if (connstatus) {
  678                 sorwakeup(head);
  679                 wakeup_one(&head->so_timeo);
  680         }
  681         return (so);
  682 }
  683 
  684 int
  685 sobind(struct socket *so, struct sockaddr *nam, struct thread *td)
  686 {
  687         int error;
  688 
  689         CURVNET_SET(so->so_vnet);
  690         error = (*so->so_proto->pr_usrreqs->pru_bind)(so, nam, td);
  691         CURVNET_RESTORE();
  692         return (error);
  693 }
  694 
  695 int
  696 sobindat(int fd, struct socket *so, struct sockaddr *nam, struct thread *td)
  697 {
  698         int error;
  699 
  700         CURVNET_SET(so->so_vnet);
  701         error = (*so->so_proto->pr_usrreqs->pru_bindat)(fd, so, nam, td);
  702         CURVNET_RESTORE();
  703         return (error);
  704 }
  705 
  706 /*
  707  * solisten() transitions a socket from a non-listening state to a listening
  708  * state, but can also be used to update the listen queue depth on an
  709  * existing listen socket.  The protocol will call back into the sockets
  710  * layer using solisten_proto_check() and solisten_proto() to check and set
  711  * socket-layer listen state.  Call backs are used so that the protocol can
  712  * acquire both protocol and socket layer locks in whatever order is required
  713  * by the protocol.
  714  *
  715  * Protocol implementors are advised to hold the socket lock across the
  716  * socket-layer test and set to avoid races at the socket layer.
  717  */
  718 int
  719 solisten(struct socket *so, int backlog, struct thread *td)
  720 {
  721         int error;
  722 
  723         CURVNET_SET(so->so_vnet);
  724         error = (*so->so_proto->pr_usrreqs->pru_listen)(so, backlog, td);
  725         CURVNET_RESTORE();
  726         return (error);
  727 }
  728 
  729 int
  730 solisten_proto_check(struct socket *so)
  731 {
  732 
  733         SOCK_LOCK_ASSERT(so);
  734 
  735         if (so->so_state & (SS_ISCONNECTED | SS_ISCONNECTING |
  736             SS_ISDISCONNECTING))
  737                 return (EINVAL);
  738         return (0);
  739 }
  740 
  741 void
  742 solisten_proto(struct socket *so, int backlog)
  743 {
  744 
  745         SOCK_LOCK_ASSERT(so);
  746 
  747         if (backlog < 0 || backlog > somaxconn)
  748                 backlog = somaxconn;
  749         so->so_qlimit = backlog;
  750         so->so_options |= SO_ACCEPTCONN;
  751 }
  752 
  753 /*
  754  * Evaluate the reference count and named references on a socket; if no
  755  * references remain, free it.  This should be called whenever a reference is
  756  * released, such as in sorele(), but also when named reference flags are
  757  * cleared in socket or protocol code.
  758  *
  759  * sofree() will free the socket if:
  760  *
  761  * - There are no outstanding file descriptor references or related consumers
  762  *   (so_count == 0).
  763  *
  764  * - The socket has been closed by user space, if ever open (SS_NOFDREF).
  765  *
  766  * - The protocol does not have an outstanding strong reference on the socket
  767  *   (SS_PROTOREF).
  768  *
  769  * - The socket is not in a completed connection queue, so a process has been
  770  *   notified that it is present.  If it is removed, the user process may
  771  *   block in accept() despite select() saying the socket was ready.
  772  */
  773 void
  774 sofree(struct socket *so)
  775 {
  776         struct protosw *pr = so->so_proto;
  777         struct socket *head;
  778 
  779         ACCEPT_LOCK_ASSERT();
  780         SOCK_LOCK_ASSERT(so);
  781 
  782         if ((so->so_state & SS_NOFDREF) == 0 || so->so_count != 0 ||
  783             (so->so_state & SS_PROTOREF) || (so->so_qstate & SQ_COMP)) {
  784                 SOCK_UNLOCK(so);
  785                 ACCEPT_UNLOCK();
  786                 return;
  787         }
  788 
  789         head = so->so_head;
  790         if (head != NULL) {
  791                 KASSERT((so->so_qstate & SQ_COMP) != 0 ||
  792                     (so->so_qstate & SQ_INCOMP) != 0,
  793                     ("sofree: so_head != NULL, but neither SQ_COMP nor "
  794                     "SQ_INCOMP"));
  795                 KASSERT((so->so_qstate & SQ_COMP) == 0 ||
  796                     (so->so_qstate & SQ_INCOMP) == 0,
  797                     ("sofree: so->so_qstate is SQ_COMP and also SQ_INCOMP"));
  798                 TAILQ_REMOVE(&head->so_incomp, so, so_list);
  799                 head->so_incqlen--;
  800                 so->so_qstate &= ~SQ_INCOMP;
  801                 so->so_head = NULL;
  802         }
  803         KASSERT((so->so_qstate & SQ_COMP) == 0 &&
  804             (so->so_qstate & SQ_INCOMP) == 0,
  805             ("sofree: so_head == NULL, but still SQ_COMP(%d) or SQ_INCOMP(%d)",
  806             so->so_qstate & SQ_COMP, so->so_qstate & SQ_INCOMP));
  807         if (so->so_options & SO_ACCEPTCONN) {
  808                 KASSERT((TAILQ_EMPTY(&so->so_comp)),
  809                     ("sofree: so_comp populated"));
  810                 KASSERT((TAILQ_EMPTY(&so->so_incomp)),
  811                     ("sofree: so_incomp populated"));
  812         }
  813         SOCK_UNLOCK(so);
  814         ACCEPT_UNLOCK();
  815 
  816         VNET_SO_ASSERT(so);
  817         if (pr->pr_flags & PR_RIGHTS && pr->pr_domain->dom_dispose != NULL)
  818                 (*pr->pr_domain->dom_dispose)(so);
  819         if (pr->pr_usrreqs->pru_detach != NULL)
  820                 (*pr->pr_usrreqs->pru_detach)(so);
  821 
  822         /*
  823          * From this point on, we assume that no other references to this
  824          * socket exist anywhere else in the stack.  Therefore, no locks need
  825          * to be acquired or held.
  826          *
  827          * We used to do a lot of socket buffer and socket locking here, as
  828          * well as invoke sorflush() and perform wakeups.  The direct call to
  829          * dom_dispose() and sbrelease_internal() are an inlining of what was
  830          * necessary from sorflush().
  831          *
  832          * Notice that the socket buffer and kqueue state are torn down
  833          * before calling pru_detach.  This means that protocols shold not
  834          * assume they can perform socket wakeups, etc, in their detach code.
  835          */
  836         sbdestroy(&so->so_snd, so);
  837         sbdestroy(&so->so_rcv, so);
  838         seldrain(&so->so_snd.sb_sel);
  839         seldrain(&so->so_rcv.sb_sel);
  840         knlist_destroy(&so->so_rcv.sb_sel.si_note);
  841         knlist_destroy(&so->so_snd.sb_sel.si_note);
  842         sodealloc(so);
  843 }
  844 
  845 /*
  846  * Close a socket on last file table reference removal.  Initiate disconnect
  847  * if connected.  Free socket when disconnect complete.
  848  *
  849  * This function will sorele() the socket.  Note that soclose() may be called
  850  * prior to the ref count reaching zero.  The actual socket structure will
  851  * not be freed until the ref count reaches zero.
  852  */
  853 int
  854 soclose(struct socket *so)
  855 {
  856         int error = 0;
  857 
  858         KASSERT(!(so->so_state & SS_NOFDREF), ("soclose: SS_NOFDREF on enter"));
  859 
  860         CURVNET_SET(so->so_vnet);
  861         funsetown(&so->so_sigio);
  862         if (so->so_state & SS_ISCONNECTED) {
  863                 if ((so->so_state & SS_ISDISCONNECTING) == 0) {
  864                         error = sodisconnect(so);
  865                         if (error) {
  866                                 if (error == ENOTCONN)
  867                                         error = 0;
  868                                 goto drop;
  869                         }
  870                 }
  871                 if (so->so_options & SO_LINGER) {
  872                         if ((so->so_state & SS_ISDISCONNECTING) &&
  873                             (so->so_state & SS_NBIO))
  874                                 goto drop;
  875                         while (so->so_state & SS_ISCONNECTED) {
  876                                 error = tsleep(&so->so_timeo,
  877                                     PSOCK | PCATCH, "soclos",
  878                                     so->so_linger * hz);
  879                                 if (error)
  880                                         break;
  881                         }
  882                 }
  883         }
  884 
  885 drop:
  886         if (so->so_proto->pr_usrreqs->pru_close != NULL)
  887                 (*so->so_proto->pr_usrreqs->pru_close)(so);
  888         ACCEPT_LOCK();
  889         if (so->so_options & SO_ACCEPTCONN) {
  890                 struct socket *sp;
  891                 /*
  892                  * Prevent new additions to the accept queues due
  893                  * to ACCEPT_LOCK races while we are draining them.
  894                  */
  895                 so->so_options &= ~SO_ACCEPTCONN;
  896                 while ((sp = TAILQ_FIRST(&so->so_incomp)) != NULL) {
  897                         TAILQ_REMOVE(&so->so_incomp, sp, so_list);
  898                         so->so_incqlen--;
  899                         sp->so_qstate &= ~SQ_INCOMP;
  900                         sp->so_head = NULL;
  901                         ACCEPT_UNLOCK();
  902                         soabort(sp);
  903                         ACCEPT_LOCK();
  904                 }
  905                 while ((sp = TAILQ_FIRST(&so->so_comp)) != NULL) {
  906                         TAILQ_REMOVE(&so->so_comp, sp, so_list);
  907                         so->so_qlen--;
  908                         sp->so_qstate &= ~SQ_COMP;
  909                         sp->so_head = NULL;
  910                         ACCEPT_UNLOCK();
  911                         soabort(sp);
  912                         ACCEPT_LOCK();
  913                 }
  914                 KASSERT((TAILQ_EMPTY(&so->so_comp)),
  915                     ("%s: so_comp populated", __func__));
  916                 KASSERT((TAILQ_EMPTY(&so->so_incomp)),
  917                     ("%s: so_incomp populated", __func__));
  918         }
  919         SOCK_LOCK(so);
  920         KASSERT((so->so_state & SS_NOFDREF) == 0, ("soclose: NOFDREF"));
  921         so->so_state |= SS_NOFDREF;
  922         sorele(so);                     /* NB: Returns with ACCEPT_UNLOCK(). */
  923         CURVNET_RESTORE();
  924         return (error);
  925 }
  926 
  927 /*
  928  * soabort() is used to abruptly tear down a connection, such as when a
  929  * resource limit is reached (listen queue depth exceeded), or if a listen
  930  * socket is closed while there are sockets waiting to be accepted.
  931  *
  932  * This interface is tricky, because it is called on an unreferenced socket,
  933  * and must be called only by a thread that has actually removed the socket
  934  * from the listen queue it was on, or races with other threads are risked.
  935  *
  936  * This interface will call into the protocol code, so must not be called
  937  * with any socket locks held.  Protocols do call it while holding their own
  938  * recursible protocol mutexes, but this is something that should be subject
  939  * to review in the future.
  940  */
  941 void
  942 soabort(struct socket *so)
  943 {
  944 
  945         /*
  946          * In as much as is possible, assert that no references to this
  947          * socket are held.  This is not quite the same as asserting that the
  948          * current thread is responsible for arranging for no references, but
  949          * is as close as we can get for now.
  950          */
  951         KASSERT(so->so_count == 0, ("soabort: so_count"));
  952         KASSERT((so->so_state & SS_PROTOREF) == 0, ("soabort: SS_PROTOREF"));
  953         KASSERT(so->so_state & SS_NOFDREF, ("soabort: !SS_NOFDREF"));
  954         KASSERT((so->so_state & SQ_COMP) == 0, ("soabort: SQ_COMP"));
  955         KASSERT((so->so_state & SQ_INCOMP) == 0, ("soabort: SQ_INCOMP"));
  956         VNET_SO_ASSERT(so);
  957 
  958         if (so->so_proto->pr_usrreqs->pru_abort != NULL)
  959                 (*so->so_proto->pr_usrreqs->pru_abort)(so);
  960         ACCEPT_LOCK();
  961         SOCK_LOCK(so);
  962         sofree(so);
  963 }
  964 
  965 int
  966 soaccept(struct socket *so, struct sockaddr **nam)
  967 {
  968         int error;
  969 
  970         SOCK_LOCK(so);
  971         KASSERT((so->so_state & SS_NOFDREF) != 0, ("soaccept: !NOFDREF"));
  972         so->so_state &= ~SS_NOFDREF;
  973         SOCK_UNLOCK(so);
  974 
  975         CURVNET_SET(so->so_vnet);
  976         error = (*so->so_proto->pr_usrreqs->pru_accept)(so, nam);
  977         CURVNET_RESTORE();
  978         return (error);
  979 }
  980 
  981 int
  982 soconnect(struct socket *so, struct sockaddr *nam, struct thread *td)
  983 {
  984 
  985         return (soconnectat(AT_FDCWD, so, nam, td));
  986 }
  987 
  988 int
  989 soconnectat(int fd, struct socket *so, struct sockaddr *nam, struct thread *td)
  990 {
  991         int error;
  992 
  993         if (so->so_options & SO_ACCEPTCONN)
  994                 return (EOPNOTSUPP);
  995 
  996         CURVNET_SET(so->so_vnet);
  997         /*
  998          * If protocol is connection-based, can only connect once.
  999          * Otherwise, if connected, try to disconnect first.  This allows
 1000          * user to disconnect by connecting to, e.g., a null address.
 1001          */
 1002         if (so->so_state & (SS_ISCONNECTED|SS_ISCONNECTING) &&
 1003             ((so->so_proto->pr_flags & PR_CONNREQUIRED) ||
 1004             (error = sodisconnect(so)))) {
 1005                 error = EISCONN;
 1006         } else {
 1007                 /*
 1008                  * Prevent accumulated error from previous connection from
 1009                  * biting us.
 1010                  */
 1011                 so->so_error = 0;
 1012                 if (fd == AT_FDCWD) {
 1013                         error = (*so->so_proto->pr_usrreqs->pru_connect)(so,
 1014                             nam, td);
 1015                 } else {
 1016                         error = (*so->so_proto->pr_usrreqs->pru_connectat)(fd,
 1017                             so, nam, td);
 1018                 }
 1019         }
 1020         CURVNET_RESTORE();
 1021 
 1022         return (error);
 1023 }
 1024 
 1025 int
 1026 soconnect2(struct socket *so1, struct socket *so2)
 1027 {
 1028         int error;
 1029 
 1030         CURVNET_SET(so1->so_vnet);
 1031         error = (*so1->so_proto->pr_usrreqs->pru_connect2)(so1, so2);
 1032         CURVNET_RESTORE();
 1033         return (error);
 1034 }
 1035 
 1036 int
 1037 sodisconnect(struct socket *so)
 1038 {
 1039         int error;
 1040 
 1041         if ((so->so_state & SS_ISCONNECTED) == 0)
 1042                 return (ENOTCONN);
 1043         if (so->so_state & SS_ISDISCONNECTING)
 1044                 return (EALREADY);
 1045         VNET_SO_ASSERT(so);
 1046         error = (*so->so_proto->pr_usrreqs->pru_disconnect)(so);
 1047         return (error);
 1048 }
 1049 
 1050 #define SBLOCKWAIT(f)   (((f) & MSG_DONTWAIT) ? 0 : SBL_WAIT)
 1051 
 1052 int
 1053 sosend_dgram(struct socket *so, struct sockaddr *addr, struct uio *uio,
 1054     struct mbuf *top, struct mbuf *control, int flags, struct thread *td)
 1055 {
 1056         long space;
 1057         ssize_t resid;
 1058         int clen = 0, error, dontroute;
 1059 
 1060         KASSERT(so->so_type == SOCK_DGRAM, ("sosend_dgram: !SOCK_DGRAM"));
 1061         KASSERT(so->so_proto->pr_flags & PR_ATOMIC,
 1062             ("sosend_dgram: !PR_ATOMIC"));
 1063 
 1064         if (uio != NULL)
 1065                 resid = uio->uio_resid;
 1066         else
 1067                 resid = top->m_pkthdr.len;
 1068         /*
 1069          * In theory resid should be unsigned.  However, space must be
 1070          * signed, as it might be less than 0 if we over-committed, and we
 1071          * must use a signed comparison of space and resid.  On the other
 1072          * hand, a negative resid causes us to loop sending 0-length
 1073          * segments to the protocol.
 1074          */
 1075         if (resid < 0) {
 1076                 error = EINVAL;
 1077                 goto out;
 1078         }
 1079 
 1080         dontroute =
 1081             (flags & MSG_DONTROUTE) && (so->so_options & SO_DONTROUTE) == 0;
 1082         if (td != NULL)
 1083                 td->td_ru.ru_msgsnd++;
 1084         if (control != NULL)
 1085                 clen = control->m_len;
 1086 
 1087         SOCKBUF_LOCK(&so->so_snd);
 1088         if (so->so_snd.sb_state & SBS_CANTSENDMORE) {
 1089                 SOCKBUF_UNLOCK(&so->so_snd);
 1090                 error = EPIPE;
 1091                 goto out;
 1092         }
 1093         if (so->so_error) {
 1094                 error = so->so_error;
 1095                 so->so_error = 0;
 1096                 SOCKBUF_UNLOCK(&so->so_snd);
 1097                 goto out;
 1098         }
 1099         if ((so->so_state & SS_ISCONNECTED) == 0) {
 1100                 /*
 1101                  * `sendto' and `sendmsg' is allowed on a connection-based
 1102                  * socket if it supports implied connect.  Return ENOTCONN if
 1103                  * not connected and no address is supplied.
 1104                  */
 1105                 if ((so->so_proto->pr_flags & PR_CONNREQUIRED) &&
 1106                     (so->so_proto->pr_flags & PR_IMPLOPCL) == 0) {
 1107                         if ((so->so_state & SS_ISCONFIRMING) == 0 &&
 1108                             !(resid == 0 && clen != 0)) {
 1109                                 SOCKBUF_UNLOCK(&so->so_snd);
 1110                                 error = ENOTCONN;
 1111                                 goto out;
 1112                         }
 1113                 } else if (addr == NULL) {
 1114                         if (so->so_proto->pr_flags & PR_CONNREQUIRED)
 1115                                 error = ENOTCONN;
 1116                         else
 1117                                 error = EDESTADDRREQ;
 1118                         SOCKBUF_UNLOCK(&so->so_snd);
 1119                         goto out;
 1120                 }
 1121         }
 1122 
 1123         /*
 1124          * Do we need MSG_OOB support in SOCK_DGRAM?  Signs here may be a
 1125          * problem and need fixing.
 1126          */
 1127         space = sbspace(&so->so_snd);
 1128         if (flags & MSG_OOB)
 1129                 space += 1024;
 1130         space -= clen;
 1131         SOCKBUF_UNLOCK(&so->so_snd);
 1132         if (resid > space) {
 1133                 error = EMSGSIZE;
 1134                 goto out;
 1135         }
 1136         if (uio == NULL) {
 1137                 resid = 0;
 1138                 if (flags & MSG_EOR)
 1139                         top->m_flags |= M_EOR;
 1140         } else {
 1141                 /*
 1142                  * Copy the data from userland into a mbuf chain.
 1143                  * If no data is to be copied in, a single empty mbuf
 1144                  * is returned.
 1145                  */
 1146                 top = m_uiotombuf(uio, M_WAITOK, space, max_hdr,
 1147                     (M_PKTHDR | ((flags & MSG_EOR) ? M_EOR : 0)));
 1148                 if (top == NULL) {
 1149                         error = EFAULT; /* only possible error */
 1150                         goto out;
 1151                 }
 1152                 space -= resid - uio->uio_resid;
 1153                 resid = uio->uio_resid;
 1154         }
 1155         KASSERT(resid == 0, ("sosend_dgram: resid != 0"));
 1156         /*
 1157          * XXXRW: Frobbing SO_DONTROUTE here is even worse without sblock
 1158          * than with.
 1159          */
 1160         if (dontroute) {
 1161                 SOCK_LOCK(so);
 1162                 so->so_options |= SO_DONTROUTE;
 1163                 SOCK_UNLOCK(so);
 1164         }
 1165         /*
 1166          * XXX all the SBS_CANTSENDMORE checks previously done could be out
 1167          * of date.  We could have received a reset packet in an interrupt or
 1168          * maybe we slept while doing page faults in uiomove() etc.  We could
 1169          * probably recheck again inside the locking protection here, but
 1170          * there are probably other places that this also happens.  We must
 1171          * rethink this.
 1172          */
 1173         VNET_SO_ASSERT(so);
 1174         error = (*so->so_proto->pr_usrreqs->pru_send)(so,
 1175             (flags & MSG_OOB) ? PRUS_OOB :
 1176         /*
 1177          * If the user set MSG_EOF, the protocol understands this flag and
 1178          * nothing left to send then use PRU_SEND_EOF instead of PRU_SEND.
 1179          */
 1180             ((flags & MSG_EOF) &&
 1181              (so->so_proto->pr_flags & PR_IMPLOPCL) &&
 1182              (resid <= 0)) ?
 1183                 PRUS_EOF :
 1184                 /* If there is more to send set PRUS_MORETOCOME */
 1185                 (flags & MSG_MORETOCOME) ||
 1186                 (resid > 0 && space > 0) ? PRUS_MORETOCOME : 0,
 1187                 top, addr, control, td);
 1188         if (dontroute) {
 1189                 SOCK_LOCK(so);
 1190                 so->so_options &= ~SO_DONTROUTE;
 1191                 SOCK_UNLOCK(so);
 1192         }
 1193         clen = 0;
 1194         control = NULL;
 1195         top = NULL;
 1196 out:
 1197         if (top != NULL)
 1198                 m_freem(top);
 1199         if (control != NULL)
 1200                 m_freem(control);
 1201         return (error);
 1202 }
 1203 
 1204 /*
 1205  * Send on a socket.  If send must go all at once and message is larger than
 1206  * send buffering, then hard error.  Lock against other senders.  If must go
 1207  * all at once and not enough room now, then inform user that this would
 1208  * block and do nothing.  Otherwise, if nonblocking, send as much as
 1209  * possible.  The data to be sent is described by "uio" if nonzero, otherwise
 1210  * by the mbuf chain "top" (which must be null if uio is not).  Data provided
 1211  * in mbuf chain must be small enough to send all at once.
 1212  *
 1213  * Returns nonzero on error, timeout or signal; callers must check for short
 1214  * counts if EINTR/ERESTART are returned.  Data and control buffers are freed
 1215  * on return.
 1216  */
 1217 int
 1218 sosend_generic(struct socket *so, struct sockaddr *addr, struct uio *uio,
 1219     struct mbuf *top, struct mbuf *control, int flags, struct thread *td)
 1220 {
 1221         long space;
 1222         ssize_t resid;
 1223         int clen = 0, error, dontroute;
 1224         int atomic = sosendallatonce(so) || top;
 1225 
 1226         if (uio != NULL)
 1227                 resid = uio->uio_resid;
 1228         else
 1229                 resid = top->m_pkthdr.len;
 1230         /*
 1231          * In theory resid should be unsigned.  However, space must be
 1232          * signed, as it might be less than 0 if we over-committed, and we
 1233          * must use a signed comparison of space and resid.  On the other
 1234          * hand, a negative resid causes us to loop sending 0-length
 1235          * segments to the protocol.
 1236          *
 1237          * Also check to make sure that MSG_EOR isn't used on SOCK_STREAM
 1238          * type sockets since that's an error.
 1239          */
 1240         if (resid < 0 || (so->so_type == SOCK_STREAM && (flags & MSG_EOR))) {
 1241                 error = EINVAL;
 1242                 goto out;
 1243         }
 1244 
 1245         dontroute =
 1246             (flags & MSG_DONTROUTE) && (so->so_options & SO_DONTROUTE) == 0 &&
 1247             (so->so_proto->pr_flags & PR_ATOMIC);
 1248         if (td != NULL)
 1249                 td->td_ru.ru_msgsnd++;
 1250         if (control != NULL)
 1251                 clen = control->m_len;
 1252 
 1253         error = sblock(&so->so_snd, SBLOCKWAIT(flags));
 1254         if (error)
 1255                 goto out;
 1256 
 1257 restart:
 1258         do {
 1259                 SOCKBUF_LOCK(&so->so_snd);
 1260                 if (so->so_snd.sb_state & SBS_CANTSENDMORE) {
 1261                         SOCKBUF_UNLOCK(&so->so_snd);
 1262                         error = EPIPE;
 1263                         goto release;
 1264                 }
 1265                 if (so->so_error) {
 1266                         error = so->so_error;
 1267                         so->so_error = 0;
 1268                         SOCKBUF_UNLOCK(&so->so_snd);
 1269                         goto release;
 1270                 }
 1271                 if ((so->so_state & SS_ISCONNECTED) == 0) {
 1272                         /*
 1273                          * `sendto' and `sendmsg' is allowed on a connection-
 1274                          * based socket if it supports implied connect.
 1275                          * Return ENOTCONN if not connected and no address is
 1276                          * supplied.
 1277                          */
 1278                         if ((so->so_proto->pr_flags & PR_CONNREQUIRED) &&
 1279                             (so->so_proto->pr_flags & PR_IMPLOPCL) == 0) {
 1280                                 if ((so->so_state & SS_ISCONFIRMING) == 0 &&
 1281                                     !(resid == 0 && clen != 0)) {
 1282                                         SOCKBUF_UNLOCK(&so->so_snd);
 1283                                         error = ENOTCONN;
 1284                                         goto release;
 1285                                 }
 1286                         } else if (addr == NULL) {
 1287                                 SOCKBUF_UNLOCK(&so->so_snd);
 1288                                 if (so->so_proto->pr_flags & PR_CONNREQUIRED)
 1289                                         error = ENOTCONN;
 1290                                 else
 1291                                         error = EDESTADDRREQ;
 1292                                 goto release;
 1293                         }
 1294                 }
 1295                 space = sbspace(&so->so_snd);
 1296                 if (flags & MSG_OOB)
 1297                         space += 1024;
 1298                 if ((atomic && resid > so->so_snd.sb_hiwat) ||
 1299                     clen > so->so_snd.sb_hiwat) {
 1300                         SOCKBUF_UNLOCK(&so->so_snd);
 1301                         error = EMSGSIZE;
 1302                         goto release;
 1303                 }
 1304                 if (space < resid + clen &&
 1305                     (atomic || space < so->so_snd.sb_lowat || space < clen)) {
 1306                         if ((so->so_state & SS_NBIO) || (flags & MSG_NBIO)) {
 1307                                 SOCKBUF_UNLOCK(&so->so_snd);
 1308                                 error = EWOULDBLOCK;
 1309                                 goto release;
 1310                         }
 1311                         error = sbwait(&so->so_snd);
 1312                         SOCKBUF_UNLOCK(&so->so_snd);
 1313                         if (error)
 1314                                 goto release;
 1315                         goto restart;
 1316                 }
 1317                 SOCKBUF_UNLOCK(&so->so_snd);
 1318                 space -= clen;
 1319                 do {
 1320                         if (uio == NULL) {
 1321                                 resid = 0;
 1322                                 if (flags & MSG_EOR)
 1323                                         top->m_flags |= M_EOR;
 1324                         } else {
 1325                                 /*
 1326                                  * Copy the data from userland into a mbuf
 1327                                  * chain.  If resid is 0, which can happen
 1328                                  * only if we have control to send, then
 1329                                  * a single empty mbuf is returned.  This
 1330                                  * is a workaround to prevent protocol send
 1331                                  * methods to panic.
 1332                                  */
 1333                                 top = m_uiotombuf(uio, M_WAITOK, space,
 1334                                     (atomic ? max_hdr : 0),
 1335                                     (atomic ? M_PKTHDR : 0) |
 1336                                     ((flags & MSG_EOR) ? M_EOR : 0));
 1337                                 if (top == NULL) {
 1338                                         error = EFAULT; /* only possible error */
 1339                                         goto release;
 1340                                 }
 1341                                 space -= resid - uio->uio_resid;
 1342                                 resid = uio->uio_resid;
 1343                         }
 1344                         if (dontroute) {
 1345                                 SOCK_LOCK(so);
 1346                                 so->so_options |= SO_DONTROUTE;
 1347                                 SOCK_UNLOCK(so);
 1348                         }
 1349                         /*
 1350                          * XXX all the SBS_CANTSENDMORE checks previously
 1351                          * done could be out of date.  We could have received
 1352                          * a reset packet in an interrupt or maybe we slept
 1353                          * while doing page faults in uiomove() etc.  We
 1354                          * could probably recheck again inside the locking
 1355                          * protection here, but there are probably other
 1356                          * places that this also happens.  We must rethink
 1357                          * this.
 1358                          */
 1359                         VNET_SO_ASSERT(so);
 1360                         error = (*so->so_proto->pr_usrreqs->pru_send)(so,
 1361                             (flags & MSG_OOB) ? PRUS_OOB :
 1362                         /*
 1363                          * If the user set MSG_EOF, the protocol understands
 1364                          * this flag and nothing left to send then use
 1365                          * PRU_SEND_EOF instead of PRU_SEND.
 1366                          */
 1367                             ((flags & MSG_EOF) &&
 1368                              (so->so_proto->pr_flags & PR_IMPLOPCL) &&
 1369                              (resid <= 0)) ?
 1370                                 PRUS_EOF :
 1371                         /* If there is more to send set PRUS_MORETOCOME. */
 1372                             (flags & MSG_MORETOCOME) ||
 1373                             (resid > 0 && space > 0) ? PRUS_MORETOCOME : 0,
 1374                             top, addr, control, td);
 1375                         if (dontroute) {
 1376                                 SOCK_LOCK(so);
 1377                                 so->so_options &= ~SO_DONTROUTE;
 1378                                 SOCK_UNLOCK(so);
 1379                         }
 1380                         clen = 0;
 1381                         control = NULL;
 1382                         top = NULL;
 1383                         if (error)
 1384                                 goto release;
 1385                 } while (resid && space > 0);
 1386         } while (resid);
 1387 
 1388 release:
 1389         sbunlock(&so->so_snd);
 1390 out:
 1391         if (top != NULL)
 1392                 m_freem(top);
 1393         if (control != NULL)
 1394                 m_freem(control);
 1395         return (error);
 1396 }
 1397 
 1398 int
 1399 sosend(struct socket *so, struct sockaddr *addr, struct uio *uio,
 1400     struct mbuf *top, struct mbuf *control, int flags, struct thread *td)
 1401 {
 1402         int error;
 1403 
 1404         CURVNET_SET(so->so_vnet);
 1405         error = so->so_proto->pr_usrreqs->pru_sosend(so, addr, uio, top,
 1406             control, flags, td);
 1407         CURVNET_RESTORE();
 1408         return (error);
 1409 }
 1410 
 1411 /*
 1412  * The part of soreceive() that implements reading non-inline out-of-band
 1413  * data from a socket.  For more complete comments, see soreceive(), from
 1414  * which this code originated.
 1415  *
 1416  * Note that soreceive_rcvoob(), unlike the remainder of soreceive(), is
 1417  * unable to return an mbuf chain to the caller.
 1418  */
 1419 static int
 1420 soreceive_rcvoob(struct socket *so, struct uio *uio, int flags)
 1421 {
 1422         struct protosw *pr = so->so_proto;
 1423         struct mbuf *m;
 1424         int error;
 1425 
 1426         KASSERT(flags & MSG_OOB, ("soreceive_rcvoob: (flags & MSG_OOB) == 0"));
 1427         VNET_SO_ASSERT(so);
 1428 
 1429         m = m_get(M_WAITOK, MT_DATA);
 1430         error = (*pr->pr_usrreqs->pru_rcvoob)(so, m, flags & MSG_PEEK);
 1431         if (error)
 1432                 goto bad;
 1433         do {
 1434                 error = uiomove(mtod(m, void *),
 1435                     (int) min(uio->uio_resid, m->m_len), uio);
 1436                 m = m_free(m);
 1437         } while (uio->uio_resid && error == 0 && m);
 1438 bad:
 1439         if (m != NULL)
 1440                 m_freem(m);
 1441         return (error);
 1442 }
 1443 
 1444 /*
 1445  * Following replacement or removal of the first mbuf on the first mbuf chain
 1446  * of a socket buffer, push necessary state changes back into the socket
 1447  * buffer so that other consumers see the values consistently.  'nextrecord'
 1448  * is the callers locally stored value of the original value of
 1449  * sb->sb_mb->m_nextpkt which must be restored when the lead mbuf changes.
 1450  * NOTE: 'nextrecord' may be NULL.
 1451  */
 1452 static __inline void
 1453 sockbuf_pushsync(struct sockbuf *sb, struct mbuf *nextrecord)
 1454 {
 1455 
 1456         SOCKBUF_LOCK_ASSERT(sb);
 1457         /*
 1458          * First, update for the new value of nextrecord.  If necessary, make
 1459          * it the first record.
 1460          */
 1461         if (sb->sb_mb != NULL)
 1462                 sb->sb_mb->m_nextpkt = nextrecord;
 1463         else
 1464                 sb->sb_mb = nextrecord;
 1465 
 1466         /*
 1467          * Now update any dependent socket buffer fields to reflect the new
 1468          * state.  This is an expanded inline of SB_EMPTY_FIXUP(), with the
 1469          * addition of a second clause that takes care of the case where
 1470          * sb_mb has been updated, but remains the last record.
 1471          */
 1472         if (sb->sb_mb == NULL) {
 1473                 sb->sb_mbtail = NULL;
 1474                 sb->sb_lastrecord = NULL;
 1475         } else if (sb->sb_mb->m_nextpkt == NULL)
 1476                 sb->sb_lastrecord = sb->sb_mb;
 1477 }
 1478 
 1479 /*
 1480  * Implement receive operations on a socket.  We depend on the way that
 1481  * records are added to the sockbuf by sbappend.  In particular, each record
 1482  * (mbufs linked through m_next) must begin with an address if the protocol
 1483  * so specifies, followed by an optional mbuf or mbufs containing ancillary
 1484  * data, and then zero or more mbufs of data.  In order to allow parallelism
 1485  * between network receive and copying to user space, as well as avoid
 1486  * sleeping with a mutex held, we release the socket buffer mutex during the
 1487  * user space copy.  Although the sockbuf is locked, new data may still be
 1488  * appended, and thus we must maintain consistency of the sockbuf during that
 1489  * time.
 1490  *
 1491  * The caller may receive the data as a single mbuf chain by supplying an
 1492  * mbuf **mp0 for use in returning the chain.  The uio is then used only for
 1493  * the count in uio_resid.
 1494  */
 1495 int
 1496 soreceive_generic(struct socket *so, struct sockaddr **psa, struct uio *uio,
 1497     struct mbuf **mp0, struct mbuf **controlp, int *flagsp)
 1498 {
 1499         struct mbuf *m, **mp;
 1500         int flags, error, offset;
 1501         ssize_t len;
 1502         struct protosw *pr = so->so_proto;
 1503         struct mbuf *nextrecord;
 1504         int moff, type = 0;
 1505         ssize_t orig_resid = uio->uio_resid;
 1506 
 1507         mp = mp0;
 1508         if (psa != NULL)
 1509                 *psa = NULL;
 1510         if (controlp != NULL)
 1511                 *controlp = NULL;
 1512         if (flagsp != NULL)
 1513                 flags = *flagsp &~ MSG_EOR;
 1514         else
 1515                 flags = 0;
 1516         if (flags & MSG_OOB)
 1517                 return (soreceive_rcvoob(so, uio, flags));
 1518         if (mp != NULL)
 1519                 *mp = NULL;
 1520         if ((pr->pr_flags & PR_WANTRCVD) && (so->so_state & SS_ISCONFIRMING)
 1521             && uio->uio_resid) {
 1522                 VNET_SO_ASSERT(so);
 1523                 (*pr->pr_usrreqs->pru_rcvd)(so, 0);
 1524         }
 1525 
 1526         error = sblock(&so->so_rcv, SBLOCKWAIT(flags));
 1527         if (error)
 1528                 return (error);
 1529 
 1530 restart:
 1531         SOCKBUF_LOCK(&so->so_rcv);
 1532         m = so->so_rcv.sb_mb;
 1533         /*
 1534          * If we have less data than requested, block awaiting more (subject
 1535          * to any timeout) if:
 1536          *   1. the current count is less than the low water mark, or
 1537          *   2. MSG_DONTWAIT is not set
 1538          */
 1539         if (m == NULL || (((flags & MSG_DONTWAIT) == 0 &&
 1540             sbavail(&so->so_rcv) < uio->uio_resid) &&
 1541             sbavail(&so->so_rcv) < so->so_rcv.sb_lowat &&
 1542             m->m_nextpkt == NULL && (pr->pr_flags & PR_ATOMIC) == 0)) {
 1543                 KASSERT(m != NULL || !sbavail(&so->so_rcv),
 1544                     ("receive: m == %p sbavail == %u",
 1545                     m, sbavail(&so->so_rcv)));
 1546                 if (so->so_error) {
 1547                         if (m != NULL)
 1548                                 goto dontblock;
 1549                         error = so->so_error;
 1550                         if ((flags & MSG_PEEK) == 0)
 1551                                 so->so_error = 0;
 1552                         SOCKBUF_UNLOCK(&so->so_rcv);
 1553                         goto release;
 1554                 }
 1555                 SOCKBUF_LOCK_ASSERT(&so->so_rcv);
 1556                 if (so->so_rcv.sb_state & SBS_CANTRCVMORE) {
 1557                         if (m == NULL) {
 1558                                 SOCKBUF_UNLOCK(&so->so_rcv);
 1559                                 goto release;
 1560                         } else
 1561                                 goto dontblock;
 1562                 }
 1563                 for (; m != NULL; m = m->m_next)
 1564                         if (m->m_type == MT_OOBDATA  || (m->m_flags & M_EOR)) {
 1565                                 m = so->so_rcv.sb_mb;
 1566                                 goto dontblock;
 1567                         }
 1568                 if ((so->so_state & (SS_ISCONNECTED|SS_ISCONNECTING)) == 0 &&
 1569                     (so->so_proto->pr_flags & PR_CONNREQUIRED)) {
 1570                         SOCKBUF_UNLOCK(&so->so_rcv);
 1571                         error = ENOTCONN;
 1572                         goto release;
 1573                 }
 1574                 if (uio->uio_resid == 0) {
 1575                         SOCKBUF_UNLOCK(&so->so_rcv);
 1576                         goto release;
 1577                 }
 1578                 if ((so->so_state & SS_NBIO) ||
 1579                     (flags & (MSG_DONTWAIT|MSG_NBIO))) {
 1580                         SOCKBUF_UNLOCK(&so->so_rcv);
 1581                         error = EWOULDBLOCK;
 1582                         goto release;
 1583                 }
 1584                 SBLASTRECORDCHK(&so->so_rcv);
 1585                 SBLASTMBUFCHK(&so->so_rcv);
 1586                 error = sbwait(&so->so_rcv);
 1587                 SOCKBUF_UNLOCK(&so->so_rcv);
 1588                 if (error)
 1589                         goto release;
 1590                 goto restart;
 1591         }
 1592 dontblock:
 1593         /*
 1594          * From this point onward, we maintain 'nextrecord' as a cache of the
 1595          * pointer to the next record in the socket buffer.  We must keep the
 1596          * various socket buffer pointers and local stack versions of the
 1597          * pointers in sync, pushing out modifications before dropping the
 1598          * socket buffer mutex, and re-reading them when picking it up.
 1599          *
 1600          * Otherwise, we will race with the network stack appending new data
 1601          * or records onto the socket buffer by using inconsistent/stale
 1602          * versions of the field, possibly resulting in socket buffer
 1603          * corruption.
 1604          *
 1605          * By holding the high-level sblock(), we prevent simultaneous
 1606          * readers from pulling off the front of the socket buffer.
 1607          */
 1608         SOCKBUF_LOCK_ASSERT(&so->so_rcv);
 1609         if (uio->uio_td)
 1610                 uio->uio_td->td_ru.ru_msgrcv++;
 1611         KASSERT(m == so->so_rcv.sb_mb, ("soreceive: m != so->so_rcv.sb_mb"));
 1612         SBLASTRECORDCHK(&so->so_rcv);
 1613         SBLASTMBUFCHK(&so->so_rcv);
 1614         nextrecord = m->m_nextpkt;
 1615         if (pr->pr_flags & PR_ADDR) {
 1616                 KASSERT(m->m_type == MT_SONAME,
 1617                     ("m->m_type == %d", m->m_type));
 1618                 orig_resid = 0;
 1619                 if (psa != NULL)
 1620                         *psa = sodupsockaddr(mtod(m, struct sockaddr *),
 1621                             M_NOWAIT);
 1622                 if (flags & MSG_PEEK) {
 1623                         m = m->m_next;
 1624                 } else {
 1625                         sbfree(&so->so_rcv, m);
 1626                         so->so_rcv.sb_mb = m_free(m);
 1627                         m = so->so_rcv.sb_mb;
 1628                         sockbuf_pushsync(&so->so_rcv, nextrecord);
 1629                 }
 1630         }
 1631 
 1632         /*
 1633          * Process one or more MT_CONTROL mbufs present before any data mbufs
 1634          * in the first mbuf chain on the socket buffer.  If MSG_PEEK, we
 1635          * just copy the data; if !MSG_PEEK, we call into the protocol to
 1636          * perform externalization (or freeing if controlp == NULL).
 1637          */
 1638         if (m != NULL && m->m_type == MT_CONTROL) {
 1639                 struct mbuf *cm = NULL, *cmn;
 1640                 struct mbuf **cme = &cm;
 1641 
 1642                 do {
 1643                         if (flags & MSG_PEEK) {
 1644                                 if (controlp != NULL) {
 1645                                         *controlp = m_copy(m, 0, m->m_len);
 1646                                         controlp = &(*controlp)->m_next;
 1647                                 }
 1648                                 m = m->m_next;
 1649                         } else {
 1650                                 sbfree(&so->so_rcv, m);
 1651                                 so->so_rcv.sb_mb = m->m_next;
 1652                                 m->m_next = NULL;
 1653                                 *cme = m;
 1654                                 cme = &(*cme)->m_next;
 1655                                 m = so->so_rcv.sb_mb;
 1656                         }
 1657                 } while (m != NULL && m->m_type == MT_CONTROL);
 1658                 if ((flags & MSG_PEEK) == 0)
 1659                         sockbuf_pushsync(&so->so_rcv, nextrecord);
 1660                 while (cm != NULL) {
 1661                         cmn = cm->m_next;
 1662                         cm->m_next = NULL;
 1663                         if (pr->pr_domain->dom_externalize != NULL) {
 1664                                 SOCKBUF_UNLOCK(&so->so_rcv);
 1665                                 VNET_SO_ASSERT(so);
 1666                                 error = (*pr->pr_domain->dom_externalize)
 1667                                     (cm, controlp, flags);
 1668                                 SOCKBUF_LOCK(&so->so_rcv);
 1669                         } else if (controlp != NULL)
 1670                                 *controlp = cm;
 1671                         else
 1672                                 m_freem(cm);
 1673                         if (controlp != NULL) {
 1674                                 orig_resid = 0;
 1675                                 while (*controlp != NULL)
 1676                                         controlp = &(*controlp)->m_next;
 1677                         }
 1678                         cm = cmn;
 1679                 }
 1680                 if (m != NULL)
 1681                         nextrecord = so->so_rcv.sb_mb->m_nextpkt;
 1682                 else
 1683                         nextrecord = so->so_rcv.sb_mb;
 1684                 orig_resid = 0;
 1685         }
 1686         if (m != NULL) {
 1687                 if ((flags & MSG_PEEK) == 0) {
 1688                         KASSERT(m->m_nextpkt == nextrecord,
 1689                             ("soreceive: post-control, nextrecord !sync"));
 1690                         if (nextrecord == NULL) {
 1691                                 KASSERT(so->so_rcv.sb_mb == m,
 1692                                     ("soreceive: post-control, sb_mb!=m"));
 1693                                 KASSERT(so->so_rcv.sb_lastrecord == m,
 1694                                     ("soreceive: post-control, lastrecord!=m"));
 1695                         }
 1696                 }
 1697                 type = m->m_type;
 1698                 if (type == MT_OOBDATA)
 1699                         flags |= MSG_OOB;
 1700         } else {
 1701                 if ((flags & MSG_PEEK) == 0) {
 1702                         KASSERT(so->so_rcv.sb_mb == nextrecord,
 1703                             ("soreceive: sb_mb != nextrecord"));
 1704                         if (so->so_rcv.sb_mb == NULL) {
 1705                                 KASSERT(so->so_rcv.sb_lastrecord == NULL,
 1706                                     ("soreceive: sb_lastercord != NULL"));
 1707                         }
 1708                 }
 1709         }
 1710         SOCKBUF_LOCK_ASSERT(&so->so_rcv);
 1711         SBLASTRECORDCHK(&so->so_rcv);
 1712         SBLASTMBUFCHK(&so->so_rcv);
 1713 
 1714         /*
 1715          * Now continue to read any data mbufs off of the head of the socket
 1716          * buffer until the read request is satisfied.  Note that 'type' is
 1717          * used to store the type of any mbuf reads that have happened so far
 1718          * such that soreceive() can stop reading if the type changes, which
 1719          * causes soreceive() to return only one of regular data and inline
 1720          * out-of-band data in a single socket receive operation.
 1721          */
 1722         moff = 0;
 1723         offset = 0;
 1724         while (m != NULL && !(m->m_flags & M_NOTAVAIL) && uio->uio_resid > 0
 1725             && error == 0) {
 1726                 /*
 1727                  * If the type of mbuf has changed since the last mbuf
 1728                  * examined ('type'), end the receive operation.
 1729                  */
 1730                 SOCKBUF_LOCK_ASSERT(&so->so_rcv);
 1731                 if (m->m_type == MT_OOBDATA || m->m_type == MT_CONTROL) {
 1732                         if (type != m->m_type)
 1733                                 break;
 1734                 } else if (type == MT_OOBDATA)
 1735                         break;
 1736                 else
 1737                     KASSERT(m->m_type == MT_DATA,
 1738                         ("m->m_type == %d", m->m_type));
 1739                 so->so_rcv.sb_state &= ~SBS_RCVATMARK;
 1740                 len = uio->uio_resid;
 1741                 if (so->so_oobmark && len > so->so_oobmark - offset)
 1742                         len = so->so_oobmark - offset;
 1743                 if (len > m->m_len - moff)
 1744                         len = m->m_len - moff;
 1745                 /*
 1746                  * If mp is set, just pass back the mbufs.  Otherwise copy
 1747                  * them out via the uio, then free.  Sockbuf must be
 1748                  * consistent here (points to current mbuf, it points to next
 1749                  * record) when we drop priority; we must note any additions
 1750                  * to the sockbuf when we block interrupts again.
 1751                  */
 1752                 if (mp == NULL) {
 1753                         SOCKBUF_LOCK_ASSERT(&so->so_rcv);
 1754                         SBLASTRECORDCHK(&so->so_rcv);
 1755                         SBLASTMBUFCHK(&so->so_rcv);
 1756                         SOCKBUF_UNLOCK(&so->so_rcv);
 1757                         error = uiomove(mtod(m, char *) + moff, (int)len, uio);
 1758                         SOCKBUF_LOCK(&so->so_rcv);
 1759                         if (error) {
 1760                                 /*
 1761                                  * The MT_SONAME mbuf has already been removed
 1762                                  * from the record, so it is necessary to
 1763                                  * remove the data mbufs, if any, to preserve
 1764                                  * the invariant in the case of PR_ADDR that
 1765                                  * requires MT_SONAME mbufs at the head of
 1766                                  * each record.
 1767                                  */
 1768                                 if (m && pr->pr_flags & PR_ATOMIC &&
 1769                                     ((flags & MSG_PEEK) == 0))
 1770                                         (void)sbdroprecord_locked(&so->so_rcv);
 1771                                 SOCKBUF_UNLOCK(&so->so_rcv);
 1772                                 goto release;
 1773                         }
 1774                 } else
 1775                         uio->uio_resid -= len;
 1776                 SOCKBUF_LOCK_ASSERT(&so->so_rcv);
 1777                 if (len == m->m_len - moff) {
 1778                         if (m->m_flags & M_EOR)
 1779                                 flags |= MSG_EOR;
 1780                         if (flags & MSG_PEEK) {
 1781                                 m = m->m_next;
 1782                                 moff = 0;
 1783                         } else {
 1784                                 nextrecord = m->m_nextpkt;
 1785                                 sbfree(&so->so_rcv, m);
 1786                                 if (mp != NULL) {
 1787                                         m->m_nextpkt = NULL;
 1788                                         *mp = m;
 1789                                         mp = &m->m_next;
 1790                                         so->so_rcv.sb_mb = m = m->m_next;
 1791                                         *mp = NULL;
 1792                                 } else {
 1793                                         so->so_rcv.sb_mb = m_free(m);
 1794                                         m = so->so_rcv.sb_mb;
 1795                                 }
 1796                                 sockbuf_pushsync(&so->so_rcv, nextrecord);
 1797                                 SBLASTRECORDCHK(&so->so_rcv);
 1798                                 SBLASTMBUFCHK(&so->so_rcv);
 1799                         }
 1800                 } else {
 1801                         if (flags & MSG_PEEK)
 1802                                 moff += len;
 1803                         else {
 1804                                 if (mp != NULL) {
 1805                                         if (flags & MSG_DONTWAIT) {
 1806                                                 *mp = m_copym(m, 0, len,
 1807                                                     M_NOWAIT);
 1808                                                 if (*mp == NULL) {
 1809                                                         /*
 1810                                                          * m_copym() couldn't
 1811                                                          * allocate an mbuf.
 1812                                                          * Adjust uio_resid back
 1813                                                          * (it was adjusted
 1814                                                          * down by len bytes,
 1815                                                          * which we didn't end
 1816                                                          * up "copying" over).
 1817                                                          */
 1818                                                         uio->uio_resid += len;
 1819                                                         break;
 1820                                                 }
 1821                                         } else {
 1822                                                 SOCKBUF_UNLOCK(&so->so_rcv);
 1823                                                 *mp = m_copym(m, 0, len,
 1824                                                     M_WAITOK);
 1825                                                 SOCKBUF_LOCK(&so->so_rcv);
 1826                                         }
 1827                                 }
 1828                                 sbcut_locked(&so->so_rcv, len);
 1829                         }
 1830                 }
 1831                 SOCKBUF_LOCK_ASSERT(&so->so_rcv);
 1832                 if (so->so_oobmark) {
 1833                         if ((flags & MSG_PEEK) == 0) {
 1834                                 so->so_oobmark -= len;
 1835                                 if (so->so_oobmark == 0) {
 1836                                         so->so_rcv.sb_state |= SBS_RCVATMARK;
 1837                                         break;
 1838                                 }
 1839                         } else {
 1840                                 offset += len;
 1841                                 if (offset == so->so_oobmark)
 1842                                         break;
 1843                         }
 1844                 }
 1845                 if (flags & MSG_EOR)
 1846                         break;
 1847                 /*
 1848                  * If the MSG_WAITALL flag is set (for non-atomic socket), we
 1849                  * must not quit until "uio->uio_resid == 0" or an error
 1850                  * termination.  If a signal/timeout occurs, return with a
 1851                  * short count but without error.  Keep sockbuf locked
 1852                  * against other readers.
 1853                  */
 1854                 while (flags & MSG_WAITALL && m == NULL && uio->uio_resid > 0 &&
 1855                     !sosendallatonce(so) && nextrecord == NULL) {
 1856                         SOCKBUF_LOCK_ASSERT(&so->so_rcv);
 1857                         if (so->so_error ||
 1858                             so->so_rcv.sb_state & SBS_CANTRCVMORE)
 1859                                 break;
 1860                         /*
 1861                          * Notify the protocol that some data has been
 1862                          * drained before blocking.
 1863                          */
 1864                         if (pr->pr_flags & PR_WANTRCVD) {
 1865                                 SOCKBUF_UNLOCK(&so->so_rcv);
 1866                                 VNET_SO_ASSERT(so);
 1867                                 (*pr->pr_usrreqs->pru_rcvd)(so, flags);
 1868                                 SOCKBUF_LOCK(&so->so_rcv);
 1869                         }
 1870                         SBLASTRECORDCHK(&so->so_rcv);
 1871                         SBLASTMBUFCHK(&so->so_rcv);
 1872                         /*
 1873                          * We could receive some data while was notifying
 1874                          * the protocol. Skip blocking in this case.
 1875                          */
 1876                         if (so->so_rcv.sb_mb == NULL) {
 1877                                 error = sbwait(&so->so_rcv);
 1878                                 if (error) {
 1879                                         SOCKBUF_UNLOCK(&so->so_rcv);
 1880                                         goto release;
 1881                                 }
 1882                         }
 1883                         m = so->so_rcv.sb_mb;
 1884                         if (m != NULL)
 1885                                 nextrecord = m->m_nextpkt;
 1886                 }
 1887         }
 1888 
 1889         SOCKBUF_LOCK_ASSERT(&so->so_rcv);
 1890         if (m != NULL && pr->pr_flags & PR_ATOMIC) {
 1891                 flags |= MSG_TRUNC;
 1892                 if ((flags & MSG_PEEK) == 0)
 1893                         (void) sbdroprecord_locked(&so->so_rcv);
 1894         }
 1895         if ((flags & MSG_PEEK) == 0) {
 1896                 if (m == NULL) {
 1897                         /*
 1898                          * First part is an inline SB_EMPTY_FIXUP().  Second
 1899                          * part makes sure sb_lastrecord is up-to-date if
 1900                          * there is still data in the socket buffer.
 1901                          */
 1902                         so->so_rcv.sb_mb = nextrecord;
 1903                         if (so->so_rcv.sb_mb == NULL) {
 1904                                 so->so_rcv.sb_mbtail = NULL;
 1905                                 so->so_rcv.sb_lastrecord = NULL;
 1906                         } else if (nextrecord->m_nextpkt == NULL)
 1907                                 so->so_rcv.sb_lastrecord = nextrecord;
 1908                 }
 1909                 SBLASTRECORDCHK(&so->so_rcv);
 1910                 SBLASTMBUFCHK(&so->so_rcv);
 1911                 /*
 1912                  * If soreceive() is being done from the socket callback,
 1913                  * then don't need to generate ACK to peer to update window,
 1914                  * since ACK will be generated on return to TCP.
 1915                  */
 1916                 if (!(flags & MSG_SOCALLBCK) &&
 1917                     (pr->pr_flags & PR_WANTRCVD)) {
 1918                         SOCKBUF_UNLOCK(&so->so_rcv);
 1919                         VNET_SO_ASSERT(so);
 1920                         (*pr->pr_usrreqs->pru_rcvd)(so, flags);
 1921                         SOCKBUF_LOCK(&so->so_rcv);
 1922                 }
 1923         }
 1924         SOCKBUF_LOCK_ASSERT(&so->so_rcv);
 1925         if (orig_resid == uio->uio_resid && orig_resid &&
 1926             (flags & MSG_EOR) == 0 && (so->so_rcv.sb_state & SBS_CANTRCVMORE) == 0) {
 1927                 SOCKBUF_UNLOCK(&so->so_rcv);
 1928                 goto restart;
 1929         }
 1930         SOCKBUF_UNLOCK(&so->so_rcv);
 1931 
 1932         if (flagsp != NULL)
 1933                 *flagsp |= flags;
 1934 release:
 1935         sbunlock(&so->so_rcv);
 1936         return (error);
 1937 }
 1938 
 1939 /*
 1940  * Optimized version of soreceive() for stream (TCP) sockets.
 1941  * XXXAO: (MSG_WAITALL | MSG_PEEK) isn't properly handled.
 1942  */
 1943 int
 1944 soreceive_stream(struct socket *so, struct sockaddr **psa, struct uio *uio,
 1945     struct mbuf **mp0, struct mbuf **controlp, int *flagsp)
 1946 {
 1947         int len = 0, error = 0, flags, oresid;
 1948         struct sockbuf *sb;
 1949         struct mbuf *m, *n = NULL;
 1950 
 1951         /* We only do stream sockets. */
 1952         if (so->so_type != SOCK_STREAM)
 1953                 return (EINVAL);
 1954         if (psa != NULL)
 1955                 *psa = NULL;
 1956         if (controlp != NULL)
 1957                 return (EINVAL);
 1958         if (flagsp != NULL)
 1959                 flags = *flagsp &~ MSG_EOR;
 1960         else
 1961                 flags = 0;
 1962         if (flags & MSG_OOB)
 1963                 return (soreceive_rcvoob(so, uio, flags));
 1964         if (mp0 != NULL)
 1965                 *mp0 = NULL;
 1966 
 1967         sb = &so->so_rcv;
 1968 
 1969         /* Prevent other readers from entering the socket. */
 1970         error = sblock(sb, SBLOCKWAIT(flags));
 1971         if (error)
 1972                 goto out;
 1973         SOCKBUF_LOCK(sb);
 1974 
 1975         /* Easy one, no space to copyout anything. */
 1976         if (uio->uio_resid == 0) {
 1977                 error = EINVAL;
 1978                 goto out;
 1979         }
 1980         oresid = uio->uio_resid;
 1981 
 1982         /* We will never ever get anything unless we are or were connected. */
 1983         if (!(so->so_state & (SS_ISCONNECTED|SS_ISDISCONNECTED))) {
 1984                 error = ENOTCONN;
 1985                 goto out;
 1986         }
 1987 
 1988 restart:
 1989         SOCKBUF_LOCK_ASSERT(&so->so_rcv);
 1990 
 1991         /* Abort if socket has reported problems. */
 1992         if (so->so_error) {
 1993                 if (sbavail(sb) > 0)
 1994                         goto deliver;
 1995                 if (oresid > uio->uio_resid)
 1996                         goto out;
 1997                 error = so->so_error;
 1998                 if (!(flags & MSG_PEEK))
 1999                         so->so_error = 0;
 2000                 goto out;
 2001         }
 2002 
 2003         /* Door is closed.  Deliver what is left, if any. */
 2004         if (sb->sb_state & SBS_CANTRCVMORE) {
 2005                 if (sbavail(sb) > 0)
 2006                         goto deliver;
 2007                 else
 2008                         goto out;
 2009         }
 2010 
 2011         /* Socket buffer is empty and we shall not block. */
 2012         if (sbavail(sb) == 0 &&
 2013             ((so->so_state & SS_NBIO) || (flags & (MSG_DONTWAIT|MSG_NBIO)))) {
 2014                 error = EAGAIN;
 2015                 goto out;
 2016         }
 2017 
 2018         /* Socket buffer got some data that we shall deliver now. */
 2019         if (sbavail(sb) > 0 && !(flags & MSG_WAITALL) &&
 2020             ((so->so_state & SS_NBIO) ||
 2021              (flags & (MSG_DONTWAIT|MSG_NBIO)) ||
 2022              sbavail(sb) >= sb->sb_lowat ||
 2023              sbavail(sb) >= uio->uio_resid ||
 2024              sbavail(sb) >= sb->sb_hiwat) ) {
 2025                 goto deliver;
 2026         }
 2027 
 2028         /* On MSG_WAITALL we must wait until all data or error arrives. */
 2029         if ((flags & MSG_WAITALL) &&
 2030             (sbavail(sb) >= uio->uio_resid || sbavail(sb) >= sb->sb_hiwat))
 2031                 goto deliver;
 2032 
 2033         /*
 2034          * Wait and block until (more) data comes in.
 2035          * NB: Drops the sockbuf lock during wait.
 2036          */
 2037         error = sbwait(sb);
 2038         if (error)
 2039                 goto out;
 2040         goto restart;
 2041 
 2042 deliver:
 2043         SOCKBUF_LOCK_ASSERT(&so->so_rcv);
 2044         KASSERT(sbavail(sb) > 0, ("%s: sockbuf empty", __func__));
 2045         KASSERT(sb->sb_mb != NULL, ("%s: sb_mb == NULL", __func__));
 2046 
 2047         /* Statistics. */
 2048         if (uio->uio_td)
 2049                 uio->uio_td->td_ru.ru_msgrcv++;
 2050 
 2051         /* Fill uio until full or current end of socket buffer is reached. */
 2052         len = min(uio->uio_resid, sbavail(sb));
 2053         if (mp0 != NULL) {
 2054                 /* Dequeue as many mbufs as possible. */
 2055                 if (!(flags & MSG_PEEK) && len >= sb->sb_mb->m_len) {
 2056                         if (*mp0 == NULL)
 2057                                 *mp0 = sb->sb_mb;
 2058                         else
 2059                                 m_cat(*mp0, sb->sb_mb);
 2060                         for (m = sb->sb_mb;
 2061                              m != NULL && m->m_len <= len;
 2062                              m = m->m_next) {
 2063                                 KASSERT(!(m->m_flags & M_NOTAVAIL),
 2064                                     ("%s: m %p not available", __func__, m));
 2065                                 len -= m->m_len;
 2066                                 uio->uio_resid -= m->m_len;
 2067                                 sbfree(sb, m);
 2068                                 n = m;
 2069                         }
 2070                         n->m_next = NULL;
 2071                         sb->sb_mb = m;
 2072                         sb->sb_lastrecord = sb->sb_mb;
 2073                         if (sb->sb_mb == NULL)
 2074                                 SB_EMPTY_FIXUP(sb);
 2075                 }
 2076                 /* Copy the remainder. */
 2077                 if (len > 0) {
 2078                         KASSERT(sb->sb_mb != NULL,
 2079                             ("%s: len > 0 && sb->sb_mb empty", __func__));
 2080 
 2081                         m = m_copym(sb->sb_mb, 0, len, M_NOWAIT);
 2082                         if (m == NULL)
 2083                                 len = 0;        /* Don't flush data from sockbuf. */
 2084                         else
 2085                                 uio->uio_resid -= len;
 2086                         if (*mp0 != NULL)
 2087                                 m_cat(*mp0, m);
 2088                         else
 2089                                 *mp0 = m;
 2090                         if (*mp0 == NULL) {
 2091                                 error = ENOBUFS;
 2092                                 goto out;
 2093                         }
 2094                 }
 2095         } else {
 2096                 /* NB: Must unlock socket buffer as uiomove may sleep. */
 2097                 SOCKBUF_UNLOCK(sb);
 2098                 error = m_mbuftouio(uio, sb->sb_mb, len);
 2099                 SOCKBUF_LOCK(sb);
 2100                 if (error)
 2101                         goto out;
 2102         }
 2103         SBLASTRECORDCHK(sb);
 2104         SBLASTMBUFCHK(sb);
 2105 
 2106         /*
 2107          * Remove the delivered data from the socket buffer unless we
 2108          * were only peeking.
 2109          */
 2110         if (!(flags & MSG_PEEK)) {
 2111                 if (len > 0)
 2112                         sbdrop_locked(sb, len);
 2113 
 2114                 /* Notify protocol that we drained some data. */
 2115                 if ((so->so_proto->pr_flags & PR_WANTRCVD) &&
 2116                     (((flags & MSG_WAITALL) && uio->uio_resid > 0) ||
 2117                      !(flags & MSG_SOCALLBCK))) {
 2118                         SOCKBUF_UNLOCK(sb);
 2119                         VNET_SO_ASSERT(so);
 2120                         (*so->so_proto->pr_usrreqs->pru_rcvd)(so, flags);
 2121                         SOCKBUF_LOCK(sb);
 2122                 }
 2123         }
 2124 
 2125         /*
 2126          * For MSG_WAITALL we may have to loop again and wait for
 2127          * more data to come in.
 2128          */
 2129         if ((flags & MSG_WAITALL) && uio->uio_resid > 0)
 2130                 goto restart;
 2131 out:
 2132         SOCKBUF_LOCK_ASSERT(sb);
 2133         SBLASTRECORDCHK(sb);
 2134         SBLASTMBUFCHK(sb);
 2135         SOCKBUF_UNLOCK(sb);
 2136         sbunlock(sb);
 2137         return (error);
 2138 }
 2139 
 2140 /*
 2141  * Optimized version of soreceive() for simple datagram cases from userspace.
 2142  * Unlike in the stream case, we're able to drop a datagram if copyout()
 2143  * fails, and because we handle datagrams atomically, we don't need to use a
 2144  * sleep lock to prevent I/O interlacing.
 2145  */
 2146 int
 2147 soreceive_dgram(struct socket *so, struct sockaddr **psa, struct uio *uio,
 2148     struct mbuf **mp0, struct mbuf **controlp, int *flagsp)
 2149 {
 2150         struct mbuf *m, *m2;
 2151         int flags, error;
 2152         ssize_t len;
 2153         struct protosw *pr = so->so_proto;
 2154         struct mbuf *nextrecord;
 2155 
 2156         if (psa != NULL)
 2157                 *psa = NULL;
 2158         if (controlp != NULL)
 2159                 *controlp = NULL;
 2160         if (flagsp != NULL)
 2161                 flags = *flagsp &~ MSG_EOR;
 2162         else
 2163                 flags = 0;
 2164 
 2165         /*
 2166          * For any complicated cases, fall back to the full
 2167          * soreceive_generic().
 2168          */
 2169         if (mp0 != NULL || (flags & MSG_PEEK) || (flags & MSG_OOB))
 2170                 return (soreceive_generic(so, psa, uio, mp0, controlp,
 2171                     flagsp));
 2172 
 2173         /*
 2174          * Enforce restrictions on use.
 2175          */
 2176         KASSERT((pr->pr_flags & PR_WANTRCVD) == 0,
 2177             ("soreceive_dgram: wantrcvd"));
 2178         KASSERT(pr->pr_flags & PR_ATOMIC, ("soreceive_dgram: !atomic"));
 2179         KASSERT((so->so_rcv.sb_state & SBS_RCVATMARK) == 0,
 2180             ("soreceive_dgram: SBS_RCVATMARK"));
 2181         KASSERT((so->so_proto->pr_flags & PR_CONNREQUIRED) == 0,
 2182             ("soreceive_dgram: P_CONNREQUIRED"));
 2183 
 2184         /*
 2185          * Loop blocking while waiting for a datagram.
 2186          */
 2187         SOCKBUF_LOCK(&so->so_rcv);
 2188         while ((m = so->so_rcv.sb_mb) == NULL) {
 2189                 KASSERT(sbavail(&so->so_rcv) == 0,
 2190                     ("soreceive_dgram: sb_mb NULL but sbavail %u",
 2191                     sbavail(&so->so_rcv)));
 2192                 if (so->so_error) {
 2193                         error = so->so_error;
 2194                         so->so_error = 0;
 2195                         SOCKBUF_UNLOCK(&so->so_rcv);
 2196                         return (error);
 2197                 }
 2198                 if (so->so_rcv.sb_state & SBS_CANTRCVMORE ||
 2199                     uio->uio_resid == 0) {
 2200                         SOCKBUF_UNLOCK(&so->so_rcv);
 2201                         return (0);
 2202                 }
 2203                 if ((so->so_state & SS_NBIO) ||
 2204                     (flags & (MSG_DONTWAIT|MSG_NBIO))) {
 2205                         SOCKBUF_UNLOCK(&so->so_rcv);
 2206                         return (EWOULDBLOCK);
 2207                 }
 2208                 SBLASTRECORDCHK(&so->so_rcv);
 2209                 SBLASTMBUFCHK(&so->so_rcv);
 2210                 error = sbwait(&so->so_rcv);
 2211                 if (error) {
 2212                         SOCKBUF_UNLOCK(&so->so_rcv);
 2213                         return (error);
 2214                 }
 2215         }
 2216         SOCKBUF_LOCK_ASSERT(&so->so_rcv);
 2217 
 2218         if (uio->uio_td)
 2219                 uio->uio_td->td_ru.ru_msgrcv++;
 2220         SBLASTRECORDCHK(&so->so_rcv);
 2221         SBLASTMBUFCHK(&so->so_rcv);
 2222         nextrecord = m->m_nextpkt;
 2223         if (nextrecord == NULL) {
 2224                 KASSERT(so->so_rcv.sb_lastrecord == m,
 2225                     ("soreceive_dgram: lastrecord != m"));
 2226         }
 2227 
 2228         KASSERT(so->so_rcv.sb_mb->m_nextpkt == nextrecord,
 2229             ("soreceive_dgram: m_nextpkt != nextrecord"));
 2230 
 2231         /*
 2232          * Pull 'm' and its chain off the front of the packet queue.
 2233          */
 2234         so->so_rcv.sb_mb = NULL;
 2235         sockbuf_pushsync(&so->so_rcv, nextrecord);
 2236 
 2237         /*
 2238          * Walk 'm's chain and free that many bytes from the socket buffer.
 2239          */
 2240         for (m2 = m; m2 != NULL; m2 = m2->m_next)
 2241                 sbfree(&so->so_rcv, m2);
 2242 
 2243         /*
 2244          * Do a few last checks before we let go of the lock.
 2245          */
 2246         SBLASTRECORDCHK(&so->so_rcv);
 2247         SBLASTMBUFCHK(&so->so_rcv);
 2248         SOCKBUF_UNLOCK(&so->so_rcv);
 2249 
 2250         if (pr->pr_flags & PR_ADDR) {
 2251                 KASSERT(m->m_type == MT_SONAME,
 2252                     ("m->m_type == %d", m->m_type));
 2253                 if (psa != NULL)
 2254                         *psa = sodupsockaddr(mtod(m, struct sockaddr *),
 2255                             M_NOWAIT);
 2256                 m = m_free(m);
 2257         }
 2258         if (m == NULL) {
 2259                 /* XXXRW: Can this happen? */
 2260                 return (0);
 2261         }
 2262 
 2263         /*
 2264          * Packet to copyout() is now in 'm' and it is disconnected from the
 2265          * queue.
 2266          *
 2267          * Process one or more MT_CONTROL mbufs present before any data mbufs
 2268          * in the first mbuf chain on the socket buffer.  We call into the
 2269          * protocol to perform externalization (or freeing if controlp ==
 2270          * NULL). In some cases there can be only MT_CONTROL mbufs without
 2271          * MT_DATA mbufs.
 2272          */
 2273         if (m->m_type == MT_CONTROL) {
 2274                 struct mbuf *cm = NULL, *cmn;
 2275                 struct mbuf **cme = &cm;
 2276 
 2277                 do {
 2278                         m2 = m->m_next;
 2279                         m->m_next = NULL;
 2280                         *cme = m;
 2281                         cme = &(*cme)->m_next;
 2282                         m = m2;
 2283                 } while (m != NULL && m->m_type == MT_CONTROL);
 2284                 while (cm != NULL) {
 2285                         cmn = cm->m_next;
 2286                         cm->m_next = NULL;
 2287                         if (pr->pr_domain->dom_externalize != NULL) {
 2288                                 error = (*pr->pr_domain->dom_externalize)
 2289                                     (cm, controlp, flags);
 2290                         } else if (controlp != NULL)
 2291                                 *controlp = cm;
 2292                         else
 2293                                 m_freem(cm);
 2294                         if (controlp != NULL) {
 2295                                 while (*controlp != NULL)
 2296                                         controlp = &(*controlp)->m_next;
 2297                         }
 2298                         cm = cmn;
 2299                 }
 2300         }
 2301         KASSERT(m == NULL || m->m_type == MT_DATA,
 2302             ("soreceive_dgram: !data"));
 2303         while (m != NULL && uio->uio_resid > 0) {
 2304                 len = uio->uio_resid;
 2305                 if (len > m->m_len)
 2306                         len = m->m_len;
 2307                 error = uiomove(mtod(m, char *), (int)len, uio);
 2308                 if (error) {
 2309                         m_freem(m);
 2310                         return (error);
 2311                 }
 2312                 if (len == m->m_len)
 2313                         m = m_free(m);
 2314                 else {
 2315                         m->m_data += len;
 2316                         m->m_len -= len;
 2317                 }
 2318         }
 2319         if (m != NULL) {
 2320                 flags |= MSG_TRUNC;
 2321                 m_freem(m);
 2322         }
 2323         if (flagsp != NULL)
 2324                 *flagsp |= flags;
 2325         return (0);
 2326 }
 2327 
 2328 int
 2329 soreceive(struct socket *so, struct sockaddr **psa, struct uio *uio,
 2330     struct mbuf **mp0, struct mbuf **controlp, int *flagsp)
 2331 {
 2332         int error;
 2333 
 2334         CURVNET_SET(so->so_vnet);
 2335         error = (so->so_proto->pr_usrreqs->pru_soreceive(so, psa, uio, mp0,
 2336             controlp, flagsp));
 2337         CURVNET_RESTORE();
 2338         return (error);
 2339 }
 2340 
 2341 int
 2342 soshutdown(struct socket *so, int how)
 2343 {
 2344         struct protosw *pr = so->so_proto;
 2345         int error, soerror_enotconn;
 2346 
 2347         if (!(how == SHUT_RD || how == SHUT_WR || how == SHUT_RDWR))
 2348                 return (EINVAL);
 2349 
 2350         soerror_enotconn = 0;
 2351         if ((so->so_state &
 2352             (SS_ISCONNECTED | SS_ISCONNECTING | SS_ISDISCONNECTING)) == 0) {
 2353                 /*
 2354                  * POSIX mandates us to return ENOTCONN when shutdown(2) is
 2355                  * invoked on a datagram sockets, however historically we would
 2356                  * actually tear socket down. This is known to be leveraged by
 2357                  * some applications to unblock process waiting in recvXXX(2)
 2358                  * by other process that it shares that socket with. Try to meet
 2359                  * both backward-compatibility and POSIX requirements by forcing
 2360                  * ENOTCONN but still asking protocol to perform pru_shutdown().
 2361                  */
 2362                 if (so->so_type != SOCK_DGRAM &&
 2363                     !(so->so_options & SO_ACCEPTCONN))
 2364                         return (ENOTCONN);
 2365                 soerror_enotconn = 1;
 2366         }
 2367 
 2368         CURVNET_SET(so->so_vnet);
 2369         if (pr->pr_usrreqs->pru_flush != NULL)
 2370                 (*pr->pr_usrreqs->pru_flush)(so, how);
 2371         if (how != SHUT_WR)
 2372                 sorflush(so);
 2373         if (how != SHUT_RD) {
 2374                 error = (*pr->pr_usrreqs->pru_shutdown)(so);
 2375                 wakeup(&so->so_timeo);
 2376                 CURVNET_RESTORE();
 2377                 return ((error == 0 && soerror_enotconn) ? ENOTCONN : error);
 2378         }
 2379         wakeup(&so->so_timeo);
 2380         CURVNET_RESTORE();
 2381 
 2382         return (soerror_enotconn ? ENOTCONN : 0);
 2383 }
 2384 
 2385 void
 2386 sorflush(struct socket *so)
 2387 {
 2388         struct sockbuf *sb = &so->so_rcv;
 2389         struct protosw *pr = so->so_proto;
 2390         struct socket aso;
 2391 
 2392         VNET_SO_ASSERT(so);
 2393 
 2394         /*
 2395          * In order to avoid calling dom_dispose with the socket buffer mutex
 2396          * held, and in order to generally avoid holding the lock for a long
 2397          * time, we make a copy of the socket buffer and clear the original
 2398          * (except locks, state).  The new socket buffer copy won't have
 2399          * initialized locks so we can only call routines that won't use or
 2400          * assert those locks.
 2401          *
 2402          * Dislodge threads currently blocked in receive and wait to acquire
 2403          * a lock against other simultaneous readers before clearing the
 2404          * socket buffer.  Don't let our acquire be interrupted by a signal
 2405          * despite any existing socket disposition on interruptable waiting.
 2406          */
 2407         socantrcvmore(so);
 2408         (void) sblock(sb, SBL_WAIT | SBL_NOINTR);
 2409 
 2410         /*
 2411          * Invalidate/clear most of the sockbuf structure, but leave selinfo
 2412          * and mutex data unchanged.
 2413          */
 2414         SOCKBUF_LOCK(sb);
 2415         bzero(&aso, sizeof(aso));
 2416         aso.so_pcb = so->so_pcb;
 2417         bcopy(&sb->sb_startzero, &aso.so_rcv.sb_startzero,
 2418             sizeof(*sb) - offsetof(struct sockbuf, sb_startzero));
 2419         bzero(&sb->sb_startzero,
 2420             sizeof(*sb) - offsetof(struct sockbuf, sb_startzero));
 2421         SOCKBUF_UNLOCK(sb);
 2422         sbunlock(sb);
 2423 
 2424         /*
 2425          * Dispose of special rights and flush the copied socket.  Don't call
 2426          * any unsafe routines (that rely on locks being initialized) on aso.
 2427          */
 2428         if (pr->pr_flags & PR_RIGHTS && pr->pr_domain->dom_dispose != NULL)
 2429                 (*pr->pr_domain->dom_dispose)(&aso);
 2430         sbrelease_internal(&aso.so_rcv, so);
 2431 }
 2432 
 2433 /*
 2434  * Wrapper for Socket established helper hook.
 2435  * Parameters: socket, context of the hook point, hook id.
 2436  */
 2437 static int inline
 2438 hhook_run_socket(struct socket *so, void *hctx, int32_t h_id)
 2439 {
 2440         struct socket_hhook_data hhook_data = {
 2441                 .so = so,
 2442                 .hctx = hctx,
 2443                 .m = NULL,
 2444                 .status = 0
 2445         };
 2446 
 2447         CURVNET_SET(so->so_vnet);
 2448         HHOOKS_RUN_IF(V_socket_hhh[h_id], &hhook_data, &so->osd);
 2449         CURVNET_RESTORE();
 2450 
 2451         /* Ugly but needed, since hhooks return void for now */
 2452         return (hhook_data.status);
 2453 }
 2454 
 2455 /*
 2456  * Perhaps this routine, and sooptcopyout(), below, ought to come in an
 2457  * additional variant to handle the case where the option value needs to be
 2458  * some kind of integer, but not a specific size.  In addition to their use
 2459  * here, these functions are also called by the protocol-level pr_ctloutput()
 2460  * routines.
 2461  */
 2462 int
 2463 sooptcopyin(struct sockopt *sopt, void *buf, size_t len, size_t minlen)
 2464 {
 2465         size_t  valsize;
 2466 
 2467         /*
 2468          * If the user gives us more than we wanted, we ignore it, but if we
 2469          * don't get the minimum length the caller wants, we return EINVAL.
 2470          * On success, sopt->sopt_valsize is set to however much we actually
 2471          * retrieved.
 2472          */
 2473         if ((valsize = sopt->sopt_valsize) < minlen)
 2474                 return EINVAL;
 2475         if (valsize > len)
 2476                 sopt->sopt_valsize = valsize = len;
 2477 
 2478         if (sopt->sopt_td != NULL)
 2479                 return (copyin(sopt->sopt_val, buf, valsize));
 2480 
 2481         bcopy(sopt->sopt_val, buf, valsize);
 2482         return (0);
 2483 }
 2484 
 2485 /*
 2486  * Kernel version of setsockopt(2).
 2487  *
 2488  * XXX: optlen is size_t, not socklen_t
 2489  */
 2490 int
 2491 so_setsockopt(struct socket *so, int level, int optname, void *optval,
 2492     size_t optlen)
 2493 {
 2494         struct sockopt sopt;
 2495 
 2496         sopt.sopt_level = level;
 2497         sopt.sopt_name = optname;
 2498         sopt.sopt_dir = SOPT_SET;
 2499         sopt.sopt_val = optval;
 2500         sopt.sopt_valsize = optlen;
 2501         sopt.sopt_td = NULL;
 2502         return (sosetopt(so, &sopt));
 2503 }
 2504 
 2505 int
 2506 sosetopt(struct socket *so, struct sockopt *sopt)
 2507 {
 2508         int     error, optval;
 2509         struct  linger l;
 2510         struct  timeval tv;
 2511         sbintime_t val;
 2512         uint32_t val32;
 2513 #ifdef MAC
 2514         struct mac extmac;
 2515 #endif
 2516 
 2517         CURVNET_SET(so->so_vnet);
 2518         error = 0;
 2519         if (sopt->sopt_level != SOL_SOCKET) {
 2520                 if (so->so_proto->pr_ctloutput != NULL) {
 2521                         error = (*so->so_proto->pr_ctloutput)(so, sopt);
 2522                         CURVNET_RESTORE();
 2523                         return (error);
 2524                 }
 2525                 error = ENOPROTOOPT;
 2526         } else {
 2527                 switch (sopt->sopt_name) {
 2528                 case SO_ACCEPTFILTER:
 2529                         error = do_setopt_accept_filter(so, sopt);
 2530                         if (error)
 2531                                 goto bad;
 2532                         break;
 2533 
 2534                 case SO_LINGER:
 2535                         error = sooptcopyin(sopt, &l, sizeof l, sizeof l);
 2536                         if (error)
 2537                                 goto bad;
 2538 
 2539                         SOCK_LOCK(so);
 2540                         so->so_linger = l.l_linger;
 2541                         if (l.l_onoff)
 2542                                 so->so_options |= SO_LINGER;
 2543                         else
 2544                                 so->so_options &= ~SO_LINGER;
 2545                         SOCK_UNLOCK(so);
 2546                         break;
 2547 
 2548                 case SO_DEBUG:
 2549                 case SO_KEEPALIVE:
 2550                 case SO_DONTROUTE:
 2551                 case SO_USELOOPBACK:
 2552                 case SO_BROADCAST:
 2553                 case SO_REUSEADDR:
 2554                 case SO_REUSEPORT:
 2555                 case SO_OOBINLINE:
 2556                 case SO_TIMESTAMP:
 2557                 case SO_BINTIME:
 2558                 case SO_NOSIGPIPE:
 2559                 case SO_NO_DDP:
 2560                 case SO_NO_OFFLOAD:
 2561                         error = sooptcopyin(sopt, &optval, sizeof optval,
 2562                             sizeof optval);
 2563                         if (error)
 2564                                 goto bad;
 2565                         SOCK_LOCK(so);
 2566                         if (optval)
 2567                                 so->so_options |= sopt->sopt_name;
 2568                         else
 2569                                 so->so_options &= ~sopt->sopt_name;
 2570                         SOCK_UNLOCK(so);
 2571                         break;
 2572 
 2573                 case SO_SETFIB:
 2574                         error = sooptcopyin(sopt, &optval, sizeof optval,
 2575                             sizeof optval);
 2576                         if (error)
 2577                                 goto bad;
 2578 
 2579                         if (optval < 0 || optval >= rt_numfibs) {
 2580                                 error = EINVAL;
 2581                                 goto bad;
 2582                         }
 2583                         if (((so->so_proto->pr_domain->dom_family == PF_INET) ||
 2584                            (so->so_proto->pr_domain->dom_family == PF_INET6) ||
 2585                            (so->so_proto->pr_domain->dom_family == PF_ROUTE)))
 2586                                 so->so_fibnum = optval;
 2587                         else
 2588                                 so->so_fibnum = 0;
 2589                         break;
 2590 
 2591                 case SO_USER_COOKIE:
 2592                         error = sooptcopyin(sopt, &val32, sizeof val32,
 2593                             sizeof val32);
 2594                         if (error)
 2595                                 goto bad;
 2596                         so->so_user_cookie = val32;
 2597                         break;
 2598 
 2599                 case SO_SNDBUF:
 2600                 case SO_RCVBUF:
 2601                 case SO_SNDLOWAT:
 2602                 case SO_RCVLOWAT:
 2603                         error = sooptcopyin(sopt, &optval, sizeof optval,
 2604                             sizeof optval);
 2605                         if (error)
 2606                                 goto bad;
 2607 
 2608                         /*
 2609                          * Values < 1 make no sense for any of these options,
 2610                          * so disallow them.
 2611                          */
 2612                         if (optval < 1) {
 2613                                 error = EINVAL;
 2614                                 goto bad;
 2615                         }
 2616 
 2617                         switch (sopt->sopt_name) {
 2618                         case SO_SNDBUF:
 2619                         case SO_RCVBUF:
 2620                                 if (sbreserve(sopt->sopt_name == SO_SNDBUF ?
 2621                                     &so->so_snd : &so->so_rcv, (u_long)optval,
 2622                                     so, curthread) == 0) {
 2623                                         error = ENOBUFS;
 2624                                         goto bad;
 2625                                 }
 2626                                 (sopt->sopt_name == SO_SNDBUF ? &so->so_snd :
 2627                                     &so->so_rcv)->sb_flags &= ~SB_AUTOSIZE;
 2628                                 break;
 2629 
 2630                         /*
 2631                          * Make sure the low-water is never greater than the
 2632                          * high-water.
 2633                          */
 2634                         case SO_SNDLOWAT:
 2635                                 SOCKBUF_LOCK(&so->so_snd);
 2636                                 so->so_snd.sb_lowat =
 2637                                     (optval > so->so_snd.sb_hiwat) ?
 2638                                     so->so_snd.sb_hiwat : optval;
 2639                                 SOCKBUF_UNLOCK(&so->so_snd);
 2640                                 break;
 2641                         case SO_RCVLOWAT:
 2642                                 SOCKBUF_LOCK(&so->so_rcv);
 2643                                 so->so_rcv.sb_lowat =
 2644                                     (optval > so->so_rcv.sb_hiwat) ?
 2645                                     so->so_rcv.sb_hiwat : optval;
 2646                                 SOCKBUF_UNLOCK(&so->so_rcv);
 2647                                 break;
 2648                         }
 2649                         break;
 2650 
 2651                 case SO_SNDTIMEO:
 2652                 case SO_RCVTIMEO:
 2653 #ifdef COMPAT_FREEBSD32
 2654                         if (SV_CURPROC_FLAG(SV_ILP32)) {
 2655                                 struct timeval32 tv32;
 2656 
 2657                                 error = sooptcopyin(sopt, &tv32, sizeof tv32,
 2658                                     sizeof tv32);
 2659                                 CP(tv32, tv, tv_sec);
 2660                                 CP(tv32, tv, tv_usec);
 2661                         } else
 2662 #endif
 2663                                 error = sooptcopyin(sopt, &tv, sizeof tv,
 2664                                     sizeof tv);
 2665                         if (error)
 2666                                 goto bad;
 2667                         if (tv.tv_sec < 0 || tv.tv_usec < 0 ||
 2668                             tv.tv_usec >= 1000000) {
 2669                                 error = EDOM;
 2670                                 goto bad;
 2671                         }
 2672                         if (tv.tv_sec > INT32_MAX)
 2673                                 val = SBT_MAX;
 2674                         else
 2675                                 val = tvtosbt(tv);
 2676                         switch (sopt->sopt_name) {
 2677                         case SO_SNDTIMEO:
 2678                                 so->so_snd.sb_timeo = val;
 2679                                 break;
 2680                         case SO_RCVTIMEO:
 2681                                 so->so_rcv.sb_timeo = val;
 2682                                 break;
 2683                         }
 2684                         break;
 2685 
 2686                 case SO_LABEL:
 2687 #ifdef MAC
 2688                         error = sooptcopyin(sopt, &extmac, sizeof extmac,
 2689                             sizeof extmac);
 2690                         if (error)
 2691                                 goto bad;
 2692                         error = mac_setsockopt_label(sopt->sopt_td->td_ucred,
 2693                             so, &extmac);
 2694 #else
 2695                         error = EOPNOTSUPP;
 2696 #endif
 2697                         break;
 2698 
 2699                 default:
 2700                         if (V_socket_hhh[HHOOK_SOCKET_OPT]->hhh_nhooks > 0)
 2701                                 error = hhook_run_socket(so, sopt,
 2702                                     HHOOK_SOCKET_OPT);
 2703                         else
 2704                                 error = ENOPROTOOPT;
 2705                         break;
 2706                 }
 2707                 if (error == 0 && so->so_proto->pr_ctloutput != NULL)
 2708                         (void)(*so->so_proto->pr_ctloutput)(so, sopt);
 2709         }
 2710 bad:
 2711         CURVNET_RESTORE();
 2712         return (error);
 2713 }
 2714 
 2715 /*
 2716  * Helper routine for getsockopt.
 2717  */
 2718 int
 2719 sooptcopyout(struct sockopt *sopt, const void *buf, size_t len)
 2720 {
 2721         int     error;
 2722         size_t  valsize;
 2723 
 2724         error = 0;
 2725 
 2726         /*
 2727          * Documented get behavior is that we always return a value, possibly
 2728          * truncated to fit in the user's buffer.  Traditional behavior is
 2729          * that we always tell the user precisely how much we copied, rather
 2730          * than something useful like the total amount we had available for
 2731          * her.  Note that this interface is not idempotent; the entire
 2732          * answer must be generated ahead of time.
 2733          */
 2734         valsize = min(len, sopt->sopt_valsize);
 2735         sopt->sopt_valsize = valsize;
 2736         if (sopt->sopt_val != NULL) {
 2737                 if (sopt->sopt_td != NULL)
 2738                         error = copyout(buf, sopt->sopt_val, valsize);
 2739                 else
 2740                         bcopy(buf, sopt->sopt_val, valsize);
 2741         }
 2742         return (error);
 2743 }
 2744 
 2745 int
 2746 sogetopt(struct socket *so, struct sockopt *sopt)
 2747 {
 2748         int     error, optval;
 2749         struct  linger l;
 2750         struct  timeval tv;
 2751 #ifdef MAC
 2752         struct mac extmac;
 2753 #endif
 2754 
 2755         CURVNET_SET(so->so_vnet);
 2756         error = 0;
 2757         if (sopt->sopt_level != SOL_SOCKET) {
 2758                 if (so->so_proto->pr_ctloutput != NULL)
 2759                         error = (*so->so_proto->pr_ctloutput)(so, sopt);
 2760                 else
 2761                         error = ENOPROTOOPT;
 2762                 CURVNET_RESTORE();
 2763                 return (error);
 2764         } else {
 2765                 switch (sopt->sopt_name) {
 2766                 case SO_ACCEPTFILTER:
 2767                         error = do_getopt_accept_filter(so, sopt);
 2768                         break;
 2769 
 2770                 case SO_LINGER:
 2771                         SOCK_LOCK(so);
 2772                         l.l_onoff = so->so_options & SO_LINGER;
 2773                         l.l_linger = so->so_linger;
 2774                         SOCK_UNLOCK(so);
 2775                         error = sooptcopyout(sopt, &l, sizeof l);
 2776                         break;
 2777 
 2778                 case SO_USELOOPBACK:
 2779                 case SO_DONTROUTE:
 2780                 case SO_DEBUG:
 2781                 case SO_KEEPALIVE:
 2782                 case SO_REUSEADDR:
 2783                 case SO_REUSEPORT:
 2784                 case SO_BROADCAST:
 2785                 case SO_OOBINLINE:
 2786                 case SO_ACCEPTCONN:
 2787                 case SO_TIMESTAMP:
 2788                 case SO_BINTIME:
 2789                 case SO_NOSIGPIPE:
 2790                         optval = so->so_options & sopt->sopt_name;
 2791 integer:
 2792                         error = sooptcopyout(sopt, &optval, sizeof optval);
 2793                         break;
 2794 
 2795                 case SO_TYPE:
 2796                         optval = so->so_type;
 2797                         goto integer;
 2798 
 2799                 case SO_PROTOCOL:
 2800                         optval = so->so_proto->pr_protocol;
 2801                         goto integer;
 2802 
 2803                 case SO_ERROR:
 2804                         SOCK_LOCK(so);
 2805                         optval = so->so_error;
 2806                         so->so_error = 0;
 2807                         SOCK_UNLOCK(so);
 2808                         goto integer;
 2809 
 2810                 case SO_SNDBUF:
 2811                         optval = so->so_snd.sb_hiwat;
 2812                         goto integer;
 2813 
 2814                 case SO_RCVBUF:
 2815                         optval = so->so_rcv.sb_hiwat;
 2816                         goto integer;
 2817 
 2818                 case SO_SNDLOWAT:
 2819                         optval = so->so_snd.sb_lowat;
 2820                         goto integer;
 2821 
 2822                 case SO_RCVLOWAT:
 2823                         optval = so->so_rcv.sb_lowat;
 2824                         goto integer;
 2825 
 2826                 case SO_SNDTIMEO:
 2827                 case SO_RCVTIMEO:
 2828                         tv = sbttotv(sopt->sopt_name == SO_SNDTIMEO ?
 2829                             so->so_snd.sb_timeo : so->so_rcv.sb_timeo);
 2830 #ifdef COMPAT_FREEBSD32
 2831                         if (SV_CURPROC_FLAG(SV_ILP32)) {
 2832                                 struct timeval32 tv32;
 2833 
 2834                                 CP(tv, tv32, tv_sec);
 2835                                 CP(tv, tv32, tv_usec);
 2836                                 error = sooptcopyout(sopt, &tv32, sizeof tv32);
 2837                         } else
 2838 #endif
 2839                                 error = sooptcopyout(sopt, &tv, sizeof tv);
 2840                         break;
 2841 
 2842                 case SO_LABEL:
 2843 #ifdef MAC
 2844                         error = sooptcopyin(sopt, &extmac, sizeof(extmac),
 2845                             sizeof(extmac));
 2846                         if (error)
 2847                                 goto bad;
 2848                         error = mac_getsockopt_label(sopt->sopt_td->td_ucred,
 2849                             so, &extmac);
 2850                         if (error)
 2851                                 goto bad;
 2852                         error = sooptcopyout(sopt, &extmac, sizeof extmac);
 2853 #else
 2854                         error = EOPNOTSUPP;
 2855 #endif
 2856                         break;
 2857 
 2858                 case SO_PEERLABEL:
 2859 #ifdef MAC
 2860                         error = sooptcopyin(sopt, &extmac, sizeof(extmac),
 2861                             sizeof(extmac));
 2862                         if (error)
 2863                                 goto bad;
 2864                         error = mac_getsockopt_peerlabel(
 2865                             sopt->sopt_td->td_ucred, so, &extmac);
 2866                         if (error)
 2867                                 goto bad;
 2868                         error = sooptcopyout(sopt, &extmac, sizeof extmac);
 2869 #else
 2870                         error = EOPNOTSUPP;
 2871 #endif
 2872                         break;
 2873 
 2874                 case SO_LISTENQLIMIT:
 2875                         optval = so->so_qlimit;
 2876                         goto integer;
 2877 
 2878                 case SO_LISTENQLEN:
 2879                         optval = so->so_qlen;
 2880                         goto integer;
 2881 
 2882                 case SO_LISTENINCQLEN:
 2883                         optval = so->so_incqlen;
 2884                         goto integer;
 2885 
 2886                 default:
 2887                         if (V_socket_hhh[HHOOK_SOCKET_OPT]->hhh_nhooks > 0)
 2888                                 error = hhook_run_socket(so, sopt,
 2889                                     HHOOK_SOCKET_OPT);
 2890                         else
 2891                                 error = ENOPROTOOPT;
 2892                         break;
 2893                 }
 2894         }
 2895 #ifdef MAC
 2896 bad:
 2897 #endif
 2898         CURVNET_RESTORE();
 2899         return (error);
 2900 }
 2901 
 2902 int
 2903 soopt_getm(struct sockopt *sopt, struct mbuf **mp)
 2904 {
 2905         struct mbuf *m, *m_prev;
 2906         int sopt_size = sopt->sopt_valsize;
 2907 
 2908         MGET(m, sopt->sopt_td ? M_WAITOK : M_NOWAIT, MT_DATA);
 2909         if (m == NULL)
 2910                 return ENOBUFS;
 2911         if (sopt_size > MLEN) {
 2912                 MCLGET(m, sopt->sopt_td ? M_WAITOK : M_NOWAIT);
 2913                 if ((m->m_flags & M_EXT) == 0) {
 2914                         m_free(m);
 2915                         return ENOBUFS;
 2916                 }
 2917                 m->m_len = min(MCLBYTES, sopt_size);
 2918         } else {
 2919                 m->m_len = min(MLEN, sopt_size);
 2920         }
 2921         sopt_size -= m->m_len;
 2922         *mp = m;
 2923         m_prev = m;
 2924 
 2925         while (sopt_size) {
 2926                 MGET(m, sopt->sopt_td ? M_WAITOK : M_NOWAIT, MT_DATA);
 2927                 if (m == NULL) {
 2928                         m_freem(*mp);
 2929                         return ENOBUFS;
 2930                 }
 2931                 if (sopt_size > MLEN) {
 2932                         MCLGET(m, sopt->sopt_td != NULL ? M_WAITOK :
 2933                             M_NOWAIT);
 2934                         if ((m->m_flags & M_EXT) == 0) {
 2935                                 m_freem(m);
 2936                                 m_freem(*mp);
 2937                                 return ENOBUFS;
 2938                         }
 2939                         m->m_len = min(MCLBYTES, sopt_size);
 2940                 } else {
 2941                         m->m_len = min(MLEN, sopt_size);
 2942                 }
 2943                 sopt_size -= m->m_len;
 2944                 m_prev->m_next = m;
 2945                 m_prev = m;
 2946         }
 2947         return (0);
 2948 }
 2949 
 2950 int
 2951 soopt_mcopyin(struct sockopt *sopt, struct mbuf *m)
 2952 {
 2953         struct mbuf *m0 = m;
 2954 
 2955         if (sopt->sopt_val == NULL)
 2956                 return (0);
 2957         while (m != NULL && sopt->sopt_valsize >= m->m_len) {
 2958                 if (sopt->sopt_td != NULL) {
 2959                         int error;
 2960 
 2961                         error = copyin(sopt->sopt_val, mtod(m, char *),
 2962                             m->m_len);
 2963                         if (error != 0) {
 2964                                 m_freem(m0);
 2965                                 return(error);
 2966                         }
 2967                 } else
 2968                         bcopy(sopt->sopt_val, mtod(m, char *), m->m_len);
 2969                 sopt->sopt_valsize -= m->m_len;
 2970                 sopt->sopt_val = (char *)sopt->sopt_val + m->m_len;
 2971                 m = m->m_next;
 2972         }
 2973         if (m != NULL) /* should be allocated enoughly at ip6_sooptmcopyin() */
 2974                 panic("ip6_sooptmcopyin");
 2975         return (0);
 2976 }
 2977 
 2978 int
 2979 soopt_mcopyout(struct sockopt *sopt, struct mbuf *m)
 2980 {
 2981         struct mbuf *m0 = m;
 2982         size_t valsize = 0;
 2983 
 2984         if (sopt->sopt_val == NULL)
 2985                 return (0);
 2986         while (m != NULL && sopt->sopt_valsize >= m->m_len) {
 2987                 if (sopt->sopt_td != NULL) {
 2988                         int error;
 2989 
 2990                         error = copyout(mtod(m, char *), sopt->sopt_val,
 2991                             m->m_len);
 2992                         if (error != 0) {
 2993                                 m_freem(m0);
 2994                                 return(error);
 2995                         }
 2996                 } else
 2997                         bcopy(mtod(m, char *), sopt->sopt_val, m->m_len);
 2998                 sopt->sopt_valsize -= m->m_len;
 2999                 sopt->sopt_val = (char *)sopt->sopt_val + m->m_len;
 3000                 valsize += m->m_len;
 3001                 m = m->m_next;
 3002         }
 3003         if (m != NULL) {
 3004                 /* enough soopt buffer should be given from user-land */
 3005                 m_freem(m0);
 3006                 return(EINVAL);
 3007         }
 3008         sopt->sopt_valsize = valsize;
 3009         return (0);
 3010 }
 3011 
 3012 /*
 3013  * sohasoutofband(): protocol notifies socket layer of the arrival of new
 3014  * out-of-band data, which will then notify socket consumers.
 3015  */
 3016 void
 3017 sohasoutofband(struct socket *so)
 3018 {
 3019 
 3020         if (so->so_sigio != NULL)
 3021                 pgsigio(&so->so_sigio, SIGURG, 0);
 3022         selwakeuppri(&so->so_rcv.sb_sel, PSOCK);
 3023 }
 3024 
 3025 int
 3026 sopoll(struct socket *so, int events, struct ucred *active_cred,
 3027     struct thread *td)
 3028 {
 3029 
 3030         /*
 3031          * We do not need to set or assert curvnet as long as everyone uses
 3032          * sopoll_generic().
 3033          */
 3034         return (so->so_proto->pr_usrreqs->pru_sopoll(so, events, active_cred,
 3035             td));
 3036 }
 3037 
 3038 int
 3039 sopoll_generic(struct socket *so, int events, struct ucred *active_cred,
 3040     struct thread *td)
 3041 {
 3042         int revents = 0;
 3043 
 3044         SOCKBUF_LOCK(&so->so_snd);
 3045         SOCKBUF_LOCK(&so->so_rcv);
 3046         if (events & (POLLIN | POLLRDNORM))
 3047                 if (soreadabledata(so))
 3048                         revents |= events & (POLLIN | POLLRDNORM);
 3049 
 3050         if (events & (POLLOUT | POLLWRNORM))
 3051                 if (sowriteable(so))
 3052                         revents |= events & (POLLOUT | POLLWRNORM);
 3053 
 3054         if (events & (POLLPRI | POLLRDBAND))
 3055                 if (so->so_oobmark || (so->so_rcv.sb_state & SBS_RCVATMARK))
 3056                         revents |= events & (POLLPRI | POLLRDBAND);
 3057 
 3058         if ((events & POLLINIGNEOF) == 0) {
 3059                 if (so->so_rcv.sb_state & SBS_CANTRCVMORE) {
 3060                         revents |= events & (POLLIN | POLLRDNORM);
 3061                         if (so->so_snd.sb_state & SBS_CANTSENDMORE)
 3062                                 revents |= POLLHUP;
 3063                 }
 3064         }
 3065 
 3066         if (revents == 0) {
 3067                 if (events & (POLLIN | POLLPRI | POLLRDNORM | POLLRDBAND)) {
 3068                         selrecord(td, &so->so_rcv.sb_sel);
 3069                         so->so_rcv.sb_flags |= SB_SEL;
 3070                 }
 3071 
 3072                 if (events & (POLLOUT | POLLWRNORM)) {
 3073                         selrecord(td, &so->so_snd.sb_sel);
 3074                         so->so_snd.sb_flags |= SB_SEL;
 3075                 }
 3076         }
 3077 
 3078         SOCKBUF_UNLOCK(&so->so_rcv);
 3079         SOCKBUF_UNLOCK(&so->so_snd);
 3080         return (revents);
 3081 }
 3082 
 3083 int
 3084 soo_kqfilter(struct file *fp, struct knote *kn)
 3085 {
 3086         struct socket *so = kn->kn_fp->f_data;
 3087         struct sockbuf *sb;
 3088 
 3089         switch (kn->kn_filter) {
 3090         case EVFILT_READ:
 3091                 if (so->so_options & SO_ACCEPTCONN)
 3092                         kn->kn_fop = &solisten_filtops;
 3093                 else
 3094                         kn->kn_fop = &soread_filtops;
 3095                 sb = &so->so_rcv;
 3096                 break;
 3097         case EVFILT_WRITE:
 3098                 kn->kn_fop = &sowrite_filtops;
 3099                 sb = &so->so_snd;
 3100                 break;
 3101         default:
 3102                 return (EINVAL);
 3103         }
 3104 
 3105         SOCKBUF_LOCK(sb);
 3106         knlist_add(&sb->sb_sel.si_note, kn, 1);
 3107         sb->sb_flags |= SB_KNOTE;
 3108         SOCKBUF_UNLOCK(sb);
 3109         return (0);
 3110 }
 3111 
 3112 /*
 3113  * Some routines that return EOPNOTSUPP for entry points that are not
 3114  * supported by a protocol.  Fill in as needed.
 3115  */
 3116 int
 3117 pru_accept_notsupp(struct socket *so, struct sockaddr **nam)
 3118 {
 3119 
 3120         return EOPNOTSUPP;
 3121 }
 3122 
 3123 int
 3124 pru_aio_queue_notsupp(struct socket *so, struct kaiocb *job)
 3125 {
 3126 
 3127         return EOPNOTSUPP;
 3128 }
 3129 
 3130 int
 3131 pru_attach_notsupp(struct socket *so, int proto, struct thread *td)
 3132 {
 3133 
 3134         return EOPNOTSUPP;
 3135 }
 3136 
 3137 int
 3138 pru_bind_notsupp(struct socket *so, struct sockaddr *nam, struct thread *td)
 3139 {
 3140 
 3141         return EOPNOTSUPP;
 3142 }
 3143 
 3144 int
 3145 pru_bindat_notsupp(int fd, struct socket *so, struct sockaddr *nam,
 3146     struct thread *td)
 3147 {
 3148 
 3149         return EOPNOTSUPP;
 3150 }
 3151 
 3152 int
 3153 pru_connect_notsupp(struct socket *so, struct sockaddr *nam, struct thread *td)
 3154 {
 3155 
 3156         return EOPNOTSUPP;
 3157 }
 3158 
 3159 int
 3160 pru_connectat_notsupp(int fd, struct socket *so, struct sockaddr *nam,
 3161     struct thread *td)
 3162 {
 3163 
 3164         return EOPNOTSUPP;
 3165 }
 3166 
 3167 int
 3168 pru_connect2_notsupp(struct socket *so1, struct socket *so2)
 3169 {
 3170 
 3171         return EOPNOTSUPP;
 3172 }
 3173 
 3174 int
 3175 pru_control_notsupp(struct socket *so, u_long cmd, caddr_t data,
 3176     struct ifnet *ifp, struct thread *td)
 3177 {
 3178 
 3179         return EOPNOTSUPP;
 3180 }
 3181 
 3182 int
 3183 pru_disconnect_notsupp(struct socket *so)
 3184 {
 3185 
 3186         return EOPNOTSUPP;
 3187 }
 3188 
 3189 int
 3190 pru_listen_notsupp(struct socket *so, int backlog, struct thread *td)
 3191 {
 3192 
 3193         return EOPNOTSUPP;
 3194 }
 3195 
 3196 int
 3197 pru_peeraddr_notsupp(struct socket *so, struct sockaddr **nam)
 3198 {
 3199 
 3200         return EOPNOTSUPP;
 3201 }
 3202 
 3203 int
 3204 pru_rcvd_notsupp(struct socket *so, int flags)
 3205 {
 3206 
 3207         return EOPNOTSUPP;
 3208 }
 3209 
 3210 int
 3211 pru_rcvoob_notsupp(struct socket *so, struct mbuf *m, int flags)
 3212 {
 3213 
 3214         return EOPNOTSUPP;
 3215 }
 3216 
 3217 int
 3218 pru_send_notsupp(struct socket *so, int flags, struct mbuf *m,
 3219     struct sockaddr *addr, struct mbuf *control, struct thread *td)
 3220 {
 3221 
 3222         return EOPNOTSUPP;
 3223 }
 3224 
 3225 int
 3226 pru_ready_notsupp(struct socket *so, struct mbuf *m, int count)
 3227 {
 3228 
 3229         return (EOPNOTSUPP);
 3230 }
 3231 
 3232 /*
 3233  * This isn't really a ``null'' operation, but it's the default one and
 3234  * doesn't do anything destructive.
 3235  */
 3236 int
 3237 pru_sense_null(struct socket *so, struct stat *sb)
 3238 {
 3239 
 3240         sb->st_blksize = so->so_snd.sb_hiwat;
 3241         return 0;
 3242 }
 3243 
 3244 int
 3245 pru_shutdown_notsupp(struct socket *so)
 3246 {
 3247 
 3248         return EOPNOTSUPP;
 3249 }
 3250 
 3251 int
 3252 pru_sockaddr_notsupp(struct socket *so, struct sockaddr **nam)
 3253 {
 3254 
 3255         return EOPNOTSUPP;
 3256 }
 3257 
 3258 int
 3259 pru_sosend_notsupp(struct socket *so, struct sockaddr *addr, struct uio *uio,
 3260     struct mbuf *top, struct mbuf *control, int flags, struct thread *td)
 3261 {
 3262 
 3263         return EOPNOTSUPP;
 3264 }
 3265 
 3266 int
 3267 pru_soreceive_notsupp(struct socket *so, struct sockaddr **paddr,
 3268     struct uio *uio, struct mbuf **mp0, struct mbuf **controlp, int *flagsp)
 3269 {
 3270 
 3271         return EOPNOTSUPP;
 3272 }
 3273 
 3274 int
 3275 pru_sopoll_notsupp(struct socket *so, int events, struct ucred *cred,
 3276     struct thread *td)
 3277 {
 3278 
 3279         return EOPNOTSUPP;
 3280 }
 3281 
 3282 static void
 3283 filt_sordetach(struct knote *kn)
 3284 {
 3285         struct socket *so = kn->kn_fp->f_data;
 3286 
 3287         SOCKBUF_LOCK(&so->so_rcv);
 3288         knlist_remove(&so->so_rcv.sb_sel.si_note, kn, 1);
 3289         if (knlist_empty(&so->so_rcv.sb_sel.si_note))
 3290                 so->so_rcv.sb_flags &= ~SB_KNOTE;
 3291         SOCKBUF_UNLOCK(&so->so_rcv);
 3292 }
 3293 
 3294 /*ARGSUSED*/
 3295 static int
 3296 filt_soread(struct knote *kn, long hint)
 3297 {
 3298         struct socket *so;
 3299 
 3300         so = kn->kn_fp->f_data;
 3301         SOCKBUF_LOCK_ASSERT(&so->so_rcv);
 3302 
 3303         kn->kn_data = sbavail(&so->so_rcv) - so->so_rcv.sb_ctl;
 3304         if (so->so_rcv.sb_state & SBS_CANTRCVMORE) {
 3305                 kn->kn_flags |= EV_EOF;
 3306                 kn->kn_fflags = so->so_error;
 3307                 return (1);
 3308         } else if (so->so_error)        /* temporary udp error */
 3309                 return (1);
 3310 
 3311         if (kn->kn_sfflags & NOTE_LOWAT) {
 3312                 if (kn->kn_data >= kn->kn_sdata)
 3313                         return 1;
 3314         } else {
 3315                 if (sbavail(&so->so_rcv) >= so->so_rcv.sb_lowat)
 3316                         return 1;
 3317         }
 3318 
 3319         /* This hook returning non-zero indicates an event, not error */
 3320         return (hhook_run_socket(so, NULL, HHOOK_FILT_SOREAD));
 3321 }
 3322 
 3323 static void
 3324 filt_sowdetach(struct knote *kn)
 3325 {
 3326         struct socket *so = kn->kn_fp->f_data;
 3327 
 3328         SOCKBUF_LOCK(&so->so_snd);
 3329         knlist_remove(&so->so_snd.sb_sel.si_note, kn, 1);
 3330         if (knlist_empty(&so->so_snd.sb_sel.si_note))
 3331                 so->so_snd.sb_flags &= ~SB_KNOTE;
 3332         SOCKBUF_UNLOCK(&so->so_snd);
 3333 }
 3334 
 3335 /*ARGSUSED*/
 3336 static int
 3337 filt_sowrite(struct knote *kn, long hint)
 3338 {
 3339         struct socket *so;
 3340 
 3341         so = kn->kn_fp->f_data;
 3342         SOCKBUF_LOCK_ASSERT(&so->so_snd);
 3343         kn->kn_data = sbspace(&so->so_snd);
 3344 
 3345         hhook_run_socket(so, kn, HHOOK_FILT_SOWRITE);
 3346 
 3347         if (so->so_snd.sb_state & SBS_CANTSENDMORE) {
 3348                 kn->kn_flags |= EV_EOF;
 3349                 kn->kn_fflags = so->so_error;
 3350                 return (1);
 3351         } else if (so->so_error)        /* temporary udp error */
 3352                 return (1);
 3353         else if (((so->so_state & SS_ISCONNECTED) == 0) &&
 3354             (so->so_proto->pr_flags & PR_CONNREQUIRED))
 3355                 return (0);
 3356         else if (kn->kn_sfflags & NOTE_LOWAT)
 3357                 return (kn->kn_data >= kn->kn_sdata);
 3358         else
 3359                 return (kn->kn_data >= so->so_snd.sb_lowat);
 3360 }
 3361 
 3362 /*ARGSUSED*/
 3363 static int
 3364 filt_solisten(struct knote *kn, long hint)
 3365 {
 3366         struct socket *so = kn->kn_fp->f_data;
 3367 
 3368         kn->kn_data = so->so_qlen;
 3369         return (!TAILQ_EMPTY(&so->so_comp));
 3370 }
 3371 
 3372 int
 3373 socheckuid(struct socket *so, uid_t uid)
 3374 {
 3375 
 3376         if (so == NULL)
 3377                 return (EPERM);
 3378         if (so->so_cred->cr_uid != uid)
 3379                 return (EPERM);
 3380         return (0);
 3381 }
 3382 
 3383 /*
 3384  * These functions are used by protocols to notify the socket layer (and its
 3385  * consumers) of state changes in the sockets driven by protocol-side events.
 3386  */
 3387 
 3388 /*
 3389  * Procedures to manipulate state flags of socket and do appropriate wakeups.
 3390  *
 3391  * Normal sequence from the active (originating) side is that
 3392  * soisconnecting() is called during processing of connect() call, resulting
 3393  * in an eventual call to soisconnected() if/when the connection is
 3394  * established.  When the connection is torn down soisdisconnecting() is
 3395  * called during processing of disconnect() call, and soisdisconnected() is
 3396  * called when the connection to the peer is totally severed.  The semantics
 3397  * of these routines are such that connectionless protocols can call
 3398  * soisconnected() and soisdisconnected() only, bypassing the in-progress
 3399  * calls when setting up a ``connection'' takes no time.
 3400  *
 3401  * From the passive side, a socket is created with two queues of sockets:
 3402  * so_incomp for connections in progress and so_comp for connections already
 3403  * made and awaiting user acceptance.  As a protocol is preparing incoming
 3404  * connections, it creates a socket structure queued on so_incomp by calling
 3405  * sonewconn().  When the connection is established, soisconnected() is
 3406  * called, and transfers the socket structure to so_comp, making it available
 3407  * to accept().
 3408  *
 3409  * If a socket is closed with sockets on either so_incomp or so_comp, these
 3410  * sockets are dropped.
 3411  *
 3412  * If higher-level protocols are implemented in the kernel, the wakeups done
 3413  * here will sometimes cause software-interrupt process scheduling.
 3414  */
 3415 void
 3416 soisconnecting(struct socket *so)
 3417 {
 3418 
 3419         SOCK_LOCK(so);
 3420         so->so_state &= ~(SS_ISCONNECTED|SS_ISDISCONNECTING);
 3421         so->so_state |= SS_ISCONNECTING;
 3422         SOCK_UNLOCK(so);
 3423 }
 3424 
 3425 void
 3426 soisconnected(struct socket *so)
 3427 {
 3428         struct socket *head;
 3429         int ret;
 3430 
 3431 restart:
 3432         ACCEPT_LOCK();
 3433         SOCK_LOCK(so);
 3434         so->so_state &= ~(SS_ISCONNECTING|SS_ISDISCONNECTING|SS_ISCONFIRMING);
 3435         so->so_state |= SS_ISCONNECTED;
 3436         head = so->so_head;
 3437         if (head != NULL && (so->so_qstate & SQ_INCOMP)) {
 3438                 if ((so->so_options & SO_ACCEPTFILTER) == 0) {
 3439                         SOCK_UNLOCK(so);
 3440                         TAILQ_REMOVE(&head->so_incomp, so, so_list);
 3441                         head->so_incqlen--;
 3442                         so->so_qstate &= ~SQ_INCOMP;
 3443                         TAILQ_INSERT_TAIL(&head->so_comp, so, so_list);
 3444                         head->so_qlen++;
 3445                         so->so_qstate |= SQ_COMP;
 3446                         ACCEPT_UNLOCK();
 3447                         sorwakeup(head);
 3448                         wakeup_one(&head->so_timeo);
 3449                 } else {
 3450                         ACCEPT_UNLOCK();
 3451                         soupcall_set(so, SO_RCV,
 3452                             head->so_accf->so_accept_filter->accf_callback,
 3453                             head->so_accf->so_accept_filter_arg);
 3454                         so->so_options &= ~SO_ACCEPTFILTER;
 3455                         ret = head->so_accf->so_accept_filter->accf_callback(so,
 3456                             head->so_accf->so_accept_filter_arg, M_NOWAIT);
 3457                         if (ret == SU_ISCONNECTED)
 3458                                 soupcall_clear(so, SO_RCV);
 3459                         SOCK_UNLOCK(so);
 3460                         if (ret == SU_ISCONNECTED)
 3461                                 goto restart;
 3462                 }
 3463                 return;
 3464         }
 3465         SOCK_UNLOCK(so);
 3466         ACCEPT_UNLOCK();
 3467         wakeup(&so->so_timeo);
 3468         sorwakeup(so);
 3469         sowwakeup(so);
 3470 }
 3471 
 3472 void
 3473 soisdisconnecting(struct socket *so)
 3474 {
 3475 
 3476         /*
 3477          * Note: This code assumes that SOCK_LOCK(so) and
 3478          * SOCKBUF_LOCK(&so->so_rcv) are the same.
 3479          */
 3480         SOCKBUF_LOCK(&so->so_rcv);
 3481         so->so_state &= ~SS_ISCONNECTING;
 3482         so->so_state |= SS_ISDISCONNECTING;
 3483         socantrcvmore_locked(so);
 3484         SOCKBUF_LOCK(&so->so_snd);
 3485         socantsendmore_locked(so);
 3486         wakeup(&so->so_timeo);
 3487 }
 3488 
 3489 void
 3490 soisdisconnected(struct socket *so)
 3491 {
 3492 
 3493         /*
 3494          * Note: This code assumes that SOCK_LOCK(so) and
 3495          * SOCKBUF_LOCK(&so->so_rcv) are the same.
 3496          */
 3497         SOCKBUF_LOCK(&so->so_rcv);
 3498         so->so_state &= ~(SS_ISCONNECTING|SS_ISCONNECTED|SS_ISDISCONNECTING);
 3499         so->so_state |= SS_ISDISCONNECTED;
 3500         socantrcvmore_locked(so);
 3501         SOCKBUF_LOCK(&so->so_snd);
 3502         sbdrop_locked(&so->so_snd, sbused(&so->so_snd));
 3503         socantsendmore_locked(so);
 3504         wakeup(&so->so_timeo);
 3505 }
 3506 
 3507 /*
 3508  * Make a copy of a sockaddr in a malloced buffer of type M_SONAME.
 3509  */
 3510 struct sockaddr *
 3511 sodupsockaddr(const struct sockaddr *sa, int mflags)
 3512 {
 3513         struct sockaddr *sa2;
 3514 
 3515         sa2 = malloc(sa->sa_len, M_SONAME, mflags);
 3516         if (sa2)
 3517                 bcopy(sa, sa2, sa->sa_len);
 3518         return sa2;
 3519 }
 3520 
 3521 /*
 3522  * Register per-socket buffer upcalls.
 3523  */
 3524 void
 3525 soupcall_set(struct socket *so, int which,
 3526     int (*func)(struct socket *, void *, int), void *arg)
 3527 {
 3528         struct sockbuf *sb;
 3529 
 3530         switch (which) {
 3531         case SO_RCV:
 3532                 sb = &so->so_rcv;
 3533                 break;
 3534         case SO_SND:
 3535                 sb = &so->so_snd;
 3536                 break;
 3537         default:
 3538                 panic("soupcall_set: bad which");
 3539         }
 3540         SOCKBUF_LOCK_ASSERT(sb);
 3541 #if 0
 3542         /* XXX: accf_http actually wants to do this on purpose. */
 3543         KASSERT(sb->sb_upcall == NULL, ("soupcall_set: overwriting upcall"));
 3544 #endif
 3545         sb->sb_upcall = func;
 3546         sb->sb_upcallarg = arg;
 3547         sb->sb_flags |= SB_UPCALL;
 3548 }
 3549 
 3550 void
 3551 soupcall_clear(struct socket *so, int which)
 3552 {
 3553         struct sockbuf *sb;
 3554 
 3555         switch (which) {
 3556         case SO_RCV:
 3557                 sb = &so->so_rcv;
 3558                 break;
 3559         case SO_SND:
 3560                 sb = &so->so_snd;
 3561                 break;
 3562         default:
 3563                 panic("soupcall_clear: bad which");
 3564         }
 3565         SOCKBUF_LOCK_ASSERT(sb);
 3566         KASSERT(sb->sb_upcall != NULL, ("soupcall_clear: no upcall to clear"));
 3567         sb->sb_upcall = NULL;
 3568         sb->sb_upcallarg = NULL;
 3569         sb->sb_flags &= ~SB_UPCALL;
 3570 }
 3571 
 3572 /*
 3573  * Create an external-format (``xsocket'') structure using the information in
 3574  * the kernel-format socket structure pointed to by so.  This is done to
 3575  * reduce the spew of irrelevant information over this interface, to isolate
 3576  * user code from changes in the kernel structure, and potentially to provide
 3577  * information-hiding if we decide that some of this information should be
 3578  * hidden from users.
 3579  */
 3580 void
 3581 sotoxsocket(struct socket *so, struct xsocket *xso)
 3582 {
 3583 
 3584         xso->xso_len = sizeof *xso;
 3585         xso->xso_so = so;
 3586         xso->so_type = so->so_type;
 3587         xso->so_options = so->so_options;
 3588         xso->so_linger = so->so_linger;
 3589         xso->so_state = so->so_state;
 3590         xso->so_pcb = so->so_pcb;
 3591         xso->xso_protocol = so->so_proto->pr_protocol;
 3592         xso->xso_family = so->so_proto->pr_domain->dom_family;
 3593         xso->so_qlen = so->so_qlen;
 3594         xso->so_incqlen = so->so_incqlen;
 3595         xso->so_qlimit = so->so_qlimit;
 3596         xso->so_timeo = so->so_timeo;
 3597         xso->so_error = so->so_error;
 3598         xso->so_pgid = so->so_sigio ? so->so_sigio->sio_pgid : 0;
 3599         xso->so_oobmark = so->so_oobmark;
 3600         sbtoxsockbuf(&so->so_snd, &xso->so_snd);
 3601         sbtoxsockbuf(&so->so_rcv, &xso->so_rcv);
 3602         xso->so_uid = so->so_cred->cr_uid;
 3603 }
 3604 
 3605 
 3606 /*
 3607  * Socket accessor functions to provide external consumers with
 3608  * a safe interface to socket state
 3609  *
 3610  */
 3611 
 3612 void
 3613 so_listeners_apply_all(struct socket *so, void (*func)(struct socket *, void *),
 3614     void *arg)
 3615 {
 3616 
 3617         TAILQ_FOREACH(so, &so->so_comp, so_list)
 3618                 func(so, arg);
 3619 }
 3620 
 3621 struct sockbuf *
 3622 so_sockbuf_rcv(struct socket *so)
 3623 {
 3624 
 3625         return (&so->so_rcv);
 3626 }
 3627 
 3628 struct sockbuf *
 3629 so_sockbuf_snd(struct socket *so)
 3630 {
 3631 
 3632         return (&so->so_snd);
 3633 }
 3634 
 3635 int
 3636 so_state_get(const struct socket *so)
 3637 {
 3638 
 3639         return (so->so_state);
 3640 }
 3641 
 3642 void
 3643 so_state_set(struct socket *so, int val)
 3644 {
 3645 
 3646         so->so_state = val;
 3647 }
 3648 
 3649 int
 3650 so_options_get(const struct socket *so)
 3651 {
 3652 
 3653         return (so->so_options);
 3654 }
 3655 
 3656 void
 3657 so_options_set(struct socket *so, int val)
 3658 {
 3659 
 3660         so->so_options = val;
 3661 }
 3662 
 3663 int
 3664 so_error_get(const struct socket *so)
 3665 {
 3666 
 3667         return (so->so_error);
 3668 }
 3669 
 3670 void
 3671 so_error_set(struct socket *so, int val)
 3672 {
 3673 
 3674         so->so_error = val;
 3675 }
 3676 
 3677 int
 3678 so_linger_get(const struct socket *so)
 3679 {
 3680 
 3681         return (so->so_linger);
 3682 }
 3683 
 3684 void
 3685 so_linger_set(struct socket *so, int val)
 3686 {
 3687 
 3688         so->so_linger = val;
 3689 }
 3690 
 3691 struct protosw *
 3692 so_protosw_get(const struct socket *so)
 3693 {
 3694 
 3695         return (so->so_proto);
 3696 }
 3697 
 3698 void
 3699 so_protosw_set(struct socket *so, struct protosw *val)
 3700 {
 3701 
 3702         so->so_proto = val;
 3703 }
 3704 
 3705 void
 3706 so_sorwakeup(struct socket *so)
 3707 {
 3708 
 3709         sorwakeup(so);
 3710 }
 3711 
 3712 void
 3713 so_sowwakeup(struct socket *so)
 3714 {
 3715 
 3716         sowwakeup(so);
 3717 }
 3718 
 3719 void
 3720 so_sorwakeup_locked(struct socket *so)
 3721 {
 3722 
 3723         sorwakeup_locked(so);
 3724 }
 3725 
 3726 void
 3727 so_sowwakeup_locked(struct socket *so)
 3728 {
 3729 
 3730         sowwakeup_locked(so);
 3731 }
 3732 
 3733 void
 3734 so_lock(struct socket *so)
 3735 {
 3736 
 3737         SOCK_LOCK(so);
 3738 }
 3739 
 3740 void
 3741 so_unlock(struct socket *so)
 3742 {
 3743 
 3744         SOCK_UNLOCK(so);
 3745 }

Cache object: be30210c1dfb20991a50707c99b07a9b


[ source navigation ] [ diff markup ] [ identifier search ] [ freetext search ] [ file search ] [ list types ] [ track identifier ]


This page is part of the FreeBSD/Linux Linux Kernel Cross-Reference, and was automatically generated using a modified version of the LXR engine.