The Design and Implementation of the FreeBSD Operating System, Second Edition
Now available: The Design and Implementation of the FreeBSD Operating System (Second Edition)


[ source navigation ] [ diff markup ] [ identifier search ] [ freetext search ] [ file search ] [ list types ] [ track identifier ]

FreeBSD/Linux Kernel Cross Reference
sys/kern/uipc_socket.c

Version: -  FREEBSD  -  FREEBSD-12-STABLE  -  FREEBSD-12-0  -  FREEBSD-11-STABLE  -  FREEBSD-11-2  -  FREEBSD-11-1  -  FREEBSD-11-0  -  FREEBSD-10-STABLE  -  FREEBSD-10-4  -  FREEBSD-10-3  -  FREEBSD-10-2  -  FREEBSD-10-1  -  FREEBSD-10-0  -  FREEBSD-9-STABLE  -  FREEBSD-9-3  -  FREEBSD-9-2  -  FREEBSD-9-1  -  FREEBSD-9-0  -  FREEBSD-8-STABLE  -  FREEBSD-8-4  -  FREEBSD-8-3  -  FREEBSD-8-2  -  FREEBSD-8-1  -  FREEBSD-8-0  -  FREEBSD-7-STABLE  -  FREEBSD-7-4  -  FREEBSD-7-3  -  FREEBSD-7-2  -  FREEBSD-7-1  -  FREEBSD-7-0  -  FREEBSD-6-STABLE  -  FREEBSD-6-4  -  FREEBSD-6-3  -  FREEBSD-6-2  -  FREEBSD-6-1  -  FREEBSD-6-0  -  FREEBSD-5-STABLE  -  FREEBSD-5-5  -  FREEBSD-5-4  -  FREEBSD-5-3  -  FREEBSD-5-2  -  FREEBSD-5-1  -  FREEBSD-5-0  -  FREEBSD-4-STABLE  -  FREEBSD-3-STABLE  -  FREEBSD22  -  linux-2.6  -  linux-2.4.22  -  MK83  -  MK84  -  PLAN9  -  DFBSD  -  NETBSD  -  NETBSD5  -  NETBSD4  -  NETBSD3  -  NETBSD20  -  OPENBSD  -  xnu-517  -  xnu-792  -  xnu-792.6.70  -  xnu-1228  -  xnu-1456.1.26  -  xnu-1699.24.8  -  xnu-2050.18.24  -  OPENSOLARIS  -  minix-3-1-1 
SearchContext: -  none  -  3  -  10 

    1 /*-
    2  * Copyright (c) 1982, 1986, 1988, 1990, 1993
    3  *      The Regents of the University of California.
    4  * Copyright (c) 2004 The FreeBSD Foundation
    5  * Copyright (c) 2004-2008 Robert N. M. Watson
    6  * All rights reserved.
    7  *
    8  * Redistribution and use in source and binary forms, with or without
    9  * modification, are permitted provided that the following conditions
   10  * are met:
   11  * 1. Redistributions of source code must retain the above copyright
   12  *    notice, this list of conditions and the following disclaimer.
   13  * 2. Redistributions in binary form must reproduce the above copyright
   14  *    notice, this list of conditions and the following disclaimer in the
   15  *    documentation and/or other materials provided with the distribution.
   16  * 4. Neither the name of the University nor the names of its contributors
   17  *    may be used to endorse or promote products derived from this software
   18  *    without specific prior written permission.
   19  *
   20  * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
   21  * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
   22  * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
   23  * ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
   24  * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
   25  * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
   26  * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
   27  * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
   28  * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
   29  * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
   30  * SUCH DAMAGE.
   31  *
   32  *      @(#)uipc_socket.c       8.3 (Berkeley) 4/15/94
   33  */
   34 
   35 /*
   36  * Comments on the socket life cycle:
   37  *
   38  * soalloc() sets of socket layer state for a socket, called only by
   39  * socreate() and sonewconn().  Socket layer private.
   40  *
   41  * sodealloc() tears down socket layer state for a socket, called only by
   42  * sofree() and sonewconn().  Socket layer private.
   43  *
   44  * pru_attach() associates protocol layer state with an allocated socket;
   45  * called only once, may fail, aborting socket allocation.  This is called
   46  * from socreate() and sonewconn().  Socket layer private.
   47  *
   48  * pru_detach() disassociates protocol layer state from an attached socket,
   49  * and will be called exactly once for sockets in which pru_attach() has
   50  * been successfully called.  If pru_attach() returned an error,
   51  * pru_detach() will not be called.  Socket layer private.
   52  *
   53  * pru_abort() and pru_close() notify the protocol layer that the last
   54  * consumer of a socket is starting to tear down the socket, and that the
   55  * protocol should terminate the connection.  Historically, pru_abort() also
   56  * detached protocol state from the socket state, but this is no longer the
   57  * case.
   58  *
   59  * socreate() creates a socket and attaches protocol state.  This is a public
   60  * interface that may be used by socket layer consumers to create new
   61  * sockets.
   62  *
   63  * sonewconn() creates a socket and attaches protocol state.  This is a
   64  * public interface  that may be used by protocols to create new sockets when
   65  * a new connection is received and will be available for accept() on a
   66  * listen socket.
   67  *
   68  * soclose() destroys a socket after possibly waiting for it to disconnect.
   69  * This is a public interface that socket consumers should use to close and
   70  * release a socket when done with it.
   71  *
   72  * soabort() destroys a socket without waiting for it to disconnect (used
   73  * only for incoming connections that are already partially or fully
   74  * connected).  This is used internally by the socket layer when clearing
   75  * listen socket queues (due to overflow or close on the listen socket), but
   76  * is also a public interface protocols may use to abort connections in
   77  * their incomplete listen queues should they no longer be required.  Sockets
   78  * placed in completed connection listen queues should not be aborted for
   79  * reasons described in the comment above the soclose() implementation.  This
   80  * is not a general purpose close routine, and except in the specific
   81  * circumstances described here, should not be used.
   82  *
   83  * sofree() will free a socket and its protocol state if all references on
   84  * the socket have been released, and is the public interface to attempt to
   85  * free a socket when a reference is removed.  This is a socket layer private
   86  * interface.
   87  *
   88  * NOTE: In addition to socreate() and soclose(), which provide a single
   89  * socket reference to the consumer to be managed as required, there are two
   90  * calls to explicitly manage socket references, soref(), and sorele().
   91  * Currently, these are generally required only when transitioning a socket
   92  * from a listen queue to a file descriptor, in order to prevent garbage
   93  * collection of the socket at an untimely moment.  For a number of reasons,
   94  * these interfaces are not preferred, and should be avoided.
   95  *
   96  * NOTE: With regard to VNETs the general rule is that callers do not set
   97  * curvnet. Exceptions to this rule include soabort(), sodisconnect(),
   98  * sofree() (and with that sorele(), sotryfree()), as well as sonewconn()
   99  * and sorflush(), which are usually called from a pre-set VNET context.
  100  * sopoll() currently does not need a VNET context to be set.
  101  */
  102 
  103 #include <sys/cdefs.h>
  104 __FBSDID("$FreeBSD: releng/11.0/sys/kern/uipc_socket.c 300419 2016-05-22 13:10:48Z bapt $");
  105 
  106 #include "opt_inet.h"
  107 #include "opt_inet6.h"
  108 #include "opt_compat.h"
  109 
  110 #include <sys/param.h>
  111 #include <sys/systm.h>
  112 #include <sys/fcntl.h>
  113 #include <sys/limits.h>
  114 #include <sys/lock.h>
  115 #include <sys/mac.h>
  116 #include <sys/malloc.h>
  117 #include <sys/mbuf.h>
  118 #include <sys/mutex.h>
  119 #include <sys/domain.h>
  120 #include <sys/file.h>                   /* for struct knote */
  121 #include <sys/hhook.h>
  122 #include <sys/kernel.h>
  123 #include <sys/khelp.h>
  124 #include <sys/event.h>
  125 #include <sys/eventhandler.h>
  126 #include <sys/poll.h>
  127 #include <sys/proc.h>
  128 #include <sys/protosw.h>
  129 #include <sys/socket.h>
  130 #include <sys/socketvar.h>
  131 #include <sys/resourcevar.h>
  132 #include <net/route.h>
  133 #include <sys/signalvar.h>
  134 #include <sys/stat.h>
  135 #include <sys/sx.h>
  136 #include <sys/sysctl.h>
  137 #include <sys/taskqueue.h>
  138 #include <sys/uio.h>
  139 #include <sys/jail.h>
  140 #include <sys/syslog.h>
  141 #include <netinet/in.h>
  142 
  143 #include <net/vnet.h>
  144 
  145 #include <security/mac/mac_framework.h>
  146 
  147 #include <vm/uma.h>
  148 
  149 #ifdef COMPAT_FREEBSD32
  150 #include <sys/mount.h>
  151 #include <sys/sysent.h>
  152 #include <compat/freebsd32/freebsd32.h>
  153 #endif
  154 
  155 static int      soreceive_rcvoob(struct socket *so, struct uio *uio,
  156                     int flags);
  157 
  158 static void     filt_sordetach(struct knote *kn);
  159 static int      filt_soread(struct knote *kn, long hint);
  160 static void     filt_sowdetach(struct knote *kn);
  161 static int      filt_sowrite(struct knote *kn, long hint);
  162 static int      filt_solisten(struct knote *kn, long hint);
  163 static int inline hhook_run_socket(struct socket *so, void *hctx, int32_t h_id);
  164 fo_kqfilter_t   soo_kqfilter;
  165 
  166 static struct filterops solisten_filtops = {
  167         .f_isfd = 1,
  168         .f_detach = filt_sordetach,
  169         .f_event = filt_solisten,
  170 };
  171 static struct filterops soread_filtops = {
  172         .f_isfd = 1,
  173         .f_detach = filt_sordetach,
  174         .f_event = filt_soread,
  175 };
  176 static struct filterops sowrite_filtops = {
  177         .f_isfd = 1,
  178         .f_detach = filt_sowdetach,
  179         .f_event = filt_sowrite,
  180 };
  181 
  182 so_gen_t        so_gencnt;      /* generation count for sockets */
  183 
  184 MALLOC_DEFINE(M_SONAME, "soname", "socket name");
  185 MALLOC_DEFINE(M_PCB, "pcb", "protocol control block");
  186 
  187 #define VNET_SO_ASSERT(so)                                              \
  188         VNET_ASSERT(curvnet != NULL,                                    \
  189             ("%s:%d curvnet is NULL, so=%p", __func__, __LINE__, (so)));
  190 
  191 VNET_DEFINE(struct hhook_head *, socket_hhh[HHOOK_SOCKET_LAST + 1]);
  192 #define V_socket_hhh            VNET(socket_hhh)
  193 
  194 /*
  195  * Limit on the number of connections in the listen queue waiting
  196  * for accept(2).
  197  * NB: The original sysctl somaxconn is still available but hidden
  198  * to prevent confusion about the actual purpose of this number.
  199  */
  200 static u_int somaxconn = SOMAXCONN;
  201 
  202 static int
  203 sysctl_somaxconn(SYSCTL_HANDLER_ARGS)
  204 {
  205         int error;
  206         int val;
  207 
  208         val = somaxconn;
  209         error = sysctl_handle_int(oidp, &val, 0, req);
  210         if (error || !req->newptr )
  211                 return (error);
  212 
  213         /*
  214          * The purpose of the UINT_MAX / 3 limit, is so that the formula
  215          *   3 * so_qlimit / 2
  216          * below, will not overflow.
  217          */
  218 
  219         if (val < 1 || val > UINT_MAX / 3)
  220                 return (EINVAL);
  221 
  222         somaxconn = val;
  223         return (0);
  224 }
  225 SYSCTL_PROC(_kern_ipc, OID_AUTO, soacceptqueue, CTLTYPE_UINT | CTLFLAG_RW,
  226     0, sizeof(int), sysctl_somaxconn, "I",
  227     "Maximum listen socket pending connection accept queue size");
  228 SYSCTL_PROC(_kern_ipc, KIPC_SOMAXCONN, somaxconn,
  229     CTLTYPE_UINT | CTLFLAG_RW | CTLFLAG_SKIP,
  230     0, sizeof(int), sysctl_somaxconn, "I",
  231     "Maximum listen socket pending connection accept queue size (compat)");
  232 
  233 static int numopensockets;
  234 SYSCTL_INT(_kern_ipc, OID_AUTO, numopensockets, CTLFLAG_RD,
  235     &numopensockets, 0, "Number of open sockets");
  236 
  237 /*
  238  * accept_mtx locks down per-socket fields relating to accept queues.  See
  239  * socketvar.h for an annotation of the protected fields of struct socket.
  240  */
  241 struct mtx accept_mtx;
  242 MTX_SYSINIT(accept_mtx, &accept_mtx, "accept", MTX_DEF);
  243 
  244 /*
  245  * so_global_mtx protects so_gencnt, numopensockets, and the per-socket
  246  * so_gencnt field.
  247  */
  248 static struct mtx so_global_mtx;
  249 MTX_SYSINIT(so_global_mtx, &so_global_mtx, "so_glabel", MTX_DEF);
  250 
  251 /*
  252  * General IPC sysctl name space, used by sockets and a variety of other IPC
  253  * types.
  254  */
  255 SYSCTL_NODE(_kern, KERN_IPC, ipc, CTLFLAG_RW, 0, "IPC");
  256 
  257 /*
  258  * Initialize the socket subsystem and set up the socket
  259  * memory allocator.
  260  */
  261 static uma_zone_t socket_zone;
  262 int     maxsockets;
  263 
  264 static void
  265 socket_zone_change(void *tag)
  266 {
  267 
  268         maxsockets = uma_zone_set_max(socket_zone, maxsockets);
  269 }
  270 
  271 static void
  272 socket_hhook_register(int subtype)
  273 {
  274         
  275         if (hhook_head_register(HHOOK_TYPE_SOCKET, subtype,
  276             &V_socket_hhh[subtype],
  277             HHOOK_NOWAIT|HHOOK_HEADISINVNET) != 0)
  278                 printf("%s: WARNING: unable to register hook\n", __func__);
  279 }
  280 
  281 static void
  282 socket_hhook_deregister(int subtype)
  283 {
  284         
  285         if (hhook_head_deregister(V_socket_hhh[subtype]) != 0)
  286                 printf("%s: WARNING: unable to deregister hook\n", __func__);
  287 }
  288 
  289 static void
  290 socket_init(void *tag)
  291 {
  292 
  293         socket_zone = uma_zcreate("socket", sizeof(struct socket), NULL, NULL,
  294             NULL, NULL, UMA_ALIGN_PTR, UMA_ZONE_NOFREE);
  295         maxsockets = uma_zone_set_max(socket_zone, maxsockets);
  296         uma_zone_set_warning(socket_zone, "kern.ipc.maxsockets limit reached");
  297         EVENTHANDLER_REGISTER(maxsockets_change, socket_zone_change, NULL,
  298             EVENTHANDLER_PRI_FIRST);
  299 }
  300 SYSINIT(socket, SI_SUB_PROTO_DOMAININIT, SI_ORDER_ANY, socket_init, NULL);
  301 
  302 static void
  303 socket_vnet_init(const void *unused __unused)
  304 {
  305         int i;
  306 
  307         /* We expect a contiguous range */
  308         for (i = 0; i <= HHOOK_SOCKET_LAST; i++)
  309                 socket_hhook_register(i);
  310 }
  311 VNET_SYSINIT(socket_vnet_init, SI_SUB_PROTO_DOMAININIT, SI_ORDER_ANY,
  312     socket_vnet_init, NULL);
  313 
  314 static void
  315 socket_vnet_uninit(const void *unused __unused)
  316 {
  317         int i;
  318 
  319         for (i = 0; i <= HHOOK_SOCKET_LAST; i++)
  320                 socket_hhook_deregister(i);
  321 }
  322 VNET_SYSUNINIT(socket_vnet_uninit, SI_SUB_PROTO_DOMAININIT, SI_ORDER_ANY,
  323     socket_vnet_uninit, NULL);
  324 
  325 /*
  326  * Initialise maxsockets.  This SYSINIT must be run after
  327  * tunable_mbinit().
  328  */
  329 static void
  330 init_maxsockets(void *ignored)
  331 {
  332 
  333         TUNABLE_INT_FETCH("kern.ipc.maxsockets", &maxsockets);
  334         maxsockets = imax(maxsockets, maxfiles);
  335 }
  336 SYSINIT(param, SI_SUB_TUNABLES, SI_ORDER_ANY, init_maxsockets, NULL);
  337 
  338 /*
  339  * Sysctl to get and set the maximum global sockets limit.  Notify protocols
  340  * of the change so that they can update their dependent limits as required.
  341  */
  342 static int
  343 sysctl_maxsockets(SYSCTL_HANDLER_ARGS)
  344 {
  345         int error, newmaxsockets;
  346 
  347         newmaxsockets = maxsockets;
  348         error = sysctl_handle_int(oidp, &newmaxsockets, 0, req);
  349         if (error == 0 && req->newptr) {
  350                 if (newmaxsockets > maxsockets &&
  351                     newmaxsockets <= maxfiles) {
  352                         maxsockets = newmaxsockets;
  353                         EVENTHANDLER_INVOKE(maxsockets_change);
  354                 } else
  355                         error = EINVAL;
  356         }
  357         return (error);
  358 }
  359 SYSCTL_PROC(_kern_ipc, OID_AUTO, maxsockets, CTLTYPE_INT|CTLFLAG_RW,
  360     &maxsockets, 0, sysctl_maxsockets, "IU",
  361     "Maximum number of sockets available");
  362 
  363 /*
  364  * Socket operation routines.  These routines are called by the routines in
  365  * sys_socket.c or from a system process, and implement the semantics of
  366  * socket operations by switching out to the protocol specific routines.
  367  */
  368 
  369 /*
  370  * Get a socket structure from our zone, and initialize it.  Note that it
  371  * would probably be better to allocate socket and PCB at the same time, but
  372  * I'm not convinced that all the protocols can be easily modified to do
  373  * this.
  374  *
  375  * soalloc() returns a socket with a ref count of 0.
  376  */
  377 static struct socket *
  378 soalloc(struct vnet *vnet)
  379 {
  380         struct socket *so;
  381 
  382         so = uma_zalloc(socket_zone, M_NOWAIT | M_ZERO);
  383         if (so == NULL)
  384                 return (NULL);
  385 #ifdef MAC
  386         if (mac_socket_init(so, M_NOWAIT) != 0) {
  387                 uma_zfree(socket_zone, so);
  388                 return (NULL);
  389         }
  390 #endif
  391         if (khelp_init_osd(HELPER_CLASS_SOCKET, &so->osd)) {
  392                 uma_zfree(socket_zone, so);
  393                 return (NULL);
  394         }
  395 
  396         SOCKBUF_LOCK_INIT(&so->so_snd, "so_snd");
  397         SOCKBUF_LOCK_INIT(&so->so_rcv, "so_rcv");
  398         sx_init(&so->so_snd.sb_sx, "so_snd_sx");
  399         sx_init(&so->so_rcv.sb_sx, "so_rcv_sx");
  400         TAILQ_INIT(&so->so_snd.sb_aiojobq);
  401         TAILQ_INIT(&so->so_rcv.sb_aiojobq);
  402         TASK_INIT(&so->so_snd.sb_aiotask, 0, soaio_snd, so);
  403         TASK_INIT(&so->so_rcv.sb_aiotask, 0, soaio_rcv, so);
  404 #ifdef VIMAGE
  405         VNET_ASSERT(vnet != NULL, ("%s:%d vnet is NULL, so=%p",
  406             __func__, __LINE__, so));
  407         so->so_vnet = vnet;
  408 #endif
  409         /* We shouldn't need the so_global_mtx */
  410         if (hhook_run_socket(so, NULL, HHOOK_SOCKET_CREATE)) {
  411                 /* Do we need more comprehensive error returns? */
  412                 uma_zfree(socket_zone, so);
  413                 return (NULL);
  414         }
  415         mtx_lock(&so_global_mtx);
  416         so->so_gencnt = ++so_gencnt;
  417         ++numopensockets;
  418 #ifdef VIMAGE
  419         vnet->vnet_sockcnt++;
  420 #endif
  421         mtx_unlock(&so_global_mtx);
  422 
  423         return (so);
  424 }
  425 
  426 /*
  427  * Free the storage associated with a socket at the socket layer, tear down
  428  * locks, labels, etc.  All protocol state is assumed already to have been
  429  * torn down (and possibly never set up) by the caller.
  430  */
  431 static void
  432 sodealloc(struct socket *so)
  433 {
  434 
  435         KASSERT(so->so_count == 0, ("sodealloc(): so_count %d", so->so_count));
  436         KASSERT(so->so_pcb == NULL, ("sodealloc(): so_pcb != NULL"));
  437 
  438         mtx_lock(&so_global_mtx);
  439         so->so_gencnt = ++so_gencnt;
  440         --numopensockets;       /* Could be below, but faster here. */
  441 #ifdef VIMAGE
  442         VNET_ASSERT(so->so_vnet != NULL, ("%s:%d so_vnet is NULL, so=%p",
  443             __func__, __LINE__, so));
  444         so->so_vnet->vnet_sockcnt--;
  445 #endif
  446         mtx_unlock(&so_global_mtx);
  447         if (so->so_rcv.sb_hiwat)
  448                 (void)chgsbsize(so->so_cred->cr_uidinfo,
  449                     &so->so_rcv.sb_hiwat, 0, RLIM_INFINITY);
  450         if (so->so_snd.sb_hiwat)
  451                 (void)chgsbsize(so->so_cred->cr_uidinfo,
  452                     &so->so_snd.sb_hiwat, 0, RLIM_INFINITY);
  453         /* remove accept filter if one is present. */
  454         if (so->so_accf != NULL)
  455                 do_setopt_accept_filter(so, NULL);
  456 #ifdef MAC
  457         mac_socket_destroy(so);
  458 #endif
  459         hhook_run_socket(so, NULL, HHOOK_SOCKET_CLOSE);
  460 
  461         crfree(so->so_cred);
  462         khelp_destroy_osd(&so->osd);
  463         sx_destroy(&so->so_snd.sb_sx);
  464         sx_destroy(&so->so_rcv.sb_sx);
  465         SOCKBUF_LOCK_DESTROY(&so->so_snd);
  466         SOCKBUF_LOCK_DESTROY(&so->so_rcv);
  467         uma_zfree(socket_zone, so);
  468 }
  469 
  470 /*
  471  * socreate returns a socket with a ref count of 1.  The socket should be
  472  * closed with soclose().
  473  */
  474 int
  475 socreate(int dom, struct socket **aso, int type, int proto,
  476     struct ucred *cred, struct thread *td)
  477 {
  478         struct protosw *prp;
  479         struct socket *so;
  480         int error;
  481 
  482         if (proto)
  483                 prp = pffindproto(dom, proto, type);
  484         else
  485                 prp = pffindtype(dom, type);
  486 
  487         if (prp == NULL) {
  488                 /* No support for domain. */
  489                 if (pffinddomain(dom) == NULL)
  490                         return (EAFNOSUPPORT);
  491                 /* No support for socket type. */
  492                 if (proto == 0 && type != 0)
  493                         return (EPROTOTYPE);
  494                 return (EPROTONOSUPPORT);
  495         }
  496         if (prp->pr_usrreqs->pru_attach == NULL ||
  497             prp->pr_usrreqs->pru_attach == pru_attach_notsupp)
  498                 return (EPROTONOSUPPORT);
  499 
  500         if (prison_check_af(cred, prp->pr_domain->dom_family) != 0)
  501                 return (EPROTONOSUPPORT);
  502 
  503         if (prp->pr_type != type)
  504                 return (EPROTOTYPE);
  505         so = soalloc(CRED_TO_VNET(cred));
  506         if (so == NULL)
  507                 return (ENOBUFS);
  508 
  509         TAILQ_INIT(&so->so_incomp);
  510         TAILQ_INIT(&so->so_comp);
  511         so->so_type = type;
  512         so->so_cred = crhold(cred);
  513         if ((prp->pr_domain->dom_family == PF_INET) ||
  514             (prp->pr_domain->dom_family == PF_INET6) ||
  515             (prp->pr_domain->dom_family == PF_ROUTE))
  516                 so->so_fibnum = td->td_proc->p_fibnum;
  517         else
  518                 so->so_fibnum = 0;
  519         so->so_proto = prp;
  520 #ifdef MAC
  521         mac_socket_create(cred, so);
  522 #endif
  523         knlist_init_mtx(&so->so_rcv.sb_sel.si_note, SOCKBUF_MTX(&so->so_rcv));
  524         knlist_init_mtx(&so->so_snd.sb_sel.si_note, SOCKBUF_MTX(&so->so_snd));
  525         so->so_count = 1;
  526         /*
  527          * Auto-sizing of socket buffers is managed by the protocols and
  528          * the appropriate flags must be set in the pru_attach function.
  529          */
  530         CURVNET_SET(so->so_vnet);
  531         error = (*prp->pr_usrreqs->pru_attach)(so, proto, td);
  532         CURVNET_RESTORE();
  533         if (error) {
  534                 KASSERT(so->so_count == 1, ("socreate: so_count %d",
  535                     so->so_count));
  536                 so->so_count = 0;
  537                 sodealloc(so);
  538                 return (error);
  539         }
  540         *aso = so;
  541         return (0);
  542 }
  543 
  544 #ifdef REGRESSION
  545 static int regression_sonewconn_earlytest = 1;
  546 SYSCTL_INT(_regression, OID_AUTO, sonewconn_earlytest, CTLFLAG_RW,
  547     &regression_sonewconn_earlytest, 0, "Perform early sonewconn limit test");
  548 #endif
  549 
  550 /*
  551  * When an attempt at a new connection is noted on a socket which accepts
  552  * connections, sonewconn is called.  If the connection is possible (subject
  553  * to space constraints, etc.) then we allocate a new structure, properly
  554  * linked into the data structure of the original socket, and return this.
  555  * Connstatus may be 0, or SS_ISCONFIRMING, or SS_ISCONNECTED.
  556  *
  557  * Note: the ref count on the socket is 0 on return.
  558  */
  559 struct socket *
  560 sonewconn(struct socket *head, int connstatus)
  561 {
  562         static struct timeval lastover;
  563         static struct timeval overinterval = { 60, 0 };
  564         static int overcount;
  565 
  566         struct socket *so;
  567         int over;
  568 
  569         ACCEPT_LOCK();
  570         over = (head->so_qlen > 3 * head->so_qlimit / 2);
  571         ACCEPT_UNLOCK();
  572 #ifdef REGRESSION
  573         if (regression_sonewconn_earlytest && over) {
  574 #else
  575         if (over) {
  576 #endif
  577                 overcount++;
  578 
  579                 if (ratecheck(&lastover, &overinterval)) {
  580                         log(LOG_DEBUG, "%s: pcb %p: Listen queue overflow: "
  581                             "%i already in queue awaiting acceptance "
  582                             "(%d occurrences)\n",
  583                             __func__, head->so_pcb, head->so_qlen, overcount);
  584 
  585                         overcount = 0;
  586                 }
  587 
  588                 return (NULL);
  589         }
  590         VNET_ASSERT(head->so_vnet != NULL, ("%s:%d so_vnet is NULL, head=%p",
  591             __func__, __LINE__, head));
  592         so = soalloc(head->so_vnet);
  593         if (so == NULL) {
  594                 log(LOG_DEBUG, "%s: pcb %p: New socket allocation failure: "
  595                     "limit reached or out of memory\n",
  596                     __func__, head->so_pcb);
  597                 return (NULL);
  598         }
  599         if ((head->so_options & SO_ACCEPTFILTER) != 0)
  600                 connstatus = 0;
  601         so->so_head = head;
  602         so->so_type = head->so_type;
  603         so->so_options = head->so_options &~ SO_ACCEPTCONN;
  604         so->so_linger = head->so_linger;
  605         so->so_state = head->so_state | SS_NOFDREF;
  606         so->so_fibnum = head->so_fibnum;
  607         so->so_proto = head->so_proto;
  608         so->so_cred = crhold(head->so_cred);
  609 #ifdef MAC
  610         mac_socket_newconn(head, so);
  611 #endif
  612         knlist_init_mtx(&so->so_rcv.sb_sel.si_note, SOCKBUF_MTX(&so->so_rcv));
  613         knlist_init_mtx(&so->so_snd.sb_sel.si_note, SOCKBUF_MTX(&so->so_snd));
  614         VNET_SO_ASSERT(head);
  615         if (soreserve(so, head->so_snd.sb_hiwat, head->so_rcv.sb_hiwat)) {
  616                 sodealloc(so);
  617                 log(LOG_DEBUG, "%s: pcb %p: soreserve() failed\n",
  618                     __func__, head->so_pcb);
  619                 return (NULL);
  620         }
  621         if ((*so->so_proto->pr_usrreqs->pru_attach)(so, 0, NULL)) {
  622                 sodealloc(so);
  623                 log(LOG_DEBUG, "%s: pcb %p: pru_attach() failed\n",
  624                     __func__, head->so_pcb);
  625                 return (NULL);
  626         }
  627         so->so_rcv.sb_lowat = head->so_rcv.sb_lowat;
  628         so->so_snd.sb_lowat = head->so_snd.sb_lowat;
  629         so->so_rcv.sb_timeo = head->so_rcv.sb_timeo;
  630         so->so_snd.sb_timeo = head->so_snd.sb_timeo;
  631         so->so_rcv.sb_flags |= head->so_rcv.sb_flags & SB_AUTOSIZE;
  632         so->so_snd.sb_flags |= head->so_snd.sb_flags & SB_AUTOSIZE;
  633         so->so_state |= connstatus;
  634         ACCEPT_LOCK();
  635         /*
  636          * The accept socket may be tearing down but we just
  637          * won a race on the ACCEPT_LOCK.
  638          * However, if sctp_peeloff() is called on a 1-to-many
  639          * style socket, the SO_ACCEPTCONN doesn't need to be set.
  640          */
  641         if (!(head->so_options & SO_ACCEPTCONN) &&
  642             ((head->so_proto->pr_protocol != IPPROTO_SCTP) ||
  643              (head->so_type != SOCK_SEQPACKET))) {
  644                 SOCK_LOCK(so);
  645                 so->so_head = NULL;
  646                 sofree(so);             /* NB: returns ACCEPT_UNLOCK'ed. */
  647                 return (NULL);
  648         }
  649         if (connstatus) {
  650                 TAILQ_INSERT_TAIL(&head->so_comp, so, so_list);
  651                 so->so_qstate |= SQ_COMP;
  652                 head->so_qlen++;
  653         } else {
  654                 /*
  655                  * Keep removing sockets from the head until there's room for
  656                  * us to insert on the tail.  In pre-locking revisions, this
  657                  * was a simple if(), but as we could be racing with other
  658                  * threads and soabort() requires dropping locks, we must
  659                  * loop waiting for the condition to be true.
  660                  */
  661                 while (head->so_incqlen > head->so_qlimit) {
  662                         struct socket *sp;
  663                         sp = TAILQ_FIRST(&head->so_incomp);
  664                         TAILQ_REMOVE(&head->so_incomp, sp, so_list);
  665                         head->so_incqlen--;
  666                         sp->so_qstate &= ~SQ_INCOMP;
  667                         sp->so_head = NULL;
  668                         ACCEPT_UNLOCK();
  669                         soabort(sp);
  670                         ACCEPT_LOCK();
  671                 }
  672                 TAILQ_INSERT_TAIL(&head->so_incomp, so, so_list);
  673                 so->so_qstate |= SQ_INCOMP;
  674                 head->so_incqlen++;
  675         }
  676         ACCEPT_UNLOCK();
  677         if (connstatus) {
  678                 sorwakeup(head);
  679                 wakeup_one(&head->so_timeo);
  680         }
  681         return (so);
  682 }
  683 
  684 int
  685 sobind(struct socket *so, struct sockaddr *nam, struct thread *td)
  686 {
  687         int error;
  688 
  689         CURVNET_SET(so->so_vnet);
  690         error = (*so->so_proto->pr_usrreqs->pru_bind)(so, nam, td);
  691         CURVNET_RESTORE();
  692         return (error);
  693 }
  694 
  695 int
  696 sobindat(int fd, struct socket *so, struct sockaddr *nam, struct thread *td)
  697 {
  698         int error;
  699 
  700         CURVNET_SET(so->so_vnet);
  701         error = (*so->so_proto->pr_usrreqs->pru_bindat)(fd, so, nam, td);
  702         CURVNET_RESTORE();
  703         return (error);
  704 }
  705 
  706 /*
  707  * solisten() transitions a socket from a non-listening state to a listening
  708  * state, but can also be used to update the listen queue depth on an
  709  * existing listen socket.  The protocol will call back into the sockets
  710  * layer using solisten_proto_check() and solisten_proto() to check and set
  711  * socket-layer listen state.  Call backs are used so that the protocol can
  712  * acquire both protocol and socket layer locks in whatever order is required
  713  * by the protocol.
  714  *
  715  * Protocol implementors are advised to hold the socket lock across the
  716  * socket-layer test and set to avoid races at the socket layer.
  717  */
  718 int
  719 solisten(struct socket *so, int backlog, struct thread *td)
  720 {
  721         int error;
  722 
  723         CURVNET_SET(so->so_vnet);
  724         error = (*so->so_proto->pr_usrreqs->pru_listen)(so, backlog, td);
  725         CURVNET_RESTORE();
  726         return (error);
  727 }
  728 
  729 int
  730 solisten_proto_check(struct socket *so)
  731 {
  732 
  733         SOCK_LOCK_ASSERT(so);
  734 
  735         if (so->so_state & (SS_ISCONNECTED | SS_ISCONNECTING |
  736             SS_ISDISCONNECTING))
  737                 return (EINVAL);
  738         return (0);
  739 }
  740 
  741 void
  742 solisten_proto(struct socket *so, int backlog)
  743 {
  744 
  745         SOCK_LOCK_ASSERT(so);
  746 
  747         if (backlog < 0 || backlog > somaxconn)
  748                 backlog = somaxconn;
  749         so->so_qlimit = backlog;
  750         so->so_options |= SO_ACCEPTCONN;
  751 }
  752 
  753 /*
  754  * Evaluate the reference count and named references on a socket; if no
  755  * references remain, free it.  This should be called whenever a reference is
  756  * released, such as in sorele(), but also when named reference flags are
  757  * cleared in socket or protocol code.
  758  *
  759  * sofree() will free the socket if:
  760  *
  761  * - There are no outstanding file descriptor references or related consumers
  762  *   (so_count == 0).
  763  *
  764  * - The socket has been closed by user space, if ever open (SS_NOFDREF).
  765  *
  766  * - The protocol does not have an outstanding strong reference on the socket
  767  *   (SS_PROTOREF).
  768  *
  769  * - The socket is not in a completed connection queue, so a process has been
  770  *   notified that it is present.  If it is removed, the user process may
  771  *   block in accept() despite select() saying the socket was ready.
  772  */
  773 void
  774 sofree(struct socket *so)
  775 {
  776         struct protosw *pr = so->so_proto;
  777         struct socket *head;
  778 
  779         ACCEPT_LOCK_ASSERT();
  780         SOCK_LOCK_ASSERT(so);
  781 
  782         if ((so->so_state & SS_NOFDREF) == 0 || so->so_count != 0 ||
  783             (so->so_state & SS_PROTOREF) || (so->so_qstate & SQ_COMP)) {
  784                 SOCK_UNLOCK(so);
  785                 ACCEPT_UNLOCK();
  786                 return;
  787         }
  788 
  789         head = so->so_head;
  790         if (head != NULL) {
  791                 KASSERT((so->so_qstate & SQ_COMP) != 0 ||
  792                     (so->so_qstate & SQ_INCOMP) != 0,
  793                     ("sofree: so_head != NULL, but neither SQ_COMP nor "
  794                     "SQ_INCOMP"));
  795                 KASSERT((so->so_qstate & SQ_COMP) == 0 ||
  796                     (so->so_qstate & SQ_INCOMP) == 0,
  797                     ("sofree: so->so_qstate is SQ_COMP and also SQ_INCOMP"));
  798                 TAILQ_REMOVE(&head->so_incomp, so, so_list);
  799                 head->so_incqlen--;
  800                 so->so_qstate &= ~SQ_INCOMP;
  801                 so->so_head = NULL;
  802         }
  803         KASSERT((so->so_qstate & SQ_COMP) == 0 &&
  804             (so->so_qstate & SQ_INCOMP) == 0,
  805             ("sofree: so_head == NULL, but still SQ_COMP(%d) or SQ_INCOMP(%d)",
  806             so->so_qstate & SQ_COMP, so->so_qstate & SQ_INCOMP));
  807         if (so->so_options & SO_ACCEPTCONN) {
  808                 KASSERT((TAILQ_EMPTY(&so->so_comp)),
  809                     ("sofree: so_comp populated"));
  810                 KASSERT((TAILQ_EMPTY(&so->so_incomp)),
  811                     ("sofree: so_incomp populated"));
  812         }
  813         SOCK_UNLOCK(so);
  814         ACCEPT_UNLOCK();
  815 
  816         VNET_SO_ASSERT(so);
  817         if (pr->pr_flags & PR_RIGHTS && pr->pr_domain->dom_dispose != NULL)
  818                 (*pr->pr_domain->dom_dispose)(so);
  819         if (pr->pr_usrreqs->pru_detach != NULL)
  820                 (*pr->pr_usrreqs->pru_detach)(so);
  821 
  822         /*
  823          * From this point on, we assume that no other references to this
  824          * socket exist anywhere else in the stack.  Therefore, no locks need
  825          * to be acquired or held.
  826          *
  827          * We used to do a lot of socket buffer and socket locking here, as
  828          * well as invoke sorflush() and perform wakeups.  The direct call to
  829          * dom_dispose() and sbrelease_internal() are an inlining of what was
  830          * necessary from sorflush().
  831          *
  832          * Notice that the socket buffer and kqueue state are torn down
  833          * before calling pru_detach.  This means that protocols shold not
  834          * assume they can perform socket wakeups, etc, in their detach code.
  835          */
  836         sbdestroy(&so->so_snd, so);
  837         sbdestroy(&so->so_rcv, so);
  838         seldrain(&so->so_snd.sb_sel);
  839         seldrain(&so->so_rcv.sb_sel);
  840         knlist_destroy(&so->so_rcv.sb_sel.si_note);
  841         knlist_destroy(&so->so_snd.sb_sel.si_note);
  842         sodealloc(so);
  843 }
  844 
  845 /*
  846  * Close a socket on last file table reference removal.  Initiate disconnect
  847  * if connected.  Free socket when disconnect complete.
  848  *
  849  * This function will sorele() the socket.  Note that soclose() may be called
  850  * prior to the ref count reaching zero.  The actual socket structure will
  851  * not be freed until the ref count reaches zero.
  852  */
  853 int
  854 soclose(struct socket *so)
  855 {
  856         int error = 0;
  857 
  858         KASSERT(!(so->so_state & SS_NOFDREF), ("soclose: SS_NOFDREF on enter"));
  859 
  860         CURVNET_SET(so->so_vnet);
  861         funsetown(&so->so_sigio);
  862         if (so->so_state & SS_ISCONNECTED) {
  863                 if ((so->so_state & SS_ISDISCONNECTING) == 0) {
  864                         error = sodisconnect(so);
  865                         if (error) {
  866                                 if (error == ENOTCONN)
  867                                         error = 0;
  868                                 goto drop;
  869                         }
  870                 }
  871                 if (so->so_options & SO_LINGER) {
  872                         if ((so->so_state & SS_ISDISCONNECTING) &&
  873                             (so->so_state & SS_NBIO))
  874                                 goto drop;
  875                         while (so->so_state & SS_ISCONNECTED) {
  876                                 error = tsleep(&so->so_timeo,
  877                                     PSOCK | PCATCH, "soclos",
  878                                     so->so_linger * hz);
  879                                 if (error)
  880                                         break;
  881                         }
  882                 }
  883         }
  884 
  885 drop:
  886         if (so->so_proto->pr_usrreqs->pru_close != NULL)
  887                 (*so->so_proto->pr_usrreqs->pru_close)(so);
  888         ACCEPT_LOCK();
  889         if (so->so_options & SO_ACCEPTCONN) {
  890                 struct socket *sp;
  891                 /*
  892                  * Prevent new additions to the accept queues due
  893                  * to ACCEPT_LOCK races while we are draining them.
  894                  */
  895                 so->so_options &= ~SO_ACCEPTCONN;
  896                 while ((sp = TAILQ_FIRST(&so->so_incomp)) != NULL) {
  897                         TAILQ_REMOVE(&so->so_incomp, sp, so_list);
  898                         so->so_incqlen--;
  899                         sp->so_qstate &= ~SQ_INCOMP;
  900                         sp->so_head = NULL;
  901                         ACCEPT_UNLOCK();
  902                         soabort(sp);
  903                         ACCEPT_LOCK();
  904                 }
  905                 while ((sp = TAILQ_FIRST(&so->so_comp)) != NULL) {
  906                         TAILQ_REMOVE(&so->so_comp, sp, so_list);
  907                         so->so_qlen--;
  908                         sp->so_qstate &= ~SQ_COMP;
  909                         sp->so_head = NULL;
  910                         ACCEPT_UNLOCK();
  911                         soabort(sp);
  912                         ACCEPT_LOCK();
  913                 }
  914                 KASSERT((TAILQ_EMPTY(&so->so_comp)),
  915                     ("%s: so_comp populated", __func__));
  916                 KASSERT((TAILQ_EMPTY(&so->so_incomp)),
  917                     ("%s: so_incomp populated", __func__));
  918         }
  919         SOCK_LOCK(so);
  920         KASSERT((so->so_state & SS_NOFDREF) == 0, ("soclose: NOFDREF"));
  921         so->so_state |= SS_NOFDREF;
  922         sorele(so);                     /* NB: Returns with ACCEPT_UNLOCK(). */
  923         CURVNET_RESTORE();
  924         return (error);
  925 }
  926 
  927 /*
  928  * soabort() is used to abruptly tear down a connection, such as when a
  929  * resource limit is reached (listen queue depth exceeded), or if a listen
  930  * socket is closed while there are sockets waiting to be accepted.
  931  *
  932  * This interface is tricky, because it is called on an unreferenced socket,
  933  * and must be called only by a thread that has actually removed the socket
  934  * from the listen queue it was on, or races with other threads are risked.
  935  *
  936  * This interface will call into the protocol code, so must not be called
  937  * with any socket locks held.  Protocols do call it while holding their own
  938  * recursible protocol mutexes, but this is something that should be subject
  939  * to review in the future.
  940  */
  941 void
  942 soabort(struct socket *so)
  943 {
  944 
  945         /*
  946          * In as much as is possible, assert that no references to this
  947          * socket are held.  This is not quite the same as asserting that the
  948          * current thread is responsible for arranging for no references, but
  949          * is as close as we can get for now.
  950          */
  951         KASSERT(so->so_count == 0, ("soabort: so_count"));
  952         KASSERT((so->so_state & SS_PROTOREF) == 0, ("soabort: SS_PROTOREF"));
  953         KASSERT(so->so_state & SS_NOFDREF, ("soabort: !SS_NOFDREF"));
  954         KASSERT((so->so_state & SQ_COMP) == 0, ("soabort: SQ_COMP"));
  955         KASSERT((so->so_state & SQ_INCOMP) == 0, ("soabort: SQ_INCOMP"));
  956         VNET_SO_ASSERT(so);
  957 
  958         if (so->so_proto->pr_usrreqs->pru_abort != NULL)
  959                 (*so->so_proto->pr_usrreqs->pru_abort)(so);
  960         ACCEPT_LOCK();
  961         SOCK_LOCK(so);
  962         sofree(so);
  963 }
  964 
  965 int
  966 soaccept(struct socket *so, struct sockaddr **nam)
  967 {
  968         int error;
  969 
  970         SOCK_LOCK(so);
  971         KASSERT((so->so_state & SS_NOFDREF) != 0, ("soaccept: !NOFDREF"));
  972         so->so_state &= ~SS_NOFDREF;
  973         SOCK_UNLOCK(so);
  974 
  975         CURVNET_SET(so->so_vnet);
  976         error = (*so->so_proto->pr_usrreqs->pru_accept)(so, nam);
  977         CURVNET_RESTORE();
  978         return (error);
  979 }
  980 
  981 int
  982 soconnect(struct socket *so, struct sockaddr *nam, struct thread *td)
  983 {
  984 
  985         return (soconnectat(AT_FDCWD, so, nam, td));
  986 }
  987 
  988 int
  989 soconnectat(int fd, struct socket *so, struct sockaddr *nam, struct thread *td)
  990 {
  991         int error;
  992 
  993         if (so->so_options & SO_ACCEPTCONN)
  994                 return (EOPNOTSUPP);
  995 
  996         CURVNET_SET(so->so_vnet);
  997         /*
  998          * If protocol is connection-based, can only connect once.
  999          * Otherwise, if connected, try to disconnect first.  This allows
 1000          * user to disconnect by connecting to, e.g., a null address.
 1001          */
 1002         if (so->so_state & (SS_ISCONNECTED|SS_ISCONNECTING) &&
 1003             ((so->so_proto->pr_flags & PR_CONNREQUIRED) ||
 1004             (error = sodisconnect(so)))) {
 1005                 error = EISCONN;
 1006         } else {
 1007                 /*
 1008                  * Prevent accumulated error from previous connection from
 1009                  * biting us.
 1010                  */
 1011                 so->so_error = 0;
 1012                 if (fd == AT_FDCWD) {
 1013                         error = (*so->so_proto->pr_usrreqs->pru_connect)(so,
 1014                             nam, td);
 1015                 } else {
 1016                         error = (*so->so_proto->pr_usrreqs->pru_connectat)(fd,
 1017                             so, nam, td);
 1018                 }
 1019         }
 1020         CURVNET_RESTORE();
 1021 
 1022         return (error);
 1023 }
 1024 
 1025 int
 1026 soconnect2(struct socket *so1, struct socket *so2)
 1027 {
 1028         int error;
 1029 
 1030         CURVNET_SET(so1->so_vnet);
 1031         error = (*so1->so_proto->pr_usrreqs->pru_connect2)(so1, so2);
 1032         CURVNET_RESTORE();
 1033         return (error);
 1034 }
 1035 
 1036 int
 1037 sodisconnect(struct socket *so)
 1038 {
 1039         int error;
 1040 
 1041         if ((so->so_state & SS_ISCONNECTED) == 0)
 1042                 return (ENOTCONN);
 1043         if (so->so_state & SS_ISDISCONNECTING)
 1044                 return (EALREADY);
 1045         VNET_SO_ASSERT(so);
 1046         error = (*so->so_proto->pr_usrreqs->pru_disconnect)(so);
 1047         return (error);
 1048 }
 1049 
 1050 #define SBLOCKWAIT(f)   (((f) & MSG_DONTWAIT) ? 0 : SBL_WAIT)
 1051 
 1052 int
 1053 sosend_dgram(struct socket *so, struct sockaddr *addr, struct uio *uio,
 1054     struct mbuf *top, struct mbuf *control, int flags, struct thread *td)
 1055 {
 1056         long space;
 1057         ssize_t resid;
 1058         int clen = 0, error, dontroute;
 1059 
 1060         KASSERT(so->so_type == SOCK_DGRAM, ("sosend_dgram: !SOCK_DGRAM"));
 1061         KASSERT(so->so_proto->pr_flags & PR_ATOMIC,
 1062             ("sosend_dgram: !PR_ATOMIC"));
 1063 
 1064         if (uio != NULL)
 1065                 resid = uio->uio_resid;
 1066         else
 1067                 resid = top->m_pkthdr.len;
 1068         /*
 1069          * In theory resid should be unsigned.  However, space must be
 1070          * signed, as it might be less than 0 if we over-committed, and we
 1071          * must use a signed comparison of space and resid.  On the other
 1072          * hand, a negative resid causes us to loop sending 0-length
 1073          * segments to the protocol.
 1074          */
 1075         if (resid < 0) {
 1076                 error = EINVAL;
 1077                 goto out;
 1078         }
 1079 
 1080         dontroute =
 1081             (flags & MSG_DONTROUTE) && (so->so_options & SO_DONTROUTE) == 0;
 1082         if (td != NULL)
 1083                 td->td_ru.ru_msgsnd++;
 1084         if (control != NULL)
 1085                 clen = control->m_len;
 1086 
 1087         SOCKBUF_LOCK(&so->so_snd);
 1088         if (so->so_snd.sb_state & SBS_CANTSENDMORE) {
 1089                 SOCKBUF_UNLOCK(&so->so_snd);
 1090                 error = EPIPE;
 1091                 goto out;
 1092         }
 1093         if (so->so_error) {
 1094                 error = so->so_error;
 1095                 so->so_error = 0;
 1096                 SOCKBUF_UNLOCK(&so->so_snd);
 1097                 goto out;
 1098         }
 1099         if ((so->so_state & SS_ISCONNECTED) == 0) {
 1100                 /*
 1101                  * `sendto' and `sendmsg' is allowed on a connection-based
 1102                  * socket if it supports implied connect.  Return ENOTCONN if
 1103                  * not connected and no address is supplied.
 1104                  */
 1105                 if ((so->so_proto->pr_flags & PR_CONNREQUIRED) &&
 1106                     (so->so_proto->pr_flags & PR_IMPLOPCL) == 0) {
 1107                         if ((so->so_state & SS_ISCONFIRMING) == 0 &&
 1108                             !(resid == 0 && clen != 0)) {
 1109                                 SOCKBUF_UNLOCK(&so->so_snd);
 1110                                 error = ENOTCONN;
 1111                                 goto out;
 1112                         }
 1113                 } else if (addr == NULL) {
 1114                         if (so->so_proto->pr_flags & PR_CONNREQUIRED)
 1115                                 error = ENOTCONN;
 1116                         else
 1117                                 error = EDESTADDRREQ;
 1118                         SOCKBUF_UNLOCK(&so->so_snd);
 1119                         goto out;
 1120                 }
 1121         }
 1122 
 1123         /*
 1124          * Do we need MSG_OOB support in SOCK_DGRAM?  Signs here may be a
 1125          * problem and need fixing.
 1126          */
 1127         space = sbspace(&so->so_snd);
 1128         if (flags & MSG_OOB)
 1129                 space += 1024;
 1130         space -= clen;
 1131         SOCKBUF_UNLOCK(&so->so_snd);
 1132         if (resid > space) {
 1133                 error = EMSGSIZE;
 1134                 goto out;
 1135         }
 1136         if (uio == NULL) {
 1137                 resid = 0;
 1138                 if (flags & MSG_EOR)
 1139                         top->m_flags |= M_EOR;
 1140         } else {
 1141                 /*
 1142                  * Copy the data from userland into a mbuf chain.
 1143                  * If no data is to be copied in, a single empty mbuf
 1144                  * is returned.
 1145                  */
 1146                 top = m_uiotombuf(uio, M_WAITOK, space, max_hdr,
 1147                     (M_PKTHDR | ((flags & MSG_EOR) ? M_EOR : 0)));
 1148                 if (top == NULL) {
 1149                         error = EFAULT; /* only possible error */
 1150                         goto out;
 1151                 }
 1152                 space -= resid - uio->uio_resid;
 1153                 resid = uio->uio_resid;
 1154         }
 1155         KASSERT(resid == 0, ("sosend_dgram: resid != 0"));
 1156         /*
 1157          * XXXRW: Frobbing SO_DONTROUTE here is even worse without sblock
 1158          * than with.
 1159          */
 1160         if (dontroute) {
 1161                 SOCK_LOCK(so);
 1162                 so->so_options |= SO_DONTROUTE;
 1163                 SOCK_UNLOCK(so);
 1164         }
 1165         /*
 1166          * XXX all the SBS_CANTSENDMORE checks previously done could be out
 1167          * of date.  We could have received a reset packet in an interrupt or
 1168          * maybe we slept while doing page faults in uiomove() etc.  We could
 1169          * probably recheck again inside the locking protection here, but
 1170          * there are probably other places that this also happens.  We must
 1171          * rethink this.
 1172          */
 1173         VNET_SO_ASSERT(so);
 1174         error = (*so->so_proto->pr_usrreqs->pru_send)(so,
 1175             (flags & MSG_OOB) ? PRUS_OOB :
 1176         /*
 1177          * If the user set MSG_EOF, the protocol understands this flag and
 1178          * nothing left to send then use PRU_SEND_EOF instead of PRU_SEND.
 1179          */
 1180             ((flags & MSG_EOF) &&
 1181              (so->so_proto->pr_flags & PR_IMPLOPCL) &&
 1182              (resid <= 0)) ?
 1183                 PRUS_EOF :
 1184                 /* If there is more to send set PRUS_MORETOCOME */
 1185                 (resid > 0 && space > 0) ? PRUS_MORETOCOME : 0,
 1186                 top, addr, control, td);
 1187         if (dontroute) {
 1188                 SOCK_LOCK(so);
 1189                 so->so_options &= ~SO_DONTROUTE;
 1190                 SOCK_UNLOCK(so);
 1191         }
 1192         clen = 0;
 1193         control = NULL;
 1194         top = NULL;
 1195 out:
 1196         if (top != NULL)
 1197                 m_freem(top);
 1198         if (control != NULL)
 1199                 m_freem(control);
 1200         return (error);
 1201 }
 1202 
 1203 /*
 1204  * Send on a socket.  If send must go all at once and message is larger than
 1205  * send buffering, then hard error.  Lock against other senders.  If must go
 1206  * all at once and not enough room now, then inform user that this would
 1207  * block and do nothing.  Otherwise, if nonblocking, send as much as
 1208  * possible.  The data to be sent is described by "uio" if nonzero, otherwise
 1209  * by the mbuf chain "top" (which must be null if uio is not).  Data provided
 1210  * in mbuf chain must be small enough to send all at once.
 1211  *
 1212  * Returns nonzero on error, timeout or signal; callers must check for short
 1213  * counts if EINTR/ERESTART are returned.  Data and control buffers are freed
 1214  * on return.
 1215  */
 1216 int
 1217 sosend_generic(struct socket *so, struct sockaddr *addr, struct uio *uio,
 1218     struct mbuf *top, struct mbuf *control, int flags, struct thread *td)
 1219 {
 1220         long space;
 1221         ssize_t resid;
 1222         int clen = 0, error, dontroute;
 1223         int atomic = sosendallatonce(so) || top;
 1224 
 1225         if (uio != NULL)
 1226                 resid = uio->uio_resid;
 1227         else
 1228                 resid = top->m_pkthdr.len;
 1229         /*
 1230          * In theory resid should be unsigned.  However, space must be
 1231          * signed, as it might be less than 0 if we over-committed, and we
 1232          * must use a signed comparison of space and resid.  On the other
 1233          * hand, a negative resid causes us to loop sending 0-length
 1234          * segments to the protocol.
 1235          *
 1236          * Also check to make sure that MSG_EOR isn't used on SOCK_STREAM
 1237          * type sockets since that's an error.
 1238          */
 1239         if (resid < 0 || (so->so_type == SOCK_STREAM && (flags & MSG_EOR))) {
 1240                 error = EINVAL;
 1241                 goto out;
 1242         }
 1243 
 1244         dontroute =
 1245             (flags & MSG_DONTROUTE) && (so->so_options & SO_DONTROUTE) == 0 &&
 1246             (so->so_proto->pr_flags & PR_ATOMIC);
 1247         if (td != NULL)
 1248                 td->td_ru.ru_msgsnd++;
 1249         if (control != NULL)
 1250                 clen = control->m_len;
 1251 
 1252         error = sblock(&so->so_snd, SBLOCKWAIT(flags));
 1253         if (error)
 1254                 goto out;
 1255 
 1256 restart:
 1257         do {
 1258                 SOCKBUF_LOCK(&so->so_snd);
 1259                 if (so->so_snd.sb_state & SBS_CANTSENDMORE) {
 1260                         SOCKBUF_UNLOCK(&so->so_snd);
 1261                         error = EPIPE;
 1262                         goto release;
 1263                 }
 1264                 if (so->so_error) {
 1265                         error = so->so_error;
 1266                         so->so_error = 0;
 1267                         SOCKBUF_UNLOCK(&so->so_snd);
 1268                         goto release;
 1269                 }
 1270                 if ((so->so_state & SS_ISCONNECTED) == 0) {
 1271                         /*
 1272                          * `sendto' and `sendmsg' is allowed on a connection-
 1273                          * based socket if it supports implied connect.
 1274                          * Return ENOTCONN if not connected and no address is
 1275                          * supplied.
 1276                          */
 1277                         if ((so->so_proto->pr_flags & PR_CONNREQUIRED) &&
 1278                             (so->so_proto->pr_flags & PR_IMPLOPCL) == 0) {
 1279                                 if ((so->so_state & SS_ISCONFIRMING) == 0 &&
 1280                                     !(resid == 0 && clen != 0)) {
 1281                                         SOCKBUF_UNLOCK(&so->so_snd);
 1282                                         error = ENOTCONN;
 1283                                         goto release;
 1284                                 }
 1285                         } else if (addr == NULL) {
 1286                                 SOCKBUF_UNLOCK(&so->so_snd);
 1287                                 if (so->so_proto->pr_flags & PR_CONNREQUIRED)
 1288                                         error = ENOTCONN;
 1289                                 else
 1290                                         error = EDESTADDRREQ;
 1291                                 goto release;
 1292                         }
 1293                 }
 1294                 space = sbspace(&so->so_snd);
 1295                 if (flags & MSG_OOB)
 1296                         space += 1024;
 1297                 if ((atomic && resid > so->so_snd.sb_hiwat) ||
 1298                     clen > so->so_snd.sb_hiwat) {
 1299                         SOCKBUF_UNLOCK(&so->so_snd);
 1300                         error = EMSGSIZE;
 1301                         goto release;
 1302                 }
 1303                 if (space < resid + clen &&
 1304                     (atomic || space < so->so_snd.sb_lowat || space < clen)) {
 1305                         if ((so->so_state & SS_NBIO) || (flags & MSG_NBIO)) {
 1306                                 SOCKBUF_UNLOCK(&so->so_snd);
 1307                                 error = EWOULDBLOCK;
 1308                                 goto release;
 1309                         }
 1310                         error = sbwait(&so->so_snd);
 1311                         SOCKBUF_UNLOCK(&so->so_snd);
 1312                         if (error)
 1313                                 goto release;
 1314                         goto restart;
 1315                 }
 1316                 SOCKBUF_UNLOCK(&so->so_snd);
 1317                 space -= clen;
 1318                 do {
 1319                         if (uio == NULL) {
 1320                                 resid = 0;
 1321                                 if (flags & MSG_EOR)
 1322                                         top->m_flags |= M_EOR;
 1323                         } else {
 1324                                 /*
 1325                                  * Copy the data from userland into a mbuf
 1326                                  * chain.  If resid is 0, which can happen
 1327                                  * only if we have control to send, then
 1328                                  * a single empty mbuf is returned.  This
 1329                                  * is a workaround to prevent protocol send
 1330                                  * methods to panic.
 1331                                  */
 1332                                 top = m_uiotombuf(uio, M_WAITOK, space,
 1333                                     (atomic ? max_hdr : 0),
 1334                                     (atomic ? M_PKTHDR : 0) |
 1335                                     ((flags & MSG_EOR) ? M_EOR : 0));
 1336                                 if (top == NULL) {
 1337                                         error = EFAULT; /* only possible error */
 1338                                         goto release;
 1339                                 }
 1340                                 space -= resid - uio->uio_resid;
 1341                                 resid = uio->uio_resid;
 1342                         }
 1343                         if (dontroute) {
 1344                                 SOCK_LOCK(so);
 1345                                 so->so_options |= SO_DONTROUTE;
 1346                                 SOCK_UNLOCK(so);
 1347                         }
 1348                         /*
 1349                          * XXX all the SBS_CANTSENDMORE checks previously
 1350                          * done could be out of date.  We could have received
 1351                          * a reset packet in an interrupt or maybe we slept
 1352                          * while doing page faults in uiomove() etc.  We
 1353                          * could probably recheck again inside the locking
 1354                          * protection here, but there are probably other
 1355                          * places that this also happens.  We must rethink
 1356                          * this.
 1357                          */
 1358                         VNET_SO_ASSERT(so);
 1359                         error = (*so->so_proto->pr_usrreqs->pru_send)(so,
 1360                             (flags & MSG_OOB) ? PRUS_OOB :
 1361                         /*
 1362                          * If the user set MSG_EOF, the protocol understands
 1363                          * this flag and nothing left to send then use
 1364                          * PRU_SEND_EOF instead of PRU_SEND.
 1365                          */
 1366                             ((flags & MSG_EOF) &&
 1367                              (so->so_proto->pr_flags & PR_IMPLOPCL) &&
 1368                              (resid <= 0)) ?
 1369                                 PRUS_EOF :
 1370                         /* If there is more to send set PRUS_MORETOCOME. */
 1371                             (resid > 0 && space > 0) ? PRUS_MORETOCOME : 0,
 1372                             top, addr, control, td);
 1373                         if (dontroute) {
 1374                                 SOCK_LOCK(so);
 1375                                 so->so_options &= ~SO_DONTROUTE;
 1376                                 SOCK_UNLOCK(so);
 1377                         }
 1378                         clen = 0;
 1379                         control = NULL;
 1380                         top = NULL;
 1381                         if (error)
 1382                                 goto release;
 1383                 } while (resid && space > 0);
 1384         } while (resid);
 1385 
 1386 release:
 1387         sbunlock(&so->so_snd);
 1388 out:
 1389         if (top != NULL)
 1390                 m_freem(top);
 1391         if (control != NULL)
 1392                 m_freem(control);
 1393         return (error);
 1394 }
 1395 
 1396 int
 1397 sosend(struct socket *so, struct sockaddr *addr, struct uio *uio,
 1398     struct mbuf *top, struct mbuf *control, int flags, struct thread *td)
 1399 {
 1400         int error;
 1401 
 1402         CURVNET_SET(so->so_vnet);
 1403         error = so->so_proto->pr_usrreqs->pru_sosend(so, addr, uio, top,
 1404             control, flags, td);
 1405         CURVNET_RESTORE();
 1406         return (error);
 1407 }
 1408 
 1409 /*
 1410  * The part of soreceive() that implements reading non-inline out-of-band
 1411  * data from a socket.  For more complete comments, see soreceive(), from
 1412  * which this code originated.
 1413  *
 1414  * Note that soreceive_rcvoob(), unlike the remainder of soreceive(), is
 1415  * unable to return an mbuf chain to the caller.
 1416  */
 1417 static int
 1418 soreceive_rcvoob(struct socket *so, struct uio *uio, int flags)
 1419 {
 1420         struct protosw *pr = so->so_proto;
 1421         struct mbuf *m;
 1422         int error;
 1423 
 1424         KASSERT(flags & MSG_OOB, ("soreceive_rcvoob: (flags & MSG_OOB) == 0"));
 1425         VNET_SO_ASSERT(so);
 1426 
 1427         m = m_get(M_WAITOK, MT_DATA);
 1428         error = (*pr->pr_usrreqs->pru_rcvoob)(so, m, flags & MSG_PEEK);
 1429         if (error)
 1430                 goto bad;
 1431         do {
 1432                 error = uiomove(mtod(m, void *),
 1433                     (int) min(uio->uio_resid, m->m_len), uio);
 1434                 m = m_free(m);
 1435         } while (uio->uio_resid && error == 0 && m);
 1436 bad:
 1437         if (m != NULL)
 1438                 m_freem(m);
 1439         return (error);
 1440 }
 1441 
 1442 /*
 1443  * Following replacement or removal of the first mbuf on the first mbuf chain
 1444  * of a socket buffer, push necessary state changes back into the socket
 1445  * buffer so that other consumers see the values consistently.  'nextrecord'
 1446  * is the callers locally stored value of the original value of
 1447  * sb->sb_mb->m_nextpkt which must be restored when the lead mbuf changes.
 1448  * NOTE: 'nextrecord' may be NULL.
 1449  */
 1450 static __inline void
 1451 sockbuf_pushsync(struct sockbuf *sb, struct mbuf *nextrecord)
 1452 {
 1453 
 1454         SOCKBUF_LOCK_ASSERT(sb);
 1455         /*
 1456          * First, update for the new value of nextrecord.  If necessary, make
 1457          * it the first record.
 1458          */
 1459         if (sb->sb_mb != NULL)
 1460                 sb->sb_mb->m_nextpkt = nextrecord;
 1461         else
 1462                 sb->sb_mb = nextrecord;
 1463 
 1464         /*
 1465          * Now update any dependent socket buffer fields to reflect the new
 1466          * state.  This is an expanded inline of SB_EMPTY_FIXUP(), with the
 1467          * addition of a second clause that takes care of the case where
 1468          * sb_mb has been updated, but remains the last record.
 1469          */
 1470         if (sb->sb_mb == NULL) {
 1471                 sb->sb_mbtail = NULL;
 1472                 sb->sb_lastrecord = NULL;
 1473         } else if (sb->sb_mb->m_nextpkt == NULL)
 1474                 sb->sb_lastrecord = sb->sb_mb;
 1475 }
 1476 
 1477 /*
 1478  * Implement receive operations on a socket.  We depend on the way that
 1479  * records are added to the sockbuf by sbappend.  In particular, each record
 1480  * (mbufs linked through m_next) must begin with an address if the protocol
 1481  * so specifies, followed by an optional mbuf or mbufs containing ancillary
 1482  * data, and then zero or more mbufs of data.  In order to allow parallelism
 1483  * between network receive and copying to user space, as well as avoid
 1484  * sleeping with a mutex held, we release the socket buffer mutex during the
 1485  * user space copy.  Although the sockbuf is locked, new data may still be
 1486  * appended, and thus we must maintain consistency of the sockbuf during that
 1487  * time.
 1488  *
 1489  * The caller may receive the data as a single mbuf chain by supplying an
 1490  * mbuf **mp0 for use in returning the chain.  The uio is then used only for
 1491  * the count in uio_resid.
 1492  */
 1493 int
 1494 soreceive_generic(struct socket *so, struct sockaddr **psa, struct uio *uio,
 1495     struct mbuf **mp0, struct mbuf **controlp, int *flagsp)
 1496 {
 1497         struct mbuf *m, **mp;
 1498         int flags, error, offset;
 1499         ssize_t len;
 1500         struct protosw *pr = so->so_proto;
 1501         struct mbuf *nextrecord;
 1502         int moff, type = 0;
 1503         ssize_t orig_resid = uio->uio_resid;
 1504 
 1505         mp = mp0;
 1506         if (psa != NULL)
 1507                 *psa = NULL;
 1508         if (controlp != NULL)
 1509                 *controlp = NULL;
 1510         if (flagsp != NULL)
 1511                 flags = *flagsp &~ MSG_EOR;
 1512         else
 1513                 flags = 0;
 1514         if (flags & MSG_OOB)
 1515                 return (soreceive_rcvoob(so, uio, flags));
 1516         if (mp != NULL)
 1517                 *mp = NULL;
 1518         if ((pr->pr_flags & PR_WANTRCVD) && (so->so_state & SS_ISCONFIRMING)
 1519             && uio->uio_resid) {
 1520                 VNET_SO_ASSERT(so);
 1521                 (*pr->pr_usrreqs->pru_rcvd)(so, 0);
 1522         }
 1523 
 1524         error = sblock(&so->so_rcv, SBLOCKWAIT(flags));
 1525         if (error)
 1526                 return (error);
 1527 
 1528 restart:
 1529         SOCKBUF_LOCK(&so->so_rcv);
 1530         m = so->so_rcv.sb_mb;
 1531         /*
 1532          * If we have less data than requested, block awaiting more (subject
 1533          * to any timeout) if:
 1534          *   1. the current count is less than the low water mark, or
 1535          *   2. MSG_DONTWAIT is not set
 1536          */
 1537         if (m == NULL || (((flags & MSG_DONTWAIT) == 0 &&
 1538             sbavail(&so->so_rcv) < uio->uio_resid) &&
 1539             sbavail(&so->so_rcv) < so->so_rcv.sb_lowat &&
 1540             m->m_nextpkt == NULL && (pr->pr_flags & PR_ATOMIC) == 0)) {
 1541                 KASSERT(m != NULL || !sbavail(&so->so_rcv),
 1542                     ("receive: m == %p sbavail == %u",
 1543                     m, sbavail(&so->so_rcv)));
 1544                 if (so->so_error) {
 1545                         if (m != NULL)
 1546                                 goto dontblock;
 1547                         error = so->so_error;
 1548                         if ((flags & MSG_PEEK) == 0)
 1549                                 so->so_error = 0;
 1550                         SOCKBUF_UNLOCK(&so->so_rcv);
 1551                         goto release;
 1552                 }
 1553                 SOCKBUF_LOCK_ASSERT(&so->so_rcv);
 1554                 if (so->so_rcv.sb_state & SBS_CANTRCVMORE) {
 1555                         if (m == NULL) {
 1556                                 SOCKBUF_UNLOCK(&so->so_rcv);
 1557                                 goto release;
 1558                         } else
 1559                                 goto dontblock;
 1560                 }
 1561                 for (; m != NULL; m = m->m_next)
 1562                         if (m->m_type == MT_OOBDATA  || (m->m_flags & M_EOR)) {
 1563                                 m = so->so_rcv.sb_mb;
 1564                                 goto dontblock;
 1565                         }
 1566                 if ((so->so_state & (SS_ISCONNECTED|SS_ISCONNECTING)) == 0 &&
 1567                     (so->so_proto->pr_flags & PR_CONNREQUIRED)) {
 1568                         SOCKBUF_UNLOCK(&so->so_rcv);
 1569                         error = ENOTCONN;
 1570                         goto release;
 1571                 }
 1572                 if (uio->uio_resid == 0) {
 1573                         SOCKBUF_UNLOCK(&so->so_rcv);
 1574                         goto release;
 1575                 }
 1576                 if ((so->so_state & SS_NBIO) ||
 1577                     (flags & (MSG_DONTWAIT|MSG_NBIO))) {
 1578                         SOCKBUF_UNLOCK(&so->so_rcv);
 1579                         error = EWOULDBLOCK;
 1580                         goto release;
 1581                 }
 1582                 SBLASTRECORDCHK(&so->so_rcv);
 1583                 SBLASTMBUFCHK(&so->so_rcv);
 1584                 error = sbwait(&so->so_rcv);
 1585                 SOCKBUF_UNLOCK(&so->so_rcv);
 1586                 if (error)
 1587                         goto release;
 1588                 goto restart;
 1589         }
 1590 dontblock:
 1591         /*
 1592          * From this point onward, we maintain 'nextrecord' as a cache of the
 1593          * pointer to the next record in the socket buffer.  We must keep the
 1594          * various socket buffer pointers and local stack versions of the
 1595          * pointers in sync, pushing out modifications before dropping the
 1596          * socket buffer mutex, and re-reading them when picking it up.
 1597          *
 1598          * Otherwise, we will race with the network stack appending new data
 1599          * or records onto the socket buffer by using inconsistent/stale
 1600          * versions of the field, possibly resulting in socket buffer
 1601          * corruption.
 1602          *
 1603          * By holding the high-level sblock(), we prevent simultaneous
 1604          * readers from pulling off the front of the socket buffer.
 1605          */
 1606         SOCKBUF_LOCK_ASSERT(&so->so_rcv);
 1607         if (uio->uio_td)
 1608                 uio->uio_td->td_ru.ru_msgrcv++;
 1609         KASSERT(m == so->so_rcv.sb_mb, ("soreceive: m != so->so_rcv.sb_mb"));
 1610         SBLASTRECORDCHK(&so->so_rcv);
 1611         SBLASTMBUFCHK(&so->so_rcv);
 1612         nextrecord = m->m_nextpkt;
 1613         if (pr->pr_flags & PR_ADDR) {
 1614                 KASSERT(m->m_type == MT_SONAME,
 1615                     ("m->m_type == %d", m->m_type));
 1616                 orig_resid = 0;
 1617                 if (psa != NULL)
 1618                         *psa = sodupsockaddr(mtod(m, struct sockaddr *),
 1619                             M_NOWAIT);
 1620                 if (flags & MSG_PEEK) {
 1621                         m = m->m_next;
 1622                 } else {
 1623                         sbfree(&so->so_rcv, m);
 1624                         so->so_rcv.sb_mb = m_free(m);
 1625                         m = so->so_rcv.sb_mb;
 1626                         sockbuf_pushsync(&so->so_rcv, nextrecord);
 1627                 }
 1628         }
 1629 
 1630         /*
 1631          * Process one or more MT_CONTROL mbufs present before any data mbufs
 1632          * in the first mbuf chain on the socket buffer.  If MSG_PEEK, we
 1633          * just copy the data; if !MSG_PEEK, we call into the protocol to
 1634          * perform externalization (or freeing if controlp == NULL).
 1635          */
 1636         if (m != NULL && m->m_type == MT_CONTROL) {
 1637                 struct mbuf *cm = NULL, *cmn;
 1638                 struct mbuf **cme = &cm;
 1639 
 1640                 do {
 1641                         if (flags & MSG_PEEK) {
 1642                                 if (controlp != NULL) {
 1643                                         *controlp = m_copy(m, 0, m->m_len);
 1644                                         controlp = &(*controlp)->m_next;
 1645                                 }
 1646                                 m = m->m_next;
 1647                         } else {
 1648                                 sbfree(&so->so_rcv, m);
 1649                                 so->so_rcv.sb_mb = m->m_next;
 1650                                 m->m_next = NULL;
 1651                                 *cme = m;
 1652                                 cme = &(*cme)->m_next;
 1653                                 m = so->so_rcv.sb_mb;
 1654                         }
 1655                 } while (m != NULL && m->m_type == MT_CONTROL);
 1656                 if ((flags & MSG_PEEK) == 0)
 1657                         sockbuf_pushsync(&so->so_rcv, nextrecord);
 1658                 while (cm != NULL) {
 1659                         cmn = cm->m_next;
 1660                         cm->m_next = NULL;
 1661                         if (pr->pr_domain->dom_externalize != NULL) {
 1662                                 SOCKBUF_UNLOCK(&so->so_rcv);
 1663                                 VNET_SO_ASSERT(so);
 1664                                 error = (*pr->pr_domain->dom_externalize)
 1665                                     (cm, controlp, flags);
 1666                                 SOCKBUF_LOCK(&so->so_rcv);
 1667                         } else if (controlp != NULL)
 1668                                 *controlp = cm;
 1669                         else
 1670                                 m_freem(cm);
 1671                         if (controlp != NULL) {
 1672                                 orig_resid = 0;
 1673                                 while (*controlp != NULL)
 1674                                         controlp = &(*controlp)->m_next;
 1675                         }
 1676                         cm = cmn;
 1677                 }
 1678                 if (m != NULL)
 1679                         nextrecord = so->so_rcv.sb_mb->m_nextpkt;
 1680                 else
 1681                         nextrecord = so->so_rcv.sb_mb;
 1682                 orig_resid = 0;
 1683         }
 1684         if (m != NULL) {
 1685                 if ((flags & MSG_PEEK) == 0) {
 1686                         KASSERT(m->m_nextpkt == nextrecord,
 1687                             ("soreceive: post-control, nextrecord !sync"));
 1688                         if (nextrecord == NULL) {
 1689                                 KASSERT(so->so_rcv.sb_mb == m,
 1690                                     ("soreceive: post-control, sb_mb!=m"));
 1691                                 KASSERT(so->so_rcv.sb_lastrecord == m,
 1692                                     ("soreceive: post-control, lastrecord!=m"));
 1693                         }
 1694                 }
 1695                 type = m->m_type;
 1696                 if (type == MT_OOBDATA)
 1697                         flags |= MSG_OOB;
 1698         } else {
 1699                 if ((flags & MSG_PEEK) == 0) {
 1700                         KASSERT(so->so_rcv.sb_mb == nextrecord,
 1701                             ("soreceive: sb_mb != nextrecord"));
 1702                         if (so->so_rcv.sb_mb == NULL) {
 1703                                 KASSERT(so->so_rcv.sb_lastrecord == NULL,
 1704                                     ("soreceive: sb_lastercord != NULL"));
 1705                         }
 1706                 }
 1707         }
 1708         SOCKBUF_LOCK_ASSERT(&so->so_rcv);
 1709         SBLASTRECORDCHK(&so->so_rcv);
 1710         SBLASTMBUFCHK(&so->so_rcv);
 1711 
 1712         /*
 1713          * Now continue to read any data mbufs off of the head of the socket
 1714          * buffer until the read request is satisfied.  Note that 'type' is
 1715          * used to store the type of any mbuf reads that have happened so far
 1716          * such that soreceive() can stop reading if the type changes, which
 1717          * causes soreceive() to return only one of regular data and inline
 1718          * out-of-band data in a single socket receive operation.
 1719          */
 1720         moff = 0;
 1721         offset = 0;
 1722         while (m != NULL && !(m->m_flags & M_NOTAVAIL) && uio->uio_resid > 0
 1723             && error == 0) {
 1724                 /*
 1725                  * If the type of mbuf has changed since the last mbuf
 1726                  * examined ('type'), end the receive operation.
 1727                  */
 1728                 SOCKBUF_LOCK_ASSERT(&so->so_rcv);
 1729                 if (m->m_type == MT_OOBDATA || m->m_type == MT_CONTROL) {
 1730                         if (type != m->m_type)
 1731                                 break;
 1732                 } else if (type == MT_OOBDATA)
 1733                         break;
 1734                 else
 1735                     KASSERT(m->m_type == MT_DATA,
 1736                         ("m->m_type == %d", m->m_type));
 1737                 so->so_rcv.sb_state &= ~SBS_RCVATMARK;
 1738                 len = uio->uio_resid;
 1739                 if (so->so_oobmark && len > so->so_oobmark - offset)
 1740                         len = so->so_oobmark - offset;
 1741                 if (len > m->m_len - moff)
 1742                         len = m->m_len - moff;
 1743                 /*
 1744                  * If mp is set, just pass back the mbufs.  Otherwise copy
 1745                  * them out via the uio, then free.  Sockbuf must be
 1746                  * consistent here (points to current mbuf, it points to next
 1747                  * record) when we drop priority; we must note any additions
 1748                  * to the sockbuf when we block interrupts again.
 1749                  */
 1750                 if (mp == NULL) {
 1751                         SOCKBUF_LOCK_ASSERT(&so->so_rcv);
 1752                         SBLASTRECORDCHK(&so->so_rcv);
 1753                         SBLASTMBUFCHK(&so->so_rcv);
 1754                         SOCKBUF_UNLOCK(&so->so_rcv);
 1755                         error = uiomove(mtod(m, char *) + moff, (int)len, uio);
 1756                         SOCKBUF_LOCK(&so->so_rcv);
 1757                         if (error) {
 1758                                 /*
 1759                                  * The MT_SONAME mbuf has already been removed
 1760                                  * from the record, so it is necessary to
 1761                                  * remove the data mbufs, if any, to preserve
 1762                                  * the invariant in the case of PR_ADDR that
 1763                                  * requires MT_SONAME mbufs at the head of
 1764                                  * each record.
 1765                                  */
 1766                                 if (m && pr->pr_flags & PR_ATOMIC &&
 1767                                     ((flags & MSG_PEEK) == 0))
 1768                                         (void)sbdroprecord_locked(&so->so_rcv);
 1769                                 SOCKBUF_UNLOCK(&so->so_rcv);
 1770                                 goto release;
 1771                         }
 1772                 } else
 1773                         uio->uio_resid -= len;
 1774                 SOCKBUF_LOCK_ASSERT(&so->so_rcv);
 1775                 if (len == m->m_len - moff) {
 1776                         if (m->m_flags & M_EOR)
 1777                                 flags |= MSG_EOR;
 1778                         if (flags & MSG_PEEK) {
 1779                                 m = m->m_next;
 1780                                 moff = 0;
 1781                         } else {
 1782                                 nextrecord = m->m_nextpkt;
 1783                                 sbfree(&so->so_rcv, m);
 1784                                 if (mp != NULL) {
 1785                                         m->m_nextpkt = NULL;
 1786                                         *mp = m;
 1787                                         mp = &m->m_next;
 1788                                         so->so_rcv.sb_mb = m = m->m_next;
 1789                                         *mp = NULL;
 1790                                 } else {
 1791                                         so->so_rcv.sb_mb = m_free(m);
 1792                                         m = so->so_rcv.sb_mb;
 1793                                 }
 1794                                 sockbuf_pushsync(&so->so_rcv, nextrecord);
 1795                                 SBLASTRECORDCHK(&so->so_rcv);
 1796                                 SBLASTMBUFCHK(&so->so_rcv);
 1797                         }
 1798                 } else {
 1799                         if (flags & MSG_PEEK)
 1800                                 moff += len;
 1801                         else {
 1802                                 if (mp != NULL) {
 1803                                         if (flags & MSG_DONTWAIT) {
 1804                                                 *mp = m_copym(m, 0, len,
 1805                                                     M_NOWAIT);
 1806                                                 if (*mp == NULL) {
 1807                                                         /*
 1808                                                          * m_copym() couldn't
 1809                                                          * allocate an mbuf.
 1810                                                          * Adjust uio_resid back
 1811                                                          * (it was adjusted
 1812                                                          * down by len bytes,
 1813                                                          * which we didn't end
 1814                                                          * up "copying" over).
 1815                                                          */
 1816                                                         uio->uio_resid += len;
 1817                                                         break;
 1818                                                 }
 1819                                         } else {
 1820                                                 SOCKBUF_UNLOCK(&so->so_rcv);
 1821                                                 *mp = m_copym(m, 0, len,
 1822                                                     M_WAITOK);
 1823                                                 SOCKBUF_LOCK(&so->so_rcv);
 1824                                         }
 1825                                 }
 1826                                 sbcut_locked(&so->so_rcv, len);
 1827                         }
 1828                 }
 1829                 SOCKBUF_LOCK_ASSERT(&so->so_rcv);
 1830                 if (so->so_oobmark) {
 1831                         if ((flags & MSG_PEEK) == 0) {
 1832                                 so->so_oobmark -= len;
 1833                                 if (so->so_oobmark == 0) {
 1834                                         so->so_rcv.sb_state |= SBS_RCVATMARK;
 1835                                         break;
 1836                                 }
 1837                         } else {
 1838                                 offset += len;
 1839                                 if (offset == so->so_oobmark)
 1840                                         break;
 1841                         }
 1842                 }
 1843                 if (flags & MSG_EOR)
 1844                         break;
 1845                 /*
 1846                  * If the MSG_WAITALL flag is set (for non-atomic socket), we
 1847                  * must not quit until "uio->uio_resid == 0" or an error
 1848                  * termination.  If a signal/timeout occurs, return with a
 1849                  * short count but without error.  Keep sockbuf locked
 1850                  * against other readers.
 1851                  */
 1852                 while (flags & MSG_WAITALL && m == NULL && uio->uio_resid > 0 &&
 1853                     !sosendallatonce(so) && nextrecord == NULL) {
 1854                         SOCKBUF_LOCK_ASSERT(&so->so_rcv);
 1855                         if (so->so_error ||
 1856                             so->so_rcv.sb_state & SBS_CANTRCVMORE)
 1857                                 break;
 1858                         /*
 1859                          * Notify the protocol that some data has been
 1860                          * drained before blocking.
 1861                          */
 1862                         if (pr->pr_flags & PR_WANTRCVD) {
 1863                                 SOCKBUF_UNLOCK(&so->so_rcv);
 1864                                 VNET_SO_ASSERT(so);
 1865                                 (*pr->pr_usrreqs->pru_rcvd)(so, flags);
 1866                                 SOCKBUF_LOCK(&so->so_rcv);
 1867                         }
 1868                         SBLASTRECORDCHK(&so->so_rcv);
 1869                         SBLASTMBUFCHK(&so->so_rcv);
 1870                         /*
 1871                          * We could receive some data while was notifying
 1872                          * the protocol. Skip blocking in this case.
 1873                          */
 1874                         if (so->so_rcv.sb_mb == NULL) {
 1875                                 error = sbwait(&so->so_rcv);
 1876                                 if (error) {
 1877                                         SOCKBUF_UNLOCK(&so->so_rcv);
 1878                                         goto release;
 1879                                 }
 1880                         }
 1881                         m = so->so_rcv.sb_mb;
 1882                         if (m != NULL)
 1883                                 nextrecord = m->m_nextpkt;
 1884                 }
 1885         }
 1886 
 1887         SOCKBUF_LOCK_ASSERT(&so->so_rcv);
 1888         if (m != NULL && pr->pr_flags & PR_ATOMIC) {
 1889                 flags |= MSG_TRUNC;
 1890                 if ((flags & MSG_PEEK) == 0)
 1891                         (void) sbdroprecord_locked(&so->so_rcv);
 1892         }
 1893         if ((flags & MSG_PEEK) == 0) {
 1894                 if (m == NULL) {
 1895                         /*
 1896                          * First part is an inline SB_EMPTY_FIXUP().  Second
 1897                          * part makes sure sb_lastrecord is up-to-date if
 1898                          * there is still data in the socket buffer.
 1899                          */
 1900                         so->so_rcv.sb_mb = nextrecord;
 1901                         if (so->so_rcv.sb_mb == NULL) {
 1902                                 so->so_rcv.sb_mbtail = NULL;
 1903                                 so->so_rcv.sb_lastrecord = NULL;
 1904                         } else if (nextrecord->m_nextpkt == NULL)
 1905                                 so->so_rcv.sb_lastrecord = nextrecord;
 1906                 }
 1907                 SBLASTRECORDCHK(&so->so_rcv);
 1908                 SBLASTMBUFCHK(&so->so_rcv);
 1909                 /*
 1910                  * If soreceive() is being done from the socket callback,
 1911                  * then don't need to generate ACK to peer to update window,
 1912                  * since ACK will be generated on return to TCP.
 1913                  */
 1914                 if (!(flags & MSG_SOCALLBCK) &&
 1915                     (pr->pr_flags & PR_WANTRCVD)) {
 1916                         SOCKBUF_UNLOCK(&so->so_rcv);
 1917                         VNET_SO_ASSERT(so);
 1918                         (*pr->pr_usrreqs->pru_rcvd)(so, flags);
 1919                         SOCKBUF_LOCK(&so->so_rcv);
 1920                 }
 1921         }
 1922         SOCKBUF_LOCK_ASSERT(&so->so_rcv);
 1923         if (orig_resid == uio->uio_resid && orig_resid &&
 1924             (flags & MSG_EOR) == 0 && (so->so_rcv.sb_state & SBS_CANTRCVMORE) == 0) {
 1925                 SOCKBUF_UNLOCK(&so->so_rcv);
 1926                 goto restart;
 1927         }
 1928         SOCKBUF_UNLOCK(&so->so_rcv);
 1929 
 1930         if (flagsp != NULL)
 1931                 *flagsp |= flags;
 1932 release:
 1933         sbunlock(&so->so_rcv);
 1934         return (error);
 1935 }
 1936 
 1937 /*
 1938  * Optimized version of soreceive() for stream (TCP) sockets.
 1939  * XXXAO: (MSG_WAITALL | MSG_PEEK) isn't properly handled.
 1940  */
 1941 int
 1942 soreceive_stream(struct socket *so, struct sockaddr **psa, struct uio *uio,
 1943     struct mbuf **mp0, struct mbuf **controlp, int *flagsp)
 1944 {
 1945         int len = 0, error = 0, flags, oresid;
 1946         struct sockbuf *sb;
 1947         struct mbuf *m, *n = NULL;
 1948 
 1949         /* We only do stream sockets. */
 1950         if (so->so_type != SOCK_STREAM)
 1951                 return (EINVAL);
 1952         if (psa != NULL)
 1953                 *psa = NULL;
 1954         if (controlp != NULL)
 1955                 return (EINVAL);
 1956         if (flagsp != NULL)
 1957                 flags = *flagsp &~ MSG_EOR;
 1958         else
 1959                 flags = 0;
 1960         if (flags & MSG_OOB)
 1961                 return (soreceive_rcvoob(so, uio, flags));
 1962         if (mp0 != NULL)
 1963                 *mp0 = NULL;
 1964 
 1965         sb = &so->so_rcv;
 1966 
 1967         /* Prevent other readers from entering the socket. */
 1968         error = sblock(sb, SBLOCKWAIT(flags));
 1969         if (error)
 1970                 goto out;
 1971         SOCKBUF_LOCK(sb);
 1972 
 1973         /* Easy one, no space to copyout anything. */
 1974         if (uio->uio_resid == 0) {
 1975                 error = EINVAL;
 1976                 goto out;
 1977         }
 1978         oresid = uio->uio_resid;
 1979 
 1980         /* We will never ever get anything unless we are or were connected. */
 1981         if (!(so->so_state & (SS_ISCONNECTED|SS_ISDISCONNECTED))) {
 1982                 error = ENOTCONN;
 1983                 goto out;
 1984         }
 1985 
 1986 restart:
 1987         SOCKBUF_LOCK_ASSERT(&so->so_rcv);
 1988 
 1989         /* Abort if socket has reported problems. */
 1990         if (so->so_error) {
 1991                 if (sbavail(sb) > 0)
 1992                         goto deliver;
 1993                 if (oresid > uio->uio_resid)
 1994                         goto out;
 1995                 error = so->so_error;
 1996                 if (!(flags & MSG_PEEK))
 1997                         so->so_error = 0;
 1998                 goto out;
 1999         }
 2000 
 2001         /* Door is closed.  Deliver what is left, if any. */
 2002         if (sb->sb_state & SBS_CANTRCVMORE) {
 2003                 if (sbavail(sb) > 0)
 2004                         goto deliver;
 2005                 else
 2006                         goto out;
 2007         }
 2008 
 2009         /* Socket buffer is empty and we shall not block. */
 2010         if (sbavail(sb) == 0 &&
 2011             ((so->so_state & SS_NBIO) || (flags & (MSG_DONTWAIT|MSG_NBIO)))) {
 2012                 error = EAGAIN;
 2013                 goto out;
 2014         }
 2015 
 2016         /* Socket buffer got some data that we shall deliver now. */
 2017         if (sbavail(sb) > 0 && !(flags & MSG_WAITALL) &&
 2018             ((so->so_state & SS_NBIO) ||
 2019              (flags & (MSG_DONTWAIT|MSG_NBIO)) ||
 2020              sbavail(sb) >= sb->sb_lowat ||
 2021              sbavail(sb) >= uio->uio_resid ||
 2022              sbavail(sb) >= sb->sb_hiwat) ) {
 2023                 goto deliver;
 2024         }
 2025 
 2026         /* On MSG_WAITALL we must wait until all data or error arrives. */
 2027         if ((flags & MSG_WAITALL) &&
 2028             (sbavail(sb) >= uio->uio_resid || sbavail(sb) >= sb->sb_hiwat))
 2029                 goto deliver;
 2030 
 2031         /*
 2032          * Wait and block until (more) data comes in.
 2033          * NB: Drops the sockbuf lock during wait.
 2034          */
 2035         error = sbwait(sb);
 2036         if (error)
 2037                 goto out;
 2038         goto restart;
 2039 
 2040 deliver:
 2041         SOCKBUF_LOCK_ASSERT(&so->so_rcv);
 2042         KASSERT(sbavail(sb) > 0, ("%s: sockbuf empty", __func__));
 2043         KASSERT(sb->sb_mb != NULL, ("%s: sb_mb == NULL", __func__));
 2044 
 2045         /* Statistics. */
 2046         if (uio->uio_td)
 2047                 uio->uio_td->td_ru.ru_msgrcv++;
 2048 
 2049         /* Fill uio until full or current end of socket buffer is reached. */
 2050         len = min(uio->uio_resid, sbavail(sb));
 2051         if (mp0 != NULL) {
 2052                 /* Dequeue as many mbufs as possible. */
 2053                 if (!(flags & MSG_PEEK) && len >= sb->sb_mb->m_len) {
 2054                         if (*mp0 == NULL)
 2055                                 *mp0 = sb->sb_mb;
 2056                         else
 2057                                 m_cat(*mp0, sb->sb_mb);
 2058                         for (m = sb->sb_mb;
 2059                              m != NULL && m->m_len <= len;
 2060                              m = m->m_next) {
 2061                                 KASSERT(!(m->m_flags & M_NOTAVAIL),
 2062                                     ("%s: m %p not available", __func__, m));
 2063                                 len -= m->m_len;
 2064                                 uio->uio_resid -= m->m_len;
 2065                                 sbfree(sb, m);
 2066                                 n = m;
 2067                         }
 2068                         n->m_next = NULL;
 2069                         sb->sb_mb = m;
 2070                         sb->sb_lastrecord = sb->sb_mb;
 2071                         if (sb->sb_mb == NULL)
 2072                                 SB_EMPTY_FIXUP(sb);
 2073                 }
 2074                 /* Copy the remainder. */
 2075                 if (len > 0) {
 2076                         KASSERT(sb->sb_mb != NULL,
 2077                             ("%s: len > 0 && sb->sb_mb empty", __func__));
 2078 
 2079                         m = m_copym(sb->sb_mb, 0, len, M_NOWAIT);
 2080                         if (m == NULL)
 2081                                 len = 0;        /* Don't flush data from sockbuf. */
 2082                         else
 2083                                 uio->uio_resid -= len;
 2084                         if (*mp0 != NULL)
 2085                                 m_cat(*mp0, m);
 2086                         else
 2087                                 *mp0 = m;
 2088                         if (*mp0 == NULL) {
 2089                                 error = ENOBUFS;
 2090                                 goto out;
 2091                         }
 2092                 }
 2093         } else {
 2094                 /* NB: Must unlock socket buffer as uiomove may sleep. */
 2095                 SOCKBUF_UNLOCK(sb);
 2096                 error = m_mbuftouio(uio, sb->sb_mb, len);
 2097                 SOCKBUF_LOCK(sb);
 2098                 if (error)
 2099                         goto out;
 2100         }
 2101         SBLASTRECORDCHK(sb);
 2102         SBLASTMBUFCHK(sb);
 2103 
 2104         /*
 2105          * Remove the delivered data from the socket buffer unless we
 2106          * were only peeking.
 2107          */
 2108         if (!(flags & MSG_PEEK)) {
 2109                 if (len > 0)
 2110                         sbdrop_locked(sb, len);
 2111 
 2112                 /* Notify protocol that we drained some data. */
 2113                 if ((so->so_proto->pr_flags & PR_WANTRCVD) &&
 2114                     (((flags & MSG_WAITALL) && uio->uio_resid > 0) ||
 2115                      !(flags & MSG_SOCALLBCK))) {
 2116                         SOCKBUF_UNLOCK(sb);
 2117                         VNET_SO_ASSERT(so);
 2118                         (*so->so_proto->pr_usrreqs->pru_rcvd)(so, flags);
 2119                         SOCKBUF_LOCK(sb);
 2120                 }
 2121         }
 2122 
 2123         /*
 2124          * For MSG_WAITALL we may have to loop again and wait for
 2125          * more data to come in.
 2126          */
 2127         if ((flags & MSG_WAITALL) && uio->uio_resid > 0)
 2128                 goto restart;
 2129 out:
 2130         SOCKBUF_LOCK_ASSERT(sb);
 2131         SBLASTRECORDCHK(sb);
 2132         SBLASTMBUFCHK(sb);
 2133         SOCKBUF_UNLOCK(sb);
 2134         sbunlock(sb);
 2135         return (error);
 2136 }
 2137 
 2138 /*
 2139  * Optimized version of soreceive() for simple datagram cases from userspace.
 2140  * Unlike in the stream case, we're able to drop a datagram if copyout()
 2141  * fails, and because we handle datagrams atomically, we don't need to use a
 2142  * sleep lock to prevent I/O interlacing.
 2143  */
 2144 int
 2145 soreceive_dgram(struct socket *so, struct sockaddr **psa, struct uio *uio,
 2146     struct mbuf **mp0, struct mbuf **controlp, int *flagsp)
 2147 {
 2148         struct mbuf *m, *m2;
 2149         int flags, error;
 2150         ssize_t len;
 2151         struct protosw *pr = so->so_proto;
 2152         struct mbuf *nextrecord;
 2153 
 2154         if (psa != NULL)
 2155                 *psa = NULL;
 2156         if (controlp != NULL)
 2157                 *controlp = NULL;
 2158         if (flagsp != NULL)
 2159                 flags = *flagsp &~ MSG_EOR;
 2160         else
 2161                 flags = 0;
 2162 
 2163         /*
 2164          * For any complicated cases, fall back to the full
 2165          * soreceive_generic().
 2166          */
 2167         if (mp0 != NULL || (flags & MSG_PEEK) || (flags & MSG_OOB))
 2168                 return (soreceive_generic(so, psa, uio, mp0, controlp,
 2169                     flagsp));
 2170 
 2171         /*
 2172          * Enforce restrictions on use.
 2173          */
 2174         KASSERT((pr->pr_flags & PR_WANTRCVD) == 0,
 2175             ("soreceive_dgram: wantrcvd"));
 2176         KASSERT(pr->pr_flags & PR_ATOMIC, ("soreceive_dgram: !atomic"));
 2177         KASSERT((so->so_rcv.sb_state & SBS_RCVATMARK) == 0,
 2178             ("soreceive_dgram: SBS_RCVATMARK"));
 2179         KASSERT((so->so_proto->pr_flags & PR_CONNREQUIRED) == 0,
 2180             ("soreceive_dgram: P_CONNREQUIRED"));
 2181 
 2182         /*
 2183          * Loop blocking while waiting for a datagram.
 2184          */
 2185         SOCKBUF_LOCK(&so->so_rcv);
 2186         while ((m = so->so_rcv.sb_mb) == NULL) {
 2187                 KASSERT(sbavail(&so->so_rcv) == 0,
 2188                     ("soreceive_dgram: sb_mb NULL but sbavail %u",
 2189                     sbavail(&so->so_rcv)));
 2190                 if (so->so_error) {
 2191                         error = so->so_error;
 2192                         so->so_error = 0;
 2193                         SOCKBUF_UNLOCK(&so->so_rcv);
 2194                         return (error);
 2195                 }
 2196                 if (so->so_rcv.sb_state & SBS_CANTRCVMORE ||
 2197                     uio->uio_resid == 0) {
 2198                         SOCKBUF_UNLOCK(&so->so_rcv);
 2199                         return (0);
 2200                 }
 2201                 if ((so->so_state & SS_NBIO) ||
 2202                     (flags & (MSG_DONTWAIT|MSG_NBIO))) {
 2203                         SOCKBUF_UNLOCK(&so->so_rcv);
 2204                         return (EWOULDBLOCK);
 2205                 }
 2206                 SBLASTRECORDCHK(&so->so_rcv);
 2207                 SBLASTMBUFCHK(&so->so_rcv);
 2208                 error = sbwait(&so->so_rcv);
 2209                 if (error) {
 2210                         SOCKBUF_UNLOCK(&so->so_rcv);
 2211                         return (error);
 2212                 }
 2213         }
 2214         SOCKBUF_LOCK_ASSERT(&so->so_rcv);
 2215 
 2216         if (uio->uio_td)
 2217                 uio->uio_td->td_ru.ru_msgrcv++;
 2218         SBLASTRECORDCHK(&so->so_rcv);
 2219         SBLASTMBUFCHK(&so->so_rcv);
 2220         nextrecord = m->m_nextpkt;
 2221         if (nextrecord == NULL) {
 2222                 KASSERT(so->so_rcv.sb_lastrecord == m,
 2223                     ("soreceive_dgram: lastrecord != m"));
 2224         }
 2225 
 2226         KASSERT(so->so_rcv.sb_mb->m_nextpkt == nextrecord,
 2227             ("soreceive_dgram: m_nextpkt != nextrecord"));
 2228 
 2229         /*
 2230          * Pull 'm' and its chain off the front of the packet queue.
 2231          */
 2232         so->so_rcv.sb_mb = NULL;
 2233         sockbuf_pushsync(&so->so_rcv, nextrecord);
 2234 
 2235         /*
 2236          * Walk 'm's chain and free that many bytes from the socket buffer.
 2237          */
 2238         for (m2 = m; m2 != NULL; m2 = m2->m_next)
 2239                 sbfree(&so->so_rcv, m2);
 2240 
 2241         /*
 2242          * Do a few last checks before we let go of the lock.
 2243          */
 2244         SBLASTRECORDCHK(&so->so_rcv);
 2245         SBLASTMBUFCHK(&so->so_rcv);
 2246         SOCKBUF_UNLOCK(&so->so_rcv);
 2247 
 2248         if (pr->pr_flags & PR_ADDR) {
 2249                 KASSERT(m->m_type == MT_SONAME,
 2250                     ("m->m_type == %d", m->m_type));
 2251                 if (psa != NULL)
 2252                         *psa = sodupsockaddr(mtod(m, struct sockaddr *),
 2253                             M_NOWAIT);
 2254                 m = m_free(m);
 2255         }
 2256         if (m == NULL) {
 2257                 /* XXXRW: Can this happen? */
 2258                 return (0);
 2259         }
 2260 
 2261         /*
 2262          * Packet to copyout() is now in 'm' and it is disconnected from the
 2263          * queue.
 2264          *
 2265          * Process one or more MT_CONTROL mbufs present before any data mbufs
 2266          * in the first mbuf chain on the socket buffer.  We call into the
 2267          * protocol to perform externalization (or freeing if controlp ==
 2268          * NULL). In some cases there can be only MT_CONTROL mbufs without
 2269          * MT_DATA mbufs.
 2270          */
 2271         if (m->m_type == MT_CONTROL) {
 2272                 struct mbuf *cm = NULL, *cmn;
 2273                 struct mbuf **cme = &cm;
 2274 
 2275                 do {
 2276                         m2 = m->m_next;
 2277                         m->m_next = NULL;
 2278                         *cme = m;
 2279                         cme = &(*cme)->m_next;
 2280                         m = m2;
 2281                 } while (m != NULL && m->m_type == MT_CONTROL);
 2282                 while (cm != NULL) {
 2283                         cmn = cm->m_next;
 2284                         cm->m_next = NULL;
 2285                         if (pr->pr_domain->dom_externalize != NULL) {
 2286                                 error = (*pr->pr_domain->dom_externalize)
 2287                                     (cm, controlp, flags);
 2288                         } else if (controlp != NULL)
 2289                                 *controlp = cm;
 2290                         else
 2291                                 m_freem(cm);
 2292                         if (controlp != NULL) {
 2293                                 while (*controlp != NULL)
 2294                                         controlp = &(*controlp)->m_next;
 2295                         }
 2296                         cm = cmn;
 2297                 }
 2298         }
 2299         KASSERT(m == NULL || m->m_type == MT_DATA,
 2300             ("soreceive_dgram: !data"));
 2301         while (m != NULL && uio->uio_resid > 0) {
 2302                 len = uio->uio_resid;
 2303                 if (len > m->m_len)
 2304                         len = m->m_len;
 2305                 error = uiomove(mtod(m, char *), (int)len, uio);
 2306                 if (error) {
 2307                         m_freem(m);
 2308                         return (error);
 2309                 }
 2310                 if (len == m->m_len)
 2311                         m = m_free(m);
 2312                 else {
 2313                         m->m_data += len;
 2314                         m->m_len -= len;
 2315                 }
 2316         }
 2317         if (m != NULL) {
 2318                 flags |= MSG_TRUNC;
 2319                 m_freem(m);
 2320         }
 2321         if (flagsp != NULL)
 2322                 *flagsp |= flags;
 2323         return (0);
 2324 }
 2325 
 2326 int
 2327 soreceive(struct socket *so, struct sockaddr **psa, struct uio *uio,
 2328     struct mbuf **mp0, struct mbuf **controlp, int *flagsp)
 2329 {
 2330         int error;
 2331 
 2332         CURVNET_SET(so->so_vnet);
 2333         error = (so->so_proto->pr_usrreqs->pru_soreceive(so, psa, uio, mp0,
 2334             controlp, flagsp));
 2335         CURVNET_RESTORE();
 2336         return (error);
 2337 }
 2338 
 2339 int
 2340 soshutdown(struct socket *so, int how)
 2341 {
 2342         struct protosw *pr = so->so_proto;
 2343         int error;
 2344 
 2345         if (!(how == SHUT_RD || how == SHUT_WR || how == SHUT_RDWR))
 2346                 return (EINVAL);
 2347         if ((so->so_state &
 2348             (SS_ISCONNECTED | SS_ISCONNECTING | SS_ISDISCONNECTING)) == 0)
 2349                 return (ENOTCONN);
 2350 
 2351         CURVNET_SET(so->so_vnet);
 2352         if (pr->pr_usrreqs->pru_flush != NULL)
 2353                 (*pr->pr_usrreqs->pru_flush)(so, how);
 2354         if (how != SHUT_WR)
 2355                 sorflush(so);
 2356         if (how != SHUT_RD) {
 2357                 error = (*pr->pr_usrreqs->pru_shutdown)(so);
 2358                 wakeup(&so->so_timeo);
 2359                 CURVNET_RESTORE();
 2360                 return (error);
 2361         }
 2362         wakeup(&so->so_timeo);
 2363         CURVNET_RESTORE();
 2364         return (0);
 2365 }
 2366 
 2367 void
 2368 sorflush(struct socket *so)
 2369 {
 2370         struct sockbuf *sb = &so->so_rcv;
 2371         struct protosw *pr = so->so_proto;
 2372         struct socket aso;
 2373 
 2374         VNET_SO_ASSERT(so);
 2375 
 2376         /*
 2377          * In order to avoid calling dom_dispose with the socket buffer mutex
 2378          * held, and in order to generally avoid holding the lock for a long
 2379          * time, we make a copy of the socket buffer and clear the original
 2380          * (except locks, state).  The new socket buffer copy won't have
 2381          * initialized locks so we can only call routines that won't use or
 2382          * assert those locks.
 2383          *
 2384          * Dislodge threads currently blocked in receive and wait to acquire
 2385          * a lock against other simultaneous readers before clearing the
 2386          * socket buffer.  Don't let our acquire be interrupted by a signal
 2387          * despite any existing socket disposition on interruptable waiting.
 2388          */
 2389         socantrcvmore(so);
 2390         (void) sblock(sb, SBL_WAIT | SBL_NOINTR);
 2391 
 2392         /*
 2393          * Invalidate/clear most of the sockbuf structure, but leave selinfo
 2394          * and mutex data unchanged.
 2395          */
 2396         SOCKBUF_LOCK(sb);
 2397         bzero(&aso, sizeof(aso));
 2398         aso.so_pcb = so->so_pcb;
 2399         bcopy(&sb->sb_startzero, &aso.so_rcv.sb_startzero,
 2400             sizeof(*sb) - offsetof(struct sockbuf, sb_startzero));
 2401         bzero(&sb->sb_startzero,
 2402             sizeof(*sb) - offsetof(struct sockbuf, sb_startzero));
 2403         SOCKBUF_UNLOCK(sb);
 2404         sbunlock(sb);
 2405 
 2406         /*
 2407          * Dispose of special rights and flush the copied socket.  Don't call
 2408          * any unsafe routines (that rely on locks being initialized) on aso.
 2409          */
 2410         if (pr->pr_flags & PR_RIGHTS && pr->pr_domain->dom_dispose != NULL)
 2411                 (*pr->pr_domain->dom_dispose)(&aso);
 2412         sbrelease_internal(&aso.so_rcv, so);
 2413 }
 2414 
 2415 /*
 2416  * Wrapper for Socket established helper hook.
 2417  * Parameters: socket, context of the hook point, hook id.
 2418  */
 2419 static int inline
 2420 hhook_run_socket(struct socket *so, void *hctx, int32_t h_id)
 2421 {
 2422         struct socket_hhook_data hhook_data = {
 2423                 .so = so,
 2424                 .hctx = hctx,
 2425                 .m = NULL,
 2426                 .status = 0
 2427         };
 2428 
 2429         CURVNET_SET(so->so_vnet);
 2430         HHOOKS_RUN_IF(V_socket_hhh[h_id], &hhook_data, &so->osd);
 2431         CURVNET_RESTORE();
 2432 
 2433         /* Ugly but needed, since hhooks return void for now */
 2434         return (hhook_data.status);
 2435 }
 2436 
 2437 /*
 2438  * Perhaps this routine, and sooptcopyout(), below, ought to come in an
 2439  * additional variant to handle the case where the option value needs to be
 2440  * some kind of integer, but not a specific size.  In addition to their use
 2441  * here, these functions are also called by the protocol-level pr_ctloutput()
 2442  * routines.
 2443  */
 2444 int
 2445 sooptcopyin(struct sockopt *sopt, void *buf, size_t len, size_t minlen)
 2446 {
 2447         size_t  valsize;
 2448 
 2449         /*
 2450          * If the user gives us more than we wanted, we ignore it, but if we
 2451          * don't get the minimum length the caller wants, we return EINVAL.
 2452          * On success, sopt->sopt_valsize is set to however much we actually
 2453          * retrieved.
 2454          */
 2455         if ((valsize = sopt->sopt_valsize) < minlen)
 2456                 return EINVAL;
 2457         if (valsize > len)
 2458                 sopt->sopt_valsize = valsize = len;
 2459 
 2460         if (sopt->sopt_td != NULL)
 2461                 return (copyin(sopt->sopt_val, buf, valsize));
 2462 
 2463         bcopy(sopt->sopt_val, buf, valsize);
 2464         return (0);
 2465 }
 2466 
 2467 /*
 2468  * Kernel version of setsockopt(2).
 2469  *
 2470  * XXX: optlen is size_t, not socklen_t
 2471  */
 2472 int
 2473 so_setsockopt(struct socket *so, int level, int optname, void *optval,
 2474     size_t optlen)
 2475 {
 2476         struct sockopt sopt;
 2477 
 2478         sopt.sopt_level = level;
 2479         sopt.sopt_name = optname;
 2480         sopt.sopt_dir = SOPT_SET;
 2481         sopt.sopt_val = optval;
 2482         sopt.sopt_valsize = optlen;
 2483         sopt.sopt_td = NULL;
 2484         return (sosetopt(so, &sopt));
 2485 }
 2486 
 2487 int
 2488 sosetopt(struct socket *so, struct sockopt *sopt)
 2489 {
 2490         int     error, optval;
 2491         struct  linger l;
 2492         struct  timeval tv;
 2493         sbintime_t val;
 2494         uint32_t val32;
 2495 #ifdef MAC
 2496         struct mac extmac;
 2497 #endif
 2498 
 2499         CURVNET_SET(so->so_vnet);
 2500         error = 0;
 2501         if (sopt->sopt_level != SOL_SOCKET) {
 2502                 if (so->so_proto->pr_ctloutput != NULL) {
 2503                         error = (*so->so_proto->pr_ctloutput)(so, sopt);
 2504                         CURVNET_RESTORE();
 2505                         return (error);
 2506                 }
 2507                 error = ENOPROTOOPT;
 2508         } else {
 2509                 switch (sopt->sopt_name) {
 2510                 case SO_ACCEPTFILTER:
 2511                         error = do_setopt_accept_filter(so, sopt);
 2512                         if (error)
 2513                                 goto bad;
 2514                         break;
 2515 
 2516                 case SO_LINGER:
 2517                         error = sooptcopyin(sopt, &l, sizeof l, sizeof l);
 2518                         if (error)
 2519                                 goto bad;
 2520 
 2521                         SOCK_LOCK(so);
 2522                         so->so_linger = l.l_linger;
 2523                         if (l.l_onoff)
 2524                                 so->so_options |= SO_LINGER;
 2525                         else
 2526                                 so->so_options &= ~SO_LINGER;
 2527                         SOCK_UNLOCK(so);
 2528                         break;
 2529 
 2530                 case SO_DEBUG:
 2531                 case SO_KEEPALIVE:
 2532                 case SO_DONTROUTE:
 2533                 case SO_USELOOPBACK:
 2534                 case SO_BROADCAST:
 2535                 case SO_REUSEADDR:
 2536                 case SO_REUSEPORT:
 2537                 case SO_OOBINLINE:
 2538                 case SO_TIMESTAMP:
 2539                 case SO_BINTIME:
 2540                 case SO_NOSIGPIPE:
 2541                 case SO_NO_DDP:
 2542                 case SO_NO_OFFLOAD:
 2543                         error = sooptcopyin(sopt, &optval, sizeof optval,
 2544                             sizeof optval);
 2545                         if (error)
 2546                                 goto bad;
 2547                         SOCK_LOCK(so);
 2548                         if (optval)
 2549                                 so->so_options |= sopt->sopt_name;
 2550                         else
 2551                                 so->so_options &= ~sopt->sopt_name;
 2552                         SOCK_UNLOCK(so);
 2553                         break;
 2554 
 2555                 case SO_SETFIB:
 2556                         error = sooptcopyin(sopt, &optval, sizeof optval,
 2557                             sizeof optval);
 2558                         if (error)
 2559                                 goto bad;
 2560 
 2561                         if (optval < 0 || optval >= rt_numfibs) {
 2562                                 error = EINVAL;
 2563                                 goto bad;
 2564                         }
 2565                         if (((so->so_proto->pr_domain->dom_family == PF_INET) ||
 2566                            (so->so_proto->pr_domain->dom_family == PF_INET6) ||
 2567                            (so->so_proto->pr_domain->dom_family == PF_ROUTE)))
 2568                                 so->so_fibnum = optval;
 2569                         else
 2570                                 so->so_fibnum = 0;
 2571                         break;
 2572 
 2573                 case SO_USER_COOKIE:
 2574                         error = sooptcopyin(sopt, &val32, sizeof val32,
 2575                             sizeof val32);
 2576                         if (error)
 2577                                 goto bad;
 2578                         so->so_user_cookie = val32;
 2579                         break;
 2580 
 2581                 case SO_SNDBUF:
 2582                 case SO_RCVBUF:
 2583                 case SO_SNDLOWAT:
 2584                 case SO_RCVLOWAT:
 2585                         error = sooptcopyin(sopt, &optval, sizeof optval,
 2586                             sizeof optval);
 2587                         if (error)
 2588                                 goto bad;
 2589 
 2590                         /*
 2591                          * Values < 1 make no sense for any of these options,
 2592                          * so disallow them.
 2593                          */
 2594                         if (optval < 1) {
 2595                                 error = EINVAL;
 2596                                 goto bad;
 2597                         }
 2598 
 2599                         switch (sopt->sopt_name) {
 2600                         case SO_SNDBUF:
 2601                         case SO_RCVBUF:
 2602                                 if (sbreserve(sopt->sopt_name == SO_SNDBUF ?
 2603                                     &so->so_snd : &so->so_rcv, (u_long)optval,
 2604                                     so, curthread) == 0) {
 2605                                         error = ENOBUFS;
 2606                                         goto bad;
 2607                                 }
 2608                                 (sopt->sopt_name == SO_SNDBUF ? &so->so_snd :
 2609                                     &so->so_rcv)->sb_flags &= ~SB_AUTOSIZE;
 2610                                 break;
 2611 
 2612                         /*
 2613                          * Make sure the low-water is never greater than the
 2614                          * high-water.
 2615                          */
 2616                         case SO_SNDLOWAT:
 2617                                 SOCKBUF_LOCK(&so->so_snd);
 2618                                 so->so_snd.sb_lowat =
 2619                                     (optval > so->so_snd.sb_hiwat) ?
 2620                                     so->so_snd.sb_hiwat : optval;
 2621                                 SOCKBUF_UNLOCK(&so->so_snd);
 2622                                 break;
 2623                         case SO_RCVLOWAT:
 2624                                 SOCKBUF_LOCK(&so->so_rcv);
 2625                                 so->so_rcv.sb_lowat =
 2626                                     (optval > so->so_rcv.sb_hiwat) ?
 2627                                     so->so_rcv.sb_hiwat : optval;
 2628                                 SOCKBUF_UNLOCK(&so->so_rcv);
 2629                                 break;
 2630                         }
 2631                         break;
 2632 
 2633                 case SO_SNDTIMEO:
 2634                 case SO_RCVTIMEO:
 2635 #ifdef COMPAT_FREEBSD32
 2636                         if (SV_CURPROC_FLAG(SV_ILP32)) {
 2637                                 struct timeval32 tv32;
 2638 
 2639                                 error = sooptcopyin(sopt, &tv32, sizeof tv32,
 2640                                     sizeof tv32);
 2641                                 CP(tv32, tv, tv_sec);
 2642                                 CP(tv32, tv, tv_usec);
 2643                         } else
 2644 #endif
 2645                                 error = sooptcopyin(sopt, &tv, sizeof tv,
 2646                                     sizeof tv);
 2647                         if (error)
 2648                                 goto bad;
 2649                         if (tv.tv_sec < 0 || tv.tv_usec < 0 ||
 2650                             tv.tv_usec >= 1000000) {
 2651                                 error = EDOM;
 2652                                 goto bad;
 2653                         }
 2654                         if (tv.tv_sec > INT32_MAX)
 2655                                 val = SBT_MAX;
 2656                         else
 2657                                 val = tvtosbt(tv);
 2658                         switch (sopt->sopt_name) {
 2659                         case SO_SNDTIMEO:
 2660                                 so->so_snd.sb_timeo = val;
 2661                                 break;
 2662                         case SO_RCVTIMEO:
 2663                                 so->so_rcv.sb_timeo = val;
 2664                                 break;
 2665                         }
 2666                         break;
 2667 
 2668                 case SO_LABEL:
 2669 #ifdef MAC
 2670                         error = sooptcopyin(sopt, &extmac, sizeof extmac,
 2671                             sizeof extmac);
 2672                         if (error)
 2673                                 goto bad;
 2674                         error = mac_setsockopt_label(sopt->sopt_td->td_ucred,
 2675                             so, &extmac);
 2676 #else
 2677                         error = EOPNOTSUPP;
 2678 #endif
 2679                         break;
 2680 
 2681                 default:
 2682                         if (V_socket_hhh[HHOOK_SOCKET_OPT]->hhh_nhooks > 0)
 2683                                 error = hhook_run_socket(so, sopt,
 2684                                     HHOOK_SOCKET_OPT);
 2685                         else
 2686                                 error = ENOPROTOOPT;
 2687                         break;
 2688                 }
 2689                 if (error == 0 && so->so_proto->pr_ctloutput != NULL)
 2690                         (void)(*so->so_proto->pr_ctloutput)(so, sopt);
 2691         }
 2692 bad:
 2693         CURVNET_RESTORE();
 2694         return (error);
 2695 }
 2696 
 2697 /*
 2698  * Helper routine for getsockopt.
 2699  */
 2700 int
 2701 sooptcopyout(struct sockopt *sopt, const void *buf, size_t len)
 2702 {
 2703         int     error;
 2704         size_t  valsize;
 2705 
 2706         error = 0;
 2707 
 2708         /*
 2709          * Documented get behavior is that we always return a value, possibly
 2710          * truncated to fit in the user's buffer.  Traditional behavior is
 2711          * that we always tell the user precisely how much we copied, rather
 2712          * than something useful like the total amount we had available for
 2713          * her.  Note that this interface is not idempotent; the entire
 2714          * answer must be generated ahead of time.
 2715          */
 2716         valsize = min(len, sopt->sopt_valsize);
 2717         sopt->sopt_valsize = valsize;
 2718         if (sopt->sopt_val != NULL) {
 2719                 if (sopt->sopt_td != NULL)
 2720                         error = copyout(buf, sopt->sopt_val, valsize);
 2721                 else
 2722                         bcopy(buf, sopt->sopt_val, valsize);
 2723         }
 2724         return (error);
 2725 }
 2726 
 2727 int
 2728 sogetopt(struct socket *so, struct sockopt *sopt)
 2729 {
 2730         int     error, optval;
 2731         struct  linger l;
 2732         struct  timeval tv;
 2733 #ifdef MAC
 2734         struct mac extmac;
 2735 #endif
 2736 
 2737         CURVNET_SET(so->so_vnet);
 2738         error = 0;
 2739         if (sopt->sopt_level != SOL_SOCKET) {
 2740                 if (so->so_proto->pr_ctloutput != NULL)
 2741                         error = (*so->so_proto->pr_ctloutput)(so, sopt);
 2742                 else
 2743                         error = ENOPROTOOPT;
 2744                 CURVNET_RESTORE();
 2745                 return (error);
 2746         } else {
 2747                 switch (sopt->sopt_name) {
 2748                 case SO_ACCEPTFILTER:
 2749                         error = do_getopt_accept_filter(so, sopt);
 2750                         break;
 2751 
 2752                 case SO_LINGER:
 2753                         SOCK_LOCK(so);
 2754                         l.l_onoff = so->so_options & SO_LINGER;
 2755                         l.l_linger = so->so_linger;
 2756                         SOCK_UNLOCK(so);
 2757                         error = sooptcopyout(sopt, &l, sizeof l);
 2758                         break;
 2759 
 2760                 case SO_USELOOPBACK:
 2761                 case SO_DONTROUTE:
 2762                 case SO_DEBUG:
 2763                 case SO_KEEPALIVE:
 2764                 case SO_REUSEADDR:
 2765                 case SO_REUSEPORT:
 2766                 case SO_BROADCAST:
 2767                 case SO_OOBINLINE:
 2768                 case SO_ACCEPTCONN:
 2769                 case SO_TIMESTAMP:
 2770                 case SO_BINTIME:
 2771                 case SO_NOSIGPIPE:
 2772                         optval = so->so_options & sopt->sopt_name;
 2773 integer:
 2774                         error = sooptcopyout(sopt, &optval, sizeof optval);
 2775                         break;
 2776 
 2777                 case SO_TYPE:
 2778                         optval = so->so_type;
 2779                         goto integer;
 2780 
 2781                 case SO_PROTOCOL:
 2782                         optval = so->so_proto->pr_protocol;
 2783                         goto integer;
 2784 
 2785                 case SO_ERROR:
 2786                         SOCK_LOCK(so);
 2787                         optval = so->so_error;
 2788                         so->so_error = 0;
 2789                         SOCK_UNLOCK(so);
 2790                         goto integer;
 2791 
 2792                 case SO_SNDBUF:
 2793                         optval = so->so_snd.sb_hiwat;
 2794                         goto integer;
 2795 
 2796                 case SO_RCVBUF:
 2797                         optval = so->so_rcv.sb_hiwat;
 2798                         goto integer;
 2799 
 2800                 case SO_SNDLOWAT:
 2801                         optval = so->so_snd.sb_lowat;
 2802                         goto integer;
 2803 
 2804                 case SO_RCVLOWAT:
 2805                         optval = so->so_rcv.sb_lowat;
 2806                         goto integer;
 2807 
 2808                 case SO_SNDTIMEO:
 2809                 case SO_RCVTIMEO:
 2810                         tv = sbttotv(sopt->sopt_name == SO_SNDTIMEO ?
 2811                             so->so_snd.sb_timeo : so->so_rcv.sb_timeo);
 2812 #ifdef COMPAT_FREEBSD32
 2813                         if (SV_CURPROC_FLAG(SV_ILP32)) {
 2814                                 struct timeval32 tv32;
 2815 
 2816                                 CP(tv, tv32, tv_sec);
 2817                                 CP(tv, tv32, tv_usec);
 2818                                 error = sooptcopyout(sopt, &tv32, sizeof tv32);
 2819                         } else
 2820 #endif
 2821                                 error = sooptcopyout(sopt, &tv, sizeof tv);
 2822                         break;
 2823 
 2824                 case SO_LABEL:
 2825 #ifdef MAC
 2826                         error = sooptcopyin(sopt, &extmac, sizeof(extmac),
 2827                             sizeof(extmac));
 2828                         if (error)
 2829                                 goto bad;
 2830                         error = mac_getsockopt_label(sopt->sopt_td->td_ucred,
 2831                             so, &extmac);
 2832                         if (error)
 2833                                 goto bad;
 2834                         error = sooptcopyout(sopt, &extmac, sizeof extmac);
 2835 #else
 2836                         error = EOPNOTSUPP;
 2837 #endif
 2838                         break;
 2839 
 2840                 case SO_PEERLABEL:
 2841 #ifdef MAC
 2842                         error = sooptcopyin(sopt, &extmac, sizeof(extmac),
 2843                             sizeof(extmac));
 2844                         if (error)
 2845                                 goto bad;
 2846                         error = mac_getsockopt_peerlabel(
 2847                             sopt->sopt_td->td_ucred, so, &extmac);
 2848                         if (error)
 2849                                 goto bad;
 2850                         error = sooptcopyout(sopt, &extmac, sizeof extmac);
 2851 #else
 2852                         error = EOPNOTSUPP;
 2853 #endif
 2854                         break;
 2855 
 2856                 case SO_LISTENQLIMIT:
 2857                         optval = so->so_qlimit;
 2858                         goto integer;
 2859 
 2860                 case SO_LISTENQLEN:
 2861                         optval = so->so_qlen;
 2862                         goto integer;
 2863 
 2864                 case SO_LISTENINCQLEN:
 2865                         optval = so->so_incqlen;
 2866                         goto integer;
 2867 
 2868                 default:
 2869                         if (V_socket_hhh[HHOOK_SOCKET_OPT]->hhh_nhooks > 0)
 2870                                 error = hhook_run_socket(so, sopt,
 2871                                     HHOOK_SOCKET_OPT);
 2872                         else
 2873                                 error = ENOPROTOOPT;
 2874                         break;
 2875                 }
 2876         }
 2877 #ifdef MAC
 2878 bad:
 2879 #endif
 2880         CURVNET_RESTORE();
 2881         return (error);
 2882 }
 2883 
 2884 int
 2885 soopt_getm(struct sockopt *sopt, struct mbuf **mp)
 2886 {
 2887         struct mbuf *m, *m_prev;
 2888         int sopt_size = sopt->sopt_valsize;
 2889 
 2890         MGET(m, sopt->sopt_td ? M_WAITOK : M_NOWAIT, MT_DATA);
 2891         if (m == NULL)
 2892                 return ENOBUFS;
 2893         if (sopt_size > MLEN) {
 2894                 MCLGET(m, sopt->sopt_td ? M_WAITOK : M_NOWAIT);
 2895                 if ((m->m_flags & M_EXT) == 0) {
 2896                         m_free(m);
 2897                         return ENOBUFS;
 2898                 }
 2899                 m->m_len = min(MCLBYTES, sopt_size);
 2900         } else {
 2901                 m->m_len = min(MLEN, sopt_size);
 2902         }
 2903         sopt_size -= m->m_len;
 2904         *mp = m;
 2905         m_prev = m;
 2906 
 2907         while (sopt_size) {
 2908                 MGET(m, sopt->sopt_td ? M_WAITOK : M_NOWAIT, MT_DATA);
 2909                 if (m == NULL) {
 2910                         m_freem(*mp);
 2911                         return ENOBUFS;
 2912                 }
 2913                 if (sopt_size > MLEN) {
 2914                         MCLGET(m, sopt->sopt_td != NULL ? M_WAITOK :
 2915                             M_NOWAIT);
 2916                         if ((m->m_flags & M_EXT) == 0) {
 2917                                 m_freem(m);
 2918                                 m_freem(*mp);
 2919                                 return ENOBUFS;
 2920                         }
 2921                         m->m_len = min(MCLBYTES, sopt_size);
 2922                 } else {
 2923                         m->m_len = min(MLEN, sopt_size);
 2924                 }
 2925                 sopt_size -= m->m_len;
 2926                 m_prev->m_next = m;
 2927                 m_prev = m;
 2928         }
 2929         return (0);
 2930 }
 2931 
 2932 int
 2933 soopt_mcopyin(struct sockopt *sopt, struct mbuf *m)
 2934 {
 2935         struct mbuf *m0 = m;
 2936 
 2937         if (sopt->sopt_val == NULL)
 2938                 return (0);
 2939         while (m != NULL && sopt->sopt_valsize >= m->m_len) {
 2940                 if (sopt->sopt_td != NULL) {
 2941                         int error;
 2942 
 2943                         error = copyin(sopt->sopt_val, mtod(m, char *),
 2944                             m->m_len);
 2945                         if (error != 0) {
 2946                                 m_freem(m0);
 2947                                 return(error);
 2948                         }
 2949                 } else
 2950                         bcopy(sopt->sopt_val, mtod(m, char *), m->m_len);
 2951                 sopt->sopt_valsize -= m->m_len;
 2952                 sopt->sopt_val = (char *)sopt->sopt_val + m->m_len;
 2953                 m = m->m_next;
 2954         }
 2955         if (m != NULL) /* should be allocated enoughly at ip6_sooptmcopyin() */
 2956                 panic("ip6_sooptmcopyin");
 2957         return (0);
 2958 }
 2959 
 2960 int
 2961 soopt_mcopyout(struct sockopt *sopt, struct mbuf *m)
 2962 {
 2963         struct mbuf *m0 = m;
 2964         size_t valsize = 0;
 2965 
 2966         if (sopt->sopt_val == NULL)
 2967                 return (0);
 2968         while (m != NULL && sopt->sopt_valsize >= m->m_len) {
 2969                 if (sopt->sopt_td != NULL) {
 2970                         int error;
 2971 
 2972                         error = copyout(mtod(m, char *), sopt->sopt_val,
 2973                             m->m_len);
 2974                         if (error != 0) {
 2975                                 m_freem(m0);
 2976                                 return(error);
 2977                         }
 2978                 } else
 2979                         bcopy(mtod(m, char *), sopt->sopt_val, m->m_len);
 2980                 sopt->sopt_valsize -= m->m_len;
 2981                 sopt->sopt_val = (char *)sopt->sopt_val + m->m_len;
 2982                 valsize += m->m_len;
 2983                 m = m->m_next;
 2984         }
 2985         if (m != NULL) {
 2986                 /* enough soopt buffer should be given from user-land */
 2987                 m_freem(m0);
 2988                 return(EINVAL);
 2989         }
 2990         sopt->sopt_valsize = valsize;
 2991         return (0);
 2992 }
 2993 
 2994 /*
 2995  * sohasoutofband(): protocol notifies socket layer of the arrival of new
 2996  * out-of-band data, which will then notify socket consumers.
 2997  */
 2998 void
 2999 sohasoutofband(struct socket *so)
 3000 {
 3001 
 3002         if (so->so_sigio != NULL)
 3003                 pgsigio(&so->so_sigio, SIGURG, 0);
 3004         selwakeuppri(&so->so_rcv.sb_sel, PSOCK);
 3005 }
 3006 
 3007 int
 3008 sopoll(struct socket *so, int events, struct ucred *active_cred,
 3009     struct thread *td)
 3010 {
 3011 
 3012         /*
 3013          * We do not need to set or assert curvnet as long as everyone uses
 3014          * sopoll_generic().
 3015          */
 3016         return (so->so_proto->pr_usrreqs->pru_sopoll(so, events, active_cred,
 3017             td));
 3018 }
 3019 
 3020 int
 3021 sopoll_generic(struct socket *so, int events, struct ucred *active_cred,
 3022     struct thread *td)
 3023 {
 3024         int revents = 0;
 3025 
 3026         SOCKBUF_LOCK(&so->so_snd);
 3027         SOCKBUF_LOCK(&so->so_rcv);
 3028         if (events & (POLLIN | POLLRDNORM))
 3029                 if (soreadabledata(so))
 3030                         revents |= events & (POLLIN | POLLRDNORM);
 3031 
 3032         if (events & (POLLOUT | POLLWRNORM))
 3033                 if (sowriteable(so))
 3034                         revents |= events & (POLLOUT | POLLWRNORM);
 3035 
 3036         if (events & (POLLPRI | POLLRDBAND))
 3037                 if (so->so_oobmark || (so->so_rcv.sb_state & SBS_RCVATMARK))
 3038                         revents |= events & (POLLPRI | POLLRDBAND);
 3039 
 3040         if ((events & POLLINIGNEOF) == 0) {
 3041                 if (so->so_rcv.sb_state & SBS_CANTRCVMORE) {
 3042                         revents |= events & (POLLIN | POLLRDNORM);
 3043                         if (so->so_snd.sb_state & SBS_CANTSENDMORE)
 3044                                 revents |= POLLHUP;
 3045                 }
 3046         }
 3047 
 3048         if (revents == 0) {
 3049                 if (events & (POLLIN | POLLPRI | POLLRDNORM | POLLRDBAND)) {
 3050                         selrecord(td, &so->so_rcv.sb_sel);
 3051                         so->so_rcv.sb_flags |= SB_SEL;
 3052                 }
 3053 
 3054                 if (events & (POLLOUT | POLLWRNORM)) {
 3055                         selrecord(td, &so->so_snd.sb_sel);
 3056                         so->so_snd.sb_flags |= SB_SEL;
 3057                 }
 3058         }
 3059 
 3060         SOCKBUF_UNLOCK(&so->so_rcv);
 3061         SOCKBUF_UNLOCK(&so->so_snd);
 3062         return (revents);
 3063 }
 3064 
 3065 int
 3066 soo_kqfilter(struct file *fp, struct knote *kn)
 3067 {
 3068         struct socket *so = kn->kn_fp->f_data;
 3069         struct sockbuf *sb;
 3070 
 3071         switch (kn->kn_filter) {
 3072         case EVFILT_READ:
 3073                 if (so->so_options & SO_ACCEPTCONN)
 3074                         kn->kn_fop = &solisten_filtops;
 3075                 else
 3076                         kn->kn_fop = &soread_filtops;
 3077                 sb = &so->so_rcv;
 3078                 break;
 3079         case EVFILT_WRITE:
 3080                 kn->kn_fop = &sowrite_filtops;
 3081                 sb = &so->so_snd;
 3082                 break;
 3083         default:
 3084                 return (EINVAL);
 3085         }
 3086 
 3087         SOCKBUF_LOCK(sb);
 3088         knlist_add(&sb->sb_sel.si_note, kn, 1);
 3089         sb->sb_flags |= SB_KNOTE;
 3090         SOCKBUF_UNLOCK(sb);
 3091         return (0);
 3092 }
 3093 
 3094 /*
 3095  * Some routines that return EOPNOTSUPP for entry points that are not
 3096  * supported by a protocol.  Fill in as needed.
 3097  */
 3098 int
 3099 pru_accept_notsupp(struct socket *so, struct sockaddr **nam)
 3100 {
 3101 
 3102         return EOPNOTSUPP;
 3103 }
 3104 
 3105 int
 3106 pru_aio_queue_notsupp(struct socket *so, struct kaiocb *job)
 3107 {
 3108 
 3109         return EOPNOTSUPP;
 3110 }
 3111 
 3112 int
 3113 pru_attach_notsupp(struct socket *so, int proto, struct thread *td)
 3114 {
 3115 
 3116         return EOPNOTSUPP;
 3117 }
 3118 
 3119 int
 3120 pru_bind_notsupp(struct socket *so, struct sockaddr *nam, struct thread *td)
 3121 {
 3122 
 3123         return EOPNOTSUPP;
 3124 }
 3125 
 3126 int
 3127 pru_bindat_notsupp(int fd, struct socket *so, struct sockaddr *nam,
 3128     struct thread *td)
 3129 {
 3130 
 3131         return EOPNOTSUPP;
 3132 }
 3133 
 3134 int
 3135 pru_connect_notsupp(struct socket *so, struct sockaddr *nam, struct thread *td)
 3136 {
 3137 
 3138         return EOPNOTSUPP;
 3139 }
 3140 
 3141 int
 3142 pru_connectat_notsupp(int fd, struct socket *so, struct sockaddr *nam,
 3143     struct thread *td)
 3144 {
 3145 
 3146         return EOPNOTSUPP;
 3147 }
 3148 
 3149 int
 3150 pru_connect2_notsupp(struct socket *so1, struct socket *so2)
 3151 {
 3152 
 3153         return EOPNOTSUPP;
 3154 }
 3155 
 3156 int
 3157 pru_control_notsupp(struct socket *so, u_long cmd, caddr_t data,
 3158     struct ifnet *ifp, struct thread *td)
 3159 {
 3160 
 3161         return EOPNOTSUPP;
 3162 }
 3163 
 3164 int
 3165 pru_disconnect_notsupp(struct socket *so)
 3166 {
 3167 
 3168         return EOPNOTSUPP;
 3169 }
 3170 
 3171 int
 3172 pru_listen_notsupp(struct socket *so, int backlog, struct thread *td)
 3173 {
 3174 
 3175         return EOPNOTSUPP;
 3176 }
 3177 
 3178 int
 3179 pru_peeraddr_notsupp(struct socket *so, struct sockaddr **nam)
 3180 {
 3181 
 3182         return EOPNOTSUPP;
 3183 }
 3184 
 3185 int
 3186 pru_rcvd_notsupp(struct socket *so, int flags)
 3187 {
 3188 
 3189         return EOPNOTSUPP;
 3190 }
 3191 
 3192 int
 3193 pru_rcvoob_notsupp(struct socket *so, struct mbuf *m, int flags)
 3194 {
 3195 
 3196         return EOPNOTSUPP;
 3197 }
 3198 
 3199 int
 3200 pru_send_notsupp(struct socket *so, int flags, struct mbuf *m,
 3201     struct sockaddr *addr, struct mbuf *control, struct thread *td)
 3202 {
 3203 
 3204         return EOPNOTSUPP;
 3205 }
 3206 
 3207 int
 3208 pru_ready_notsupp(struct socket *so, struct mbuf *m, int count)
 3209 {
 3210 
 3211         return (EOPNOTSUPP);
 3212 }
 3213 
 3214 /*
 3215  * This isn't really a ``null'' operation, but it's the default one and
 3216  * doesn't do anything destructive.
 3217  */
 3218 int
 3219 pru_sense_null(struct socket *so, struct stat *sb)
 3220 {
 3221 
 3222         sb->st_blksize = so->so_snd.sb_hiwat;
 3223         return 0;
 3224 }
 3225 
 3226 int
 3227 pru_shutdown_notsupp(struct socket *so)
 3228 {
 3229 
 3230         return EOPNOTSUPP;
 3231 }
 3232 
 3233 int
 3234 pru_sockaddr_notsupp(struct socket *so, struct sockaddr **nam)
 3235 {
 3236 
 3237         return EOPNOTSUPP;
 3238 }
 3239 
 3240 int
 3241 pru_sosend_notsupp(struct socket *so, struct sockaddr *addr, struct uio *uio,
 3242     struct mbuf *top, struct mbuf *control, int flags, struct thread *td)
 3243 {
 3244 
 3245         return EOPNOTSUPP;
 3246 }
 3247 
 3248 int
 3249 pru_soreceive_notsupp(struct socket *so, struct sockaddr **paddr,
 3250     struct uio *uio, struct mbuf **mp0, struct mbuf **controlp, int *flagsp)
 3251 {
 3252 
 3253         return EOPNOTSUPP;
 3254 }
 3255 
 3256 int
 3257 pru_sopoll_notsupp(struct socket *so, int events, struct ucred *cred,
 3258     struct thread *td)
 3259 {
 3260 
 3261         return EOPNOTSUPP;
 3262 }
 3263 
 3264 static void
 3265 filt_sordetach(struct knote *kn)
 3266 {
 3267         struct socket *so = kn->kn_fp->f_data;
 3268 
 3269         SOCKBUF_LOCK(&so->so_rcv);
 3270         knlist_remove(&so->so_rcv.sb_sel.si_note, kn, 1);
 3271         if (knlist_empty(&so->so_rcv.sb_sel.si_note))
 3272                 so->so_rcv.sb_flags &= ~SB_KNOTE;
 3273         SOCKBUF_UNLOCK(&so->so_rcv);
 3274 }
 3275 
 3276 /*ARGSUSED*/
 3277 static int
 3278 filt_soread(struct knote *kn, long hint)
 3279 {
 3280         struct socket *so;
 3281 
 3282         so = kn->kn_fp->f_data;
 3283         SOCKBUF_LOCK_ASSERT(&so->so_rcv);
 3284 
 3285         kn->kn_data = sbavail(&so->so_rcv) - so->so_rcv.sb_ctl;
 3286         if (so->so_rcv.sb_state & SBS_CANTRCVMORE) {
 3287                 kn->kn_flags |= EV_EOF;
 3288                 kn->kn_fflags = so->so_error;
 3289                 return (1);
 3290         } else if (so->so_error)        /* temporary udp error */
 3291                 return (1);
 3292 
 3293         if (kn->kn_sfflags & NOTE_LOWAT) {
 3294                 if (kn->kn_data >= kn->kn_sdata)
 3295                         return 1;
 3296         } else {
 3297                 if (sbavail(&so->so_rcv) >= so->so_rcv.sb_lowat)
 3298                         return 1;
 3299         }
 3300 
 3301         /* This hook returning non-zero indicates an event, not error */
 3302         return (hhook_run_socket(so, NULL, HHOOK_FILT_SOREAD));
 3303 }
 3304 
 3305 static void
 3306 filt_sowdetach(struct knote *kn)
 3307 {
 3308         struct socket *so = kn->kn_fp->f_data;
 3309 
 3310         SOCKBUF_LOCK(&so->so_snd);
 3311         knlist_remove(&so->so_snd.sb_sel.si_note, kn, 1);
 3312         if (knlist_empty(&so->so_snd.sb_sel.si_note))
 3313                 so->so_snd.sb_flags &= ~SB_KNOTE;
 3314         SOCKBUF_UNLOCK(&so->so_snd);
 3315 }
 3316 
 3317 /*ARGSUSED*/
 3318 static int
 3319 filt_sowrite(struct knote *kn, long hint)
 3320 {
 3321         struct socket *so;
 3322 
 3323         so = kn->kn_fp->f_data;
 3324         SOCKBUF_LOCK_ASSERT(&so->so_snd);
 3325         kn->kn_data = sbspace(&so->so_snd);
 3326 
 3327         hhook_run_socket(so, kn, HHOOK_FILT_SOWRITE);
 3328 
 3329         if (so->so_snd.sb_state & SBS_CANTSENDMORE) {
 3330                 kn->kn_flags |= EV_EOF;
 3331                 kn->kn_fflags = so->so_error;
 3332                 return (1);
 3333         } else if (so->so_error)        /* temporary udp error */
 3334                 return (1);
 3335         else if (((so->so_state & SS_ISCONNECTED) == 0) &&
 3336             (so->so_proto->pr_flags & PR_CONNREQUIRED))
 3337                 return (0);
 3338         else if (kn->kn_sfflags & NOTE_LOWAT)
 3339                 return (kn->kn_data >= kn->kn_sdata);
 3340         else
 3341                 return (kn->kn_data >= so->so_snd.sb_lowat);
 3342 }
 3343 
 3344 /*ARGSUSED*/
 3345 static int
 3346 filt_solisten(struct knote *kn, long hint)
 3347 {
 3348         struct socket *so = kn->kn_fp->f_data;
 3349 
 3350         kn->kn_data = so->so_qlen;
 3351         return (!TAILQ_EMPTY(&so->so_comp));
 3352 }
 3353 
 3354 int
 3355 socheckuid(struct socket *so, uid_t uid)
 3356 {
 3357 
 3358         if (so == NULL)
 3359                 return (EPERM);
 3360         if (so->so_cred->cr_uid != uid)
 3361                 return (EPERM);
 3362         return (0);
 3363 }
 3364 
 3365 /*
 3366  * These functions are used by protocols to notify the socket layer (and its
 3367  * consumers) of state changes in the sockets driven by protocol-side events.
 3368  */
 3369 
 3370 /*
 3371  * Procedures to manipulate state flags of socket and do appropriate wakeups.
 3372  *
 3373  * Normal sequence from the active (originating) side is that
 3374  * soisconnecting() is called during processing of connect() call, resulting
 3375  * in an eventual call to soisconnected() if/when the connection is
 3376  * established.  When the connection is torn down soisdisconnecting() is
 3377  * called during processing of disconnect() call, and soisdisconnected() is
 3378  * called when the connection to the peer is totally severed.  The semantics
 3379  * of these routines are such that connectionless protocols can call
 3380  * soisconnected() and soisdisconnected() only, bypassing the in-progress
 3381  * calls when setting up a ``connection'' takes no time.
 3382  *
 3383  * From the passive side, a socket is created with two queues of sockets:
 3384  * so_incomp for connections in progress and so_comp for connections already
 3385  * made and awaiting user acceptance.  As a protocol is preparing incoming
 3386  * connections, it creates a socket structure queued on so_incomp by calling
 3387  * sonewconn().  When the connection is established, soisconnected() is
 3388  * called, and transfers the socket structure to so_comp, making it available
 3389  * to accept().
 3390  *
 3391  * If a socket is closed with sockets on either so_incomp or so_comp, these
 3392  * sockets are dropped.
 3393  *
 3394  * If higher-level protocols are implemented in the kernel, the wakeups done
 3395  * here will sometimes cause software-interrupt process scheduling.
 3396  */
 3397 void
 3398 soisconnecting(struct socket *so)
 3399 {
 3400 
 3401         SOCK_LOCK(so);
 3402         so->so_state &= ~(SS_ISCONNECTED|SS_ISDISCONNECTING);
 3403         so->so_state |= SS_ISCONNECTING;
 3404         SOCK_UNLOCK(so);
 3405 }
 3406 
 3407 void
 3408 soisconnected(struct socket *so)
 3409 {
 3410         struct socket *head;
 3411         int ret;
 3412 
 3413 restart:
 3414         ACCEPT_LOCK();
 3415         SOCK_LOCK(so);
 3416         so->so_state &= ~(SS_ISCONNECTING|SS_ISDISCONNECTING|SS_ISCONFIRMING);
 3417         so->so_state |= SS_ISCONNECTED;
 3418         head = so->so_head;
 3419         if (head != NULL && (so->so_qstate & SQ_INCOMP)) {
 3420                 if ((so->so_options & SO_ACCEPTFILTER) == 0) {
 3421                         SOCK_UNLOCK(so);
 3422                         TAILQ_REMOVE(&head->so_incomp, so, so_list);
 3423                         head->so_incqlen--;
 3424                         so->so_qstate &= ~SQ_INCOMP;
 3425                         TAILQ_INSERT_TAIL(&head->so_comp, so, so_list);
 3426                         head->so_qlen++;
 3427                         so->so_qstate |= SQ_COMP;
 3428                         ACCEPT_UNLOCK();
 3429                         sorwakeup(head);
 3430                         wakeup_one(&head->so_timeo);
 3431                 } else {
 3432                         ACCEPT_UNLOCK();
 3433                         soupcall_set(so, SO_RCV,
 3434                             head->so_accf->so_accept_filter->accf_callback,
 3435                             head->so_accf->so_accept_filter_arg);
 3436                         so->so_options &= ~SO_ACCEPTFILTER;
 3437                         ret = head->so_accf->so_accept_filter->accf_callback(so,
 3438                             head->so_accf->so_accept_filter_arg, M_NOWAIT);
 3439                         if (ret == SU_ISCONNECTED)
 3440                                 soupcall_clear(so, SO_RCV);
 3441                         SOCK_UNLOCK(so);
 3442                         if (ret == SU_ISCONNECTED)
 3443                                 goto restart;
 3444                 }
 3445                 return;
 3446         }
 3447         SOCK_UNLOCK(so);
 3448         ACCEPT_UNLOCK();
 3449         wakeup(&so->so_timeo);
 3450         sorwakeup(so);
 3451         sowwakeup(so);
 3452 }
 3453 
 3454 void
 3455 soisdisconnecting(struct socket *so)
 3456 {
 3457 
 3458         /*
 3459          * Note: This code assumes that SOCK_LOCK(so) and
 3460          * SOCKBUF_LOCK(&so->so_rcv) are the same.
 3461          */
 3462         SOCKBUF_LOCK(&so->so_rcv);
 3463         so->so_state &= ~SS_ISCONNECTING;
 3464         so->so_state |= SS_ISDISCONNECTING;
 3465         socantrcvmore_locked(so);
 3466         SOCKBUF_LOCK(&so->so_snd);
 3467         socantsendmore_locked(so);
 3468         wakeup(&so->so_timeo);
 3469 }
 3470 
 3471 void
 3472 soisdisconnected(struct socket *so)
 3473 {
 3474 
 3475         /*
 3476          * Note: This code assumes that SOCK_LOCK(so) and
 3477          * SOCKBUF_LOCK(&so->so_rcv) are the same.
 3478          */
 3479         SOCKBUF_LOCK(&so->so_rcv);
 3480         so->so_state &= ~(SS_ISCONNECTING|SS_ISCONNECTED|SS_ISDISCONNECTING);
 3481         so->so_state |= SS_ISDISCONNECTED;
 3482         socantrcvmore_locked(so);
 3483         SOCKBUF_LOCK(&so->so_snd);
 3484         sbdrop_locked(&so->so_snd, sbused(&so->so_snd));
 3485         socantsendmore_locked(so);
 3486         wakeup(&so->so_timeo);
 3487 }
 3488 
 3489 /*
 3490  * Make a copy of a sockaddr in a malloced buffer of type M_SONAME.
 3491  */
 3492 struct sockaddr *
 3493 sodupsockaddr(const struct sockaddr *sa, int mflags)
 3494 {
 3495         struct sockaddr *sa2;
 3496 
 3497         sa2 = malloc(sa->sa_len, M_SONAME, mflags);
 3498         if (sa2)
 3499                 bcopy(sa, sa2, sa->sa_len);
 3500         return sa2;
 3501 }
 3502 
 3503 /*
 3504  * Register per-socket buffer upcalls.
 3505  */
 3506 void
 3507 soupcall_set(struct socket *so, int which,
 3508     int (*func)(struct socket *, void *, int), void *arg)
 3509 {
 3510         struct sockbuf *sb;
 3511 
 3512         switch (which) {
 3513         case SO_RCV:
 3514                 sb = &so->so_rcv;
 3515                 break;
 3516         case SO_SND:
 3517                 sb = &so->so_snd;
 3518                 break;
 3519         default:
 3520                 panic("soupcall_set: bad which");
 3521         }
 3522         SOCKBUF_LOCK_ASSERT(sb);
 3523 #if 0
 3524         /* XXX: accf_http actually wants to do this on purpose. */
 3525         KASSERT(sb->sb_upcall == NULL, ("soupcall_set: overwriting upcall"));
 3526 #endif
 3527         sb->sb_upcall = func;
 3528         sb->sb_upcallarg = arg;
 3529         sb->sb_flags |= SB_UPCALL;
 3530 }
 3531 
 3532 void
 3533 soupcall_clear(struct socket *so, int which)
 3534 {
 3535         struct sockbuf *sb;
 3536 
 3537         switch (which) {
 3538         case SO_RCV:
 3539                 sb = &so->so_rcv;
 3540                 break;
 3541         case SO_SND:
 3542                 sb = &so->so_snd;
 3543                 break;
 3544         default:
 3545                 panic("soupcall_clear: bad which");
 3546         }
 3547         SOCKBUF_LOCK_ASSERT(sb);
 3548         KASSERT(sb->sb_upcall != NULL, ("soupcall_clear: no upcall to clear"));
 3549         sb->sb_upcall = NULL;
 3550         sb->sb_upcallarg = NULL;
 3551         sb->sb_flags &= ~SB_UPCALL;
 3552 }
 3553 
 3554 /*
 3555  * Create an external-format (``xsocket'') structure using the information in
 3556  * the kernel-format socket structure pointed to by so.  This is done to
 3557  * reduce the spew of irrelevant information over this interface, to isolate
 3558  * user code from changes in the kernel structure, and potentially to provide
 3559  * information-hiding if we decide that some of this information should be
 3560  * hidden from users.
 3561  */
 3562 void
 3563 sotoxsocket(struct socket *so, struct xsocket *xso)
 3564 {
 3565 
 3566         xso->xso_len = sizeof *xso;
 3567         xso->xso_so = so;
 3568         xso->so_type = so->so_type;
 3569         xso->so_options = so->so_options;
 3570         xso->so_linger = so->so_linger;
 3571         xso->so_state = so->so_state;
 3572         xso->so_pcb = so->so_pcb;
 3573         xso->xso_protocol = so->so_proto->pr_protocol;
 3574         xso->xso_family = so->so_proto->pr_domain->dom_family;
 3575         xso->so_qlen = so->so_qlen;
 3576         xso->so_incqlen = so->so_incqlen;
 3577         xso->so_qlimit = so->so_qlimit;
 3578         xso->so_timeo = so->so_timeo;
 3579         xso->so_error = so->so_error;
 3580         xso->so_pgid = so->so_sigio ? so->so_sigio->sio_pgid : 0;
 3581         xso->so_oobmark = so->so_oobmark;
 3582         sbtoxsockbuf(&so->so_snd, &xso->so_snd);
 3583         sbtoxsockbuf(&so->so_rcv, &xso->so_rcv);
 3584         xso->so_uid = so->so_cred->cr_uid;
 3585 }
 3586 
 3587 
 3588 /*
 3589  * Socket accessor functions to provide external consumers with
 3590  * a safe interface to socket state
 3591  *
 3592  */
 3593 
 3594 void
 3595 so_listeners_apply_all(struct socket *so, void (*func)(struct socket *, void *),
 3596     void *arg)
 3597 {
 3598 
 3599         TAILQ_FOREACH(so, &so->so_comp, so_list)
 3600                 func(so, arg);
 3601 }
 3602 
 3603 struct sockbuf *
 3604 so_sockbuf_rcv(struct socket *so)
 3605 {
 3606 
 3607         return (&so->so_rcv);
 3608 }
 3609 
 3610 struct sockbuf *
 3611 so_sockbuf_snd(struct socket *so)
 3612 {
 3613 
 3614         return (&so->so_snd);
 3615 }
 3616 
 3617 int
 3618 so_state_get(const struct socket *so)
 3619 {
 3620 
 3621         return (so->so_state);
 3622 }
 3623 
 3624 void
 3625 so_state_set(struct socket *so, int val)
 3626 {
 3627 
 3628         so->so_state = val;
 3629 }
 3630 
 3631 int
 3632 so_options_get(const struct socket *so)
 3633 {
 3634 
 3635         return (so->so_options);
 3636 }
 3637 
 3638 void
 3639 so_options_set(struct socket *so, int val)
 3640 {
 3641 
 3642         so->so_options = val;
 3643 }
 3644 
 3645 int
 3646 so_error_get(const struct socket *so)
 3647 {
 3648 
 3649         return (so->so_error);
 3650 }
 3651 
 3652 void
 3653 so_error_set(struct socket *so, int val)
 3654 {
 3655 
 3656         so->so_error = val;
 3657 }
 3658 
 3659 int
 3660 so_linger_get(const struct socket *so)
 3661 {
 3662 
 3663         return (so->so_linger);
 3664 }
 3665 
 3666 void
 3667 so_linger_set(struct socket *so, int val)
 3668 {
 3669 
 3670         so->so_linger = val;
 3671 }
 3672 
 3673 struct protosw *
 3674 so_protosw_get(const struct socket *so)
 3675 {
 3676 
 3677         return (so->so_proto);
 3678 }
 3679 
 3680 void
 3681 so_protosw_set(struct socket *so, struct protosw *val)
 3682 {
 3683 
 3684         so->so_proto = val;
 3685 }
 3686 
 3687 void
 3688 so_sorwakeup(struct socket *so)
 3689 {
 3690 
 3691         sorwakeup(so);
 3692 }
 3693 
 3694 void
 3695 so_sowwakeup(struct socket *so)
 3696 {
 3697 
 3698         sowwakeup(so);
 3699 }
 3700 
 3701 void
 3702 so_sorwakeup_locked(struct socket *so)
 3703 {
 3704 
 3705         sorwakeup_locked(so);
 3706 }
 3707 
 3708 void
 3709 so_sowwakeup_locked(struct socket *so)
 3710 {
 3711 
 3712         sowwakeup_locked(so);
 3713 }
 3714 
 3715 void
 3716 so_lock(struct socket *so)
 3717 {
 3718 
 3719         SOCK_LOCK(so);
 3720 }
 3721 
 3722 void
 3723 so_unlock(struct socket *so)
 3724 {
 3725 
 3726         SOCK_UNLOCK(so);
 3727 }

Cache object: 9972b35db67451c565afe485aed35889


[ source navigation ] [ diff markup ] [ identifier search ] [ freetext search ] [ file search ] [ list types ] [ track identifier ]


This page is part of the FreeBSD/Linux Linux Kernel Cross-Reference, and was automatically generated using a modified version of the LXR engine.