The Design and Implementation of the FreeBSD Operating System, Second Edition
Now available: The Design and Implementation of the FreeBSD Operating System (Second Edition)


[ source navigation ] [ diff markup ] [ identifier search ] [ freetext search ] [ file search ] [ list types ] [ track identifier ]

FreeBSD/Linux Kernel Cross Reference
sys/vm/uma_int.h

Version: -  FREEBSD  -  FREEBSD-13-STABLE  -  FREEBSD-13-0  -  FREEBSD-12-STABLE  -  FREEBSD-12-0  -  FREEBSD-11-STABLE  -  FREEBSD-11-0  -  FREEBSD-10-STABLE  -  FREEBSD-10-0  -  FREEBSD-9-STABLE  -  FREEBSD-9-0  -  FREEBSD-8-STABLE  -  FREEBSD-8-0  -  FREEBSD-7-STABLE  -  FREEBSD-7-0  -  FREEBSD-6-STABLE  -  FREEBSD-6-0  -  FREEBSD-5-STABLE  -  FREEBSD-5-0  -  FREEBSD-4-STABLE  -  FREEBSD-3-STABLE  -  FREEBSD22  -  l41  -  OPENBSD  -  linux-2.6  -  MK84  -  PLAN9  -  xnu-8792 
SearchContext: -  none  -  3  -  10 

    1 /*-
    2  * SPDX-License-Identifier: BSD-2-Clause-FreeBSD
    3  *
    4  * Copyright (c) 2002-2019 Jeffrey Roberson <jeff@FreeBSD.org>
    5  * Copyright (c) 2004, 2005 Bosko Milekic <bmilekic@FreeBSD.org>
    6  * All rights reserved.
    7  *
    8  * Redistribution and use in source and binary forms, with or without
    9  * modification, are permitted provided that the following conditions
   10  * are met:
   11  * 1. Redistributions of source code must retain the above copyright
   12  *    notice unmodified, this list of conditions, and the following
   13  *    disclaimer.
   14  * 2. Redistributions in binary form must reproduce the above copyright
   15  *    notice, this list of conditions and the following disclaimer in the
   16  *    documentation and/or other materials provided with the distribution.
   17  *
   18  * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
   19  * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
   20  * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
   21  * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
   22  * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
   23  * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
   24  * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
   25  * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
   26  * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
   27  * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
   28  *
   29  * $FreeBSD$
   30  *
   31  */
   32 
   33 #include <sys/counter.h>
   34 #include <sys/_bitset.h>
   35 #include <sys/_domainset.h>
   36 #include <sys/_task.h>
   37 
   38 /* 
   39  * This file includes definitions, structures, prototypes, and inlines that
   40  * should not be used outside of the actual implementation of UMA.
   41  */
   42 
   43 /* 
   44  * The brief summary;  Zones describe unique allocation types.  Zones are
   45  * organized into per-CPU caches which are filled by buckets.  Buckets are
   46  * organized according to memory domains.  Buckets are filled from kegs which
   47  * are also organized according to memory domains.  Kegs describe a unique
   48  * allocation type, backend memory provider, and layout.  Kegs are associated
   49  * with one or more zones and zones reference one or more kegs.  Kegs provide
   50  * slabs which are virtually contiguous collections of pages.  Each slab is
   51  * broken down int one or more items that will satisfy an individual allocation.
   52  *
   53  * Allocation is satisfied in the following order:
   54  * 1) Per-CPU cache
   55  * 2) Per-domain cache of buckets
   56  * 3) Slab from any of N kegs
   57  * 4) Backend page provider
   58  *
   59  * More detail on individual objects is contained below:
   60  *
   61  * Kegs contain lists of slabs which are stored in either the full bin, empty
   62  * bin, or partially allocated bin, to reduce fragmentation.  They also contain
   63  * the user supplied value for size, which is adjusted for alignment purposes
   64  * and rsize is the result of that.  The Keg also stores information for
   65  * managing a hash of page addresses that maps pages to uma_slab_t structures
   66  * for pages that don't have embedded uma_slab_t's.
   67  *
   68  * Keg slab lists are organized by memory domain to support NUMA allocation
   69  * policies.  By default allocations are spread across domains to reduce the
   70  * potential for hotspots.  Special keg creation flags may be specified to
   71  * prefer location allocation.  However there is no strict enforcement as frees
   72  * may happen on any CPU and these are returned to the CPU-local cache
   73  * regardless of the originating domain.
   74  *  
   75  * The uma_slab_t may be embedded in a UMA_SLAB_SIZE chunk of memory or it may
   76  * be allocated off the page from a special slab zone.  The free list within a
   77  * slab is managed with a bitmask.  For item sizes that would yield more than
   78  * 10% memory waste we potentially allocate a separate uma_slab_t if this will
   79  * improve the number of items per slab that will fit.  
   80  *
   81  * The only really gross cases, with regards to memory waste, are for those
   82  * items that are just over half the page size.   You can get nearly 50% waste,
   83  * so you fall back to the memory footprint of the power of two allocator. I
   84  * have looked at memory allocation sizes on many of the machines available to
   85  * me, and there does not seem to be an abundance of allocations at this range
   86  * so at this time it may not make sense to optimize for it.  This can, of 
   87  * course, be solved with dynamic slab sizes.
   88  *
   89  * Kegs may serve multiple Zones but by far most of the time they only serve
   90  * one.  When a Zone is created, a Keg is allocated and setup for it.  While
   91  * the backing Keg stores slabs, the Zone caches Buckets of items allocated
   92  * from the slabs.  Each Zone is equipped with an init/fini and ctor/dtor
   93  * pair, as well as with its own set of small per-CPU caches, layered above
   94  * the Zone's general Bucket cache.
   95  *
   96  * The PCPU caches are protected by critical sections, and may be accessed
   97  * safely only from their associated CPU, while the Zones backed by the same
   98  * Keg all share a common Keg lock (to coalesce contention on the backing
   99  * slabs).  The backing Keg typically only serves one Zone but in the case of
  100  * multiple Zones, one of the Zones is considered the Primary Zone and all
  101  * Zone-related stats from the Keg are done in the Primary Zone.  For an
  102  * example of a Multi-Zone setup, refer to the Mbuf allocation code.
  103  */
  104 
  105 /*
  106  *      This is the representation for normal (Non OFFPAGE slab)
  107  *
  108  *      i == item
  109  *      s == slab pointer
  110  *
  111  *      <----------------  Page (UMA_SLAB_SIZE) ------------------>
  112  *      ___________________________________________________________
  113  *     | _  _  _  _  _  _  _  _  _  _  _  _  _  _  _   ___________ |
  114  *     ||i||i||i||i||i||i||i||i||i||i||i||i||i||i||i| |slab header||
  115  *     ||_||_||_||_||_||_||_||_||_||_||_||_||_||_||_| |___________|| 
  116  *     |___________________________________________________________|
  117  *
  118  *
  119  *      This is an OFFPAGE slab. These can be larger than UMA_SLAB_SIZE.
  120  *
  121  *      ___________________________________________________________
  122  *     | _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _   |
  123  *     ||i||i||i||i||i||i||i||i||i||i||i||i||i||i||i||i||i||i||i|  |
  124  *     ||_||_||_||_||_||_||_||_||_||_||_||_||_||_||_||_||_||_||_|  |
  125  *     |___________________________________________________________|
  126  *       ___________    ^
  127  *      |slab header|   |
  128  *      |___________|---*
  129  *
  130  */
  131 
  132 #ifndef VM_UMA_INT_H
  133 #define VM_UMA_INT_H
  134 
  135 #define UMA_SLAB_SIZE   PAGE_SIZE       /* How big are our slabs? */
  136 #define UMA_SLAB_MASK   (PAGE_SIZE - 1) /* Mask to get back to the page */
  137 #define UMA_SLAB_SHIFT  PAGE_SHIFT      /* Number of bits PAGE_MASK */
  138 
  139 /* Max waste percentage before going to off page slab management */
  140 #define UMA_MAX_WASTE   10
  141 
  142 /* Max size of a CACHESPREAD slab. */
  143 #define UMA_CACHESPREAD_MAX_SIZE        (128 * 1024)
  144 
  145 /*
  146  * These flags must not overlap with the UMA_ZONE flags specified in uma.h.
  147  */
  148 #define UMA_ZFLAG_OFFPAGE       0x00200000      /*
  149                                                  * Force the slab structure
  150                                                  * allocation off of the real
  151                                                  * memory.
  152                                                  */
  153 #define UMA_ZFLAG_HASH          0x00400000      /*
  154                                                  * Use a hash table instead of
  155                                                  * caching information in the
  156                                                  * vm_page.
  157                                                  */
  158 #define UMA_ZFLAG_VTOSLAB       0x00800000      /*
  159                                                  * Zone uses vtoslab for
  160                                                  * lookup.
  161                                                  */
  162 #define UMA_ZFLAG_CTORDTOR      0x01000000      /* Zone has ctor/dtor set. */
  163 #define UMA_ZFLAG_LIMIT         0x02000000      /* Zone has limit set. */
  164 #define UMA_ZFLAG_CACHE         0x04000000      /* uma_zcache_create()d it */
  165 #define UMA_ZFLAG_RECLAIMING    0x08000000      /* Running zone_reclaim(). */
  166 #define UMA_ZFLAG_BUCKET        0x10000000      /* Bucket zone. */
  167 #define UMA_ZFLAG_INTERNAL      0x20000000      /* No offpage no PCPU. */
  168 #define UMA_ZFLAG_TRASH         0x40000000      /* Add trash ctor/dtor. */
  169 
  170 #define UMA_ZFLAG_INHERIT                                               \
  171     (UMA_ZFLAG_OFFPAGE | UMA_ZFLAG_HASH | UMA_ZFLAG_VTOSLAB |           \
  172      UMA_ZFLAG_BUCKET | UMA_ZFLAG_INTERNAL)
  173 
  174 #define PRINT_UMA_ZFLAGS        "\2"   \
  175     "\37TRASH"                          \
  176     "\36INTERNAL"                       \
  177     "\35BUCKET"                         \
  178     "\34RECLAIMING"                     \
  179     "\33CACHE"                          \
  180     "\32LIMIT"                          \
  181     "\31CTORDTOR"                       \
  182     "\30VTOSLAB"                        \
  183     "\27HASH"                           \
  184     "\26OFFPAGE"                        \
  185     "\23SMR"                            \
  186     "\22ROUNDROBIN"                     \
  187     "\21FIRSTTOUCH"                     \
  188     "\20PCPU"                           \
  189     "\17NODUMP"                         \
  190     "\16CACHESPREAD"                    \
  191     "\14MAXBUCKET"                      \
  192     "\13NOBUCKET"                       \
  193     "\12SECONDARY"                      \
  194     "\11NOTPAGE"                        \
  195     "\10VM"                             \
  196     "\7MTXCLASS"                        \
  197     "\6NOFREE"                          \
  198     "\5MALLOC"                          \
  199     "\4NOTOUCH"                         \
  200     "\3CONTIG"                          \
  201     "\2ZINIT"
  202 
  203 /*
  204  * Hash table for freed address -> slab translation.
  205  *
  206  * Only zones with memory not touchable by the allocator use the
  207  * hash table.  Otherwise slabs are found with vtoslab().
  208  */
  209 #define UMA_HASH_SIZE_INIT      32              
  210 
  211 #define UMA_HASH(h, s) ((((uintptr_t)s) >> UMA_SLAB_SHIFT) & (h)->uh_hashmask)
  212 
  213 #define UMA_HASH_INSERT(h, s, mem)                                      \
  214         LIST_INSERT_HEAD(&(h)->uh_slab_hash[UMA_HASH((h),               \
  215             (mem))], slab_tohashslab(s), uhs_hlink)
  216 
  217 #define UMA_HASH_REMOVE(h, s)                                           \
  218         LIST_REMOVE(slab_tohashslab(s), uhs_hlink)
  219 
  220 LIST_HEAD(slabhashhead, uma_hash_slab);
  221 
  222 struct uma_hash {
  223         struct slabhashhead     *uh_slab_hash;  /* Hash table for slabs */
  224         u_int           uh_hashsize;    /* Current size of the hash table */
  225         u_int           uh_hashmask;    /* Mask used during hashing */
  226 };
  227 
  228 /*
  229  * Align field or structure to cache 'sector' in intel terminology.  This
  230  * is more efficient with adjacent line prefetch.
  231  */
  232 #if defined(__amd64__) || defined(__powerpc64__)
  233 #define UMA_SUPER_ALIGN (CACHE_LINE_SIZE * 2)
  234 #else
  235 #define UMA_SUPER_ALIGN CACHE_LINE_SIZE
  236 #endif
  237 
  238 #define UMA_ALIGN       __aligned(UMA_SUPER_ALIGN)
  239 
  240 /*
  241  * The uma_bucket structure is used to queue and manage buckets divorced
  242  * from per-cpu caches.  They are loaded into uma_cache_bucket structures
  243  * for use.
  244  */
  245 struct uma_bucket {
  246         STAILQ_ENTRY(uma_bucket)        ub_link; /* Link into the zone */
  247         int16_t         ub_cnt;                 /* Count of items in bucket. */
  248         int16_t         ub_entries;             /* Max items. */
  249         smr_seq_t       ub_seq;                 /* SMR sequence number. */
  250         void            *ub_bucket[];           /* actual allocation storage */
  251 };
  252 
  253 typedef struct uma_bucket * uma_bucket_t;
  254 
  255 /*
  256  * The uma_cache_bucket structure is statically allocated on each per-cpu
  257  * cache.  Its use reduces branches and cache misses in the fast path.
  258  */
  259 struct uma_cache_bucket {
  260         uma_bucket_t    ucb_bucket;
  261         int16_t         ucb_cnt;
  262         int16_t         ucb_entries;
  263         uint32_t        ucb_spare;
  264 };
  265 
  266 typedef struct uma_cache_bucket * uma_cache_bucket_t;
  267 
  268 /*
  269  * The uma_cache structure is allocated for each cpu for every zone
  270  * type.  This optimizes synchronization out of the allocator fast path.
  271  */
  272 struct uma_cache {
  273         struct uma_cache_bucket uc_freebucket;  /* Bucket we're freeing to */
  274         struct uma_cache_bucket uc_allocbucket; /* Bucket to allocate from */
  275         struct uma_cache_bucket uc_crossbucket; /* cross domain bucket */
  276         uint64_t                uc_allocs;      /* Count of allocations */
  277         uint64_t                uc_frees;       /* Count of frees */
  278 } UMA_ALIGN;
  279 
  280 typedef struct uma_cache * uma_cache_t;
  281 
  282 LIST_HEAD(slabhead, uma_slab);
  283 
  284 /*
  285  * The cache structure pads perfectly into 64 bytes so we use spare
  286  * bits from the embedded cache buckets to store information from the zone
  287  * and keep all fast-path allocations accessing a single per-cpu line.
  288  */
  289 static inline void
  290 cache_set_uz_flags(uma_cache_t cache, uint32_t flags)
  291 {
  292 
  293         cache->uc_freebucket.ucb_spare = flags;
  294 }
  295 
  296 static inline void
  297 cache_set_uz_size(uma_cache_t cache, uint32_t size)
  298 {
  299 
  300         cache->uc_allocbucket.ucb_spare = size;
  301 }
  302 
  303 static inline uint32_t
  304 cache_uz_flags(uma_cache_t cache)
  305 {
  306 
  307         return (cache->uc_freebucket.ucb_spare);
  308 }
  309 
  310 static inline uint32_t
  311 cache_uz_size(uma_cache_t cache)
  312 {
  313 
  314         return (cache->uc_allocbucket.ucb_spare);
  315 }
  316 
  317 /*
  318  * Per-domain slab lists.  Embedded in the kegs.
  319  */
  320 struct uma_domain {
  321         struct mtx_padalign ud_lock;    /* Lock for the domain lists. */
  322         struct slabhead ud_part_slab;   /* partially allocated slabs */
  323         struct slabhead ud_free_slab;   /* completely unallocated slabs */
  324         struct slabhead ud_full_slab;   /* fully allocated slabs */
  325         uint32_t        ud_pages;       /* Total page count */
  326         uint32_t        ud_free_items;  /* Count of items free in all slabs */
  327         uint32_t        ud_free_slabs;  /* Count of free slabs */
  328 } __aligned(CACHE_LINE_SIZE);
  329 
  330 typedef struct uma_domain * uma_domain_t;
  331 
  332 /*
  333  * Keg management structure
  334  *
  335  * TODO: Optimize for cache line size
  336  *
  337  */
  338 struct uma_keg {
  339         struct uma_hash uk_hash;
  340         LIST_HEAD(,uma_zone)    uk_zones;       /* Keg's zones */
  341 
  342         struct domainset_ref uk_dr;     /* Domain selection policy. */
  343         uint32_t        uk_align;       /* Alignment mask */
  344         uint32_t        uk_reserve;     /* Number of reserved items. */
  345         uint32_t        uk_size;        /* Requested size of each item */
  346         uint32_t        uk_rsize;       /* Real size of each item */
  347 
  348         uma_init        uk_init;        /* Keg's init routine */
  349         uma_fini        uk_fini;        /* Keg's fini routine */
  350         uma_alloc       uk_allocf;      /* Allocation function */
  351         uma_free        uk_freef;       /* Free routine */
  352 
  353         u_long          uk_offset;      /* Next free offset from base KVA */
  354         vm_offset_t     uk_kva;         /* Zone base KVA */
  355 
  356         uint32_t        uk_pgoff;       /* Offset to uma_slab struct */
  357         uint16_t        uk_ppera;       /* pages per allocation from backend */
  358         uint16_t        uk_ipers;       /* Items per slab */
  359         uint32_t        uk_flags;       /* Internal flags */
  360 
  361         /* Least used fields go to the last cache line. */
  362         const char      *uk_name;               /* Name of creating zone. */
  363         LIST_ENTRY(uma_keg)     uk_link;        /* List of all kegs */
  364 
  365         /* Must be last, variable sized. */
  366         struct uma_domain       uk_domain[];    /* Keg's slab lists. */
  367 };
  368 typedef struct uma_keg  * uma_keg_t;
  369 
  370 /*
  371  * Free bits per-slab.
  372  */
  373 #define SLAB_MAX_SETSIZE        (PAGE_SIZE / UMA_SMALLEST_UNIT)
  374 #define SLAB_MIN_SETSIZE        _BITSET_BITS
  375 BITSET_DEFINE(noslabbits, 0);
  376 
  377 /*
  378  * The slab structure manages a single contiguous allocation from backing
  379  * store and subdivides it into individually allocatable items.
  380  */
  381 struct uma_slab {
  382         LIST_ENTRY(uma_slab)    us_link;        /* slabs in zone */
  383         uint16_t        us_freecount;           /* How many are free? */
  384         uint8_t         us_flags;               /* Page flags see uma.h */
  385         uint8_t         us_domain;              /* Backing NUMA domain. */
  386         struct noslabbits us_free;              /* Free bitmask, flexible. */
  387 };
  388 _Static_assert(sizeof(struct uma_slab) == __offsetof(struct uma_slab, us_free),
  389     "us_free field must be last");
  390 _Static_assert(MAXMEMDOM < 255,
  391     "us_domain field is not wide enough");
  392 
  393 typedef struct uma_slab * uma_slab_t;
  394 
  395 /*
  396  * Slab structure with a full sized bitset and hash link for both
  397  * HASH and OFFPAGE zones.
  398  */
  399 struct uma_hash_slab {
  400         LIST_ENTRY(uma_hash_slab) uhs_hlink;    /* Link for hash table */
  401         uint8_t                 *uhs_data;      /* First item */
  402         struct uma_slab         uhs_slab;       /* Must be last. */
  403 };
  404 
  405 typedef struct uma_hash_slab * uma_hash_slab_t;
  406 
  407 static inline uma_hash_slab_t
  408 slab_tohashslab(uma_slab_t slab)
  409 {
  410 
  411         return (__containerof(slab, struct uma_hash_slab, uhs_slab));
  412 }
  413 
  414 static inline void *
  415 slab_data(uma_slab_t slab, uma_keg_t keg)
  416 {
  417 
  418         if ((keg->uk_flags & UMA_ZFLAG_OFFPAGE) == 0)
  419                 return ((void *)((uintptr_t)slab - keg->uk_pgoff));
  420         else
  421                 return (slab_tohashslab(slab)->uhs_data);
  422 }
  423 
  424 static inline void *
  425 slab_item(uma_slab_t slab, uma_keg_t keg, int index)
  426 {
  427         uintptr_t data;
  428 
  429         data = (uintptr_t)slab_data(slab, keg);
  430         return ((void *)(data + keg->uk_rsize * index));
  431 }
  432 
  433 static inline int
  434 slab_item_index(uma_slab_t slab, uma_keg_t keg, void *item)
  435 {
  436         uintptr_t data;
  437 
  438         data = (uintptr_t)slab_data(slab, keg);
  439         return (((uintptr_t)item - data) / keg->uk_rsize);
  440 }
  441 
  442 STAILQ_HEAD(uma_bucketlist, uma_bucket);
  443 
  444 struct uma_zone_domain {
  445         struct uma_bucketlist uzd_buckets; /* full buckets */
  446         uma_bucket_t    uzd_cross;      /* Fills from cross buckets. */
  447         long            uzd_nitems;     /* total item count */
  448         long            uzd_imax;       /* maximum item count this period */
  449         long            uzd_imin;       /* minimum item count this period */
  450         long            uzd_wss;        /* working set size estimate */
  451         smr_seq_t       uzd_seq;        /* Lowest queued seq. */
  452         struct mtx      uzd_lock;       /* Lock for the domain */
  453 } __aligned(CACHE_LINE_SIZE);
  454 
  455 typedef struct uma_zone_domain * uma_zone_domain_t;
  456 
  457 /*
  458  * Zone structure - per memory type.
  459  */
  460 struct uma_zone {
  461         /* Offset 0, used in alloc/free fast/medium fast path and const. */
  462         uint32_t        uz_flags;       /* Flags inherited from kegs */
  463         uint32_t        uz_size;        /* Size inherited from kegs */
  464         uma_ctor        uz_ctor;        /* Constructor for each allocation */
  465         uma_dtor        uz_dtor;        /* Destructor */
  466         smr_t           uz_smr;         /* Safe memory reclaim context. */
  467         uint64_t        uz_max_items;   /* Maximum number of items to alloc */
  468         uint64_t        uz_bucket_max;  /* Maximum bucket cache size */
  469         uint16_t        uz_bucket_size; /* Number of items in full bucket */
  470         uint16_t        uz_bucket_size_max; /* Maximum number of bucket items */
  471         uint32_t        uz_sleepers;    /* Threads sleeping on limit */
  472         counter_u64_t   uz_xdomain;     /* Total number of cross-domain frees */
  473 
  474         /* Offset 64, used in bucket replenish. */
  475         uma_keg_t       uz_keg;         /* This zone's keg if !CACHE */
  476         uma_import      uz_import;      /* Import new memory to cache. */
  477         uma_release     uz_release;     /* Release memory from cache. */
  478         void            *uz_arg;        /* Import/release argument. */
  479         uma_init        uz_init;        /* Initializer for each item */
  480         uma_fini        uz_fini;        /* Finalizer for each item. */
  481         volatile uint64_t uz_items;     /* Total items count & sleepers */
  482         uint64_t        uz_sleeps;      /* Total number of alloc sleeps */
  483 
  484         /* Offset 128 Rare stats, misc read-only. */
  485         LIST_ENTRY(uma_zone) uz_link;   /* List of all zones in keg */
  486         counter_u64_t   uz_allocs;      /* Total number of allocations */
  487         counter_u64_t   uz_frees;       /* Total number of frees */
  488         counter_u64_t   uz_fails;       /* Total number of alloc failures */
  489         const char      *uz_name;       /* Text name of the zone */
  490         char            *uz_ctlname;    /* sysctl safe name string. */
  491         int             uz_namecnt;     /* duplicate name count. */
  492         uint16_t        uz_bucket_size_min; /* Min number of items in bucket */
  493         uint16_t        uz_pad0;
  494 
  495         /* Offset 192, rare read-only. */
  496         struct sysctl_oid *uz_oid;      /* sysctl oid pointer. */
  497         const char      *uz_warning;    /* Warning to print on failure */
  498         struct timeval  uz_ratecheck;   /* Warnings rate-limiting */
  499         struct task     uz_maxaction;   /* Task to run when at limit */
  500 
  501         /* Offset 256. */
  502         struct mtx      uz_cross_lock;  /* Cross domain free lock */
  503 
  504         /*
  505          * This HAS to be the last item because we adjust the zone size
  506          * based on NCPU and then allocate the space for the zones.
  507          */
  508         struct uma_cache        uz_cpu[]; /* Per cpu caches */
  509 
  510         /* domains follow here. */
  511 };
  512 
  513 /*
  514  * Macros for interpreting the uz_items field.  20 bits of sleeper count
  515  * and 44 bit of item count.
  516  */
  517 #define UZ_ITEMS_SLEEPER_SHIFT  44LL
  518 #define UZ_ITEMS_SLEEPERS_MAX   ((1 << (64 - UZ_ITEMS_SLEEPER_SHIFT)) - 1)
  519 #define UZ_ITEMS_COUNT_MASK     ((1LL << UZ_ITEMS_SLEEPER_SHIFT) - 1)
  520 #define UZ_ITEMS_COUNT(x)       ((x) & UZ_ITEMS_COUNT_MASK)
  521 #define UZ_ITEMS_SLEEPERS(x)    ((x) >> UZ_ITEMS_SLEEPER_SHIFT)
  522 #define UZ_ITEMS_SLEEPER        (1LL << UZ_ITEMS_SLEEPER_SHIFT)
  523 
  524 #define ZONE_ASSERT_COLD(z)                                             \
  525         KASSERT(uma_zone_get_allocs((z)) == 0,                          \
  526             ("zone %s initialization after use.", (z)->uz_name))
  527 
  528 /* Domains are contiguous after the last CPU */
  529 #define ZDOM_GET(z, n)                                                  \
  530         (&((uma_zone_domain_t)&(z)->uz_cpu[mp_maxid + 1])[n])
  531 
  532 #undef  UMA_ALIGN
  533 
  534 #ifdef _KERNEL
  535 /* Internal prototypes */
  536 static __inline uma_slab_t hash_sfind(struct uma_hash *hash, uint8_t *data);
  537 
  538 /* Lock Macros */
  539 
  540 #define KEG_LOCKPTR(k, d)       (struct mtx *)&(k)->uk_domain[(d)].ud_lock
  541 #define KEG_LOCK_INIT(k, d, lc)                                         \
  542         do {                                                            \
  543                 if ((lc))                                               \
  544                         mtx_init(KEG_LOCKPTR(k, d), (k)->uk_name,       \
  545                             (k)->uk_name, MTX_DEF | MTX_DUPOK);         \
  546                 else                                                    \
  547                         mtx_init(KEG_LOCKPTR(k, d), (k)->uk_name,       \
  548                             "UMA zone", MTX_DEF | MTX_DUPOK);           \
  549         } while (0)
  550 
  551 #define KEG_LOCK_FINI(k, d)     mtx_destroy(KEG_LOCKPTR(k, d))
  552 #define KEG_LOCK(k, d)                                                  \
  553         ({ mtx_lock(KEG_LOCKPTR(k, d)); KEG_LOCKPTR(k, d); })
  554 #define KEG_UNLOCK(k, d)        mtx_unlock(KEG_LOCKPTR(k, d))
  555 #define KEG_LOCK_ASSERT(k, d)   mtx_assert(KEG_LOCKPTR(k, d), MA_OWNED)
  556 
  557 #define KEG_GET(zone, keg) do {                                 \
  558         (keg) = (zone)->uz_keg;                                 \
  559         KASSERT((void *)(keg) != NULL,                          \
  560             ("%s: Invalid zone %p type", __func__, (zone)));    \
  561         } while (0)
  562 
  563 #define KEG_ASSERT_COLD(k)                                              \
  564         KASSERT(uma_keg_get_allocs((k)) == 0,                           \
  565             ("keg %s initialization after use.", (k)->uk_name))
  566 
  567 #define ZDOM_LOCK_INIT(z, zdom, lc)                                     \
  568         do {                                                            \
  569                 if ((lc))                                               \
  570                         mtx_init(&(zdom)->uzd_lock, (z)->uz_name,       \
  571                             (z)->uz_name, MTX_DEF | MTX_DUPOK);         \
  572                 else                                                    \
  573                         mtx_init(&(zdom)->uzd_lock, (z)->uz_name,       \
  574                             "UMA zone", MTX_DEF | MTX_DUPOK);           \
  575         } while (0)
  576 #define ZDOM_LOCK_FINI(z)       mtx_destroy(&(z)->uzd_lock)
  577 #define ZDOM_LOCK_ASSERT(z)     mtx_assert(&(z)->uzd_lock, MA_OWNED)
  578 
  579 #define ZDOM_LOCK(z)    mtx_lock(&(z)->uzd_lock)
  580 #define ZDOM_OWNED(z)   (mtx_owner(&(z)->uzd_lock) != NULL)
  581 #define ZDOM_UNLOCK(z)  mtx_unlock(&(z)->uzd_lock)
  582 
  583 #define ZONE_LOCK(z)    ZDOM_LOCK(ZDOM_GET((z), 0))
  584 #define ZONE_UNLOCK(z)  ZDOM_UNLOCK(ZDOM_GET((z), 0))
  585 
  586 #define ZONE_CROSS_LOCK_INIT(z)                                 \
  587         mtx_init(&(z)->uz_cross_lock, "UMA Cross", NULL, MTX_DEF)
  588 #define ZONE_CROSS_LOCK(z)      mtx_lock(&(z)->uz_cross_lock)
  589 #define ZONE_CROSS_UNLOCK(z)    mtx_unlock(&(z)->uz_cross_lock)
  590 #define ZONE_CROSS_LOCK_FINI(z) mtx_destroy(&(z)->uz_cross_lock)
  591 
  592 /*
  593  * Find a slab within a hash table.  This is used for OFFPAGE zones to lookup
  594  * the slab structure.
  595  *
  596  * Arguments:
  597  *      hash  The hash table to search.
  598  *      data  The base page of the item.
  599  *
  600  * Returns:
  601  *      A pointer to a slab if successful, else NULL.
  602  */
  603 static __inline uma_slab_t
  604 hash_sfind(struct uma_hash *hash, uint8_t *data)
  605 {
  606         uma_hash_slab_t slab;
  607         u_int hval;
  608 
  609         hval = UMA_HASH(hash, data);
  610 
  611         LIST_FOREACH(slab, &hash->uh_slab_hash[hval], uhs_hlink) {
  612                 if ((uint8_t *)slab->uhs_data == data)
  613                         return (&slab->uhs_slab);
  614         }
  615         return (NULL);
  616 }
  617 
  618 static __inline uma_slab_t
  619 vtoslab(vm_offset_t va)
  620 {
  621         vm_page_t p;
  622 
  623         p = PHYS_TO_VM_PAGE(pmap_kextract(va));
  624         return (p->plinks.uma.slab);
  625 }
  626 
  627 static __inline void
  628 vtozoneslab(vm_offset_t va, uma_zone_t *zone, uma_slab_t *slab)
  629 {
  630         vm_page_t p;
  631 
  632         p = PHYS_TO_VM_PAGE(pmap_kextract(va));
  633         *slab = p->plinks.uma.slab;
  634         *zone = p->plinks.uma.zone;
  635 }
  636 
  637 static __inline void
  638 vsetzoneslab(vm_offset_t va, uma_zone_t zone, uma_slab_t slab)
  639 {
  640         vm_page_t p;
  641 
  642         p = PHYS_TO_VM_PAGE(pmap_kextract(va));
  643         p->plinks.uma.slab = slab;
  644         p->plinks.uma.zone = zone;
  645 }
  646 
  647 extern unsigned long uma_kmem_limit;
  648 extern unsigned long uma_kmem_total;
  649 
  650 /* Adjust bytes under management by UMA. */
  651 static inline void
  652 uma_total_dec(unsigned long size)
  653 {
  654 
  655         atomic_subtract_long(&uma_kmem_total, size);
  656 }
  657 
  658 static inline void
  659 uma_total_inc(unsigned long size)
  660 {
  661 
  662         if (atomic_fetchadd_long(&uma_kmem_total, size) > uma_kmem_limit)
  663                 uma_reclaim_wakeup();
  664 }
  665 
  666 /*
  667  * The following two functions may be defined by architecture specific code
  668  * if they can provide more efficient allocation functions.  This is useful
  669  * for using direct mapped addresses.
  670  */
  671 void *uma_small_alloc(uma_zone_t zone, vm_size_t bytes, int domain,
  672     uint8_t *pflag, int wait);
  673 void uma_small_free(void *mem, vm_size_t size, uint8_t flags);
  674 
  675 /* Set a global soft limit on UMA managed memory. */
  676 void uma_set_limit(unsigned long limit);
  677 
  678 #endif /* _KERNEL */
  679 
  680 #endif /* VM_UMA_INT_H */

Cache object: 809fd8d06a76dba5256b8b94811f07c9


[ source navigation ] [ diff markup ] [ identifier search ] [ freetext search ] [ file search ] [ list types ] [ track identifier ]


This page is part of the FreeBSD/Linux Linux Kernel Cross-Reference, and was automatically generated using a modified version of the LXR engine.