Now available: The Design and Implementation of the FreeBSD Operating System (Second Edition)
[ source navigation ] [ diff markup ] [ identifier search ] [ freetext search ] [ file search ] [ list types ] [ track identifier ]
FreeBSD/Linux Kernel Cross Reference
sys/uvm/uvm_loan.c
Version:
- FREEBSD - FREEBSD-13-STABLE - FREEBSD-13-0 - FREEBSD-12-STABLE - FREEBSD-12-0 - FREEBSD-11-STABLE - FREEBSD-11-0 - FREEBSD-10-STABLE - FREEBSD-10-0 - FREEBSD-9-STABLE - FREEBSD-9-0 - FREEBSD-8-STABLE - FREEBSD-8-0 - FREEBSD-7-STABLE - FREEBSD-7-0 - FREEBSD-6-STABLE - FREEBSD-6-0 - FREEBSD-5-STABLE - FREEBSD-5-0 - FREEBSD-4-STABLE - FREEBSD-3-STABLE - FREEBSD22 - l41 - OPENBSD - linux-2.6 - MK84 - PLAN9 - xnu-8792
SearchContext: - none - 3 - 10
SearchContext: - none - 3 - 10
1 /* $NetBSD: uvm_loan.c,v 1.50 2004/03/24 07:50:48 junyoung Exp $ */ 2 3 /* 4 * 5 * Copyright (c) 1997 Charles D. Cranor and Washington University. 6 * All rights reserved. 7 * 8 * Redistribution and use in source and binary forms, with or without 9 * modification, are permitted provided that the following conditions 10 * are met: 11 * 1. Redistributions of source code must retain the above copyright 12 * notice, this list of conditions and the following disclaimer. 13 * 2. Redistributions in binary form must reproduce the above copyright 14 * notice, this list of conditions and the following disclaimer in the 15 * documentation and/or other materials provided with the distribution. 16 * 3. All advertising materials mentioning features or use of this software 17 * must display the following acknowledgement: 18 * This product includes software developed by Charles D. Cranor and 19 * Washington University. 20 * 4. The name of the author may not be used to endorse or promote products 21 * derived from this software without specific prior written permission. 22 * 23 * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR 24 * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES 25 * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. 26 * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, 27 * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT 28 * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 29 * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 30 * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 31 * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF 32 * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 33 * 34 * from: Id: uvm_loan.c,v 1.1.6.4 1998/02/06 05:08:43 chs Exp 35 */ 36 37 /* 38 * uvm_loan.c: page loanout handler 39 */ 40 41 #include <sys/cdefs.h> 42 __KERNEL_RCSID(0, "$NetBSD: uvm_loan.c,v 1.50 2004/03/24 07:50:48 junyoung Exp $"); 43 44 #include <sys/param.h> 45 #include <sys/systm.h> 46 #include <sys/kernel.h> 47 #include <sys/proc.h> 48 #include <sys/malloc.h> 49 #include <sys/mman.h> 50 51 #include <uvm/uvm.h> 52 53 /* 54 * "loaned" pages are pages which are (read-only, copy-on-write) loaned 55 * from the VM system to other parts of the kernel. this allows page 56 * copying to be avoided (e.g. you can loan pages from objs/anons to 57 * the mbuf system). 58 * 59 * there are 3 types of loans possible: 60 * O->K uvm_object page to wired kernel page (e.g. mbuf data area) 61 * A->K anon page to wired kernel page (e.g. mbuf data area) 62 * O->A uvm_object to anon loan (e.g. vnode page to an anon) 63 * note that it possible to have an O page loaned to both an A and K 64 * at the same time. 65 * 66 * loans are tracked by pg->loan_count. an O->A page will have both 67 * a uvm_object and a vm_anon, but PQ_ANON will not be set. this sort 68 * of page is considered "owned" by the uvm_object (not the anon). 69 * 70 * each loan of a page to the kernel bumps the pg->wire_count. the 71 * kernel mappings for these pages will be read-only and wired. since 72 * the page will also be wired, it will not be a candidate for pageout, 73 * and thus will never be pmap_page_protect()'d with VM_PROT_NONE. a 74 * write fault in the kernel to one of these pages will not cause 75 * copy-on-write. instead, the page fault is considered fatal. this 76 * is because the kernel mapping will have no way to look up the 77 * object/anon which the page is owned by. this is a good side-effect, 78 * since a kernel write to a loaned page is an error. 79 * 80 * owners that want to free their pages and discover that they are 81 * loaned out simply "disown" them (the page becomes an orphan). these 82 * pages should be freed when the last loan is dropped. in some cases 83 * an anon may "adopt" an orphaned page. 84 * 85 * locking: to read pg->loan_count either the owner or the page queues 86 * must be locked. to modify pg->loan_count, both the owner of the page 87 * and the PQs must be locked. pg->flags is (as always) locked by 88 * the owner of the page. 89 * 90 * note that locking from the "loaned" side is tricky since the object 91 * getting the loaned page has no reference to the page's owner and thus 92 * the owner could "die" at any time. in order to prevent the owner 93 * from dying the page queues should be locked. this forces us to sometimes 94 * use "try" locking. 95 * 96 * loans are typically broken by the following events: 97 * 1. user-level xwrite fault to a loaned page 98 * 2. pageout of clean+inactive O->A loaned page 99 * 3. owner frees page (e.g. pager flush) 100 * 101 * note that loaning a page causes all mappings of the page to become 102 * read-only (via pmap_page_protect). this could have an unexpected 103 * effect on normal "wired" pages if one is not careful (XXX). 104 */ 105 106 /* 107 * local prototypes 108 */ 109 110 static int uvm_loananon(struct uvm_faultinfo *, void ***, 111 int, struct vm_anon *); 112 static int uvm_loanentry(struct uvm_faultinfo *, void ***, int); 113 static int uvm_loanuobj(struct uvm_faultinfo *, void ***, 114 int, vaddr_t); 115 static int uvm_loanzero(struct uvm_faultinfo *, void ***, int); 116 static void uvm_unloananon(struct vm_anon **, int); 117 static void uvm_unloanpage(struct vm_page **, int); 118 static void uvm_loanpage(struct vm_page **, int); 119 120 121 /* 122 * inlines 123 */ 124 125 /* 126 * uvm_loanentry: loan out pages in a map entry (helper fn for uvm_loan()) 127 * 128 * => "ufi" is the result of a successful map lookup (meaning that 129 * on entry the map is locked by the caller) 130 * => we may unlock and then relock the map if needed (for I/O) 131 * => we put our output result in "output" 132 * => we always return with the map unlocked 133 * => possible return values: 134 * -1 == error, map is unlocked 135 * 0 == map relock error (try again!), map is unlocked 136 * >0 == number of pages we loaned, map is unlocked 137 */ 138 139 static __inline int 140 uvm_loanentry(ufi, output, flags) 141 struct uvm_faultinfo *ufi; 142 void ***output; 143 int flags; 144 { 145 vaddr_t curaddr = ufi->orig_rvaddr; 146 vsize_t togo = ufi->size; 147 struct vm_aref *aref = &ufi->entry->aref; 148 struct uvm_object *uobj = ufi->entry->object.uvm_obj; 149 struct vm_anon *anon; 150 int rv, result = 0; 151 152 /* 153 * lock us the rest of the way down (we unlock before return) 154 */ 155 if (aref->ar_amap) 156 amap_lock(aref->ar_amap); 157 if (uobj) 158 simple_lock(&uobj->vmobjlock); 159 160 /* 161 * loop until done 162 */ 163 while (togo) { 164 165 /* 166 * find the page we want. check the anon layer first. 167 */ 168 169 if (aref->ar_amap) { 170 anon = amap_lookup(aref, curaddr - ufi->entry->start); 171 } else { 172 anon = NULL; 173 } 174 175 /* locked: map, amap, uobj */ 176 if (anon) { 177 rv = uvm_loananon(ufi, output, flags, anon); 178 } else if (uobj) { 179 rv = uvm_loanuobj(ufi, output, flags, curaddr); 180 } else if (UVM_ET_ISCOPYONWRITE(ufi->entry)) { 181 rv = uvm_loanzero(ufi, output, flags); 182 } else { 183 uvmfault_unlockall(ufi, aref->ar_amap, uobj, NULL); 184 rv = -1; 185 } 186 /* locked: if (rv > 0) => map, amap, uobj [o.w. unlocked] */ 187 188 /* total failure */ 189 if (rv < 0) 190 return (-1); 191 192 /* relock failed, need to do another lookup */ 193 if (rv == 0) 194 return (result); 195 196 /* 197 * got it... advance to next page 198 */ 199 200 result++; 201 togo -= PAGE_SIZE; 202 curaddr += PAGE_SIZE; 203 } 204 205 /* 206 * unlock what we locked, unlock the maps and return 207 */ 208 209 if (aref->ar_amap) 210 amap_unlock(aref->ar_amap); 211 if (uobj) 212 simple_unlock(&uobj->vmobjlock); 213 uvmfault_unlockmaps(ufi, FALSE); 214 return (result); 215 } 216 217 /* 218 * normal functions 219 */ 220 221 /* 222 * uvm_loan: loan pages in a map out to anons or to the kernel 223 * 224 * => map should be unlocked 225 * => start and len should be multiples of PAGE_SIZE 226 * => result is either an array of anon's or vm_pages (depending on flags) 227 * => flag values: UVM_LOAN_TOANON - loan to anons 228 * UVM_LOAN_TOPAGE - loan to wired kernel page 229 * one and only one of these flags must be set! 230 * => returns 0 (success), or an appropriate error number 231 */ 232 233 int 234 uvm_loan(map, start, len, v, flags) 235 struct vm_map *map; 236 vaddr_t start; 237 vsize_t len; 238 void *v; 239 int flags; 240 { 241 struct uvm_faultinfo ufi; 242 void **result, **output; 243 int rv, error; 244 245 /* 246 * ensure that one and only one of the flags is set 247 */ 248 249 KASSERT(((flags & UVM_LOAN_TOANON) == 0) ^ 250 ((flags & UVM_LOAN_TOPAGE) == 0)); 251 KASSERT((map->flags & VM_MAP_INTRSAFE) == 0); 252 253 /* 254 * "output" is a pointer to the current place to put the loaned page. 255 */ 256 257 result = v; 258 output = &result[0]; /* start at the beginning ... */ 259 260 /* 261 * while we've got pages to do 262 */ 263 264 while (len > 0) { 265 266 /* 267 * fill in params for a call to uvmfault_lookup 268 */ 269 270 ufi.orig_map = map; 271 ufi.orig_rvaddr = start; 272 ufi.orig_size = len; 273 274 /* 275 * do the lookup, the only time this will fail is if we hit on 276 * an unmapped region (an error) 277 */ 278 279 if (!uvmfault_lookup(&ufi, FALSE)) { 280 error = ENOENT; 281 goto fail; 282 } 283 284 /* 285 * map now locked. now do the loanout... 286 */ 287 288 rv = uvm_loanentry(&ufi, &output, flags); 289 if (rv < 0) { 290 /* all unlocked due to error */ 291 error = EINVAL; 292 goto fail; 293 } 294 295 /* 296 * done! the map is unlocked. advance, if possible. 297 * 298 * XXXCDC: could be recoded to hold the map lock with 299 * smarter code (but it only happens on map entry 300 * boundaries, so it isn't that bad). 301 */ 302 303 if (rv) { 304 rv <<= PAGE_SHIFT; 305 len -= rv; 306 start += rv; 307 } 308 } 309 return 0; 310 311 fail: 312 /* 313 * failed to complete loans. drop any loans and return failure code. 314 * map is already unlocked. 315 */ 316 317 if (output - result) { 318 if (flags & UVM_LOAN_TOANON) { 319 uvm_unloananon((struct vm_anon **)result, 320 output - result); 321 } else { 322 uvm_unloanpage((struct vm_page **)result, 323 output - result); 324 } 325 } 326 return (error); 327 } 328 329 /* 330 * uvm_loananon: loan a page from an anon out 331 * 332 * => called with map, amap, uobj locked 333 * => return value: 334 * -1 = fatal error, everything is unlocked, abort. 335 * 0 = lookup in ufi went stale, everything unlocked, relookup and 336 * try again 337 * 1 = got it, everything still locked 338 */ 339 340 int 341 uvm_loananon(ufi, output, flags, anon) 342 struct uvm_faultinfo *ufi; 343 void ***output; 344 int flags; 345 struct vm_anon *anon; 346 { 347 struct vm_page *pg; 348 int error; 349 350 /* 351 * if we are loaning to "another" anon then it is easy, we just 352 * bump the reference count on the current anon and return a 353 * pointer to it (it becomes copy-on-write shared). 354 */ 355 356 if (flags & UVM_LOAN_TOANON) { 357 simple_lock(&anon->an_lock); 358 pg = anon->u.an_page; 359 if (pg && (pg->pqflags & PQ_ANON) != 0 && anon->an_ref == 1) { 360 pmap_page_protect(pg, VM_PROT_READ); 361 } 362 anon->an_ref++; 363 **output = anon; 364 (*output)++; 365 simple_unlock(&anon->an_lock); 366 return (1); 367 } 368 369 /* 370 * we are loaning to a kernel-page. we need to get the page 371 * resident so we can wire it. uvmfault_anonget will handle 372 * this for us. 373 */ 374 375 simple_lock(&anon->an_lock); 376 error = uvmfault_anonget(ufi, ufi->entry->aref.ar_amap, anon); 377 378 /* 379 * if we were unable to get the anon, then uvmfault_anonget has 380 * unlocked everything and returned an error code. 381 */ 382 383 if (error) { 384 385 /* need to refault (i.e. refresh our lookup) ? */ 386 if (error == ERESTART) { 387 return (0); 388 } 389 390 /* "try again"? sleep a bit and retry ... */ 391 if (error == EAGAIN) { 392 tsleep(&lbolt, PVM, "loanagain", 0); 393 return (0); 394 } 395 396 /* otherwise flag it as an error */ 397 return (-1); 398 } 399 400 /* 401 * we have the page and its owner locked: do the loan now. 402 */ 403 404 pg = anon->u.an_page; 405 uvm_lock_pageq(); 406 KASSERT(pg->wire_count == 0); 407 if (pg->loan_count == 0) { 408 pmap_page_protect(pg, VM_PROT_READ); 409 } 410 pg->loan_count++; 411 uvm_pagedequeue(pg); 412 uvm_unlock_pageq(); 413 **output = pg; 414 (*output)++; 415 416 /* unlock anon and return success */ 417 if (pg->uobject) /* XXXCDC: what if this is our uobj? bad */ 418 simple_unlock(&pg->uobject->vmobjlock); 419 simple_unlock(&anon->an_lock); 420 return (1); 421 } 422 423 /* 424 * uvm_loanpage: loan out pages to kernel (->K) 425 * 426 * => page's owner should be locked. 427 * => caller should own the pages. 428 */ 429 static void 430 uvm_loanpage(pgpp, npages) 431 struct vm_page **pgpp; 432 int npages; 433 { 434 int i; 435 436 for (i = 0; i < npages; i++) { 437 struct vm_page *pg = pgpp[i]; 438 439 KASSERT(pg->uobject != NULL); 440 KASSERT(!(pg->flags & (PG_RELEASED|PG_PAGEOUT))); 441 LOCK_ASSERT(simple_lock_held(&pg->uobject->vmobjlock)); 442 KASSERT(pg->flags & PG_BUSY); 443 444 uvm_lock_pageq(); 445 if (pg->loan_count == 0) { 446 pmap_page_protect(pg, VM_PROT_READ); 447 } 448 pg->loan_count++; 449 uvm_pagedequeue(pg); 450 uvm_unlock_pageq(); 451 if (pg->flags & PG_WANTED) { 452 wakeup(pg); 453 } 454 pg->flags &= ~(PG_WANTED|PG_BUSY); 455 UVM_PAGE_OWN(pg, NULL); 456 } 457 } 458 459 /* 460 * XXX UBC temp limit 461 * number of pages to get at once. 462 * should be <= MAX_READ_AHEAD in genfs_vnops.c 463 */ 464 #define UVM_LOAN_GET_CHUNK 16 465 466 /* 467 * uvm_loanuobjpages: loan pages from a uobj out (O->K) 468 * 469 * => uobj shouldn't be locked. (we'll lock it) 470 */ 471 int 472 uvm_loanuobjpages(uobj, pgoff, orignpages, origpgpp) 473 struct uvm_object *uobj; 474 voff_t pgoff; 475 int orignpages; 476 struct vm_page **origpgpp; 477 { 478 int ndone; 479 struct vm_page **pgpp; 480 int error; 481 int i; 482 struct simplelock *slock; 483 484 pgpp = origpgpp; 485 for (ndone = 0; ndone < orignpages; ) { 486 int npages; 487 int npendloan = 0xdead; /* XXX gcc */ 488 reget: 489 npages = MIN(UVM_LOAN_GET_CHUNK, orignpages - ndone); 490 simple_lock(&uobj->vmobjlock); 491 error = (*uobj->pgops->pgo_get)(uobj, 492 pgoff + (ndone << PAGE_SHIFT), pgpp, &npages, 0, 493 VM_PROT_READ, 0, PGO_SYNCIO); 494 if (error == EAGAIN) { 495 tsleep(&lbolt, PVM, "nfsread", 0); 496 continue; 497 } 498 if (error) { 499 uvm_unloan(origpgpp, ndone, UVM_LOAN_TOPAGE); 500 return error; 501 } 502 503 KASSERT(npages > 0); 504 505 /* loan and unbusy pages */ 506 slock = NULL; 507 for (i = 0; i < npages; i++) { 508 struct simplelock *nextslock; /* slock for next page */ 509 struct vm_page *pg = *pgpp; 510 511 /* XXX assuming that the page is owned by uobj */ 512 KASSERT(pg->uobject != NULL); 513 nextslock = &pg->uobject->vmobjlock; 514 515 if (slock != nextslock) { 516 if (slock) { 517 KASSERT(npendloan > 0); 518 uvm_loanpage(pgpp - npendloan, 519 npendloan); 520 simple_unlock(slock); 521 } 522 slock = nextslock; 523 simple_lock(slock); 524 npendloan = 0; 525 } 526 527 if (pg->flags & PG_RELEASED) { 528 /* 529 * release pages and try again. 530 */ 531 simple_unlock(slock); 532 for (; i < npages; i++) { 533 pg = pgpp[i]; 534 slock = &pg->uobject->vmobjlock; 535 536 simple_lock(slock); 537 uvm_lock_pageq(); 538 uvm_page_unbusy(&pg, 1); 539 uvm_unlock_pageq(); 540 simple_unlock(slock); 541 } 542 goto reget; 543 } 544 545 npendloan++; 546 pgpp++; 547 ndone++; 548 KASSERT(pgpp - origpgpp == ndone); 549 } 550 KASSERT(slock != NULL); 551 KASSERT(npendloan > 0); 552 uvm_loanpage(pgpp - npendloan, npendloan); 553 simple_unlock(slock); 554 } 555 556 return 0; 557 } 558 559 /* 560 * uvm_loanuobj: loan a page from a uobj out 561 * 562 * => called with map, amap, uobj locked 563 * => return value: 564 * -1 = fatal error, everything is unlocked, abort. 565 * 0 = lookup in ufi went stale, everything unlocked, relookup and 566 * try again 567 * 1 = got it, everything still locked 568 */ 569 570 static int 571 uvm_loanuobj(ufi, output, flags, va) 572 struct uvm_faultinfo *ufi; 573 void ***output; 574 int flags; 575 vaddr_t va; 576 { 577 struct vm_amap *amap = ufi->entry->aref.ar_amap; 578 struct uvm_object *uobj = ufi->entry->object.uvm_obj; 579 struct vm_page *pg; 580 struct vm_anon *anon; 581 int error, npages; 582 boolean_t locked; 583 584 /* 585 * first we must make sure the page is resident. 586 * 587 * XXXCDC: duplicate code with uvm_fault(). 588 */ 589 590 if (uobj->pgops->pgo_get) { /* try locked pgo_get */ 591 npages = 1; 592 pg = NULL; 593 error = (*uobj->pgops->pgo_get)(uobj, 594 va - ufi->entry->start + ufi->entry->offset, 595 &pg, &npages, 0, VM_PROT_READ, MADV_NORMAL, PGO_LOCKED); 596 } else { 597 error = EIO; /* must have pgo_get op */ 598 } 599 600 /* 601 * check the result of the locked pgo_get. if there is a problem, 602 * then we fail the loan. 603 */ 604 605 if (error && error != EBUSY) { 606 uvmfault_unlockall(ufi, amap, uobj, NULL); 607 return (-1); 608 } 609 610 /* 611 * if we need to unlock for I/O, do so now. 612 */ 613 614 if (error == EBUSY) { 615 uvmfault_unlockall(ufi, amap, NULL, NULL); 616 617 /* locked: uobj */ 618 npages = 1; 619 error = (*uobj->pgops->pgo_get)(uobj, 620 va - ufi->entry->start + ufi->entry->offset, 621 &pg, &npages, 0, VM_PROT_READ, MADV_NORMAL, PGO_SYNCIO); 622 /* locked: <nothing> */ 623 624 if (error) { 625 if (error == EAGAIN) { 626 tsleep(&lbolt, PVM, "fltagain2", 0); 627 return (0); 628 } 629 return (-1); 630 } 631 632 /* 633 * pgo_get was a success. attempt to relock everything. 634 */ 635 636 locked = uvmfault_relock(ufi); 637 if (locked && amap) 638 amap_lock(amap); 639 simple_lock(&uobj->vmobjlock); 640 641 /* 642 * verify that the page has not be released and re-verify 643 * that amap slot is still free. if there is a problem we 644 * drop our lock (thus force a lookup refresh/retry). 645 */ 646 647 if ((pg->flags & PG_RELEASED) != 0 || 648 (locked && amap && amap_lookup(&ufi->entry->aref, 649 ufi->orig_rvaddr - ufi->entry->start))) { 650 if (locked) 651 uvmfault_unlockall(ufi, amap, NULL, NULL); 652 locked = FALSE; 653 } 654 655 /* 656 * didn't get the lock? release the page and retry. 657 */ 658 659 if (locked == FALSE) { 660 if (pg->flags & PG_WANTED) { 661 wakeup(pg); 662 } 663 if (pg->flags & PG_RELEASED) { 664 uvm_lock_pageq(); 665 uvm_pagefree(pg); 666 uvm_unlock_pageq(); 667 return (0); 668 } 669 uvm_lock_pageq(); 670 uvm_pageactivate(pg); 671 uvm_unlock_pageq(); 672 pg->flags &= ~(PG_BUSY|PG_WANTED); 673 UVM_PAGE_OWN(pg, NULL); 674 simple_unlock(&uobj->vmobjlock); 675 return (0); 676 } 677 } 678 679 /* 680 * at this point we have the page we want ("pg") marked PG_BUSY for us 681 * and we have all data structures locked. do the loanout. page can 682 * not be PG_RELEASED (we caught this above). 683 */ 684 685 if ((flags & UVM_LOAN_TOANON) == 0) { 686 uvm_loanpage(&pg, 1); 687 **output = pg; 688 (*output)++; 689 return (1); 690 } 691 692 /* 693 * must be a loan to an anon. check to see if there is already 694 * an anon associated with this page. if so, then just return 695 * a reference to this object. the page should already be 696 * mapped read-only because it is already on loan. 697 */ 698 699 if (pg->uanon) { 700 anon = pg->uanon; 701 simple_lock(&anon->an_lock); 702 anon->an_ref++; 703 simple_unlock(&anon->an_lock); 704 if (pg->flags & PG_WANTED) { 705 wakeup(pg); 706 } 707 pg->flags &= ~(PG_WANTED|PG_BUSY); 708 UVM_PAGE_OWN(pg, NULL); 709 **output = anon; 710 (*output)++; 711 return (1); 712 } 713 714 /* 715 * need to allocate a new anon 716 */ 717 718 anon = uvm_analloc(); 719 if (anon == NULL) { 720 if (pg->flags & PG_WANTED) { 721 wakeup(pg); 722 } 723 pg->flags &= ~(PG_WANTED|PG_BUSY); 724 UVM_PAGE_OWN(pg, NULL); 725 uvmfault_unlockall(ufi, amap, uobj, NULL); 726 return (-1); 727 } 728 anon->u.an_page = pg; 729 pg->uanon = anon; 730 uvm_lock_pageq(); 731 if (pg->loan_count == 0) { 732 pmap_page_protect(pg, VM_PROT_READ); 733 } 734 pg->loan_count++; 735 uvm_pageactivate(pg); 736 uvm_unlock_pageq(); 737 if (pg->flags & PG_WANTED) { 738 wakeup(pg); 739 } 740 pg->flags &= ~(PG_WANTED|PG_BUSY); 741 UVM_PAGE_OWN(pg, NULL); 742 simple_unlock(&anon->an_lock); 743 **output = anon; 744 (*output)++; 745 return (1); 746 } 747 748 /* 749 * uvm_loanzero: loan a zero-fill page out 750 * 751 * => called with map, amap, uobj locked 752 * => return value: 753 * -1 = fatal error, everything is unlocked, abort. 754 * 0 = lookup in ufi went stale, everything unlocked, relookup and 755 * try again 756 * 1 = got it, everything still locked 757 */ 758 759 static struct uvm_object uvm_loanzero_object; 760 761 static int 762 uvm_loanzero(ufi, output, flags) 763 struct uvm_faultinfo *ufi; 764 void ***output; 765 int flags; 766 { 767 struct vm_anon *anon; 768 struct vm_page *pg; 769 struct uvm_object *uobj = ufi->entry->object.uvm_obj; 770 struct vm_amap *amap = ufi->entry->aref.ar_amap; 771 772 again: 773 simple_lock(&uvm_loanzero_object.vmobjlock); 774 775 /* 776 * first, get ahold of our single zero page. 777 */ 778 779 if (__predict_false((pg = 780 TAILQ_FIRST(&uvm_loanzero_object.memq)) == NULL)) { 781 while ((pg = uvm_pagealloc(&uvm_loanzero_object, 0, NULL, 782 UVM_PGA_ZERO)) == NULL) { 783 simple_unlock(&uvm_loanzero_object.vmobjlock); 784 uvmfault_unlockall(ufi, amap, uobj, NULL); 785 uvm_wait("loanzero"); 786 if (!uvmfault_relock(ufi)) { 787 return (0); 788 } 789 if (amap) { 790 amap_lock(amap); 791 } 792 if (uobj) { 793 simple_lock(&uobj->vmobjlock); 794 } 795 goto again; 796 } 797 798 /* got a zero'd page. */ 799 pg->flags &= ~(PG_WANTED|PG_BUSY|PG_FAKE); 800 pg->flags |= PG_RDONLY; 801 uvm_lock_pageq(); 802 uvm_pageactivate(pg); 803 uvm_unlock_pageq(); 804 UVM_PAGE_OWN(pg, NULL); 805 } 806 807 if ((flags & UVM_LOAN_TOANON) == 0) { /* loaning to kernel-page */ 808 uvm_lock_pageq(); 809 pg->loan_count++; 810 uvm_pagedequeue(pg); 811 uvm_unlock_pageq(); 812 simple_unlock(&uvm_loanzero_object.vmobjlock); 813 **output = pg; 814 (*output)++; 815 return (1); 816 } 817 818 /* 819 * loaning to an anon. check to see if there is already an anon 820 * associated with this page. if so, then just return a reference 821 * to this object. 822 */ 823 824 if (pg->uanon) { 825 anon = pg->uanon; 826 simple_lock(&anon->an_lock); 827 anon->an_ref++; 828 simple_unlock(&anon->an_lock); 829 simple_unlock(&uvm_loanzero_object.vmobjlock); 830 **output = anon; 831 (*output)++; 832 return (1); 833 } 834 835 /* 836 * need to allocate a new anon 837 */ 838 839 anon = uvm_analloc(); 840 if (anon == NULL) { 841 /* out of swap causes us to fail */ 842 simple_unlock(&uvm_loanzero_object.vmobjlock); 843 uvmfault_unlockall(ufi, amap, uobj, NULL); 844 return (-1); 845 } 846 anon->u.an_page = pg; 847 pg->uanon = anon; 848 uvm_lock_pageq(); 849 pg->loan_count++; 850 uvm_pageactivate(pg); 851 uvm_unlock_pageq(); 852 simple_unlock(&uvm_loanzero_object.vmobjlock); 853 **output = anon; 854 (*output)++; 855 return (1); 856 } 857 858 859 /* 860 * uvm_unloananon: kill loans on anons (basically a normal ref drop) 861 * 862 * => we expect all our resources to be unlocked 863 */ 864 865 static void 866 uvm_unloananon(aloans, nanons) 867 struct vm_anon **aloans; 868 int nanons; 869 { 870 struct vm_anon *anon; 871 872 while (nanons-- > 0) { 873 int refs; 874 875 anon = *aloans++; 876 simple_lock(&anon->an_lock); 877 refs = --anon->an_ref; 878 simple_unlock(&anon->an_lock); 879 880 if (refs == 0) { 881 uvm_anfree(anon); 882 } 883 } 884 } 885 886 /* 887 * uvm_unloanpage: kill loans on pages loaned out to the kernel 888 * 889 * => we expect all our resources to be unlocked 890 */ 891 892 static void 893 uvm_unloanpage(ploans, npages) 894 struct vm_page **ploans; 895 int npages; 896 { 897 struct vm_page *pg; 898 struct simplelock *slock; 899 900 uvm_lock_pageq(); 901 while (npages-- > 0) { 902 pg = *ploans++; 903 904 /* 905 * do a little dance to acquire the object or anon lock 906 * as appropriate. we are locking in the wrong order, 907 * so we have to do a try-lock here. 908 */ 909 910 slock = NULL; 911 while (pg->uobject != NULL || pg->uanon != NULL) { 912 if (pg->uobject != NULL) { 913 slock = &pg->uobject->vmobjlock; 914 } else { 915 slock = &pg->uanon->an_lock; 916 } 917 if (simple_lock_try(slock)) { 918 break; 919 } 920 uvm_unlock_pageq(); 921 uvm_lock_pageq(); 922 slock = NULL; 923 } 924 925 /* 926 * drop our loan. if page is owned by an anon but 927 * PQ_ANON is not set, the page was loaned to the anon 928 * from an object which dropped ownership, so resolve 929 * this by turning the anon's loan into real ownership 930 * (ie. decrement loan_count again and set PQ_ANON). 931 * after all this, if there are no loans left, put the 932 * page back a paging queue (if the page is owned by 933 * an anon) or free it (if the page is now unowned). 934 */ 935 936 KASSERT(pg->loan_count > 0); 937 pg->loan_count--; 938 if (pg->uobject == NULL && pg->uanon != NULL && 939 (pg->pqflags & PQ_ANON) == 0) { 940 KASSERT(pg->loan_count > 0); 941 pg->loan_count--; 942 pg->pqflags |= PQ_ANON; 943 } 944 if (pg->loan_count == 0) { 945 if (pg->uobject == NULL && pg->uanon == NULL) { 946 KASSERT((pg->flags & PG_BUSY) == 0); 947 uvm_pagefree(pg); 948 } else { 949 uvm_pageactivate(pg); 950 } 951 } else if (pg->loan_count == 1 && pg->uobject != NULL && 952 pg->uanon != NULL) { 953 uvm_pageactivate(pg); 954 } 955 if (slock != NULL) { 956 simple_unlock(slock); 957 } 958 } 959 uvm_unlock_pageq(); 960 } 961 962 /* 963 * uvm_unloan: kill loans on pages or anons. 964 */ 965 966 void 967 uvm_unloan(void *v, int npages, int flags) 968 { 969 if (flags & UVM_LOAN_TOANON) { 970 uvm_unloananon(v, npages); 971 } else { 972 uvm_unloanpage(v, npages); 973 } 974 } 975 976 /* 977 * Minimal pager for uvm_loanzero_object. We need to provide a "put" 978 * method, because the page can end up on a paging queue, and the 979 * page daemon will want to call pgo_put when it encounters the page 980 * on the inactive list. 981 */ 982 983 static int 984 ulz_put(struct uvm_object *uobj, voff_t start, voff_t stop, int flags) 985 { 986 struct vm_page *pg; 987 988 KDASSERT(uobj == &uvm_loanzero_object); 989 990 /* 991 * Don't need to do any work here if we're not freeing pages. 992 */ 993 994 if ((flags & PGO_FREE) == 0) { 995 simple_unlock(&uobj->vmobjlock); 996 return 0; 997 } 998 999 /* 1000 * we don't actually want to ever free the uvm_loanzero_page, so 1001 * just reactivate or dequeue it. 1002 */ 1003 1004 pg = TAILQ_FIRST(&uobj->memq); 1005 KASSERT(pg != NULL); 1006 KASSERT(TAILQ_NEXT(pg, listq) == NULL); 1007 1008 uvm_lock_pageq(); 1009 if (pg->uanon) 1010 uvm_pageactivate(pg); 1011 else 1012 uvm_pagedequeue(pg); 1013 uvm_unlock_pageq(); 1014 1015 simple_unlock(&uobj->vmobjlock); 1016 return 0; 1017 } 1018 1019 static struct uvm_pagerops ulz_pager = { 1020 NULL, /* init */ 1021 NULL, /* reference */ 1022 NULL, /* detach */ 1023 NULL, /* fault */ 1024 NULL, /* get */ 1025 ulz_put, /* put */ 1026 }; 1027 1028 /* 1029 * uvm_loan_init(): initialize the uvm_loan() facility. 1030 */ 1031 1032 void 1033 uvm_loan_init(void) 1034 { 1035 1036 simple_lock_init(&uvm_loanzero_object.vmobjlock); 1037 TAILQ_INIT(&uvm_loanzero_object.memq); 1038 uvm_loanzero_object.pgops = &ulz_pager; 1039 } 1040 1041 /* 1042 * uvm_loanbreak: break loan on a uobj page 1043 * 1044 * => called with uobj locked 1045 * => the page should be busy 1046 * => return value: 1047 * newly allocated page if succeeded 1048 */ 1049 struct vm_page * 1050 uvm_loanbreak(struct vm_page *uobjpage) 1051 { 1052 struct vm_page *pg; 1053 #ifdef DIAGNOSTIC 1054 struct uvm_object *uobj = uobjpage->uobject; 1055 #endif 1056 1057 KASSERT(uobj != NULL); 1058 LOCK_ASSERT(simple_lock_held(&uobj->vmobjlock)); 1059 KASSERT(uobjpage->flags & PG_BUSY); 1060 1061 /* alloc new un-owned page */ 1062 pg = uvm_pagealloc(NULL, 0, NULL, 0); 1063 if (pg == NULL) 1064 return NULL; 1065 1066 /* 1067 * copy the data from the old page to the new 1068 * one and clear the fake/clean flags on the 1069 * new page (keep it busy). force a reload 1070 * of the old page by clearing it from all 1071 * pmaps. then lock the page queues to 1072 * rename the pages. 1073 */ 1074 1075 uvm_pagecopy(uobjpage, pg); /* old -> new */ 1076 pg->flags &= ~(PG_FAKE|PG_CLEAN); 1077 pmap_page_protect(uobjpage, VM_PROT_NONE); 1078 if (uobjpage->flags & PG_WANTED) 1079 wakeup(uobjpage); 1080 /* uobj still locked */ 1081 uobjpage->flags &= ~(PG_WANTED|PG_BUSY); 1082 UVM_PAGE_OWN(uobjpage, NULL); 1083 1084 uvm_lock_pageq(); 1085 1086 /* 1087 * replace uobjpage with new page. 1088 */ 1089 1090 uvm_pagereplace(uobjpage, pg); 1091 1092 /* 1093 * if the page is no longer referenced by 1094 * an anon (i.e. we are breaking an O->K 1095 * loan), then remove it from any pageq's. 1096 */ 1097 if (uobjpage->uanon == NULL) 1098 uvm_pagedequeue(uobjpage); 1099 1100 /* 1101 * at this point we have absolutely no 1102 * control over uobjpage 1103 */ 1104 1105 /* install new page */ 1106 uvm_pageactivate(pg); 1107 uvm_unlock_pageq(); 1108 1109 /* 1110 * done! loan is broken and "pg" is 1111 * PG_BUSY. it can now replace uobjpage. 1112 */ 1113 1114 return pg; 1115 }
Cache object: 1c86ab7753754939e13ced878ff5b5d5
[ source navigation ] [ diff markup ] [ identifier search ] [ freetext search ] [ file search ] [ list types ] [ track identifier ]