Now available: The Design and Implementation of the FreeBSD Operating System (Second Edition)
[ source navigation ] [ diff markup ] [ identifier search ] [ freetext search ] [ file search ] [ list types ] [ track identifier ]
FreeBSD/Linux Kernel Cross Reference
sys/vm/memory_object.c
Version:
- FREEBSD - FREEBSD-13-STABLE - FREEBSD-13-0 - FREEBSD-12-STABLE - FREEBSD-12-0 - FREEBSD-11-STABLE - FREEBSD-11-0 - FREEBSD-10-STABLE - FREEBSD-10-0 - FREEBSD-9-STABLE - FREEBSD-9-0 - FREEBSD-8-STABLE - FREEBSD-8-0 - FREEBSD-7-STABLE - FREEBSD-7-0 - FREEBSD-6-STABLE - FREEBSD-6-0 - FREEBSD-5-STABLE - FREEBSD-5-0 - FREEBSD-4-STABLE - FREEBSD-3-STABLE - FREEBSD22 - l41 - OPENBSD - linux-2.6 - MK84 - PLAN9 - xnu-8792
SearchContext: - none - 3 - 10
SearchContext: - none - 3 - 10
1 /* 2 * Mach Operating System 3 * Copyright (c) 1992,1991,1990,1989,1988,1987 Carnegie Mellon University 4 * All Rights Reserved. 5 * 6 * Permission to use, copy, modify and distribute this software and its 7 * documentation is hereby granted, provided that both the copyright 8 * notice and this permission notice appear in all copies of the 9 * software, derivative works or modified versions, and any portions 10 * thereof, and that both notices appear in supporting documentation. 11 * 12 * CARNEGIE MELLON ALLOWS FREE USE OF THIS SOFTWARE IN ITS "AS IS" 13 * CONDITION. CARNEGIE MELLON DISCLAIMS ANY LIABILITY OF ANY KIND FOR 14 * ANY DAMAGES WHATSOEVER RESULTING FROM THE USE OF THIS SOFTWARE. 15 * 16 * Carnegie Mellon requests users of this software to return to 17 * 18 * Software Distribution Coordinator or Software.Distribution@CS.CMU.EDU 19 * School of Computer Science 20 * Carnegie Mellon University 21 * Pittsburgh PA 15213-3890 22 * 23 * any improvements or extensions that they make and grant Carnegie Mellon 24 * the rights to redistribute these changes. 25 */ 26 /* 27 * HISTORY 28 * $Log: memory_object.c,v $ 29 * Revision 2.31 93/01/14 18:00:42 danner 30 * Removed unneeded cast in argument to thread_wakeup/assert_wait. 31 * [92/12/30 dbg] 32 * 33 * Unlock object around call to vm_map_copy_invoke_cont in 34 * memory_object_data_supply. 35 * [92/09/22 dbg] 36 * 64bit cleanup. 37 * [92/12/01 af] 38 * 39 * Unlock object around call to vm_map_copy_invoke_cont in 40 * memory_object_data_supply. 41 * [92/09/22 dbg] 42 * 43 * Revision 2.30 92/08/03 17:59:55 jfriedl 44 * removed silly prototypes 45 * [92/08/02 jfriedl] 46 * 47 * Revision 2.29 92/05/21 17:25:17 jfriedl 48 * Added stuff to quiet gcc warnings. 49 * [92/05/16 jfriedl] 50 * 51 * Revision 2.28 92/03/10 16:29:58 jsb 52 * Add MEMORY_OBJECT_COPY_TEMPORARY case to 53 * memory_object_set_attributes_common. 54 * [92/03/06 16:57:38 jsb] 55 * 56 * NORMA_VM: don't define cover and backwards compatibility routines. 57 * [92/03/06 16:54:53 jsb] 58 * 59 * [David L. Black 92/02/22 17:05:17 dlb@osf.org] 60 * Implement no change of page lock functionality in 61 * memory_object_lock_request. 62 * 63 * Revision 2.27 92/02/23 19:50:36 elf 64 * Bug fix to avoid duplicate vm_map_copy_t deallocate in 65 * error case of memory_object_data_supply. 66 * [92/02/19 17:38:17 dlb] 67 * 68 * Revision 2.25.2.2 92/02/18 19:20:01 jeffreyh 69 * Cleaned up incorrect comment 70 * [92/02/18 jeffreyh] 71 * 72 * Revision 2.25.2.1 92/01/21 21:55:10 jsb 73 * Created memory_object_set_attributes_common which provides 74 * functionality common to memory_object_set_attributes, 75 * memory_object_ready, and memory_object_change_attributes. 76 * Fixed memory_object_change_attributes to set use_old_pageout 77 * false instead of true. 78 * [92/01/21 18:40:20 jsb] 79 * 80 * Revision 2.26 92/01/23 15:21:33 rpd 81 * Fixed memory_object_change_attributes. 82 * Created memory_object_set_attributes_common. 83 * [92/01/18 rpd] 84 * 85 * Revision 2.25 91/10/09 16:19:18 af 86 * Fixed assertion in memory_object_data_supply. 87 * [91/10/06 rpd] 88 * 89 * Added vm_page_deactivate_hint. 90 * [91/09/29 rpd] 91 * 92 * Revision 2.24 91/08/28 11:17:50 jsb 93 * Continuation bug fix. 94 * [91/08/05 17:42:57 dlb] 95 * 96 * Add vm_map_copy continuation support to memory_object_data_supply. 97 * [91/07/30 14:13:59 dlb] 98 * 99 * Turn on page lists by default: gut and remove body of memory_object_ 100 * data_provided -- it's now a wrapper for memory_object_data_supply. 101 * Precious page support: 102 * Extensive modifications to to memory_object_lock_{request,page} 103 * Implement old version wrapper xxx_memory_object_lock_request 104 * Maintain use_old_pageout field in vm objects. 105 * Add memory_object_ready, memory_object_change_attributes. 106 * [91/07/03 14:11:35 dlb] 107 * 108 * Revision 2.23 91/08/03 18:19:51 jsb 109 * For now, use memory_object_data_supply iff NORMA_IPC. 110 * [91/07/04 13:14:50 jsb] 111 * 112 * Revision 2.22 91/07/31 18:20:56 dbg 113 * Removed explicit data_dealloc argument to 114 * memory_object_data_supply. MiG now handles a user-specified 115 * dealloc flag. 116 * [91/07/29 dbg] 117 * 118 * Revision 2.21 91/07/01 09:17:25 jsb 119 * Fixed remaining MACH_PORT references. 120 * 121 * Revision 2.20 91/07/01 08:31:54 jsb 122 * Changed mach_port_t to ipc_port_t in memory_object_data_supply. 123 * 124 * Revision 2.19 91/07/01 08:26:53 jsb 125 * 21-Jun-91 David L. Black (dlb) at Open Software Foundation 126 * Add memory_object_data_supply. 127 * [91/06/29 16:36:43 jsb] 128 * 129 * Revision 2.18 91/06/25 11:06:45 rpd 130 * Fixed includes to avoid norma files unless they are really needed. 131 * [91/06/25 rpd] 132 * 133 * Revision 2.17 91/06/25 10:33:04 rpd 134 * Changed memory_object_t to ipc_port_t where appropriate. 135 * [91/05/28 rpd] 136 * 137 * Revision 2.16 91/06/17 15:48:55 jsb 138 * NORMA_VM: include xmm_server_rename.h, for interposition. 139 * [91/06/17 11:09:52 jsb] 140 * 141 * Revision 2.15 91/05/18 14:39:33 rpd 142 * Fixed memory_object_lock_page to handle fictitious pages. 143 * [91/04/06 rpd] 144 * Changed memory_object_data_provided, etc, 145 * to allow for fictitious pages. 146 * [91/03/29 rpd] 147 * Added vm/memory_object.h. 148 * [91/03/22 rpd] 149 * 150 * Revision 2.14 91/05/14 17:47:54 mrt 151 * Correcting copyright 152 * 153 * Revision 2.13 91/03/16 15:04:30 rpd 154 * Removed the old version of memory_object_data_provided. 155 * [91/03/11 rpd] 156 * Fixed memory_object_data_provided to return success 157 * iff it consumes the copy object. 158 * [91/02/10 rpd] 159 * 160 * Revision 2.12 91/02/05 17:57:25 mrt 161 * Changed to new Mach copyright 162 * [91/02/01 16:30:46 mrt] 163 * 164 * Revision 2.11 91/01/08 16:44:11 rpd 165 * Added continuation argument to thread_block. 166 * [90/12/08 rpd] 167 * 168 * Revision 2.10 90/10/25 14:49:30 rwd 169 * Clean and not flush pages to lock request get moved to the 170 * inactive queue. 171 * [90/10/24 rwd] 172 * 173 * Revision 2.9 90/08/27 22:15:49 dbg 174 * Fix error in initial assumptions: vm_pageout_setup must take a 175 * BUSY page, to prevent the page from being scrambled by pagein. 176 * [90/07/26 dbg] 177 * 178 * Revision 2.8 90/08/06 15:08:16 rwd 179 * Fix locking problems in memory_object_lock_request. 180 * [90/07/12 rwd] 181 * Fix memory_object_lock_request to only send contiguous pages as 182 * one message. If dirty pages were seperated by absent pages, 183 * then the wrong thing was done. 184 * [90/07/11 rwd] 185 * 186 * Revision 2.7 90/06/19 23:01:38 rpd 187 * Bring old single_page version of memory_object_data_provided up 188 * to date. 189 * [90/06/05 dbg] 190 * 191 * Correct object locking in memory_object_lock_request. 192 * [90/06/05 dbg] 193 * 194 * Revision 2.6 90/06/02 15:10:14 rpd 195 * Changed memory_object_lock_request/memory_object_lock_completed calls 196 * to allow both send and send-once right reply-to ports. 197 * [90/05/31 rpd] 198 * 199 * Added memory_manager_default_port. 200 * [90/04/29 rpd] 201 * Converted to new IPC. Purged MACH_XP_FPD. 202 * [90/03/26 23:11:14 rpd] 203 * 204 * Revision 2.5 90/05/29 18:38:29 rwd 205 * New memory_object_lock_request from dbg. 206 * [90/05/18 13:04:36 rwd] 207 * 208 * Picked up rfr MACH_PAGEMAP changes. 209 * [90/04/12 13:45:43 rwd] 210 * 211 * Revision 2.4 90/05/03 15:58:23 dbg 212 * Pass should_flush to vm_pageout_page: don't flush page if not 213 * requested. 214 * [90/03/28 dbg] 215 * 216 * Revision 2.3 90/02/22 20:05:10 dbg 217 * Pick up changes from mainline: 218 * 219 * Fix placeholder page handling in memory_object_data_provided. 220 * Old code was calling zalloc while holding a lock. 221 * [89/12/13 19:58:28 dlb] 222 * 223 * Don't clear busy flags on any pages in memory_object_lock_page 224 * (from memory_object_lock_request)!! Implemented by changing 225 * PAGE_WAKEUP to not clear busy flag and using PAGE_WAKEUP_DONE 226 * when it must be cleared. See vm/vm_page.h. With dbg's help. 227 * [89/12/13 dlb] 228 * 229 * Don't activate fictitious pages after freeing them in 230 * memory_object_data_{unavailable,error}. Add missing lock and 231 * unlock of page queues when activating pages in same routines. 232 * [89/12/11 dlb] 233 * Retry lookup after using CLEAN_DIRTY_PAGES in 234 * memory_object_lock_request(). Also delete old version of 235 * memory_object_data_provided(). From mwyoung. 236 * [89/11/17 dlb] 237 * 238 * Save all page-cleaning operations until it becomes necessary 239 * to block in memory_object_lock_request(). 240 * [89/09/30 18:07:16 mwyoung] 241 * 242 * Split out a per-page routine for lock requests. 243 * [89/08/20 19:47:42 mwyoung] 244 * 245 * Verify that the user memory used in 246 * memory_object_data_provided() is valid, even if it won't 247 * be used to fill a page request. 248 * [89/08/01 14:58:21 mwyoung] 249 * 250 * Make memory_object_data_provided atomic, interruptible, 251 * and serializable when handling multiple pages. Add 252 * optimization for single-page operations. 253 * [89/05/12 16:06:13 mwyoung] 254 * 255 * Simplify lock/clean/flush sequences memory_object_lock_request. 256 * Correct error in call to pmap_page_protect() there. 257 * Make error/absent pages eligible for pageout. 258 * [89/04/22 mwyoung] 259 * 260 * Revision 2.2 89/09/08 11:28:10 dbg 261 * Pass keep_wired argument to vm_move. Disabled 262 * host_set_memory_object_default. 263 * [89/07/14 dbg] 264 * 265 * 28-Apr-89 David Golub (dbg) at Carnegie-Mellon University 266 * Clean up fast_pager_data option. Remove pager_data_provided_inline. 267 * 268 * Revision 2.18 89/04/23 13:25:30 gm0w 269 * Fixed typo to pmap_page_protect in memory_object_lock_request(). 270 * [89/04/23 gm0w] 271 * 272 * Revision 2.17 89/04/22 15:35:09 gm0w 273 * Commented out check/uprintf if memory_object_data_unavailable 274 * was called on a permanent object. 275 * [89/04/14 gm0w] 276 * 277 * Revision 2.16 89/04/18 21:24:24 mwyoung 278 * Recent history: 279 * Add vm_set_default_memory_manager(), 280 * memory_object_get_attributes(). 281 * Whenever asked to clean a page, use pmap_is_modified, even 282 * if not flushing the data. 283 * Handle fictitious pages when accepting data (or error or 284 * unavailable). 285 * Avoid waiting in memory_object_data_error(). 286 * 287 * Previous history has been integrated into the documentation below. 288 * [89/04/18 mwyoung] 289 * 290 */ 291 /* 292 * File: vm/memory_object.c 293 * Author: Michael Wayne Young 294 * 295 * External memory management interface control functions. 296 */ 297 298 /* 299 * Interface dependencies: 300 */ 301 302 #include <mach/std_types.h> /* For pointer_t */ 303 #include <mach/mach_types.h> 304 305 #include <mach/kern_return.h> 306 #include <vm/vm_object.h> 307 #include <mach/memory_object.h> 308 #include <mach/memory_object_user.h> 309 #include <mach/memory_object_default.h> 310 #include <mach/boolean.h> 311 #include <mach/vm_prot.h> 312 #include <mach/message.h> 313 314 /* 315 * Implementation dependencies: 316 */ 317 #include <vm/memory_object.h> 318 #include <vm/vm_page.h> 319 #include <vm/vm_pageout.h> 320 #include <vm/pmap.h> /* For copy_to_phys, pmap_clear_modify */ 321 #include <kern/thread.h> /* For current_thread() */ 322 #include <kern/host.h> 323 #include <vm/vm_kern.h> /* For kernel_map, vm_move */ 324 #include <vm/vm_map.h> /* For vm_map_pageable */ 325 #include <ipc/ipc_port.h> 326 327 #include <norma_vm.h> 328 #include <norma_ipc.h> 329 #if NORMA_VM 330 #include <norma/xmm_server_rename.h> 331 #endif NORMA_VM 332 #include <mach_pagemap.h> 333 #if MACH_PAGEMAP 334 #include <vm/vm_external.h> 335 #endif MACH_PAGEMAP 336 337 typedef int memory_object_lock_result_t; /* moved from below */ 338 339 340 ipc_port_t memory_manager_default = IP_NULL; 341 decl_simple_lock_data(,memory_manager_default_lock) 342 343 /* 344 * Important note: 345 * All of these routines gain a reference to the 346 * object (first argument) as part of the automatic 347 * argument conversion. Explicit deallocation is necessary. 348 */ 349 350 #if !NORMA_VM 351 /* 352 * If successful, destroys the map copy object. 353 */ 354 kern_return_t memory_object_data_provided(object, offset, data, data_cnt, 355 lock_value) 356 vm_object_t object; 357 vm_offset_t offset; 358 pointer_t data; 359 unsigned int data_cnt; 360 vm_prot_t lock_value; 361 { 362 return memory_object_data_supply(object, offset, (vm_map_copy_t) data, 363 data_cnt, lock_value, FALSE, IP_NULL, 364 0); 365 } 366 #endif !NORMA_VM 367 368 369 kern_return_t memory_object_data_supply(object, offset, data_copy, data_cnt, 370 lock_value, precious, reply_to, reply_to_type) 371 register 372 vm_object_t object; 373 register 374 vm_offset_t offset; 375 vm_map_copy_t data_copy; 376 unsigned int data_cnt; 377 vm_prot_t lock_value; 378 boolean_t precious; 379 ipc_port_t reply_to; 380 mach_msg_type_name_t reply_to_type; 381 { 382 kern_return_t result = KERN_SUCCESS; 383 vm_offset_t error_offset = 0; 384 register 385 vm_page_t m; 386 register 387 vm_page_t data_m; 388 vm_size_t original_length; 389 vm_offset_t original_offset; 390 vm_page_t *page_list; 391 boolean_t was_absent; 392 vm_map_copy_t orig_copy = data_copy; 393 394 /* 395 * Look for bogus arguments 396 */ 397 398 if (object == VM_OBJECT_NULL) { 399 return(KERN_INVALID_ARGUMENT); 400 } 401 402 if (lock_value & ~VM_PROT_ALL) { 403 vm_object_deallocate(object); 404 return(KERN_INVALID_ARGUMENT); 405 } 406 407 if ((data_cnt % PAGE_SIZE) != 0) { 408 vm_object_deallocate(object); 409 return(KERN_INVALID_ARGUMENT); 410 } 411 412 /* 413 * Adjust the offset from the memory object to the offset 414 * within the vm_object. 415 */ 416 417 original_length = data_cnt; 418 original_offset = offset; 419 420 assert(data_copy->type == VM_MAP_COPY_PAGE_LIST); 421 page_list = &data_copy->cpy_page_list[0]; 422 423 vm_object_lock(object); 424 vm_object_paging_begin(object); 425 offset -= object->paging_offset; 426 427 /* 428 * Loop over copy stealing pages for pagein. 429 */ 430 431 for (; data_cnt > 0 ; data_cnt -= PAGE_SIZE, offset += PAGE_SIZE) { 432 433 assert(data_copy->cpy_npages > 0); 434 data_m = *page_list; 435 436 if (data_m == VM_PAGE_NULL || data_m->tabled || 437 data_m->error || data_m->absent || data_m->fictitious) { 438 439 panic("Data_supply: bad page"); 440 } 441 442 /* 443 * Look up target page and check its state. 444 */ 445 446 retry_lookup: 447 m = vm_page_lookup(object,offset); 448 if (m == VM_PAGE_NULL) { 449 was_absent = FALSE; 450 } 451 else { 452 if (m->absent && m->busy) { 453 454 /* 455 * Page was requested. Free the busy 456 * page waiting for it. Insertion 457 * of new page happens below. 458 */ 459 460 VM_PAGE_FREE(m); 461 was_absent = TRUE; 462 } 463 else { 464 465 /* 466 * Have to wait for page that is busy and 467 * not absent. This is probably going to 468 * be an error, but go back and check. 469 */ 470 if (m->busy) { 471 PAGE_ASSERT_WAIT(m, FALSE); 472 vm_object_unlock(object); 473 thread_block((void (*)()) 0); 474 vm_object_lock(object); 475 goto retry_lookup; 476 } 477 478 /* 479 * Page already present; error. 480 * This is an error if data is precious. 481 */ 482 result = KERN_MEMORY_PRESENT; 483 error_offset = offset + object->paging_offset; 484 485 break; 486 } 487 } 488 489 /* 490 * Ok to pagein page. Target object now has no page 491 * at offset. Set the page parameters, then drop 492 * in new page and set up pageout state. Object is 493 * still locked here. 494 * 495 * Must clear busy bit in page before inserting it. 496 * Ok to skip wakeup logic because nobody else 497 * can possibly know about this page. 498 */ 499 500 data_m->busy = FALSE; 501 data_m->dirty = FALSE; 502 pmap_clear_modify(data_m->phys_addr); 503 504 data_m->page_lock = lock_value; 505 data_m->unlock_request = VM_PROT_NONE; 506 data_m->precious = precious; 507 508 vm_page_lock_queues(); 509 vm_page_insert(data_m, object, offset); 510 511 if (was_absent) 512 vm_page_activate(data_m); 513 else 514 vm_page_deactivate(data_m); 515 516 vm_page_unlock_queues(); 517 518 /* 519 * Null out this page list entry, and advance to next 520 * page. 521 */ 522 523 *page_list++ = VM_PAGE_NULL; 524 525 if (--(data_copy->cpy_npages) == 0 && 526 vm_map_copy_has_cont(data_copy)) { 527 vm_map_copy_t new_copy; 528 529 vm_object_unlock(object); 530 531 vm_map_copy_invoke_cont(data_copy, &new_copy, &result); 532 533 if (result == KERN_SUCCESS) { 534 535 /* 536 * Consume on success requires that 537 * we keep the original vm_map_copy 538 * around in case something fails. 539 * Free the old copy if it's not the original 540 */ 541 if (data_copy != orig_copy) { 542 vm_map_copy_discard(data_copy); 543 } 544 545 if ((data_copy = new_copy) != VM_MAP_COPY_NULL) 546 page_list = &data_copy->cpy_page_list[0]; 547 548 vm_object_lock(object); 549 } 550 else { 551 vm_object_lock(object); 552 error_offset = offset + object->paging_offset + 553 PAGE_SIZE; 554 break; 555 } 556 } 557 } 558 559 /* 560 * Send reply if one was requested. 561 */ 562 vm_object_paging_end(object); 563 vm_object_unlock(object); 564 565 if (vm_map_copy_has_cont(data_copy)) 566 vm_map_copy_abort_cont(data_copy); 567 568 if (IP_VALID(reply_to)) { 569 memory_object_supply_completed( 570 reply_to, reply_to_type, 571 object->pager_request, 572 original_offset, 573 original_length, 574 result, 575 error_offset); 576 } 577 578 vm_object_deallocate(object); 579 580 /* 581 * Consume on success: The final data copy must be 582 * be discarded if it is not the original. The original 583 * gets discarded only if this routine succeeds. 584 */ 585 if (data_copy != orig_copy) 586 vm_map_copy_discard(data_copy); 587 if (result == KERN_SUCCESS) 588 vm_map_copy_discard(orig_copy); 589 590 591 return(result); 592 } 593 594 kern_return_t memory_object_data_error(object, offset, size, error_value) 595 vm_object_t object; 596 vm_offset_t offset; 597 vm_size_t size; 598 kern_return_t error_value; 599 { 600 if (object == VM_OBJECT_NULL) 601 return(KERN_INVALID_ARGUMENT); 602 603 if (size != round_page(size)) 604 return(KERN_INVALID_ARGUMENT); 605 606 #ifdef lint 607 /* Error value is ignored at this time */ 608 error_value++; 609 #endif 610 611 vm_object_lock(object); 612 offset -= object->paging_offset; 613 614 while (size != 0) { 615 register vm_page_t m; 616 617 m = vm_page_lookup(object, offset); 618 if ((m != VM_PAGE_NULL) && m->busy && m->absent) { 619 m->error = TRUE; 620 m->absent = FALSE; 621 vm_object_absent_release(object); 622 623 PAGE_WAKEUP_DONE(m); 624 625 vm_page_lock_queues(); 626 vm_page_activate(m); 627 vm_page_unlock_queues(); 628 } 629 630 size -= PAGE_SIZE; 631 offset += PAGE_SIZE; 632 } 633 vm_object_unlock(object); 634 635 vm_object_deallocate(object); 636 return(KERN_SUCCESS); 637 } 638 639 kern_return_t memory_object_data_unavailable(object, offset, size) 640 vm_object_t object; 641 vm_offset_t offset; 642 vm_size_t size; 643 { 644 #if MACH_PAGEMAP 645 vm_external_t existence_info = VM_EXTERNAL_NULL; 646 #endif MACH_PAGEMAP 647 648 if (object == VM_OBJECT_NULL) 649 return(KERN_INVALID_ARGUMENT); 650 651 if (size != round_page(size)) 652 return(KERN_INVALID_ARGUMENT); 653 654 #if MACH_PAGEMAP 655 if ((offset == 0) && (size > VM_EXTERNAL_LARGE_SIZE) && 656 (object->existence_info == VM_EXTERNAL_NULL)) { 657 existence_info = vm_external_create(VM_EXTERNAL_SMALL_SIZE); 658 } 659 #endif MACH_PAGEMAP 660 661 vm_object_lock(object); 662 #if MACH_PAGEMAP 663 if (existence_info != VM_EXTERNAL_NULL) { 664 object->existence_info = existence_info; 665 } 666 if ((offset == 0) && (size > VM_EXTERNAL_LARGE_SIZE)) { 667 vm_object_unlock(object); 668 vm_object_deallocate(object); 669 return(KERN_SUCCESS); 670 } 671 #endif MACH_PAGEMAP 672 offset -= object->paging_offset; 673 674 while (size != 0) { 675 register vm_page_t m; 676 677 /* 678 * We're looking for pages that are both busy and 679 * absent (waiting to be filled), converting them 680 * to just absent. 681 * 682 * Pages that are just busy can be ignored entirely. 683 */ 684 685 m = vm_page_lookup(object, offset); 686 if ((m != VM_PAGE_NULL) && m->busy && m->absent) { 687 PAGE_WAKEUP_DONE(m); 688 689 vm_page_lock_queues(); 690 vm_page_activate(m); 691 vm_page_unlock_queues(); 692 } 693 size -= PAGE_SIZE; 694 offset += PAGE_SIZE; 695 } 696 697 vm_object_unlock(object); 698 699 vm_object_deallocate(object); 700 return(KERN_SUCCESS); 701 } 702 703 /* 704 * Routine: memory_object_lock_page 705 * 706 * Description: 707 * Perform the appropriate lock operations on the 708 * given page. See the description of 709 * "memory_object_lock_request" for the meanings 710 * of the arguments. 711 * 712 * Returns an indication that the operation 713 * completed, blocked, or that the page must 714 * be cleaned. 715 */ 716 717 #define MEMORY_OBJECT_LOCK_RESULT_DONE 0 718 #define MEMORY_OBJECT_LOCK_RESULT_MUST_BLOCK 1 719 #define MEMORY_OBJECT_LOCK_RESULT_MUST_CLEAN 2 720 #define MEMORY_OBJECT_LOCK_RESULT_MUST_RETURN 3 721 722 memory_object_lock_result_t memory_object_lock_page(m, should_return, 723 should_flush, prot) 724 vm_page_t m; 725 memory_object_return_t should_return; 726 boolean_t should_flush; 727 vm_prot_t prot; 728 { 729 /* 730 * Don't worry about pages for which the kernel 731 * does not have any data. 732 */ 733 734 if (m->absent) 735 return(MEMORY_OBJECT_LOCK_RESULT_DONE); 736 737 /* 738 * If we cannot change access to the page, 739 * either because a mapping is in progress 740 * (busy page) or because a mapping has been 741 * wired, then give up. 742 */ 743 744 if (m->busy) 745 return(MEMORY_OBJECT_LOCK_RESULT_MUST_BLOCK); 746 747 assert(!m->fictitious); 748 749 if (m->wire_count != 0) { 750 /* 751 * If no change would take place 752 * anyway, return successfully. 753 * 754 * No change means: 755 * Not flushing AND 756 * No change to page lock [2 checks] AND 757 * Don't need to send page to manager 758 * 759 * Don't need to send page to manager means: 760 * No clean or return request OR ( 761 * Page is not dirty [2 checks] AND ( 762 * Page is not precious OR 763 * No request to return precious pages )) 764 * 765 * Now isn't that straightforward and obvious ?? ;-) 766 * 767 * XXX This doesn't handle sending a copy of a wired 768 * XXX page to the pager, but that will require some 769 * XXX significant surgery. 770 */ 771 772 if (!should_flush && 773 ((m->page_lock == prot) || (prot == VM_PROT_NO_CHANGE)) && 774 ((should_return == MEMORY_OBJECT_RETURN_NONE) || 775 (!m->dirty && !pmap_is_modified(m->phys_addr) && 776 (!m->precious || 777 should_return != MEMORY_OBJECT_RETURN_ALL)))) { 778 /* 779 * Restart page unlock requests, 780 * even though no change took place. 781 * [Memory managers may be expecting 782 * to see new requests.] 783 */ 784 m->unlock_request = VM_PROT_NONE; 785 PAGE_WAKEUP(m); 786 787 return(MEMORY_OBJECT_LOCK_RESULT_DONE); 788 } 789 790 return(MEMORY_OBJECT_LOCK_RESULT_MUST_BLOCK); 791 } 792 793 /* 794 * If the page is to be flushed, allow 795 * that to be done as part of the protection. 796 */ 797 798 if (should_flush) 799 prot = VM_PROT_ALL; 800 801 /* 802 * Set the page lock. 803 * 804 * If we are decreasing permission, do it now; 805 * let the fault handler take care of increases 806 * (pmap_page_protect may not increase protection). 807 */ 808 809 if (prot != VM_PROT_NO_CHANGE) { 810 if ((m->page_lock ^ prot) & prot) { 811 pmap_page_protect(m->phys_addr, VM_PROT_ALL & ~prot); 812 } 813 m->page_lock = prot; 814 815 /* 816 * Restart any past unlock requests, even if no 817 * change resulted. If the manager explicitly 818 * requested no protection change, then it is assumed 819 * to be remembering past requests. 820 */ 821 822 m->unlock_request = VM_PROT_NONE; 823 PAGE_WAKEUP(m); 824 } 825 826 /* 827 * Handle cleaning. 828 */ 829 830 if (should_return != MEMORY_OBJECT_RETURN_NONE) { 831 /* 832 * Check whether the page is dirty. If 833 * write permission has not been removed, 834 * this may have unpredictable results. 835 */ 836 837 if (!m->dirty) 838 m->dirty = pmap_is_modified(m->phys_addr); 839 840 if (m->dirty || (m->precious && 841 should_return == MEMORY_OBJECT_RETURN_ALL)) { 842 /* 843 * If we weren't planning 844 * to flush the page anyway, 845 * we may need to remove the 846 * page from the pageout 847 * system and from physical 848 * maps now. 849 */ 850 851 vm_page_lock_queues(); 852 VM_PAGE_QUEUES_REMOVE(m); 853 vm_page_unlock_queues(); 854 855 if (!should_flush) 856 pmap_page_protect(m->phys_addr, 857 VM_PROT_NONE); 858 859 /* 860 * Cleaning a page will cause 861 * it to be flushed. 862 */ 863 864 if (m->dirty) 865 return(MEMORY_OBJECT_LOCK_RESULT_MUST_CLEAN); 866 else 867 return(MEMORY_OBJECT_LOCK_RESULT_MUST_RETURN); 868 } 869 } 870 871 /* 872 * Handle flushing 873 */ 874 875 if (should_flush) { 876 VM_PAGE_FREE(m); 877 } else { 878 extern boolean_t vm_page_deactivate_hint; 879 880 /* 881 * XXX Make clean but not flush a paging hint, 882 * and deactivate the pages. This is a hack 883 * because it overloads flush/clean with 884 * implementation-dependent meaning. This only 885 * happens to pages that are already clean. 886 */ 887 888 if (vm_page_deactivate_hint && 889 (should_return != MEMORY_OBJECT_RETURN_NONE)) { 890 vm_page_lock_queues(); 891 vm_page_deactivate(m); 892 vm_page_unlock_queues(); 893 } 894 } 895 896 return(MEMORY_OBJECT_LOCK_RESULT_DONE); 897 } 898 899 /* 900 * Routine: memory_object_lock_request [user interface] 901 * 902 * Description: 903 * Control use of the data associated with the given 904 * memory object. For each page in the given range, 905 * perform the following operations, in order: 906 * 1) restrict access to the page (disallow 907 * forms specified by "prot"); 908 * 2) return data to the manager (if "should_return" 909 * is RETURN_DIRTY and the page is dirty, or 910 * "should_return" is RETURN_ALL and the page 911 * is either dirty or precious); and, 912 * 3) flush the cached copy (if "should_flush" 913 * is asserted). 914 * The set of pages is defined by a starting offset 915 * ("offset") and size ("size"). Only pages with the 916 * same page alignment as the starting offset are 917 * considered. 918 * 919 * A single acknowledgement is sent (to the "reply_to" 920 * port) when these actions are complete. If successful, 921 * the naked send right for reply_to is consumed. 922 */ 923 924 kern_return_t 925 memory_object_lock_request(object, offset, size, 926 should_return, should_flush, prot, 927 reply_to, reply_to_type) 928 register vm_object_t object; 929 register vm_offset_t offset; 930 register vm_size_t size; 931 memory_object_return_t should_return; 932 boolean_t should_flush; 933 vm_prot_t prot; 934 ipc_port_t reply_to; 935 mach_msg_type_name_t reply_to_type; 936 { 937 register vm_page_t m; 938 vm_offset_t original_offset = offset; 939 vm_size_t original_size = size; 940 vm_offset_t paging_offset = 0; 941 vm_object_t new_object = VM_OBJECT_NULL; 942 vm_offset_t new_offset = 0; 943 vm_offset_t last_offset = offset; 944 int page_lock_result; 945 int pageout_action = 0; /* '=0' to quiet lint */ 946 947 #define DATA_WRITE_MAX 32 948 vm_page_t holding_pages[DATA_WRITE_MAX]; 949 950 /* 951 * Check for bogus arguments. 952 */ 953 if (object == VM_OBJECT_NULL || 954 ((prot & ~VM_PROT_ALL) != 0 && prot != VM_PROT_NO_CHANGE)) 955 return (KERN_INVALID_ARGUMENT); 956 957 size = round_page(size); 958 959 /* 960 * Lock the object, and acquire a paging reference to 961 * prevent the memory_object and control ports from 962 * being destroyed. 963 */ 964 965 vm_object_lock(object); 966 vm_object_paging_begin(object); 967 offset -= object->paging_offset; 968 969 /* 970 * To avoid blocking while scanning for pages, save 971 * dirty pages to be cleaned all at once. 972 * 973 * XXXO A similar strategy could be used to limit the 974 * number of times that a scan must be restarted for 975 * other reasons. Those pages that would require blocking 976 * could be temporarily collected in another list, or 977 * their offsets could be recorded in a small array. 978 */ 979 980 /* 981 * XXX NOTE: May want to consider converting this to a page list 982 * XXX vm_map_copy interface. Need to understand object 983 * XXX coalescing implications before doing so. 984 */ 985 986 #define PAGEOUT_PAGES \ 987 MACRO_BEGIN \ 988 vm_map_copy_t copy; \ 989 register int i; \ 990 register vm_page_t hp; \ 991 \ 992 vm_object_unlock(object); \ 993 \ 994 (void) vm_map_copyin_object(new_object, 0, new_offset, ©); \ 995 \ 996 if (object->use_old_pageout) { \ 997 assert(pageout_action == MEMORY_OBJECT_LOCK_RESULT_MUST_CLEAN); \ 998 (void) memory_object_data_write( \ 999 object->pager, \ 1000 object->pager_request, \ 1001 paging_offset, \ 1002 (pointer_t) copy, \ 1003 new_offset); \ 1004 } \ 1005 else { \ 1006 (void) memory_object_data_return( \ 1007 object->pager, \ 1008 object->pager_request, \ 1009 paging_offset, \ 1010 (pointer_t) copy, \ 1011 new_offset, \ 1012 (pageout_action == MEMORY_OBJECT_LOCK_RESULT_MUST_CLEAN), \ 1013 !should_flush); \ 1014 } \ 1015 \ 1016 vm_object_lock(object); \ 1017 \ 1018 for (i = 0; i < atop(new_offset); i++) { \ 1019 hp = holding_pages[i]; \ 1020 if (hp != VM_PAGE_NULL) \ 1021 VM_PAGE_FREE(hp); \ 1022 } \ 1023 \ 1024 new_object = VM_OBJECT_NULL; \ 1025 MACRO_END 1026 1027 for (; 1028 size != 0; 1029 size -= PAGE_SIZE, offset += PAGE_SIZE) 1030 { 1031 /* 1032 * Limit the number of pages to be cleaned at once. 1033 */ 1034 if (new_object != VM_OBJECT_NULL && 1035 new_offset >= PAGE_SIZE * DATA_WRITE_MAX) 1036 { 1037 PAGEOUT_PAGES; 1038 } 1039 1040 while ((m = vm_page_lookup(object, offset)) != VM_PAGE_NULL) { 1041 switch ((page_lock_result = memory_object_lock_page(m, 1042 should_return, 1043 should_flush, 1044 prot))) 1045 { 1046 case MEMORY_OBJECT_LOCK_RESULT_DONE: 1047 /* 1048 * End of a cluster of dirty pages. 1049 */ 1050 if (new_object != VM_OBJECT_NULL) { 1051 PAGEOUT_PAGES; 1052 continue; 1053 } 1054 break; 1055 1056 case MEMORY_OBJECT_LOCK_RESULT_MUST_BLOCK: 1057 /* 1058 * Since it is necessary to block, 1059 * clean any dirty pages now. 1060 */ 1061 if (new_object != VM_OBJECT_NULL) { 1062 PAGEOUT_PAGES; 1063 continue; 1064 } 1065 1066 PAGE_ASSERT_WAIT(m, FALSE); 1067 vm_object_unlock(object); 1068 thread_block((void (*)()) 0); 1069 vm_object_lock(object); 1070 continue; 1071 1072 case MEMORY_OBJECT_LOCK_RESULT_MUST_CLEAN: 1073 case MEMORY_OBJECT_LOCK_RESULT_MUST_RETURN: 1074 /* 1075 * The clean and return cases are similar. 1076 * 1077 * Mark the page busy since we unlock the 1078 * object below. 1079 */ 1080 m->busy = TRUE; 1081 1082 /* 1083 * if this would form a discontiguous block, 1084 * clean the old pages and start anew. 1085 * 1086 * NOTE: The first time through here, new_object 1087 * is null, hiding the fact that pageout_action 1088 * is not initialized. 1089 */ 1090 if (new_object != VM_OBJECT_NULL && 1091 (last_offset != offset || 1092 pageout_action != page_lock_result)) { 1093 PAGEOUT_PAGES; 1094 } 1095 1096 vm_object_unlock(object); 1097 1098 /* 1099 * If we have not already allocated an object 1100 * for a range of pages to be written, do so 1101 * now. 1102 */ 1103 if (new_object == VM_OBJECT_NULL) { 1104 new_object = vm_object_allocate(original_size); 1105 new_offset = 0; 1106 paging_offset = m->offset + 1107 object->paging_offset; 1108 pageout_action = page_lock_result; 1109 } 1110 1111 /* 1112 * Move or copy the dirty page into the 1113 * new object. 1114 */ 1115 m = vm_pageout_setup(m, 1116 m->offset + object->paging_offset, 1117 new_object, 1118 new_offset, 1119 should_flush); 1120 1121 /* 1122 * Save the holding page if there is one. 1123 */ 1124 holding_pages[atop(new_offset)] = m; 1125 new_offset += PAGE_SIZE; 1126 last_offset = offset + PAGE_SIZE; 1127 1128 vm_object_lock(object); 1129 break; 1130 } 1131 break; 1132 } 1133 } 1134 1135 /* 1136 * We have completed the scan for applicable pages. 1137 * Clean any pages that have been saved. 1138 */ 1139 if (new_object != VM_OBJECT_NULL) { 1140 PAGEOUT_PAGES; 1141 } 1142 1143 if (IP_VALID(reply_to)) { 1144 vm_object_unlock(object); 1145 1146 /* consumes our naked send-once/send right for reply_to */ 1147 (void) memory_object_lock_completed(reply_to, reply_to_type, 1148 object->pager_request, original_offset, original_size); 1149 1150 vm_object_lock(object); 1151 } 1152 1153 vm_object_paging_end(object); 1154 vm_object_unlock(object); 1155 vm_object_deallocate(object); 1156 1157 return (KERN_SUCCESS); 1158 } 1159 1160 #if !NORMA_VM 1161 /* 1162 * Old version of memory_object_lock_request. 1163 */ 1164 kern_return_t 1165 xxx_memory_object_lock_request(object, offset, size, 1166 should_clean, should_flush, prot, 1167 reply_to, reply_to_type) 1168 register vm_object_t object; 1169 register vm_offset_t offset; 1170 register vm_size_t size; 1171 boolean_t should_clean; 1172 boolean_t should_flush; 1173 vm_prot_t prot; 1174 ipc_port_t reply_to; 1175 mach_msg_type_name_t reply_to_type; 1176 { 1177 register int should_return; 1178 1179 if (should_clean) 1180 should_return = MEMORY_OBJECT_RETURN_DIRTY; 1181 else 1182 should_return = MEMORY_OBJECT_RETURN_NONE; 1183 1184 return(memory_object_lock_request(object,offset,size, 1185 should_return, should_flush, prot, 1186 reply_to, reply_to_type)); 1187 } 1188 #endif !NORMA_VM 1189 1190 kern_return_t 1191 memory_object_set_attributes_common(object, object_ready, may_cache, 1192 copy_strategy, use_old_pageout) 1193 vm_object_t object; 1194 boolean_t object_ready; 1195 boolean_t may_cache; 1196 memory_object_copy_strategy_t copy_strategy; 1197 boolean_t use_old_pageout; 1198 { 1199 if (object == VM_OBJECT_NULL) 1200 return(KERN_INVALID_ARGUMENT); 1201 1202 /* 1203 * Verify the attributes of importance 1204 */ 1205 1206 switch(copy_strategy) { 1207 case MEMORY_OBJECT_COPY_NONE: 1208 case MEMORY_OBJECT_COPY_CALL: 1209 case MEMORY_OBJECT_COPY_DELAY: 1210 case MEMORY_OBJECT_COPY_TEMPORARY: 1211 break; 1212 default: 1213 vm_object_deallocate(object); 1214 return(KERN_INVALID_ARGUMENT); 1215 } 1216 1217 if (object_ready) 1218 object_ready = TRUE; 1219 if (may_cache) 1220 may_cache = TRUE; 1221 1222 vm_object_lock(object); 1223 1224 /* 1225 * Wake up anyone waiting for the ready attribute 1226 * to become asserted. 1227 */ 1228 1229 if (object_ready && !object->pager_ready) { 1230 object->use_old_pageout = use_old_pageout; 1231 vm_object_wakeup(object, VM_OBJECT_EVENT_PAGER_READY); 1232 } 1233 1234 /* 1235 * Copy the attributes 1236 */ 1237 1238 object->can_persist = may_cache; 1239 object->pager_ready = object_ready; 1240 if (copy_strategy == MEMORY_OBJECT_COPY_TEMPORARY) { 1241 object->temporary = TRUE; 1242 } else { 1243 object->copy_strategy = copy_strategy; 1244 } 1245 1246 vm_object_unlock(object); 1247 1248 vm_object_deallocate(object); 1249 1250 return(KERN_SUCCESS); 1251 } 1252 1253 #if !NORMA_VM 1254 1255 /* 1256 * XXX rpd claims that reply_to could be obviated in favor of a client 1257 * XXX stub that made change_attributes an RPC. Need investigation. 1258 */ 1259 1260 kern_return_t memory_object_change_attributes(object, may_cache, 1261 copy_strategy, reply_to, reply_to_type) 1262 vm_object_t object; 1263 boolean_t may_cache; 1264 memory_object_copy_strategy_t copy_strategy; 1265 ipc_port_t reply_to; 1266 mach_msg_type_name_t reply_to_type; 1267 { 1268 kern_return_t result; 1269 1270 /* 1271 * Do the work and throw away our object reference. It 1272 * is important that the object reference be deallocated 1273 * BEFORE sending the reply. The whole point of the reply 1274 * is that it shows up after the terminate message that 1275 * may be generated by setting the object uncacheable. 1276 * 1277 * XXX may_cache may become a tri-valued variable to handle 1278 * XXX uncache if not in use. 1279 */ 1280 result = memory_object_set_attributes_common(object, TRUE, 1281 may_cache, copy_strategy, 1282 FALSE); 1283 1284 if (IP_VALID(reply_to)) { 1285 1286 /* consumes our naked send-once/send right for reply_to */ 1287 (void) memory_object_change_completed(reply_to, reply_to_type, 1288 may_cache, copy_strategy); 1289 1290 } 1291 1292 return(result); 1293 } 1294 1295 kern_return_t 1296 memory_object_set_attributes(object, object_ready, may_cache, copy_strategy) 1297 vm_object_t object; 1298 boolean_t object_ready; 1299 boolean_t may_cache; 1300 memory_object_copy_strategy_t copy_strategy; 1301 { 1302 return memory_object_set_attributes_common(object, object_ready, 1303 may_cache, copy_strategy, 1304 TRUE); 1305 } 1306 1307 kern_return_t memory_object_ready(object, may_cache, copy_strategy) 1308 vm_object_t object; 1309 boolean_t may_cache; 1310 memory_object_copy_strategy_t copy_strategy; 1311 { 1312 return memory_object_set_attributes_common(object, TRUE, 1313 may_cache, copy_strategy, 1314 FALSE); 1315 } 1316 #endif !NORMA_VM 1317 1318 kern_return_t memory_object_get_attributes(object, object_ready, 1319 may_cache, copy_strategy) 1320 vm_object_t object; 1321 boolean_t *object_ready; 1322 boolean_t *may_cache; 1323 memory_object_copy_strategy_t *copy_strategy; 1324 { 1325 if (object == VM_OBJECT_NULL) 1326 return(KERN_INVALID_ARGUMENT); 1327 1328 vm_object_lock(object); 1329 *may_cache = object->can_persist; 1330 *object_ready = object->pager_ready; 1331 *copy_strategy = object->copy_strategy; 1332 vm_object_unlock(object); 1333 1334 vm_object_deallocate(object); 1335 1336 return(KERN_SUCCESS); 1337 } 1338 1339 /* 1340 * If successful, consumes the supplied naked send right. 1341 */ 1342 kern_return_t vm_set_default_memory_manager(host, default_manager) 1343 host_t host; 1344 ipc_port_t *default_manager; 1345 { 1346 ipc_port_t current_manager; 1347 ipc_port_t new_manager; 1348 ipc_port_t returned_manager; 1349 1350 if (host == HOST_NULL) 1351 return(KERN_INVALID_HOST); 1352 1353 new_manager = *default_manager; 1354 simple_lock(&memory_manager_default_lock); 1355 current_manager = memory_manager_default; 1356 1357 if (new_manager == IP_NULL) { 1358 /* 1359 * Retrieve the current value. 1360 */ 1361 1362 returned_manager = ipc_port_copy_send(current_manager); 1363 } else { 1364 /* 1365 * Retrieve the current value, 1366 * and replace it with the supplied value. 1367 * We consume the supplied naked send right. 1368 */ 1369 1370 returned_manager = current_manager; 1371 memory_manager_default = new_manager; 1372 1373 /* 1374 * In case anyone's been waiting for a memory 1375 * manager to be established, wake them up. 1376 */ 1377 1378 thread_wakeup((event_t) &memory_manager_default); 1379 } 1380 1381 simple_unlock(&memory_manager_default_lock); 1382 1383 *default_manager = returned_manager; 1384 return(KERN_SUCCESS); 1385 } 1386 1387 /* 1388 * Routine: memory_manager_default_reference 1389 * Purpose: 1390 * Returns a naked send right for the default 1391 * memory manager. The returned right is always 1392 * valid (not IP_NULL or IP_DEAD). 1393 */ 1394 1395 ipc_port_t memory_manager_default_reference() 1396 { 1397 ipc_port_t current_manager; 1398 1399 simple_lock(&memory_manager_default_lock); 1400 1401 while (current_manager = ipc_port_copy_send(memory_manager_default), 1402 !IP_VALID(current_manager)) { 1403 thread_sleep((event_t) &memory_manager_default, 1404 simple_lock_addr(memory_manager_default_lock), 1405 FALSE); 1406 simple_lock(&memory_manager_default_lock); 1407 } 1408 1409 simple_unlock(&memory_manager_default_lock); 1410 1411 return current_manager; 1412 } 1413 1414 /* 1415 * Routine: memory_manager_default_port 1416 * Purpose: 1417 * Returns true if the receiver for the port 1418 * is the default memory manager. 1419 * 1420 * This is a hack to let ds_read_done 1421 * know when it should keep memory wired. 1422 */ 1423 1424 boolean_t memory_manager_default_port(port) 1425 ipc_port_t port; 1426 { 1427 ipc_port_t current; 1428 boolean_t result; 1429 1430 simple_lock(&memory_manager_default_lock); 1431 current = memory_manager_default; 1432 if (IP_VALID(current)) { 1433 /* 1434 * There is no point in bothering to lock 1435 * both ports, which would be painful to do. 1436 * If the receive rights are moving around, 1437 * we might be inaccurate. 1438 */ 1439 1440 result = port->ip_receiver == current->ip_receiver; 1441 } else 1442 result = FALSE; 1443 simple_unlock(&memory_manager_default_lock); 1444 1445 return result; 1446 } 1447 1448 void memory_manager_default_init() 1449 { 1450 memory_manager_default = IP_NULL; 1451 simple_lock_init(&memory_manager_default_lock); 1452 }
Cache object: 8f4f039ac30a918661c4e88743dc7839
[ source navigation ] [ diff markup ] [ identifier search ] [ freetext search ] [ file search ] [ list types ] [ track identifier ]