|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
1. What is the OpenSolaris Project? 3. Planning the OpenSolaris Environment Development Environment Configuration Network Auto-Configuration Daemon Web Server Virtualization With Zones Creating ZFS Storage Pools and File Systems Creating a Mirrored ZFS Storage Pool Creating ZFS File Systems as Home Directories Creating a RAID-Z Configuration Userland Consolidations and Descriptions 5. Core Features of the Solaris OS Development Process and Coding Style 7. Getting Started With DTrace 8. Debugging Applications With DTrace 9. Debugging C++ Applications With DTrace Using DTrace to Profile and Debug A C++ Program 10. Managing Memory with DTrace and MDB Using DTrace and MDB to Examine Virtual Memory |
Part No: 819-5580-13 March, 2008 4150 Network Circle Copyright 2008 SMI Sun Microsystems, Inc. has intellectual property rights relating to technology embodied in the product that is described in this document. In particular, and without limitation, these intellectual property rights may include one or more U.S. patents or pending patent applications in the U.S. and in other countries. U.S. Government Rights – Commercial software. Government users are subject to the Sun Microsystems, Inc. standard license agreement and applicable provisions of the FAR and its supplements. This distribution may include materials developed by third parties. Parts of the product may be derived from Berkeley BSD systems, licensed from the University of California. UNIX is a registered trademark in the U.S. and other countries, exclusively licensed through X/Open Company, Ltd. Sun, Sun Microsystems, the Sun logo, the Solaris logo, the Java Coffee Cup logo, docs.sun.com, Java, and Solaris are trademarks or registered trademarks of Sun Microsystems, Inc. in the U.S. and other countries. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. in the U.S. and other countries. Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc. The OPEN LOOK and SunTM Graphical User Interface was developed by Sun Microsystems, Inc. for its users and licensees. Sun acknowledges the pioneering efforts of Xerox in researching and developing the concept of visual or graphical user interfaces for the computer industry. Sun holds a non-exclusive license from Xerox to the Xerox Graphical User Interface, which license also covers Sun's licensees who implement OPEN LOOK GUIs and otherwise comply with Sun's written license agreements. Products covered by and information contained in this publication are controlled by U.S. Export Control laws and may be subject to the export or import laws in other countries. Nuclear, missile, chemical or biological weapons or nuclear maritime end uses or end users, whether direct or indirect, are strictly prohibited. Export or reexport to countries subject to U.S. embargo or to entities identified on U.S. export exclusion lists, including, but not limited to, the denied persons and specially designated nationals lists is strictly prohibited. DOCUMENTATION IS PROVIDED “AS IS” AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID. Sun Microsystems, Inc. détient les droits de propriété intellectuelle relatifs à la technologie incorporée dans le produit qui est décrit dans ce document. En particulier, et ce sans limitation, ces droits de propriété intellectuelle peuvent inclure un ou plusieurs brevets américains ou des applications de brevet en attente aux Etats-Unis et dans d'autres pays. Cette distribution peut comprendre des composants développés par des tierces personnes. Certaines composants de ce produit peuvent être dérivées du logiciel Berkeley BSD, licenciés par l'Université de Californie. UNIX est une marque déposée aux Etats-Unis et dans d'autres pays; elle est licenciée exclusivement par X/Open Company, Ltd. Sun, Sun Microsystems, le logo Sun, le logo Solaris, le logo Java Coffee Cup, docs.sun.com, Java et Solaris sont des marques de fabrique ou des marques déposées de Sun Microsystems, Inc. aux Etats-Unis et dans d'autres pays. Toutes les marques SPARC sont utilisées sous licence et sont des marques de fabrique ou des marques déposées de SPARC International, Inc. aux Etats-Unis et dans d'autres pays. Les produits portant les marques SPARC sont basés sur une architecture développée par Sun Microsystems, Inc. L'interface d'utilisation graphique OPEN LOOK et Sun a été développée par Sun Microsystems, Inc. pour ses utilisateurs et licenciés. Sun reconnaît les efforts de pionniers de Xerox pour la recherche et le développement du concept des interfaces d'utilisation visuelle ou graphique pour l'industrie de l'informatique. Sun détient une licence non exclusive de Xerox sur l'interface d'utilisation graphique Xerox, cette licence couvrant également les licenciés de Sun qui mettent en place l'interface d'utilisation graphique OPEN LOOK et qui, en outre, se conforment aux licences écrites de Sun. Les produits qui font l'objet de cette publication et les informations qu'il contient sont régis par la legislation américaine en matière de contrôle des exportations et peuvent être soumis au droit d'autres pays dans le domaine des exportations et importations. Les utilisations finales, ou utilisateurs finaux, pour des armes nucléaires, des missiles, des armes chimiques ou biologiques ou pour le nucléaire maritime, directement ou indirectement, sont strictement interdites. Les exportations ou réexportations vers des pays sous embargo des Etats-Unis, ou vers des entités figurant sur les listes d'exclusion d'exportation américaines, y compris, mais de manière non exclusive, la liste de personnes qui font objet d'un ordre de ne pas participer, d'une façon directe ou indirecte, aux exportations des produits ou des services qui sont régis par la legislation américaine en matière de contrôle des exportations et la liste de ressortissants spécifiquement designés, sont rigoureusement interdites. LA DOCUMENTATION EST FOURNIE "EN L'ETAT" ET TOUTES AUTRES CONDITIONS, DECLARATIONS ET GARANTIES EXPRESSES OU TACITES SONT FORMELLEMENT EXCLUES, DANS LA MESURE AUTORISEE PAR LA LOI APPLICABLE, Y COMPRIS NOTAMMENT TOUTE GARANTIE IMPLICITE RELATIVE A LA QUALITE MARCHANDE, A L'APTITUDE A UNE UTILISATION PARTICULIERE OU A L'ABSENCE DE CONTREFACON. Module 1What is the OpenSolaris Project?ObjectivesThe objective of this course is to learn about operating system computing by using the SolarisTM Operating System source code that is freely available through the OpenSolaris project. Tip - To receive a free OpenSolaris Starter Kit that includes training materials, source code, and developer tools, register online at http://get.opensolaris.org. We'll start by showing you the user groups, portals, and documentation you will use to get started with UNIX® computing. Next, we'll show you where to go to access the code, communities, discussions, projects, and source browser for the OpenSolaris project. Then, we'll give you steps to configure zones, ZFS, networking, and the environment. Finally, we'll demonstrate debugging processes, applications, page faults, and device drivers with DTrace in the lab exercises. Note - Go to http://opensolaris.org/os/community/edu/curriculum_development/general_test/ to download the scripts, source code, and makefile for CCtest used later in this document to demonstrate C++ debugging with DTrace. The OpenSolaris project was launched on June 14, 2005 to create a community development effort using the Solaris OS code as a starting point. It is a nexus for a community development effort where contributors from Sun and elsewhere can collaborate on developing and improving operating system technology. The OpenSolaris source code will find a variety of uses, including being the basis for future versions of the Solaris OS product, other operating system projects, third-party products and distributions of interest to the community. The OpenSolaris project is currently sponsored by Sun Microsystems, Inc. In the first two years, over 60,000 participants have become registered members. The engineering community is continually growing and changing to meet the needs of developers, system administrators, and end users of the Solaris Operating System. Teaching with the OpenSolaris project provides the following advantages over instructional operating systems:
The Internationalization and Localization Community is helping to translate the OpenSolaris English web site into many languages. So far, eight country portals are under development, as follows:
Portals for Germany, Russia, Czech Republic, Spain, Korea, and Mexico are planned. See the OpenSolaris Portals project to get involved, or chat on one of the seven OpenSolaris chat rooms using IRC at irc.freenode.net. See http://opensolaris.org/os/chat/ Web Resources for OpenSolarisYou can download the OpenSolaris source, view the license terms and access instructions for building source and installing the pre-built archives at: http://www.opensolaris.org/os/downloads. The icons in the upper-right of the OpenSolaris web pages link you to discussions, communities, projects, downloads, and source browser resources. In addition, the OpenSolaris web site provides search across all of the site content and aggregated blogs. DiscussionsDiscussions provide you with access to the experts who are working on new open source technologies. Discussions also provide an archive of previous conversations that you can reference for answers to your questions. See http://www.opensolaris.org/os/discussions for the complete list of forums to which you can subscribe. CommunitiesCommunities provide connections to other participants with similar interests in the OpenSolaris project. Communities form around interest groups, technologies, support, tools, and user groups, for example:
These are only a few of 30 communities actively working on OpenSolaris. See http://www.opensolaris.org/os/communities for the complete list. Sun intends to have non-Sun contributors and wants to foster collaborations across industrial and academic affiliations. The OpenSolaris project will empower and expand the existing Solaris community. The OpenSolaris project will allow for the creation of entirely new communities. ProjectsProjects hosted on the http://www.opensolaris.org/ web site are collaborative efforts that produce objects such as code changes, documents, graphics, or joint-authored products. Projects have code repositories and committers and may live within a community or independently. New projects are initiated by participants by request on the discussions. Projects that are submitted and accepted by at least one other interested participant in the sponsoring community are given space on the projects page to get started. See http://www.opensolaris.org/os/projects for the current list of new projects. Projects give you the opportunity to share files, disk space, and an email alias. You can collaborate with other engineers across the globe to work on a new technology or an improvement to existing technology.
Centralized and distributed source repositories are hosted on the opensolaris.org web site. The centralized source management model uses the Subversion (SVN) source control management program. Repositories managed in a distributed fashion use the Mercurial (hg) source control management program. The creation of a source repository on opensolaris.org is completed by a Project Leader by using the Project web pages. Developers with commit rights access repositories through their opensolaris.org accounts. Commit rights are managed by Project Leaders. If you need an account, you may sign up to acquire one. Additionally, you will have to provide a Secure Shell (SSH) public key. Refer to the tools community for the most recent source control information, downloads and instructions http://opensolaris.org/os/community/tools OpenGrokOpenGrok™ is the fast and usable source code search and cross reference engine used in OpenSolaris. See http://cvs.opensolaris.org/source to try it out! The first project to be hosted on opensolaris.org was OpenGrok. See http://www.opensolaris.org/os/project/opengrok to find out about the ongoing development project. Take an online tour of the source and you'll discover cleanly written, extensively commented code that reads like a book. If you're interested in working on an OpenSolaris project, you can download the complete codebase. If you just need to know how some features work in the Solaris OS, the source code browser provides a convenient alternative. OpenGrok understands various program file formats and version control histories like SCCS, RCS, and CVS, so that you can better understand the open source. Module 2OpenSolaris AdvocacyObjectivesThe Advocates Community exists to help people around the world get involved in the OpenSolaris Community. We welcome participation from people of all languages and cultures and people with all levels of technical and non-technical skills. Everyone has something to contribute. See http://opensolaris.org/os/community/advocacy/ In the Advocates community you will find independent user group projects, presentations, news, articles, blogs, technical & non-technical content, videos and podcasts, events and conferences, community metrics, swag, badges, buttons, and a variety of other promotional projects. Why Use OpenSolaris?This section describes practical reasons to consider choosing to use OpenSolaris as your development platform. PriceSince the availability of the Solaris 10 Operating System in January 2005, its popularity has exploded. As of July 2007, in excess of 8.7 million copies were registered, more than all of the previous versions of the Solaris OS combined. Further fueling this frenzy was the release of OpenSolaris in June 2005. Given this surge in the number of users, more developers (commercial and open-source alike) are seeing the Solaris operating system as a viable target for their software. One of the reasons the Solaris OS enjoyed a huge popularity boost was its price: $0 for everyone, for any use (commercial and non-commercial), on any machine (using both SPARC and x86 platforms). Another reason was Sun's promise (and delivering on that promise) of making the Solaris source code available under an OSI-approved open-source license, the Common Development and Distribution License (CDDL). Innovative Core FeaturesHowever, the most important reason for the popularity of the Solaris OS is arguably the vast wealth of features it offers. In no particular order, these include the following:
All of these features build on what long-time Solaris OS users have come to expect: rock-solid stability, huge scalability, high performance, and guaranteed backwards compatibility. The last of these is especially important to commercial software developers, because maintenance is usually the largest expense associated with a piece of software. With its backwards compatibility guarantee, software vendors know that (provided they use only published APIs) software built for Solaris N will run correctly on Solaris N+1 and subsequent versions. (Contrast this with some other operating systems, where incompatible changes to system components -- for example, libraries -- are made without regard to the effect on applications. The net effect is application breakage, resulting in increased maintenance costs and frustration for application vendors and users.) Hardware Platform NeutralityThe preceding paragraphs give some reasons why we should develop for the Solaris OS, but there are additional reasons to develop on the Solaris platform. One is that Solaris is a multi-OSOplatform OS, supporting both SPARC and x86 architectures (a community-driven port to Power is in the works, too). Although there was an issue a few years ago with the Solaris OS for x86 platforms, the fact that Sun has introduced a range of AMD-based servers and workstations demonstrates the company's commitment to x86 technology. From the developer's perspective, the Solaris versions for SPARC and x86 platforms have the same feature set and APIs. This means that developers can concentrate on the other issues endemic to cross-platform development, like CPU endianness. The SPARC platform is big-endian and x86 is little-endian, so an application that is developed and tested on the Solaris platform has a high probability of being free from endian-related problems. The Solaris OS also supports both 32-bit and 64-bit applications on both platforms, thus helping to eliminate bugs due to assumptions about word size. Perhaps the most compelling reason to develop software on the Solaris OS is the wealth of professional-grade development tools available for it. Development ToolsOne of the most important features of an OS from a developer's point of view is the variety and quality of the development tools available. Compilers and debuggers are the most obvious examples of these tools, but other examples include code checkers (to ensure that our code is free from subtle errors the compiler might not catch), cross-reference generators (to see which functions reference other functions and variables), and performance analyzers. The Sun Studio suite is the product of choice for Solaris OS developers. Available as a free download from the http://developers.sun.com web site, Sun Studio software is a collection of professional-grade compilers and tools. It includes C, C++, and FORTRAN compilers, code analysis tools, an integrated development environment (IDE), the dbx source-level debugger, and editors. Other tools included with Studio software are cscope (an interactive source browser), ctrace (a tool to generate a self-tracing version of our programs), cxref (a C program cross-referencer), dmake (for distributed parallel makes), and lint (the C program checker). The Solaris OS ships with the GNU C compiler, gcc, and its companion source-level debugger, gdb. The Solaris OS also ships with the very powerful modular debugger, mdb. However, mdb is not a source-level debugger. It is most useful when we are debugging kernel code, or performing post-mortem analysis on programs for which the source is not available. See the Solaris Modular Debugger Guide and Solaris Performance and Tools by McDougall, Mauro, and Gregg for more information about mdb. AcknowledgmentsThe following members of the OpenSolaris Community reviewed and provided feedback on this document:
The following OpenSolaris community members provided excellent new content:
Many thanks also go to Steven Cogorno, David Comay, Teresa Giacomini, Stephen Hahn, Patrick Finch, and Sue Weber for their work to make the initial version possible. To participate in future reviews of this document, use the instructions at the following URL: http://www.opensolaris.org/os/community/documentation/reviews Module 3Planning the OpenSolaris EnvironmentObjectivesThe objective of this module is to understand the system requirements, support information, and documentation available for the OpenSolaris project installation and configuration. Additional Resources
There is no substitute for hands-on experience with operating system code and direct access to kernel modules. The unique challenges of kernel development and access to root privileges for a system are made simpler by the tools, forums, and documentation provided for the OpenSolaris project. Tip - To receive an OpenSolaris Starter Kit that includes training materials, source code, and developer tools, go to http://get.opensolaris.org. The OpenSolaris 64-bit kernel is seamless: 32-bit applications run unmodified on it. One may alternate between the 32-bit and 64-bit kernel with only a reboot. All of the architectures supported by the OpenSolaris project are built from the source code basis. The 64-bit kernel isn't a separate version or variant of the system. 32-bit and 64-bit applications and libraries coexist seamlessly. Consider the following features of OpenSolaris as you plan your development environment: Table 3-1 Configurable Lab Component Support
Problem: machines are underutilized; utilization can be increased through virtualization with Zones. Each zone looks, feels, and smells like its own machine, you can even reboot them! Most other virtualization technologies virtualize at the hardware layer. Zones are a new facility in OpenSolaris that instead virtualizes at the operating system layer. Refer to Module 4 for more information about how Zones and Branded Zones enable kernel and user mode development of Solaris and Linux applications without impacting developers in separate zones. Participation in the OpenSolaris project can improve overall performance across your network with the latest technologies. Your lab environment becomes self-sustaining when hosted on OpenSolaris because you are always running the latest and greatest environment, empowered to update it yourself. NetworkingThe OpenSolaris project meets future networking challenges by radically improving your network performance without requiring changes to your existing applications.
Find out more about ongoing networking developments from the OpenSolaris Networking Community: http://www.opensolaris.org/os/community/networking. Network Auto-Configuration DaemonThe Solaris Express Developer Edition 5/07 release booting process runs the nwamd daemon. This daemon implements an alternate instance of the SMF service, svc:/network/physical, which enables automated network configuration with minimal intervention. The nwamd daemon monitors the Ethernet port and automatically enables DHCP on the appropriate IP interface. If no cable is plugged into a wired network, the nwamd daemon conducts a wireless scan and sends queries to the user for a WiFi access point to connect to. You don't need to spend extensive amounts of time manually configuring the interfaces on your systems. Automatic configuration also aids you in administration, because you can reconfigure network addresses with minimal intervention. To view your NWAM status, type the following command in a terminal window. # svcs nwam STATE STIME FMRI online 11:29:50 svc:/network/physical:nwam The OpenSolaris Network Auto-Magic Phase 0 page and nwamd man page contain further details, including instructions for turning off the nwamd daemon, if preferred. For more information and a link to the nwamd(1M) man page, see http://www.opensolaris.org/os/project/nwam. Zone OverviewA zone can be thought of as a container in which one or more applications run isolated from all other applications on the system. Most software that runs on OpenSolaris will run unmodified in a zone. Since zones do not change the OpenSolaris Application Programming Interface (APIs) or Application Binary Interface (ABI), recompiling an application is not necessary in order to run it inside a zone. Zones AdministrationZone administration consists of the following commands:
The following global scope properties are used with zones:
This lab exercise will introduce you to creating zones. SummaryThis exercise uses detailed examples to help you understand the process of creating, installing, and booting a zone. Note - This procedure does not apply to an lx branded zone. To Create, Install, and Boot a Zone
Each zone has its own characteristics, for example, zonename, IP addresses, hostname, naming services, root and non-root users. By default, the OS runs in a global zone. The administrator can virtualize the execution environment by defining one or more non-global zones. Network services can be run limiting the damage possible in the event of security violation. Since zones are implemented in software, they aren't limited to granularity defined by hardware boundaries. Instead zones offer sub-CPU granularity. Creating Non-Global ZonesThis lab exercise will demonstrate how to support two different sets of web server user groups on one physical host. SummarySimultaneous access to both web servers will be configured so that each web server and system will be protected should one become compromised. Creating Two Non-Global Zones
The end user sees each zone as a different system. Each web server has it's own name service:
A malicious attack on one web server is contained to that zone. Port conflicts are no longer a problem! Creating ZFS Storage Pools and File SystemsEach ZFS storage pool is comprised of one or more virtual devices, which describe the layout of physical storage and its fault characteristics. The most basic building block for a storage pool is a piece of physical storage. This can be any block device of at least 128 Mbytes in size. Typically, this is a hard drive that is visible to the system in the /dev/dsk directory. A storage device can be a whole disk (c0t0d0) or an individual slice (c0t0d0s7). The recommended mode of operation is to use an entire disk, in which case the disk does not need to be specially formatted. ZFS formats the disk using an EFI label to contain a single, large slice. In this module, we'll start by learning about mirrored storage pool configuration. Then we'll show you how to create a RAID-Z configuration. ZFS uses the concept of storage pools to manage physical storage. Historically, file systems were constructed on top of a single physical device. To address multiple devices and provide for data redundancy, the concept of a volume manager was introduced to provide the image of a single device so that file systems would not have to be modified to take advantage of multiple devices. This design added another layer of complexity and ultimately prevented certain file system advances, because the file system had no control over the physical placement of data on the virtualized volumes. Application issues a read. ZFS mirror tries the first disk. Checksum reveals that the block is corrupt on disk. ZFS tries the second disk. Checksum indicates that the block is good. ZFS returns good data to the application and repairs the damaged block on the first disk. Creating a Mirrored ZFS Storage PoolThe objective of this lab exercise is to create and list a mirrored storage pool using the zpool command. For information about determining whether a ZFS mirrored storage pool configuration or a RAID-Z storage pool configuration is right for your environment, go to: http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide SummaryZFS is easy, so let's get on with it! It's time to create your first pool: To Create Mirrored Storage Pools
The objective of this lab exercise is to learn how to set up ZFS file systems as several home directories. By using ZFS file system features, available in the OpenSolaris project, you might be able to simplify your kernel development environment by implementing snapshots and their rollback features. SummaryIn this lab, we'll use the zfs command to create a filesystem and set its mountpoint. To Create ZFS File Systems as Home Directories
The objective of this lab exercise is to introduce you to the RAID-Z configuration. SummaryYou might want to create a RAID-Z configuration as an alternative to a mirrored storage pool configuration if you need to maximize disk space. To Create a RAID-Z Configuration
The objective of this module is to introduce you to the userland consolidations of OpenSolaris. In general, you can think of userland consolidations as existing outside of the kernel and as components with which users interact. Each of the following consolidations deliver source files to the opensolaris.org web site or download center. To access each consolidation, refer to the following URL: http://opensolaris.org/os/downloads/ Userland Consolidations and Descriptions
The objective of this module is to describe the core features of the Solaris OS and how the features have fundamentally changed operating system computing. Additional ResourcesOpenSolaris Development Process; http://www.opensolaris.org/os/community/onnv/os_dev_process/ C Style and Coding Standards for SunOS; http://www.opensolaris.org/os/community/documentation/getting_started_docs/ Development Process and Coding StyleThe development process steps and the coding style that is used by the OS/Net consolidation (ON) are used to deliver the core Operating System and Networking components to Solaris. ON contains the source for the kernel and all platforms (on all architectures), the bulk of the drivers, filesystems, core libraries, and basic commands that you'd expect to find on a Solaris system.The development process for the OpenSolaris project follows the following high-level steps:
The Integration phase is to make sure everything that was supposed to be done has in fact been done, which means conducting reviews for code, documentation, and completeness. Sometimes, the integrated change needs to be communicated by sending heads-up messages to appropriate communities and possibly presenting a transfer of information (TOI) to a support organization to help them understand the change. The formal process document for OpenSolaris describes the previous steps in greater detail, with flow charts that illustrate the development phases. That document also details the following design principles and core values that are to be applied to source code development for the OpenSolaris project:
Refer to http://www.opensolaris.org/os/community/onnv/os_dev_process/ for more detailed information about the process that is used for collaborative development of OpenSolaris code. Like many projects, OpenSolaris enforces a coding style on contributed code, regardless of its source. This style is described in detail at http://opensolaris.org/os/community/onnv/. This coding style is very similar to that used by the Linux kernel, BSD systems, and many other non-GNU projects (the GNU project uses its own unique coding style). Also, examine the files in usr/src/prototypes; these provide examples of the correct general layout and style for most types of source files. Two tools for checking many elements of the coding style are available as part of the OpenSolaris distribution. These tools are cstyle(1) for verifying compliance of C code with most style guidelines, and hdrchk(1) for checking the style of C and C++ headers. There are style mistakes that cannot be caught by any reasonable tool, and others that cannot be caught by the particular implementations. Improving the accuracy and completeness of these tools is an ongoing task. OverviewNow that you have considered the development environment, processes, and values applied to engineering by OpenSolaris developers, let's discuss in more depth, features of the operating system that exemplify performance, security, serviceability, and manageability:
The "FireEngine" approach in Solaris 10 merges all protocol layers into one STREAMS module that is fully multithreaded. Inside the merged module, instead of per-data structure locks, a per-CPU synchronization mechanism called "vertical perimeter" is used. The "vertical perimeter" is implemented using a serialization queue abstraction called "squeue." Each squeue is bound to a CPU, and each connection is in turn bound to a squeue that provides any synchronization and mutual exclusion needed for the connection-specific data structures. SynchronizationSince the stack is fully multithreaded (barring the per-CPU serialization enforced by the vertical perimeter), it uses a reference-based scheme to ensure that the connection instance is available when needed. For an established TCP connection, three references are guaranteed to be on it. Each protocol layer has a reference on the instance (one each for TCP and IP) and the classifier itself has a reference since it is an established connection. Each time a packet arrives for the connection and the classifier looks up the connection instance, an extra reference is placed, which is dropped when the protocol layer finishes processing that packet. TCP, IP, and UDPThe Solaris 10 OS provides the same view for TCP as previous releases -- that is, TCP appears as a clone device but it is actually a composite, with the TCP and IP code merged into a single D_MP STREAMS module. The operational part of TCP is fully protected by the vertical perimeter that entered through the squeue primitives. FireEngine changes the interface between TCP and IP from the existing STREAMS- based message passing interface to a functional call-based interface, both in the control and data paths. There is a fully multithreaded UDP module running under the same protection domain as IP. Though UDP and IP are running in the same protection domain, they are still separate STREAMS modules. Therefore, STREAMS plumbing is kept unchanged and a UDP module instance is always pushed above IP. The Solaris 10 platform allows for the following plumbing modes:
Solaris 10 software introduces a new device driver framework called GLDv3 along with the new stack. Most of the major device drivers were ported to this framework, and all future and 10Gb device drivers will be based on this framework. This framework also provided a STREAMS-based DLPI layer for backward compatibility (to allow external, non-IP modules to continue to work). GLDv3 architecture virtualizes Layer two of the network stack. A one-to-one correspondence between network interfaces and devices no longer exists. Refer to the Nemo Project hosted on opensolaris.org for more information about the framework, the MAC services module, and the Data-Link Services module. VirtualizationProject Crossbow creates virtual stacks around any service (HTTP, HTTPS, FTP, NFS, etc.), protocol (TCP, UDP, SCTP, etc.), or Solaris Containers technology. The virtual stacks are separated by means of a H/W classification engine such that traffic for one stack does not impact other virtual stacks. Each virtual stack can be assigned its own priority and bandwidth on a shared NIC without causing performance degradation to the system or the service/container. The architecture dynamically manages priority and bandwidth resources, and can provide better defense against denial-of-service attacks directed at a particular service or container by isolating the impact to just that service or container. Least PrivilegeUNIX® has historically had an all-or-nothing privilege model that imposes the following restrictions:
In the Solaris OS we've developed fine-grained privileges. Fine-grained privileges allows applications and users to run with just the privileges they need. The least privilege allows students to be granted the privileges that they need to complete their course work, participate in research, and maintain a portion of the campus or department infrastructure. Packet FilteringSolaris IP Filter provides stateful packet filtering and network address translation (NAT). Solaris IP Filter is derived from the open source IP Filter software. IP Filter can filter by IP address, port, protocol, or network interface according to filter rules. IP FilterThe Packet Filtering Hooks (PFHooks) API has been introduced since Solaris 10 Update 4, to replace the STREAMS-based implementation of IP Filter. Using the PFHooks framework, the performance of firewall software like IP Filter is significantly improved. PFhooks also provides the ability to intercept loopback and inter-zone traffic. Third-party firewall software is developed and registered with the PFHooks API using the net_register_hook(info, event, hook); hook. Enabling Simple Packet FiltersThe objective of this exercise is to learn about packet filtering. Solaris IP Filter is installed with the Solaris operating system. However, packet filtering is not enabled by default. IP Filter can filter by IP address, port, protocol, or network interface according to filter rules. Following is an example filter rule: block in on ce0 proto tcp from 192.168.0.0/16 to any port = 23 To use Solaris IP Filter, simply enter your filter rules in the /etc/ipf/ipf.conf file. Then, enable and restart the svc:network/ipfilter service by using the svcadm command. Note - You can also use the ipf command to work with the rule sets. Solaris IP Filter can perform network address translation (NAT) for a source address or a destination address according to NAT rules. Following is an example of a NAT rule: map ce0 192.168.1.0/24 -> 10.1.0.0/16 To use network address translation, enter your NAT rules in the /etc/ipf/ipnat.conf file. Then, enable and restart the svc:/network/ipfilter service by using the svcadm command. Note - You can also use the ipnat command to work with rule sets. Sample Packet Filtering Rules This section includes various examples of filtering rule syntax. Invoke the rules by adding them to the /etc/ipf/ipf.conf file. Then, enable Solaris IP Filter and reboot your machine as detailed in the previous exercise. Log all inbound packets on le0 which has IP options present. log in on le0 from any to any with ipopts Block any inbound packets on le0 which are fragmented and too short on which to do any meaningful comparison. This actually only applies to TCP packets which can be missing the flags/ports (depending on which part of the fragment you see). block in log quick on le0 from any to any with short frag Log all inbound TCP packets with the SYN flag (only) set. Note - If it was an inbound TCP packet with the SYN flag set and it had IP options present, this rule and the above rule would cause it to be logged twice. log in on le0 proto tcp from any to any flags S/SA Block and log any inbound ICMP unreachables. block in log on le0 proto icmp from any to any icmp-type unreach Block and log any inbound UDP packets on le0 which are going to port 2049 (the NFS port). block in log on le0 proto udp from any to any port = 2049 Quickly allow any packets to/from a particular pair of hosts pass in quick from any to 10.1.3.2/32 pass in quick from any to 10.1.0.13/32 pass in quick from 10.1.3.2/32 to any pass in quick from 10.1.0.13/32 to any Block (and stop matching) any packet with IP options present. block in quick on le0 from any to any with ipopts Allow any packet through pass in from any to any Block any inbound UDP packets destined for these subnets. block in on le0 proto udp from any to 10.1.3.0/24 block in on le0 proto udp from any to 10.1.1.0/24 block in on le0 proto udp from any to 10.1.2.0/24 Block any inbound TCP packets with only the SYN flag set that are destined for these subnets. block in on le0 proto tcp from any to 10.1.3.0/24 flags S/SA block in on le0 proto tcp from any to 10.1.2.0/24 flags S/SA block in on le0 proto tcp from any to 10.1.1.0/24 flags S/SA Block any inbound ICMP packets destined for these subnets. block in on le0 proto icmp from any to 10.1.3.0/24 block in on le0 proto icmp from any to 10.1.1.0/24 block in on le0 proto icmp from any to 10.1.2.0/24Zones A zone is a virtual operating system abstraction that provides a protected environment in which applications run. The applications are protected from each other to provide software fault isolation. To ease the labor of managing multiple applications and their environments, they co-exist within one operating system instance, and are usually managed as one entity. A small number of applications which are normally run as root or with certain privileges may not run inside a zone if they rely on being able to access or change some global resource. An example might be the ability to change the system's time-of-day clock. The few applications which fall into this category may need applications to run properly inside a zone or in some cases, should continue to be used within the global zone. Here are some guidelines:
Super-user in a zone can't affect or obtain privileges in other zones. This allows students a safe sandbox in which to experiment. Zones can be used as instructional tool or infrastructure component For example, you can allocate each student an IP address and a zone and allow them all to safely share one machine. Zones can be combined with the resource management facilities which are present in OpenSolaris to provide more complete, isolated environments. While the zone supplies the security, name space and fault isolation, the resource management facilities can be used to prevent processes in one zone from using too much of a system resource or to guarantee them a certain service level. Together, zones and resource management are often referred to as containers. See http://opensolaris.org/os/community/zones/faq for answers to a large number of common questions about zones and links to the latest administration documentation. Zones provide protected environments for Solaris applications.Separate and protected run-time environments are available through the OpenSolaris project, by using BrandZ. Branded Zones (BrandZ)BrandZ is a framework that extends the zones infrastructure to create Branded Zones, which are zones that contain non-native operating environments. A branded zone may be as simple as an environment where the standard Solaris utilities are replaced by their GNU equivalents, or as complex as a complete Linux user space. BrandZ extends the Zones infrastructure in user space in the following ways:
BrandZ provides a set of interposition points in the kernel:
The lx brand enables Linux binary applications to run unmodified on Solaris, within zones that are running a complete Linux user space. The lx brand enables user-level Linux software to run on a machine with a OpenSolaris kernel, and includes the tools necessary to install a CentOS or Red Hat Enterprise Linux distribution inside a zone on a Solaris system. The lx brand will run on x86/x64 systems booted with either a 32-bit or 64-bit kernel. Regardless of the underlying kernel, only 32-bit Linux applications are able to run. This feature is only available for x86 and AMD x64 architectures at this time. However, porting to SPARC might be an interesting community project because BrandZ lx is still very much a work in progress. Refer to http://opensolaris.org/os/community/brandz/install for the installation requirements and instructions. The OpenSolaris project addresses the unique challenges of operating system development and testing for application performance using features like zones. Zones NetworkingSolaris zones can be designated as one of the following:
Exclusive-IP zones have their own IP stacks and may have their own physical interfaces. An exclusive-IP zone may also have its own VLAN interfaces. The configuration of exclusive-IP zones is the same as that of a physical machine. Shared-IP zones share the IP stack with the global zone, so shared-IP zones are shielded from the configuration details for devices, routing and so on. Each shared-IP zone can be assigned IPv4/IPv6 addresses. Each shared-IP zone also has its own port space. Applications can bind to INADDR_ANY and will only receive traffic for that zone. Both type of zones cannot see the traffic of other zones. Packets coming from a zone have a source address belonging to that zone. A shared-IP zone can only send packets on an interface on which it has an address. A shared-IP zone can only use a default router if it is directly reachable from the zone. The default router has to be in the same IP subnet as the zone. Shared-IP zones cannot change their network configuration or routing table and cannot see the configuration of other zones. /dev/ip is not present in the shared-IP zone. SNMP agents must open /dev/arp instead. Multiple shared-IP zones can share a broadcast address and may join the same multi-cast group. Shared-IP zones have the following networking limitations:
Exclusive-IP zones do not have the above limitations, and can change their network configuration or routing table inside the zone. /dev/ip is present in the exclusive-IP zone. Zones Identity, CPU Visibility, and PackagingEach zone controls its node name, timezone, and naming services like LDAP and NIS. The sysidtool can set this up. Separate /etc/passwd files mean that root privileges can be delegated to the zone. User IDs may map to different names when domains differ. By default, all zones see all CPUs. Restricted view is enabled automatically when resource pools are enabled. Zones can add their own packages. Patches can be made to those packages. System Patches are applied in the global zone. Then, in non-global zones the zone will automatically boot -s to apply the patch. The SUNW_PKG_ALLZONES package should be kept consistent between the global zone and all non-global zones. The SUNW_PKG_HOLLOW causes package name to appear in non-global zones (NGZ) for dependency purposes but the contents are not installed. Zones DevicesEach zone has its own devices. Zones see a subset of safe pseudo devices in their /dev directory. Applications reference the logical path to a device presented in /dev. The /dev directory exists in non-global zones, the /devices directory does not. Devices like random, console, and null are safe, but others like /dev/kmem are not. Zones can modify the permissions of their devices but cannot issue mknod(2). Physical device files like those for raw disks can be put in a zone with caution. Devices maybe shared among zones, but need careful security concerns before doing this. For example, you might have devices that you want to assign to specific zones. Allowing unprivileged users to access block devices could permit those devices to be used to cause system panic, bus resets, or other adverse effects. Placing a physical device into more than one zone can create a covert channel between zones. Global zone applications that use such a device risk the possibility of compromised data or data corruption by a non-global zone. Predictive Self-HealingPredictive self-healing was implemented in two ways in the Solaris 10 OS. This section describes the new Fault Management Architecture and Services Management Facility that make up the self-healing technology. Fault Management Architecture (FMA)The Solaris OS provides a new architecture, FMA, for building resilient error handlers, error telemetry, automated diagnosis software, response agents, and a consistent model of system failures for a management stack. Many parts of Solaris are already participating in FMA, including the CPU and Memory error handling for UltraSPARC III and IV, the UltraSPARC PCI HBAs, and Opteron. A variety of projects are underway, including full support for CPU, Memory, and I/O faults on Opteron, conversion of key device drivers, and integration with various management stacks. When a subsystem is converted to participate in Fault Management, error handling is made resilient so that the system can continue to operate despite some underlying failure, and telemetry events are produced that drive automated diagnosis and response. The Fault Management tools and architecture enable development of self-healing content for software and hardware failures, for both microscopic and macroscopic system resources, all with a unified, simple view for administrators and system management software. The Fault Manager associates diagnosis state with persistent identifiers corresponding to the system resources, such as hardware serial numbers. As a result, the Fault Manager automatically updates this state after most repair actions, without requiring any manual intervention. See http://opensolaris.org/os/community/fm for information about how to participate in the Fault Management community or to download the Fault Management MIB that is currently in development. The legacy UNIX failure model was simply to leave error handling up to each subsystem author, and simply provide the ability to emit an error message for a human to the system log in a non-standard format. Dynamic Tracing (DTrace)DTrace provides a powerful infrastructure to permit administrators, developers, and service personnel to concisely answer arbitrary questions about the behavior of the operating system and user programs. DTrace enables you to do the following: Historically, it has been difficult to look inside of complicated software systems with no way to see interactions between processes and no way to observe kernel activity. This makes it difficult to understand even a single application. DTrace is a new facility in the OpenSolaris project for systemic dynamic instrumentation. By using a special-purpose control language, DTrace can give concise answers to arbitrary questions about the system.
Find the DTrace community pages here http://opensolaris.org/os/community/dtrace. Note - Go to http://opensolaris.org/os/community/edu/curriculum_development/general_test/ to download the scripts, source code, and makefile for CCtest used later in this document to demonstrate C++ debugging with DTrace. In addition to DTrace, the OpenSolaris project provides debugging facilities for low-level types of development, for example, device driver development. This gives students the means to take software apart and understand its inner workings. It enables computer science educators to show principles from the classroom on a real, production machine. It allows researchers to better and more quickly understand and improve software systems. Modular Debugger (MDB)MDB is a debugger designed to facilitate analysis of problems that require low-level debugging facilities, examination of core files, and knowledge of assembly language to diagnose and correct. Generally, kernel and device developers rely on mdb to determine why and where their code went wrong. MDB is available as two commands that share common features: mdb and kmdb. You can use the mdb command interactively or in scripts to debug live user processes, user process core files, kernel crash dumps, the live operating system, object files, and other files. You can use the kmdb command to debug the live operating system kernel and device drivers when you also need to control and halt the execution of the kernel. There is an active community for MDB, where you can ask the experts or review previous conversations and common questions. See http://opensolaris.org/os/community/mdb ZFS File SystemZFS filesystems are not constrained to specific devices, so they can be created easily and quickly like directories. They grow automatically within the space allocated to the storage pool. Checksumming and Data RecoveryWith ZFS, all data and metadata is checksummed using a user-selectable algorithm. All checksumming and data recovery is done at the file system layer, and is transparent to applications. In addition, ZFS provides for self-healing data. ZFS supports storage pools with varying levels of data redundancy, including mirroring and a variation on RAID-5. When a bad data block is detected, ZFS fetches the correct data from another redundant copy, and repairs the bad data, replacing it with the good copy. Traditional file systems that do provide checksumming have performed it on a per-block basis, out of necessity due to the volume management layer and traditional file system design. The traditional design means that certain failure modes, such as writing a complete block to an incorrect location, can result in properly checksummed data that is actually incorrect. ZFS checksums are stored in a way such that these failure modes are detected and can be recovered from gracefully. Old way: create one filesystem, such as /export/home, to manage many user subdirectories. ZFS way: just create one filesystem per user. ZFS presents a pooled storage model that eliminates the concept of volumes and the associated problems of partitions, provisioning, wasted bandwidth, and stranded storage. Thousands of file systems can draw from a common storage pool, each one consuming only as much space as it actually needs. The combined I/O bandwidth of all devices in the pool is available to all filesystems at all times. Each storage pool is comprised of one or more virtual devices, which describe the layout of physical storage and its fault characteristics. See http://opensolaris.org/os/community/zfs/demos/basics for 100 Mirrored Filesystems in 5 Minutes, a demonstration of administering ZFS file systems. RAID-ZIn addition to pooled storage, ZFS provides redundant mirrored and RAID-Z data redundancy configurations. A RAID-Z configuration is a virtual device that stores data and parity on multiple disks, similar to RAID-5. All traditional RAID-5-like algorithms including RAID-4. RAID-5. RAID-6, RDP, and EVEN-ODD, for example, suffer from a problem known as the "RAID-5 write hole": if only part of a RAID-5 stripe is written, and power is lost before all blocks have made it to disk, the parity will remain out of sync with data. The parity is therefore useless forever, unless a subsequent full-stripe write overwrites it. In RAID-Z, ZFS uses variable-width RAID stripes so that all writes are full-stripe writes. This feature is only possible because ZFS integrates filesystem and device management in such a way that the filesystem's metadata has enough information about the underlying data replication model to handle variable-width RAID stripes. RAID-Z is the world's first software-only solution to the RAID-5 write hole. Services Management Facility (SMF)SMF creates a supported, unified model for management of an enormous number of services, such as email delivery, ftp requests, and remote command execution in the OpenSolaris project. The smf(5) framework replaces (in a compatible manner) the existing init.d(4) startup mechanism and includes an enhanced inetd(1M). SMF gives developers the following:
See http://opensolaris.org/os/community/smf/scfdot to see a graph of the SMF services and their dependencies on an x86 system freshly installed with the Solaris OS Nevada. Module 6Programming ConceptsObjectivesThis module provides a high-level description of the fundamental concepts of the OpenSolaris programming environment, as follows:
The basic unit of workload is the process. Process IDs (PIDs) are numbered sequentially throughout the system. By default, each user is assigned by the system administrator to a project, which is a network-wide administrative identifier. Each successful login to a project creates a new task, which is a grouping mechanism for processes. A task contains the login process as well as subsequent child processes. The resource pools facility brings together process-bindable resources into a common abstraction called a pool. Processor sets and other entities are configured, grouped, and labelled such that workload components are associated with a subset of a system's total resources. When the pools facility is disabled, all processes belong to the same pool, pool_default, and processor sets are managed through the pset() system call. When the pools facility is enabled, processor sets must be managed by using the pools facility. New pools can be created and associated with processor sets. Processes may be bound to pools that have non-empty resource sets. If we search OpenGrok for pool.c, we find extensive code comments that describe relationships between tasks, pools, projects, and processes, as follows: The operation that binds tasks and projects to pools is atomic. That is, either all processes in a given task or a project will be bound to a new pool, or (in case of an error) they will be all left bound to the old pool. Processes in a given task or a given project can only be bound to different pools if they were rebound individually one by one as single processes. Threads or LWPs of the same process do not have pool bindings, and are bound to the same resource sets associated with the resource pool of that process. Processes can optionally be run inside a zone. Zones are set up by system administrators, often for security purposes, in order to isolate groups of users or processes from one another. Threaded ProgrammingNow that we've learned about processes in the context of tasks, projects, resource pools, zones, and branded zones, let's discuss processes in the context of threads. Traditional UNIX already supports the concept of threads. Each process contains a single thread, so programming with multiple processes is programming with multiple threads. But, a process is also an address space, and creating a process involves creating a new address space. Creating a thread is less expensive than creating a new process because the newly created thread uses the current process address space. The time that is required to switch between threads is less than the time required to switch between processes. A switch between threads is faster because no switching between address spaces occurs. Communication between the threads of one process is simple because the threads share everything, inlcuding a common address space and open file descriptors. So, data produced by one thread is immediately available to all the other threads. The libraries are libpthread for POSIX threads, and libthread for OpenSolaris threads. Multithreading provides flexibility by decoupling kernel-level and user-level resources. In OpenSolaris, multithreading support for both sets of interfaces is provided by the standard C library. Use pthread_create(3C) to add a new thread of control to the current process. int pthread_create(pthread_t *tid, const pthread_attr_t *tattr,
void*(*start_routine)(void *), void *arg);The pthread_create() function is called with attr that has the necessary state behavior. start_routine is the function with which the new thread begins execution. When start_routine returns, the thread exits with the exit status set to the value returned by start_routine. pthread_create() returns zero when the call completes successfully. Any other return value indicates that an error occurred. Go to /on/usr/src/lib/libc/spec/threads.spec in OpenGrok for the complete list of pthread functions and declarations. Thread synchronization enables you to control program flow and access to shared data for concurrently executing threads. The four synchronization objects are mutex locks, read/write locks, condition variables, and semaphores.
Synchronization objects are variables in memory that you access just like data. Threads in different processes can communicate with each other through synchronization objects that are placed in threads-controlled shared memory. The threads can communicate with each other even though the threads in different processes are generally invisible to each other. Synchronization objects can also be placed in files. The synchronization objects can have lifetimes beyond the life of the creating process. We can use OpenGrok to find libthread in the source code tree, and the second most relevant result is found in mutex.c, accompanied by the following code comment excerpt: We can use OpenGrok to find libthread in the source code tree, and the second most relevant result is found in mutex.c, accompanied by the following code comment excerpt: Implementation of all threads interfaces between ld.so.1 and libthread. In a non-threaded environment all thread interfaces are vectored to noops. When called via _ld_concurrency() from libthread these vectors are reassigned to real threads interfaces. Two models are supported: TI_VERSION == 1 Under this model libthread provides rw_rwlock/rw_unlock, through which we vector all rt_mutex_lock/rt_mutex_unlock calls. Under lib/libthread these interfaces provided _sigon/_sigoff (unlike lwp/libthread that provided signal blocking via bind_guard/bind_clear. TI_VERSION == 2 Under this model only libthreads bind_guard/bind_clear and thr_self interfaces are used. Both libthreads block signals under the bind_guard/bind_clear interfaces. Lower level locking is derived from internally bound _lwp_ interfaces. This removes recursive problems encountered when obtaining locking interfaces from libthread. The use of mutexes over reader/writer locks also enables the use of condition variables for controlling thread concurrency (allows access to objects only after their .init has completed). Now that you understand a bit about how synchronization objects are defined in multi-threaded programming, let's learn how these objects are managed by using scheduling classes. CPU SchedulingProcesses run in a scheduling class with a separate scheduling policy applied to each class, as follows:
A scheduling class is maintained for each lightweight process (LWP). Threads have the scheduling class and priority of their underlying LWPs. Each LWP in a process can have a unique scheduling class and priority that are visible to the kernel. Thread priorities regulate contention for synchronization objects. The RT and TS scheduling classes both call priocntl(2) to set the priority level of processes or LWPs within a process. Using OpenGrok to search the code base for priocntl, we find the variables that are used in the RT and TS scheduling classes in the rtsched.c file as follows: We can use OpenGrok to quickly find the file and view its comments. 27 #pragma ident "@(#)rtsched.c 1.10 05/06/08 SMI"
28
29 #include "lint.h"
30 #include "thr_uberdata.h"
31 #include <sched.h>
32 #include <sys/priocntl.h>
33 #include <sys/rtpriocntl.h>
34 #include <sys/tspriocntl.h>
35 #include <sys/rt.h>
36 #include <sys/ts.h>
37
38 /*
39 * The following variables are used for caching information
40 * for priocntl TS and RT scheduling classs.
41 */
42 struct pcclass ts_class, rt_class;
43
44 static rtdpent_t *rt_dptbl; /* RT class parameter table */
45 static int rt_rrmin;
46 static int rt_rrmax;
47 static int rt_fifomin;
48 static int rt_fifomax;
49 static int rt_othermin;
50 static int rt_othermax;
...Typing the man priocntl command in a terminal window shows the details of each scheduling class and describes attributes and usage. For example: We can use the man command to view detailed man pages for more usage information and an explanation of the command options. % man priocntl
Reformatting page. Please Wait... done
User Commands priocntl(1)
NAME
priocntl - display or set scheduling parameters of specified
process(es)
SYNOPSIS
priocntl -l
priocntl -d [-i idtype] [idlist]
priocntl -s [-c class] [ class-specific options] [-
i idtype] [idlist]
priocntl -e [-c class] [ class-specific options] command
[argument(s)]
DESCRIPTION
The priocntl command displays or sets scheduling parameters
of the specified process(es). It can also be used to display
the current configuration information for the system's pro-
cess scheduler or execute a command with specified schedul-
ing parameters.
Processes fall into distinct classes with a separate
scheduling policy applied to each class. The process classes
currently supported are the real-time class, time-sharing
class, interactive class, fair-share class, and the fixed
priority class. The characteristics of these classes and the
class-specific options they accept are described below in
the USAGE section under the headings Real-Time Class, Time-
Sharing Class, Inter-Active Class, Fair-Share Class, and
--More--(4%)Kernel OverviewNow that you have a high-level understanding of processes, threads, and scheduling, let's discuss the kernel and how kernel modules are different from user programs. The Solaris kernel does the following:
The following section discusses several important differences between kernel modules and user programs. Execution Differences Between Kernel Modules and User ProgramsThe following characteristics of kernel modules highlight important differences between the execution of kernel modules and the execution of user programs:
The following characteristics of kernel modules highlight important differences between the structure of kernel modules and the structure of user programs:
Debugging processes at all levels of the development stack is a key part of writing kernel modules. A full search for libthread in OpenGrok, reveals the following code comments in the mdb_tdb.c file that describe the connection between multi-threaded debugging and how mdb works: Again, use OpenGrok to quickly find the file and view its code comments, as excerpted here: In order to properly debug multi-threaded programs, the proc target must be able to query and modify information such as a thread's register set using either the native LWP services provided by libproc (if the process is not linked with libthread), or using the services provided by libthread_db (if the process is linked with libthread). Additionally, a process may begin life as a single-threaded process and then later dlopen() libthread, so we must be prepared to switch modes on-the-fly. There are also two possible libthread implementations (one in /usr/lib and one in /usr/lib/lwp) so we cannot link mdb against libthread_db directly; instead, we must dlopen the appropriate libthread_db on-the-fly based on which libthread.so the victim process has open. Finally, mdb is designed so that multiple targets can be active simultaneously, so we could even have *both* libthread_db's open at the same time. This might happen if you were looking at two multi-threaded user processes inside of a crash dump, one using /usr/lib/libthread.so and the other using /usr/lib/lwp/libthread.so. To meet these requirements, we implement a libthread_db "cache" in this file. The proc target calls mdb_tdb_load() with the pathname of a libthread_db to load, and if it is not already open, we dlopen() it, look up the symbols we need to reference, and fill in an ops vector which we return to the caller. Once an object is loaded, we don't bother unloading it unless the entire cache is explicitly flushed. This mechanism also has the nice property that we don't bother loading libthread_db until we need it, so the debugger starts up faster. The following mdb commands can be used to access the LWPs of a multi-threaded program:
DTrace probes are constructed in a manner similar to MDB queries. We'll continue the hands-on lab exercises with DTrace and then add MDB when the debugging becomes more complex. Module 7Getting Started With DTraceObjectivesThe objective of this lab is to introduce you to DTrace using a probe script for a system call using DTrace. The DTrace provider includes three probes, BEGIN, END, and ERROR. BEGIN is the first probe to fire. All BEGIN clauses will fire before any other probe fires. BEGIN is typically used to initialize. END will fire after all other probes are completed and can be used to output results. ERROR fires under an error condition and is used for error handling. Additional Resources
Completion of the lab exercise will result in basic understanding of DTrace probes. SummaryWe're going to start learning DTrace by building some very simple requests using the probe named BEGIN, which fires once each time you start a new tracing request. You can use the dtrace(1M) utility's -n option to enable a probe using its string name. To Enable a Simple DTrace Probe
The objective of this lab is to explore probes in more detail and to show you how to list the probes on a system. SummaryIn the preceding examples, you learned to use two simple probes named BEGIN and END. But where did these probes come from? DTrace probes come from a set of kernel modules called providers, each of which performs a particular kind of instrumentation to create probes. For example, the syscall provider provides probes in every system call and the fbt provider provides probes into every function in the kernel. When you use DTrace, each provider is given an opportunity to publish the probes it can provide to the DTrace framework. You can then enable and bind your tracing actions to any of the probes that have been published. To List Traceable Probes
Now that you understand a little bit about naming, enabling, and listing probes, you're ready to write the DTrace version of everyone's first program, "Hello, World." SummaryThis lab demonstrates that, in addition to constructing DTrace experiments on the command line, you can also write them in text files using the D programming language. To Write a DTrace Program
Each D program consists of a series of clauses, each clause describing one or more probes to enable, and an optional set of actions to perform when the probe fires. The actions are listed as a series of statements enclosed in braces { } following the probe name. Each statement ends with a semicolon (;). Your first statement uses the function trace() to indicate that DTrace should record the specified argument, the string “hello, world”, when the BEGIN probe fires, and then print it out. The second statement uses the function exit() to indicate that DTrace should cease tracing and exit the dtrace command. DTrace provides a set of useful functions like trace() and exit() for you to call in your D programs. To call a function, you specify its name followed by a parenthesized list of arguments. The complete set of D functions is described in Solaris Dynamic Tracing Guide. By now, if you're familiar with the C programming language, you've probably realized from the name and our examples that DTrace's D programming language is very similar to C and awk(1). Indeed, D is derived from a large subset of C combined with a special set of functions and variables to help make tracing easy. If you've written a C program before, you will be able to immediately transfer most of your knowledge to building tracing programs in D. If you've never written a C program before, learning D is still very easy. But first, let's take a step back from language rules and learn more about how DTrace works, and then we'll return to learning how to build more interesting D programs. Module 8Debugging Applications With DTraceObjectivesThe objective of this module is to use DTrace to monitor application events. Additional ResourcesApplication Packaging Developer’s Guide. Sun Microsystems, Inc., 2005. Enabling User Mode ProbesDTrace allows you to dynamically add probes into user level functions. The user code does not need any recompilation, special flags, or even a restart. DTrace probes can be turned on just by calling the provider. The pid provider is extremely flexible and allows you to instrument any instruction in user land including entry and exit. The pid provider creates probes on the fly when they are needed. This is why they do not appear in the dtrace -l listing. You can use the pid provider to trace Function Boundaries or any arbitrary instruction in a given function. A probe description has the following syntax: pid:mod:function:name
In this exercise we will learn to use DTrace on user applications. SummaryThis lab builds on the use of a process ID in the probe description to trace the associated application. The steps increase in complexity to the end of the exercise, increasing the amount and depth of information about the application behavior that is output. To DTrace gcalctool
The examples in this module demonstrate the use of DTrace to diagnose C++ application errors. These examples are also used to compare DTrace with other application debugging tools, including Sun Studio 10 software and mdb. The DTrace command compiles the D language Script. DTrace instructs the provider to enable the probes. The intermediate code is checked for safety (like Java). The compiled code is executed in the kernel by DTrace. As soon as the D program exits all instrumentation is removed. Using DTrace to Profile and Debug A C++ ProgramA sample program CCtest was created to demonstrate an error common to C++ applications -- the memory leak. In many cases, a memory leak occurs when an object is created, but never destroyed, and such is the case with the program contained in this module. There is no limit (except system resources) on the number of D scripts that can be run simultaneously. Different users can debug the system simultaneously without causing data corruption or collision issues. When debugging a C++ program, you may notice that your compiler converts some C++ names into mangled, semi-intelligible strings of characters and digits. This name mangling is an implementation detail required for support of C++ function overloading, to provide valid external names for C++ function names that include special characters, and to distinguish instances of the same name declared in different namespaces and classes. For example, using nm to extract the symbol table from a sample program named CCtest produces the following output: # /usr/ccs/bin/nm CCtest ... [61] | 134549248| 53|FUNC |GLOB |0 |9 |__1cJTestClass2T5B6M_v_ [85] | 134549301| 47|FUNC |GLOB |0 |9 |__1cJTestClass2T6M_v_ [76] | 134549136| 37|FUNC |GLOB |0 |9 |__1cJTestClass2t5B6M_v_ [62] | 134549173| 71|FUNC |GLOB |0 |9 |__1cJTestClass2t5B6Mpc_v_ [64] | 134549136| 37|FUNC |GLOB |0 |9 |__1cJTestClass2t6M_v_ [89] | 134549173| 71|FUNC |GLOB |0 |9 |__1cJTestClass2t6Mpc_v_ [80] | 134616000| 16|OBJT |GLOB |0 |18 |__1cJTestClassG__vtbl_ [91] | 134549348| 16|FUNC |GLOB |0 |9 |__1cJTestClassJClassName6kM_pc_ ... Note - Go to http://opensolaris.org/os/community/edu/curriculum_development/general_test/ to download the scripts, source code, and makefile for CCtest. From this output, you may correctly assume that a number of these mangled symbols are associated with a class named TestClass, but you cannot readily determine whether these symbols are associated with constructors, destructors, or class functions. The Sun Studio compiler includes the following three utilities that can be used to translate the mangled symbols to their C++ counterparts: nm -C, dem, and c++filt. Note - Sun Studio 10 software is used here, but the examples were tested with both Sun Studio 9 and 10. If your C++ application was compiled with gcc/g++, you have an additional choice for demangling your application -- in addition to c++filt, which recognizes both Sun Studio and GNU mangled names, the open source gc++filt found in /usr/sfw/bin can be used to demangle the symbols contained in your g++ application. Examples: Sun Studio symbols without c++filt: # nm CCtest | grep TestClass [65] | 134549280| 37|FUNC |GLOB |0 |9 |__1cJTestClass2t6M_v_ [56] | 134549352| 54|FUNC |GLOB |0 |9 |__1cJTestClass2t6Mi_v_ [92] | 134549317| 35|FUNC |GLOB |0 |9 |__1cJTestClass2t6Mpc_v_ ... Sun Studio symbols with c++filt: # nm CCtest | grep TestClass | c++filt [65] | 134549280| 37|FUNC |GLOB |0 |9 |TestClass::TestClass() [56] | 134549352| 54|FUNC |GLOB |0 |9 |TestClass::TestClass(int) [92] | 134549317| 35|FUNC |GLOB |0 |9 |TestClass::TestClass(char*) ... g++ symbols without gc++filt: [86] | 134550070| 41|FUNC |GLOB |0 |12 |_ZN9TestClassC1EPc
[110] | 134550180| 68|FUNC |GLOB |0 |12 |_ZN9TestClassC1Ei
[114] | 134549984| 43|FUNC |GLOB |0 |12 |_ZN9TestClassC1Ev
...g++ symbols with gc++filt: # nm gCCtest | grep TestClass | gc++filt
[86] | 134550070| 41|FUNC |GLOB |0 |12 |TestClass::TestClass(char*)
[110] | 134550180| 68|FUNC |GLOB |0 |12 |TestClass::TestClass(int)
[114] | 134549984| 43|FUNC |GLOB |0 |12 |TestClass::TestClass()
...And finally, displaying symbols with nm -C: [64] | 134549344| 71|FUNC |GLOB |0 |9 |TestClass::TestClass()
[__1cJTestClass2t6M_v_]
[87] | 134549424| 70|FUNC |GLOB |0 |9 |TestClass::TestClass(const char*)
[__1cJTestClass2t6Mpkc_v_]
[57] | 134549504| 95|FUNC |GLOB |0 |9 |TestClass::TestClass(int)
[__1cJTestClass2t6Mi_v_]Let's use this information to create a DTrace script to perform an aggregation on the object calls associated with our test program. We can use the DTrace pid provider to enable probes associated with our mangled C++ symbols. To test our constructor/destructor theory, let's start by counting the following:
Use the following script to extract the symbols corresponding to the new() and delete() functions from the CCtest program: # dem `nm CCtest | awk -F\| '{ print $NF; }'` | egrep "new|delete"
__1c2k6Fpv_v_ == void operator delete(void*)
__1c2n6FI_pv_ == void*operator new(unsigned)The corresponding DTrace script is used to enable probes on new() and delete() (saved as CCagg.d): #!/usr/sbin/dtrace -s
pid$1::__1c2n6FI_pv_:
{
@n[probefunc] = count();
}
pid$1::__1c2k6Fpv_v_:
{
@d[probefunc] = count();
}
END
{
printa(@n);
printa(@d);
}Start the CCtest program in one window, then execute the script we just created in another window as follows: # dtrace -s ./CCagg.d `pgrep CCtest` | c++filt The DTrace output is piped through c++filt to demangle the C++ symbols, with the following caution. Caution - You can't exit the DTrace script with a ^C as you would do normally because c++filt will be killed along with DTrace and you're left with no output. To display the output of this command, go to another window on your system and type: # pkill dtrace Use this sequence of steps for the rest of the exercises: Window 1: # ./CCtest Window 2: # dtrace -s scriptname | c++filt Window 3: # pkill dtrace The output of our aggregation script in window 2 should look like this: void*operator new(unsigned) 12 void operator delete(void*) 8 So, we may be on the right track with the theory that we are creating more objects than we are deleting. Let's check the memory addresses of our objects and attempt to match the instances of new() and delete(). The DTrace argument variables are used to display the addresses associated with our objects. Since a pointer to the object is contained in the return value of new(), we should see the same pointer value as arg0 in the call to delete(). With a slight modification to our initial script, we now have the following script, named CCaddr.d: #!/usr/sbin/dtrace -s
#pragma D option quiet
/*
__1c2k6Fpv_v_ == void operator delete(void*)
__1c2n6FI_pv_ == void*operator new(unsigned)
*/
/* return from new() */
pid$1::__1c2n6FI_pv_:return
{
printf("%s: %x\n", probefunc, arg1);
}
/* call to delete() */ pid$1::__1c2k6Fpv_v_:entry
{
printf("%s: %x\n", probefunc, arg0);
}Execute this script: # dtrace -s ./CCaddr.d `pgrep CCtest` | c++filt Wait for a bit, then type this in window 3: # pkill dtrace Our output looks like a repeating pattern of three calls to new() and two calls to delete(): void*operator new(unsigned): 809e480 void*operator new(unsigned): 8068a70 void*operator new(unsigned): 809e4a0 void operator delete(void*): 8068a70 void operator delete(void*): 809e4a0 As you inspect the repeating output, a pattern emerges. It seems that the first new() of the repeating pattern does not have a corresponding call to delete(). At this point we have identified the source of the memory leak! Let's continue with DTrace and see what else we can learn from this information. We still do not know what type of class is associated with the object created at address 809e480. Including a call to ustack() on entry to new() provides a hint. Here's the modification to our previous script, renamed CCstack.d: #!/usr/sbin/dtrace -s
#pragma D option quiet
/*
__1c2k6Fpv_v_ == void operator delete(void*)
__1c2n6FI_pv_ == void*operator new(unsigned)
*/
pid$1::__1c2n6FI_pv_:entry
{
ustack();
}
pid$1::__1c2n6FI_pv_:return
{
printf("%s: %x\n", probefunc, arg1);
}
pid$1::__1c2k6Fpv_v_:entry
{
printf("%s: %x\n", probefunc, arg0);
}Execute CCstack.d in Window 2, then type pkill dtrace in Window 3 to print the following output: # dtrace -s ./CCstack.d `pgrep CCtest` | c++filt
libCrun.so.1`void*operator new(unsigned)
CCtest`main+0x19
CCtest`0x8050cda
void*operator new(unsigned): 80a2bd0
libCrun.so.1`void*operator new(unsigned)
CCtest`main+0x57
CCtest`0x8050cda
void*operator new(unsigned): 8068a70
libCrun.so.1`void*operator new(unsigned)
CCtest`main+0x9a
CCtest`0x8050cda
void*operator new(unsigned): 80a2bf0
void operator delete(void*): 8068a70
void operator delete(void*): 80a2bf0The ustack() data tells us that new() is called from main+0x19, main+0x57, and main+0x9a -- we're interested in the object associated with the first call to new(), at main+0x19. To determine the type of constructor called at main+0x19, we can use mdb as follows: # gcore `pgrep CCtest`
gcore: core.1478 dumped
# mdb core.1478
Loading modules: [ libc.so.1 ld.so.1 ]
> main::dis
main: pushl %ebp
main+1: movl %esp,%ebp
main+3: subl $0x38,%esp
main+6: movl %esp,-0x2c(%ebp)
main+9: movl %ebx,-0x30(%ebp)
main+0xc: movl %esi,-0x34(%ebp)
main+0xf: movl %edi,-0x38(%ebp)
main+0x12: pushl $0x8
main+0x14: call -0x2e4 <PLT=libCrun.so.1`__1c2n6FI_pv_>
main+0x19: addl $0x4,%esp
main+0x1c: movl %eax,-0x10(%ebp)
main+0x1f: movl -0x10(%ebp),%eax
main+0x22: pushl %eax
main+0x23: call +0x1d5 <__1cJTestClass2t5B6M_v_>
...Our constructor is called after the call to new, at offset main+0x23. So, we have identified a call to the constructor __1cJTestClass2t5B6M_v_ that is never destroyed. Using dem to demangle this symbol produces: # dem __1cJTestClass2t5B6M_v_ __1cJTestClass2t5B6M_v_ == TestClass::TestClass #Nvariant 1() Thus, a call to new TestClass() at main+0x19 is the cause of the memory leak. Examining the CCtest.cc source file reveals: ... t = new TestClass(); cout << t->ClassName(); t = new TestClass((const char *)"Hello."); cout << t->ClassName(); tt = new TestClass((const char *)"Goodbye."); cout << tt->ClassName(); delete(t); delete(tt); ... It's clear that the first use of the variable t = new TestClass(); is overwritten by the second use: t = new TestClass((const char *)"Hello.");. The memory leak has been identified and a fix can be implemented. The DTrace pid provider allows you to enable a probe at any instruction associated with a process that is being examined. This example is intended to model the DTrace approach to interactive process debugging. DTrace features used in this example include: aggregations, displaying function arguments and return values, and viewing the user call stack. The dem and c++filt commands in Sun Studio software and the gc++filt in gcc were used to extract the function probes from the program symbol table and display the DTrace output in a source-compatible format. Source files created for this example: Example 9-1 TestClass.hclass TestClass
{
public:
TestClass();
TestClass(const char *name);
TestClass(int i);
virtual ~TestClass();
virtual char *ClassName() const;
private:
char *str;
};
TestClass.cc:
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <unistd.h>
#include "TestClass.h"
TestClass::TestClass() {
str=strdup("empty.");
}
TestClass::TestClass(const char *name) {
str=strdup(name);
}
TestClass::TestClass(int i) {
str=(char *)malloc(128);
sprintf(str, "Integer = %d", i);
}
TestClass::~TestClass() {
if ( str )
free(str);
}
char *TestClass::ClassName() const {
return str;
}Example 9-2 CCtest.cc#include <iostream.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include "TestClass.h"
int main(int argc, char **argv)
{
TestClass *t;
TestClass *tt;
while (1) {
t = new TestClass();
cout << t->ClassName();
t = new TestClass((const char *)"Hello.");
cout << t->ClassName();
tt = new TestClass((const char *)"Goodbye.");
cout << tt->ClassName();
delete(t);
delete(tt);
sleep(1);
}
}Example 9-3 MakefileOBJS=CCtest.o TestClass.o
PROGS=CCtest
CC=CC
all: $(PROGS)
echo "Done."
clean:
rm $(OBJS) $(PROGS)
CCtest: $(OBJS)
$(CC) -o CCtest $(OBJS)
.cc.o:
$(CC) $(CFLAGS) -c $<Module 10Managing Memory with DTrace and MDBObjectivesThis module will build on what we've learned about using DTrace to observe processes by examining a page fault. Then, we'll incorporate low-level debugging with MDB to find the problem in the code. Additional ResourcesSolaris Modular Debugger Guide Sun Microsystems, Inc., 2007. Software Memory ManagementOpenSolaris memory management uses software constructs called segments to manage virtual memory of processes as well as the kernel itself. Most of the data structures involved in the software side of memory management are defined in /usr/include/vm/*.h. In this module, we'll examine the code and data structures used to handle page faults. The particular fault shown in this module is a major page fault, that is, it results in I/O on the disk. By contrast, a minor page fault does not result in I/O. For example, paging in a page of code for an executable is a major fault. Faulting a new heap page is a minor fault. Heap pages can simply be allocated and zeroed out (no need to access the disk). Using DTrace and MDB to Examine Virtual MemoryThe objective of this lab is to examine a page fault using DTrace and MDB. SummaryWe'll start with a DTrace script to trace the actions of a single page fault for a given process. The script prints the user virtual address that caused the fault, and then traces every function that is called from the time of the fault until the page fault handler returns. We'll use the output of the script to determine what source code needs to be examined for more detail. Note - In this module, we've added text to the extensive code output to guide the exercise. Look for the <----symbol to find associated text in the output. DTracing a Page Fault for a Single Process
The objective of this module is to learn about how you can use DTrace to debug your driver development projects by reviewing a case study. Porting the smbfs Driver from Linux to the Solaris OSThis case study focuses on leveraging the DTrace capability for device driver development. Historically, debugging a device driver required that a developer use function calls like cmn_err() to log diagnostic information to the /var/adm/messages file. This cumbersome process requires guesswork, re-compilation, and system reboots to uncover software coding errors. Developers with a talent for assembly language can use adb and create custom modules in C for mdb to diagnose software errors. However, historical approaches to kernel development and debugging are quite time-consuming. DTrace provides a diagnostic short-cut. Instead of sifting through the /var/adm/messages file or pages of truss output, DTrace can be used to capture information on only the events that you as a developer wish to view. The magnitude of the benefit provided by DTrace can best be provided through a few simple examples. First, create an smbfs driver template based on Sun's nfs driver. After the driver compiles successfully, test that the driver can be loaded and unloaded successfully. First copy the prototype driver to /usr/kernel/fs and attempt to modload it by hand: # modload /usr/kernel/fs/smbfs
can't load module: Out of memory or no room in system tablesAnd the /var/adm/messages file contains: genunix: [ID 104096 kern.warning] WARNING: system call missing from bind file Searching for the system call missing message, reveals it is in the function mod_getsysent() in the file modconf.c, on a failed call to mod_getsysnum. Instead of manually searching the flow of mod_getsysnum() from source file to source file, here's a simple DTrace script to enable all entry and return events in the fbt (Function Boundary Tracing) provider once mod_getsynum() is entered. #!/usr/sbin/dtrace -s
#pragma D option flowindent
fbt::mod_getsysnum:entry
/execname == "modload"/
{
self->follow = 1;
}
fbt::mod_getsysnum:return
{
self->follow = 0;
trace(arg1);
}
fbt:::entry
/self->follow/
{
}
fbt:::return
/self->follow/
{
trace(arg1);
}
Note - trace(arg1) displays the function's return value. Executing this script and running the modload command in another window produces the following output: # ./mod_getsysnum.d
dtrace: script './mod_getsysnum.d' matched 35750 probes
CPU FUNCTION
0 -> mod_getsysnum
0 -> find_mbind
0 -> nm_hash
0 <- nm_hash 41
0 -> strcmp
0 <- strcmp 4294967295
0 -> strcmp
0 <- strcmp 7
0 <- find_mbind 0
0 <- mod_getsysnum 4294967295Thus either find_mbind() returning '0', or nm_hash() returning '41' is the culprit. A quick look at find_mbind() reveals that a return value of 0 indicates an error state. Viewing the source to find_mbind() in /usr/src/uts/common/os/modsubr.c, reveals that we're searching for a char string in a hash table. Let's use DTrace to display the contents of the search string and hash table. To view the contents of the search string we add a strcmp() trace to our previous mod_getsysnum.d script: fbt::strcmp:entry
{
printf("name:%s, hash:%s", stringof(arg0),
stringof(arg1));
}Here are the results of our next attempt to load our driver: # ./mod_getsysnum.d
dtrace: script './mod_getsysnum.d' matched 35751 probes
CPU FUNCTION
0 -> mod_getsysnum
0 -> find_mbind
0 -> nm_hash
0 <- nm_hash 41
0 -> strcmp
0 | strcmp:entry name:smbfs,
hash:timer_getoverrun
0 <- strcmp 4294967295
0 -> strcmp
0 | strcmp:entry name:smbfs,
hash:lwp_sema_post
0 <- strcmp 7
0 <- find_mbind 0
0 <- mod_getsysnum 4294967295So we're looking for smbfs in a hash table, and it's not present. How does smbfs get into this hash table? Let's return to find_mbind() and observe that the hash table variable sb_hashtab is passed to the failing nm_hash() function. A quick search of the source code reveals that sb_hashtab is initialized with a call to read_binding_file(), which takes as its arguments a config file, the hash table, and a function pointer. A few more clicks on our source code browser reveal the contents of the config file to be defined as /etc/name_to_sysnum in the file /usr/src/uts/common/os/modctl.c. It looks like we forgot to include a configuration entry for my driver. Add the following to the /etc/name_to_sysnum file and reboot. 'smbfs 177'
(read_binding_file() is read once at boot time.)After rebooting the driver can be loaded successfully. # modload /usr/kernel/fs/smbfs Verify that the driver is loaded with the modinfo command: # modinfo | grep smbfs
160 feb21a58 351ac 177 1 smbfs (SMBFS syscall,client,comm)
160 feb21a58 351ac 24 1 smbfs (network filesystem)
160 feb21a58 351ac 25 1 smbfs (network filesystem version 2)
160 feb21a58 351ac 26 1 smbfs (network filesystem version 3)
Note - Remember that this driver was based on an nfs template, which explains this output. Let's make sure we can also unload the module: # modunload -i 160
can't unload the module: Device busyThis is most likely due to an EBUSY errno return value. But now, since the smbfs driver is a loaded module, we have access to all of the smbfs functions: # dtrace -l fbt:smbfs:: | wc -l
1002This is amazing! Without any special coding, we now have access to 1002 entry and return events contained in the driver. These 1002 function handles allow us to debug my work without a special 'instrumented code' version of the driver! Let's monitor all smbfs calls when modunload is called, using this simple DTrace script: #!/usr/sbin/dtrace -s
#pragma D option flowindent
fbt:smbfs::entry
{
}
fbt:smbfs::return
{
trace(arg1);
}It seems that the smbfs code is not being accessed by modunload. So, let's use DTrace to look at modunload with this script: #!/usr/sbin/dtrace -s
#pragma D option flowindent
fbt::modunload:entry
{
self->follow = 1;
trace(execname);
trace(arg0);
}
fbt::modunload:return
{
self->follow = 0;
trace(arg1);
}
fbt:::entry
/self->follow/
{
}
fbt:::return
/self->follow/
{
trace(arg1);
}
Here's the output of this script:
# ./modunload.d
dtrace: script './modunload.d' matched 36695 probes
CPU FUNCTION
0 -> modunload modunload 160
0 | modunload:entry
0 -> mod_hold_by_id
0 -> mod_circdep
0 <- mod_circdep 0
0 -> mod_hold_by_modctl
0 <- mod_hold_by_modctl 0
0 <- mod_hold_by_id 3602566648
0 -> moduninstall
0 <- moduninstall 16
0 -> mod_release_mod
0 -> mod_release
0 <- mod_release 3602566648
0 <- mod_release_mod 3602566648
0 <- modunload 16Observe that the EBUSY return value '16' is coming from moduninstall. Let's take a look at the source code for moduninstall. moduninstall returns EBUSY in a few locations, so let's look at the following possibilities:
We can't directly access all of these possibilities, but let's approach them from a process of elimination. We'll use the following script to display the contents of the various structures and return values in moduninstall: #!/usr/sbin/dtrace -s
#pragma D option flowindent
fbt::moduninstall:entry
{
self->follow = 1;
printf("mod_prim:%d\n",
((struct modctl *)arg0)->mod_prim);
printf("mod_ref:%d\n",
((struct modctl *)arg0)->mod_ref);
printf("mod_nenabled:%d\n",
((struct modctl *)arg0)->mod_nenabled);
printf("mod_loadflags:%d\n",
((struct modctl *)arg0)->mod_loadflags);
}
fbt::moduninstall:return
{
self->follow = 0;
trace(arg1);
}
fbt::kobj_lookup:entry
/self->follow/
{
}
fbt::kobj_lookup:return
/self->follow/
{
trace(arg1);
}
fbt::detach_driver:entry
/self->follow/
{
}
fbt::detach_driver:return
/self->follow/
{
trace(arg1);
}
This script produces the following output:
# ./moduninstall.d
dtrace: script './moduninstall.d' matched 6 probes
CPU FUNCTION
0 -> moduninstall
mod_prim:0
mod_ref:0
mod_nenabled:0
mod_loadflags:1
0 -> detach_driver
0 <- detach_driver 0
0 -> kobj_lookup
0 <- kobj_lookup 4273103456
0 <- moduninstall 16Comparing this output to the code tells us that the failure is not due to the mp structure values or the return values from detach_driver() of kobj_lookup(). Thus, by a process of elimination, it must be the status returned via the status = (*func)(); call, which calls the smbfs _fini() routine. And here's what the smbfs _fini() routine contains: int _fini(void)
{
/* don't allow module to be unloaded */
return (EBUSY);
}Changing the return value to '0' and recompiling the code results in a driver that we can now load and unload, thus we have completed the objectives of this exercise. We've used the Function Boundary Tracing provider exclusively in these examples. Note that fbt is only one of DTrace's many providers. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|