#pragma ident	"@(#)flowadm.1m.txt	1.16	08/03/03 SMI"


System Administration Commands			      flowadm(1M)



NAME
     flowadm - administer bandwidth resource control and priority
              for protocols, services, Containers and Virtual machines

SYNOPSIS

     flowadm add-flow [-t] [-R root-dir] -l link 
         -a attr=value[,...] -p prop=value[,...] flow-name

     flowadm remove-flow [-t] [-R root-dir] {-l link | flow-name}

     flowadm set-flowprop [-t] [-R root-dir] -p prop=value[,...] flow-name

     flowadm reset-flowprop [-t] [-R root-dir] [-p prop,...] flow-name

     flowadm show-flowprop [-cP] [-l link] [-p prop,...] [flow-name]

     flowadm show-flow [-pP] [-S] [-s [-i interval]] [-l link] [flow-name]

     flowadm show-usage [-d] -f <filename>
         [[-s <time>] [-e  <time>] -p ] [-l <link>] [flow-name>]

     flowadm -?

DESCRIPTION

     The flowadm command is used to create, modify, remove and show
     networking bandwidth and associated resources for a type of traffic on 
     a particular link.

CONCEPTS

     The flowadm command allows users to manage networking bandwidth
     resource for a transport, service, or a virtual machine. The
     service is specified as a combination of transport and port
     while a virtual machine is specified by its MAC address or an
     IP address. The command can also be used on Virtual NICs
     which are created for a virtual machine or Solaris Containers
     to allow the Virtual machine or Container administrator to
     further sub divide the bandwidth allocated to it when the VNIC
     was created.

     A flow is defined as a set of attributes based on Layer 2, 3 and
     4 headers which can be used to identify a protocol, service, or a
     virtual machine. When a flow is identified based on flow
     attributes, separate kernel resources including layer 2, 3, and 4
     queues, their processing threads, etc are uniquely created for it
     such that other traffic has minimal or zero impact on it. The 
     processing of the packets for a given flow can be restricted to
     a given CPU or processor set such that all the resources 
     necessary for processing the packets of the flow are dedicated to 
     it. The in bound packets for the identified transport, service or
     virtual machine are separated by the NIC hardware (when possible)
     which utilizes separate FIFOs and DMA channels so that bandwidth
     limits can be enforced with near zero performance impact.

     The flowadm command can be used to identify a flow without
     imposing any bandwidth resource control. This would result in 
     the traffic type getting its own hardware and software resources
     and queues so that its isolated from rest of the networking
     traffic for more deterministic behavior.


SUBCOMMANDS


     flowadm add-flow [-t] [-R root-dir] -l link 
         -a attr=value[,...] -p prop=value[,...] flow-name

	 Adds a flow to the system identified by its flow attributes and
         properties.
	 

	 As part of identifying a particular flow, its bandwidth
	 resource can be limited and its relative priority to other 
         traffic can be specified. If no bandwidth limit or priority is
	 specified, the traffic still gets its unique layer 2, 3, and 4
	 queues and processing threads including NIC hardware resources
	 (when supported) so that the selected traffic can be separated
	 from others and can flow with minimal impact from other traffic.

         -t, --temporary

             The changes are temporary and will not persist across reboots.


         -R root-dir, --root-dir=root-dir

             Specifies an alternate root  directory  where  flowadm should 
             apply persistent creation.


         -l <link>, --link=<link>

             Specify the link to which the flow will be added.


         -a attr=value,..., --attr attr=value,...
 
             A comma-separated list of attributes to be set  to  the
             specified values.


         -p prop=value,..., --prop prop=value,...
 
             A comma-separated list of properties to be set  to  the
             specified values.


     flowadm remove-flow [-t] [-R root-dir] {-l link|flow-name}

	 Remove an existing flow identified by its link and flow-name. 

         -t, --temporary

             The changes are temporary and will not persist across reboots.


         -R root-dir, --root-dir=root-dir

             Specifies an alternate root  directory  where  flowadm should 
             apply persistent removal.


         -l <link>, --link=<link> | flow-name

             If a link is specified, remove all flows from that link.  If
             a flow-name is specified, remove only that flow.


     flowadm set-flowprop [-t] [-R root-dir] -p prop=value[,...] flow-name

	 Set values of one or more properties on the flow specifed by 
         name.  The complete list of properties can be retrieved using 
         show-flow

         -t, --temporary

             The changes are temporary and will not persist across reboots.


         -R root-dir, --root-dir=root-dir

             Specifies an alternate root  directory  where  flowadm should 
             apply persistent creation.

         -p prop=value,..., --prop prop=value,...
 
             A comma-separated list of properties to be set  to  the
             specified values.


     flowadm reset-flowprop [-t] [-R root-dir] -p prop, ... flow-name

         Resets one or more properties to their default values  on  the
         specified specified flow . If no properties are specified,
         all properties are reset.

         -t, --temporary

             Specifies that the resets are  temporary.  Temporary
             resets last until the next reboot.

         -R root-dir, --root-dir=root-dir

             Specifies an alternate root  directory  where  flowadm
             should apply persistent creation.

         -p prop, ..., --prop=prop, ...

             A comma-separated list of properties to reset.


     flowadm show-flowprop [-cP] [-l link] [-p prop, ...] [flow-name]

         Show the current or persistent values  of  one  or  more
         properties,  either for all flows, flows on a specified link,
         or for the specified flow name.
         By default, current values are shown. If no  properties are 
         specified, all available flow properties are displayed. For  
         each  property,  the  following  fields are displayed:
 
         FLOW        The name of the flow.

         PROPERTY    The name of the property.

         VALUE       The current (or persistent) property  value.
                     The  value  is  shown  as "--", if it is not
                     set, and "?", if the value is unknown.  Per-
                     sistent values that are not set or have been
                     reset will be shown as "--" and will use the
                     system DEFAULT value (if any).
 
         DEFAULT     The default value of the  property.  If  the
                     property  has  no  default  value,  "--"  is
                     shown.

         POSSIBLE    A comma-separated list  of  the  values  the
                     property  may  have.  If  the  values span a
                     numeric range, min - max  may  be  shown  as
                     shorthand.   If   the  possible  values  are
                     unknown or unbounded, "--" is shown.

         Flow properties are documented in the FLOW PROPERTIES section.

         -c, --parseable

             Display using a stable machine-parseable format.

         -P, --persistent

             Display persistent flow property information

         -p prop, ..., --prop=prop, ...

             A comma-separated list of properties to show.

     flowadm show-flow [-pP] [-S] [-s [-i interval]] [-l link | flow-name]


         Show flow configuration information (the  default)  or
         statistics,  either for all flows, all flows on a link
         or for the specified flow-name.

         -p, --parseable
 
             Display using a stable machine-parseable format.


         -P, --persistent

            Display persistent flow property information.
 

	 -S, --continuous

             Continuously display network utilization by flow in a manner 
             similar to the way that prstat(1m) displays cpu utilization
             by process.
 

         -s, --statistics
 
             Displays flow statistics.
 
 
         -i interval, --interval=interval
 
             Used with the -s option to specify an  interval,  in
             seconds, at which statistics should be displayed. If
             this option is not specified, statistics  will  only
             be displayed once.


         -l link, --link=link | flow-name

             Display information for all flows on the named link or information
             for the named flow.


     flowadm show-usage [-d] -f <filename> 
         [[-s <time>] [-e <time>] -p][-l <link> | <flow-name> ]
 
      
         Show the historical network flow usage from a stored extended
         accounting file.  Configuration and enabling of net accounting 
         through acctadm(1m) is required.  The default output will be
         the summary of flow usage for the entire period of time in
         which extended accounting was enabled.
  
         -d
 
         Display the dates for which there is logging information. The
	 date is in the format DD/MM/YYYY.

         -f <filename>
 
         Read extended accounting records of network flow usage from the
         <filename>. 
 
         -s <time>, -e <time>
 
         Start and stop times for data display.  Time is in the format
	 YYYY.MM.DD,hh:mm:ss.

         -p
 
         When specified with -s and/or -e, outputs flow usage data in a
	 form suitable for input to gnuplot.
 
         -l <link-name>

         If specified, only print the network flow usage from the named
         link.  Otherwise, display network usage from all links.

         <flow-name>
 
         If specified, only print the network flow usage from the named
         flow.  Otherwise, display network usage from all flows.
 
	  
  FLOW ATTRIBUTES

     The flow attribute that identify a flow is a comma separated list
     of one or more keyword,value pair from below

     sap      Identifies a flow by the SAP  <value>. The <value>
	      may be specified in decimal, hexadecimal (preceded by
	      the 0x or 0X prefix) or octal (preceded by the prefix 0).

     ip_version={4|6}
             Version of IP protocol.
	     If unspecified, ip_version will be inferred from local_ip
	     and remote_ip attributes. In the absence of local_ip and
	     remote_ip and ip_version attributes, default ip_version
	     will be considered as IPV4. 

      local_ip[/prefix_len]
 	      Identifies a network flow by the local IP address.
 	      <value> must be a IPv4 address in dotted-decimal notation
 	      or an IPv6 address in colon-separated notation.
	      prefix_len is optional.
	      If specified, it describes the netmask for a subnet address, 
	      following the same notation convention of ifconfig(1M) and
	      route(1M) addresses.
	      If unspecified, given IP address will be considered as
	      host address for which default prefix length for a
	      IPv4 address is /32 and for IPv6 it is /128.
		
      remote_ip[/prefix_len]
 	      Identifies a network flow by the remote IP address.
 	      <value> must be a IPv4 address in dotted-decimal notation
 	      or an IPv6 address in colon-separated notation.
	      If specified, it describes the netmask for a subnet address, 
	      following the same notation convention of ifconfig(1M) and
	      route(1M) addresses.
	      If unspecified, given IP address will be considered as
	      host address for which default prefix length for a
	      IPv4 address is /32 and for IPv6 it is /128.

     transport={tcp|udp|icmpv6|sctp|<transport number>}
	     Identifies a layer 4 protocol to be used. Is typically used
	     in combination with local_port to identify the service which 
 	     needs special attention.

     local_port
	     Identifies a service specified by the local port.

     remote_port
	     Identifies a service identified by the remote port

     dsfield
	     Identifies a value in the 8-bit differentiated services
	     field (as defined in RFC 2474).

     dsfield_mask
	     Specifies a mask to be used along with the dsfield value.
	     The mask is used to state the bits of interest in the
	     differentiated services field when comparing with the
	     dsfield value. A '0' in a bit position indicates that
	     the bit value needs to be ignored and a '1' indicates
	     otherwise. Thus, the mask can range from 1 to 255. When
	     a mask is not specified, the value is defaulted to 255,
	     i.e. check all the bits in the dsfield. A mask of 0 is
	     invalid.


 RESTRICTIONS
      Individual flow restrictions

          Restrictions on individual flows do not require knowledge of other
          flows that have been added to the link.

          Duplicate attributes
	       An attribute may only be listed once for each flow.

	       Not Valid:
	       #flowadm add-flow -l vnic1 -a local_port=80,local_port=8080
	                           httpflow

          ip_version, local_ip and remote_ip
 	       Flows which specify both local_ip and remote_ip but have
 	       addresses from different versions are not allowed.

 	       Flows which specify an ip_version which conflicts with the
 	       ip_version of local_ip or remote_ip are not allowed.

	       Valid:
 	       #flowadm add-flow -l vnic1 -a ip_version=6 v6flow
 	       #flowadm add-flow -l vnic1 -a ip_version=4,local_ip=10.10.10.1 
                                  v4flow

 	       Not Valid:
 	       #flowadm add-flow -l vnic1 -a local_ip=10.10.10.1,
                                 remote_ip=::1:2:3 flow1
 	       #flowadm add-flow -l vnic1 -a ip_version=6,local_ip=10.100.100.1
                                 flow1
 	       #flowadm add-flow -l vnic1 -a ip_version=6,ip_version=4 flow1
	
           transport, local_port and remote_port
 	       TCP, UDP and SCTP flows can be specified with a local or 
 	       remote port.  An ICMPv6 flow which specifies a port will 
 	       not be allowed.

 	       If a local or remote port is specifed, the transport must
 	       be either TCP, UDP or SCTP.
 
 	       Valid:
 	       #flowadm add-flow -l e1000g0 -a transport=udp udpflow
 	       #flowadm add-flow -l e1000g0 -a transport=tcp,local_port=80
                                 udp80flow

 	       Not valid:
 	       #flowadm add-flow -l e1000g0 -a local_port=25 flow25
 	       #flowadm add-flow -l e1000g0 -a transport=icmpv6,port=16 flow16
  
      Flow restrictions per link

          Disjoint flows
 
 	  Two flows are considered disjoint if they specify a different
          value for the same attribute.  Disjoint flows are allowed.

	      #flowadm add-flow -l bge0 -a transport=tcp flow1
 	      #flowadm add-flow -l bge0 -a transport=udp flow2

          Enclosed flows
 
          
          A flow is considered to be enclosed by another flow if all of the 
          flow's packets to the other flow.  The properties of the flow which 
          is more specific will be applied.  In the example, all packets 
          that match flow2 will also match flow1.  As flow2 is more 
          specific, its resource constraint (maxbw=200) will be used.

 	      #flowadm add-flow -l bge0 -a transport=tcp -p maxbw=300 tcpflow
 	      #flowadm add-flow -l bge0 -p transport=tcp,local_port=80 
                                -p maxbw=200 tcp80flow 
       
          Intersecting flows

 	  Two flows are intersecting if a packet can belong to both
 	  flows, but other packets can belong only one of the flows.
 	  The creation of a flow which intersects an existing flow 
          will not be allowed.

 	      #flowadm add-flow -l nxge0 -a transport=tcp -p maxbw=100 tcpflow
 	      #flowadm add-flow -l nxge0 -a local_ip=10.100.100.80  
                               -p maxbw=200 localipflow

	  CPU bindings and fanout

	  A flow on a primary MAC client (i.e. bge0, e1000g1 etc), can have
	  a CPU list or fanout specified as per the following:

		o  If the underlying link has a CPU list specified, the
		   flow's CPU list must be equal to or a subset of the link's
		   CPU list.

		o  If the underlying link has a fanout, a flow can
		   have a fanout less than or equal to that of the link's
		   fanout. Moreover, a flow cannot have a CPU list
		   specified in this case.

		o  If the underlying link has a CPU list specified, a
		   flow can have a fanout that is less than or equal to
		   the number of CPUs in the link's CPU list.

		o  If the underlying link doesn't have a fanout or CPU
		   list, a flow can specify any fanout or CPU list.

	  A flow on a MAC client other than the primary (say a VNIC), cannot
	  have a fanout or CPU list.


      Flow restrictions per zone

          Flow names

 	        Within a zone, no 2 flows may have the same name.  Conversely,
                after adding the flow with the link specified, the link will
                not be required for display, modifcation or deletion of the 
                flow.

          Links with exclusive IP stacks
           
                The global zone is prevented from creating flows on top of
                a data-link which is assigned to a non-global zone.

 FLOW PROPERTIES

     The following flow properties are supported. Note  that
     the ability to set a given property to a given value depends
     on the driver and hardware.
 

     maxbw   Sets the full duplex bandwidth for the flow. The
             bandwidth is specified as an integer with one of the
             scale suffixes(K, M, or G for Kbps, Mbps, and Gbps).
             The minimum value of maxbw is 1.2 Mpbs.

 
     priority
             Sets the relative priority for the  flow. The value may
             be given as one of the tokens  rt(real time), high, normal,
             or low.

     cpus    Bind the processing of packets for a given flow to
             a processor or a set of processors. The value can be
             a comma separated list of one or more processor id.
             If the list consists of more than one processor,
             the processing will spread out to all the processors
             although connection to processor affinity and packet
             ordering for packets of any individual connection will
             be maintained.
 
             The processor or set of processors are not exclusively
             reserved  for the network device. Only the kernel
             threads and interrupt  associated with processing of
             the device are bound to the processor or the set of
             processor specified. In case, it is desired that
             processors be dedicated to the device, psrset(1M) can
             be used to create a processor_set and then
             specifiying the processors from  the processor_set to
             bind the device to.
 
             If the flow was already bound to processor or set
             of processors  due to a previous operation, the binding
             will be removed and the  new set of processors will be
             used instead.

	     Processor (or sets) can be specified only for a flow that
	     is on a physical NIC (see RESTRICTIONS for more on this).

EXAMPLES

	Example 1: Create a policy around mission critical port 443 traffic
	which is https service.

	To create a policy around inbound https traffic on a https server
	so that https gets it dedicated NIC hardware and kernel TCP/IP
	resources. The flow-name specified is https-1 which is used to
	later modify of delete the policy.

	# flowadm add-flow -l bge0 -a transport=TCP local_port=443 https-1

	Example 2: Modify an existing policy to add bandwidth resource control

	To modify https-1 policy to add bandwidth control, give it a 
	high priority and bind the processing of the flow to CPU# 9
	
	# flowadm set-flowprop -p maxbw=500Mbps,priority=high,cpus=9
	 https-1

	Example 3: Limit the bandwidth usage of UDP protocol

	To create a policy for UDP protocol so that it can not consume more
	than 100Mbps of available bandwidth. The flow-name is called
	limit-udp-1.

	# flowadm add-flow -l bge0 -a transport=UDP -p maxbw=100Mbps
	 priority=low limit-udp-1

	Example 4: Set a policy for EF PHB (DSCP value of 101110 from
	RFC 2598) with bandwidth of 500 Mbps and high priority. The dsfield
	value for this flow will be 46 (101110) with the dsfield_mask
	being 252 (since we want to ignore the 2 lest significant bits).

	# flowadm add-flow -l bge0 -a dsfield=46,dsfield_mask=252 
	 -p maxbw=500Mbps,priority=high efphb-flow

	Example 5: Show flow usage

	Flow usage statistics can be stored using the extended accounting
	facility acctadm(1m).

	# acctadm -e extended -f /var/log/net.log net

	# acctadm net
	    Network accounting: active
	    Network accounting file: /var/log/net.log
	    Tracked Network resources: extended
	    Untracked Network resources: none

	The historical data which was saved can be retrieved in summary
	form using the show-usage option of flowadm.


	Display summary information

	# flowadm show-usage -f /var/log/net.log
	Bytes       Packets   Errors    Duration       Bandwidth      Link/Flow
	________________________________________________________________________
	546908      1031      0         100           43.75 Kbps        flowtcp
	0           0         0         0              0.00 bps         flowudp

	Display dates for which logging information is available

	# flowadm show-usage -d -f /var/log/net.log
	02/19/2008

	Display logging information for 'flowtcp' starting at
	02/19/2008,10:38:46 and ending at 02/19/2008,10:40:06

	# flowadm show-usage -s02/19/2008,10:38:46 -e02/19/2008,10:40:06
	    -f /var/log/net.log flowtcp
	Start Time  End Time    In Bytes     Out Bytes     Bandwidth    Device
	______________________________________________________________________
	10:38:46    10:39:06    1546         6539          3.23 Kbps    flowtcp
	10:39:06    10:39:26    3586         9922          5.40 Kbps    flowtcp
	10:39:26    10:39:46    240          216          182.40 bps    flowtcp
	10:39:46    10:40:06    0            0              0.00 bps    flowtcp

	Output the same information as above as a plotfile

	# flowadm show-usage -s02/19/2008,12:28:04 -e02/19/2008,12:32:04
	    -p -f /var/log/net.log flowtcp
	# Time tcp-flow
	10:39:06 3.23
	10:39:26 5.40
	10:39:46 0.18
	10:40:06 0.00


EXIT STATUS
     The following exit	values are returned:
     0	      All actions were performed successfully.

     >0	      An error occurred.



ATTRIBUTES
     See attributes(5) for descriptions	of the	following  attri-
     butes:
     ____________________________________________________________
    |	    ATTRIBUTE TYPE	  |	  ATTRIBUTE VALUE	|
    |_____________________________|_____________________________|
    |_____________________________|_____________________________|
    | Interface	Stability	  | Evolving			|
    |_____________________________|_____________________________|

SEE ALSO
     dladm(1M), attributes(5), dlpi(7P), ifconfig(1M), route(1M),  acctadm(1m)

NOTES
     The configuration of all policies will persist across reboot.