[MidoNet-dev] Draft Proposal: rate-limiting in Midonet

Guillermo Ontañón guillermo at midokura.jp
Fri Feb 22 11:48:31 UTC 2013

On Thu, Feb 21, 2013 at 9:01 AM, Dan Mihai Dumitriu <dan at midokura.com>wrote:

> This looks good to Me Guillermo!
> On Wed, Feb 20, 2013 at 3:00 AM, Guillermo Ontañón <guillermo at midokura.jp>wrote:
>> Hi devs,
>> Below is my draft spec for Midonet's rate-limiting feature, looking
>> forward to discussing it:
>> Feature Description
>> -------------------
>> Add a rate-limiting feature targetted at cloud service providers, who
>> may want to limit the bandwidth available to their tenants in various
>> ways.
>> Use cases
>> ---------
>> There are three use cases, with varying degrees of importance/
>> usefulness, that Midonet could support:
>> Use case 1: Limit the uplink egress bandwitdh
>> Two reasons why this may make sense for a cloud service provider:
>> * Pricing options based on uplink speed.
>> * Resource sharing. The uplink is a shared resource and likely
>>   oversubscribed, rate limiting provides a limit into how much a given
>>   subscriber can affect resource allocation to the rest of subscribers.
>> Use case 2: Limit east-west traffic
>> Though much less important than use case #1. The motivations are
>> similar.
>> Use case 3. Limit the global egress bandwidth for a VM
>> Reasons for doing this:
>> * Having a global bandwidth limit for the VM makes it less necessary to
>>   explicitly support the 'east-west' use case. If that's the case,
>>   east-west bandwidth for a VM would be the global limit minus its
>>   current uplink usage.
>> * Fair sharing of the network link in a host among all guests. Note
>>   that, this is not real or perfect 'sharing' if the limits of the local
>>   VMs aren't somehow balanced to fully utilize the host's link.
>> * Limiting the guest's bandwidth to ensure services running on the host
>>   have guaranteed bandwidth.
>> Design/implementation implications of use cases
>> -----------------------------------------------
>> Use cases #1 and #2 clearly require that traffic be mapped to a rate
>> limiting queue on a per-flow basis. With the difference that use case #1
>> could have simpler design/implementations because all the rate-limited
>> traffic goes through a few edge nodes. We will call this flow-based
>> rate-limiting.
>> Use case #3 is very clearly the simplest from all points of view,
>> blindly rate limiting all traffic that ingresses the virtual topology
>> from a given VM vport. We will call this port-based rate-limiting.
>> Use cases #1 and #2 also pose the question of whether the rate-limiting
>> is applied to each VM (or rather, vport) independently or it can be
>> applied to each tenant or arbitrary group of VMs as a whole.
>> The latter, together with flow-based limiting, is a full generalisation
>> of the problem because it requires a distributed rate-limiting queue (on
>> top of the per-flow classification).  This is, for the time being, out
>> of the scope for this feature.
>> Furthermore, in a vport-local scenario, rate-limiting may be applied on
>> the ingress vport, on egress, or both.
>> Features supported in the Quantum NVP QoS feature
>> -------------------------------------------------
>> The NVP QoS feature for Quantum [0] is in code-review [1] state. It
>> offers:
>>   * Putting a rate-limiting queue in a VMs vport (no flow matching, not
>>     a distributed queue).
>>   * The settings for port's queue may be chosen individually or inherit
>>     from the queue set for the 'network'. So an admin may just configure
>>     one queue and have a whole set of VMs use those settings.
>>   * A rate-limiting queue can set:
>>     * max bandwidth rate
>>     * min bandwidth rate ???
>>     * qos_marking (unclear what it does, could it mark the packet for
>>       the kernel to custom further qos/processing on the packet?)
> Not sure what this is.  It might be setting the mark field on the skb,
> which could then be matched by linux tc.

Yep, that's what I understand, but I want to make 100% sure.

>>      * dscp (unclear too)
> Sets the DSCP field of the outer ip header.  This could be matched by the
> underlay switches and mapped to a traffic class, which needs to have been
> setup a priori.  This is useful as well.  (I might put this into a
> different action though.)

So we'd have three new non-terminating actions: QUEUE, QOS_MARK, DSCP.

If we have separate actions, maybe the Queue object is not needed until
when/if we do more fancy qos, for now it would only have a 'max-bandwidth'
field and that could be set with the QOS action, or rather RATE_LIMIT.


>> Proposed Rate Limiting feature for Midonet Diyari 13.06
>> -------------------------------------------------------
>> Midonet will support ingress vport-local flow-based rate-limiting.
>> Meaning that:
>>   * Queues are local to a vport and applied on ingress only, a flow may
>>     only traverse one queue.
>>   * Flows are assigned to queues independently.
>> Therefore, all of the use cases discussed above would be covered,
>> although in a non-distributed manner.
>> Flow classification will use the existing chains/rules infrastructure.
>> Only one extension is needed: a new QOS action for rules. Upon matching
>> on a rule with a QOS action, Chain.apply() will:
>>   * Is the flow already marked for QOS by an eearlier rule?
>>     - Yes -> do nothing.
>>     - No  -> mark the flow with the queue id given by the rule
>>   * Continue processing on the next rule (this is a non-terminating
>>     rule)
>> QOS rules will have a queueId UUID attribute. Queue objects need not
>> exist per-vport, their configuration will be used to create a queue
>> when traffic ingressing a vport is marked with a given queue id, but a
>> queue object is not bound to a vport it can be reused by rules
>> regardless of ingress port.
>> Necessary REST API changes
>> --------------------------
>> Changes in RuleDto:
>>   * 'flowAction' will get a new allowed value: 'queue'.
>>   * New field: queueId. It MUST have a value if flowAction is 'queue',
>>     and that value must be id of a valid QueueDto object.
>> A new QueueDto is created, with fields:
>>   * id
>>   * maxRateKbps Maximum ingress (into the virtual topology - egress from
>>     the pov of the VM) bandwidth.
>>   * qosMark (to-be-discussed) - this would mark the packet for matching
>>     in processing by the linux qos stack, letting admins setup more
>>     sophisticated QoS behaviour. This would use OVS skb-mark action.
>> A new top-level URL for the QueueDto:
>>   * /queues/ - ApplicationDto would get a new getQueues() method.
>>   * /queues/:queue_id/ (URL for a Queue - POST, DELETE, GET)
>> Regards,
>> -- Guillermo Ontañón
>> _______________________________________________
>> MidoNet-dev mailing list
>> MidoNet-dev at lists.midonet.org
>> http://lists.midonet.org/listinfo/midonet-dev

-- Guillermo Ontañón
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.midonet.org/pipermail/midonet-dev/attachments/20130222/990b6b79/attachment-0001.html>

More information about the MidoNet-dev mailing list