API stabilization patches (rev 3)
by Michal Fojtik
Hi,
I added few more patches to go:
4/7: Fixed missing ID elements in storage_volumes and hardware_profiles
5/7: Added comments to feature for brief listing. I want to keep these
features commented here just for testing in future.
6/7: This will be a little bit longer patch :)
I removed old client completely and rewrote it from scratch. I used
some meta magic to get auto-generated classes and methods.
This approach will allow us to add a new elements/attribute in future to API
and 'client' library will update itself.
Also REXML was removed completely and replaced by faster Nokogiri.
Just one small thing is still missing here. I don't have '.to_plain' methods
implemented yet, so command line client will not work... (Need +1 day ;)
7/7: As a bonus, *all* RSpec tests now pass nicely.
There are +2 incompatibilities:
instance.instance_profile => instance.hardware_profile
realm.limit => NoMethod exception (when no limit is set for realm)
-- Michal
PS: Apply this patch to latest 'master'.
14 years
[PATCH aggregator] Added DataService
by Martyn Taylor
From: martyntaylor <mtaylor(a)redhat.com>
---
src/app/services/data_service.rb | 112 +++++++++++++++++++++++
src/spec/services/data_service_spec.rb | 157 ++++++++++++++++++++++++++++++++
2 files changed, 269 insertions(+), 0 deletions(-)
create mode 100644 src/app/services/data_service.rb
create mode 100644 src/spec/services/data_service_spec.rb
diff --git a/src/app/services/data_service.rb b/src/app/services/data_service.rb
new file mode 100644
index 0000000..6465f2e
--- /dev/null
+++ b/src/app/services/data_service.rb
@@ -0,0 +1,112 @@
+class DataService
+
+ QoSDataPoint = Struct.new(:time, :average, :max, :min)
+ QuotaUsagePoint = Struct.new(:used, :max)
+ TotalQuotaUsagePoint = Struct.new(:name, :no_instances)
+
+ # This will return array of data points between start and end, if there is a data point where the interval start + interval end
+ # is greater than the end time, it will be ignored
+ # Example:
+ # start = 12.30, end = 12.32, interval = 45secs
+ # Intervals: 12.30.00 - 12.30.45, 12.30.45 - 12.31.30 will be returned. Interval 12.31.30 - 12.32.15 will not
+ def self.qos_task_submission_stats(start_time, end_time, interval_length, parent, action)
+
+ instances = []
+
+ if parent.class == Provider
+ cloud_accounts = CloudAccount.find(:all, :conditions => {:provider_id => parent.id})
+ cloud_accounts.each do |cloud_account|
+ instances.concat(instances)
+ end
+ elsif parent.class == Pool || parent.class == CloudAccount
+ instances = parent.instances
+ else
+ return nil
+ end
+
+ return calculate_qos_task_submission_stats(start_time, end_time, interval_length, instances, action)
+ end
+
+ def self.tasks_submissions_mean_max_min(time, tasks)
+
+ first_pass = true
+
+ total_time = nil
+ maximum_time = nil
+ minimum_time = nil
+
+ tasks.each do |task|
+
+ if(first_pass == true)
+ total_time = task.submission_time
+ maximum_time = task.submission_time
+ minimum_time = task.submission_time
+ first_pass = false
+ else
+ total_time += task.submission_time
+
+ if task.submission_time > maximum_time
+ maximum_time = task.submission_time
+ end
+
+ if task.submission_time< minimum_time
+ minimum_time = task.submission_time
+ end
+ end
+
+ end
+ average_time = total_time / tasks.length
+
+ return QoSDataPoint.new(time, average_time, maximum_time, minimum_time)
+ end
+
+ # Returns the Used and Maximum Number of Instances in Quota
+ def self.quota_utilisation(parent)
+ quota = parent.quota
+ if quota
+ return QuotaUsagePoint.new(quota.total_instances, quota.maximum_total_instances)
+ end
+ end
+
+ def self.total_quota_utilisation(provider)
+ data_points = []
+ free_instances = 0
+
+ cloud_accounts = CloudAccount.find(:all, :conditions => {:provider_id => provider.id})
+ cloud_accounts.each do |cloud_account|
+ quota = cloud_account.quota
+ if quota
+ data_points << TotalQuotaUsagePoint.new(cloud_account.username, quota.total_instances)
+ free_instances += (quota.maximum_total_instances - quota.total_instances)
+ end
+ end
+ data_points << TotalQuotaUsagePoint.new("free", free_instances)
+ return data_points
+ end
+
+ private
+ def self.calculate_qos_task_submission_stats(start_time, end_time, interval_length, instances, action)
+
+ data = []
+ until start_time > (end_time - interval_length) do
+ interval_time = start_time + interval_length
+
+ tasks = Task.find(:all, :conditions => { :time_submitted => start_time..interval_time,
+ :time_started => start_time..Time.now,
+ :failure_code => nil,
+ :action => action,
+ :task_target_id => instances
+ })
+ if tasks.length > 0
+ data << tasks_submissions_mean_max_min(start_time, tasks)
+ else
+ data << QoSDataPoint.new(start_time, 0, 0, 0)
+ end
+
+ start_time = interval_time
+ end
+
+ return data
+ end
+
+end
diff --git a/src/spec/services/data_service_spec.rb b/src/spec/services/data_service_spec.rb
new file mode 100644
index 0000000..f69f8e8
--- /dev/null
+++ b/src/spec/services/data_service_spec.rb
@@ -0,0 +1,157 @@
+require 'spec_helper'
+
+describe DataService do
+
+ it "should calculate the quota usage for a provider with a numbner of cloud accounts" do
+ client = mock('DeltaCloud', :null_object => true)
+ provider = Factory.build(:mock_provider)
+ provider.stub!(:connect).and_return(client)
+ provider.save!
+
+ data = [[25, 10], [40, 20], [20, 20]]
+ free = 0
+ for i in 0..2
+ cloud_account = Factory.build(:cloud_account, :provider => provider, :username => "username" + i.to_s)
+ cloud_account.stub!(:valid_credentials?).and_return(true)
+ cloud_account.save!
+
+ quota = Factory(:quota, :maximum_total_instances => data[i][0], :total_instances => data[i][1])
+ cloud_account.quota_id = quota.id
+ cloud_account.save!
+
+ free += (data[i][0] - data[i][1])
+ end
+
+ data_points = DataService.total_quota_utilisation(provider)
+ data_points[0].should == DataService::TotalQuotaUsagePoint.new("username0", data[0][1])
+ data_points[1].should == DataService::TotalQuotaUsagePoint.new("username1", data[1][1])
+ data_points[2].should == DataService::TotalQuotaUsagePoint.new("username2", data[2][1])
+ data_points[3].should == DataService::TotalQuotaUsagePoint.new("free", free)
+
+ end
+
+ it "should calculate the total number of instances and maximum numbder of instances of a cloud account" do
+ client = mock('DeltaCloud', :null_object => true)
+ provider = Factory.build(:mock_provider)
+ provider.stub!(:connect).and_return(client)
+ provider.save!
+
+ cloud_account = Factory.build(:cloud_account, :provider => provider)
+ cloud_account.stub!(:valid_credentials?).and_return(true)
+ cloud_account.save!
+
+ quota = Factory(:quota, :maximum_total_instances => 50, :total_instances => 20)
+ cloud_account.quota_id = quota.id
+
+ data_point = DataService.quota_utilisation(cloud_account)
+ data_point.should == DataService::QuotaUsagePoint.new(20, 50)
+ end
+
+ it "should calculate the average, max and min task submission times" do
+ tasks = []
+ instance = Factory :instance
+
+ for i in 1..10 do
+ time = Time.utc(2010,"jan",1,20,15,1)
+ task = Task.new(:instance => instance, :type => "InstanceTask", :state => Task::STATE_PENDING, :failure_code => nil)
+ task.time_submitted = time
+ time += i
+ task.time_started = time
+ task.save
+ tasks << task
+ end
+
+ data_point = DataService.tasks_submissions_mean_max_min(Time.now, tasks)
+
+ data_point.average.should == 5.5
+ data_point.min.should == 1
+ data_point.max.should == 10
+ end
+
+ it "should create data points for the average, max and min task submission times between two times at given intervals" do
+ pool = Factory :pool
+ instance = Factory(:instance, :pool_id => pool.id)
+
+ expected_averages = [ 20, 40, 60, 80, 100]
+ no_intervals = expected_averages.length
+ interval_length = 30
+
+ end_time = Time.utc(2010,"jan",1,20,15,1)
+ start_time = end_time - (interval_length * no_intervals)
+
+ generate_tasks(start_time, interval_length, instance, expected_averages)
+
+ data_points = DataService.qos_task_submission_stats(start_time, end_time, interval_length, pool, InstanceTask::ACTION_CREATE)
+
+ for i in 0...data_points.length
+ average_time = expected_averages[i]
+ dp = data_points[i]
+
+ dp.average.should == average_time
+ # The multiplications could be set as static numbers but are left as calculations for easier understanding of code
+ dp.max.should == (average_time / 10) * 2 * 9
+ dp.min.should == (average_time / 10) * 2
+ end
+ end
+
+ it "should create data points for mean, max and min task submission times at given intervals for a provider with multiple accounts" do
+ pool = Factory :pool
+
+ expected_averages = []
+ expected_averages[0] = [ 20, 40, 60, 80, 100]
+ expected_averages[1] = [ 40, 60, 80, 100, 120]
+ expected_averages[2] = [ 60, 80, 100, 120, 140]
+
+ no_intervals = expected_averages.length
+ interval_length = 30
+ end_time = Time.utc(2010,"jan",1,20,15,1)
+ start_time = end_time - (interval_length * no_intervals)
+
+ client = mock('DeltaCloud', :null_object => true)
+ provider = Factory.build(:mock_provider)
+ provider.stub!(:connect).and_return(client)
+ provider.save!
+
+ cloud_accounts = []
+ expected_averages.each do |expected_average|
+ cloud_account = Factory.build(:cloud_account, :provider => provider, :username => "username" + expected_average[0].to_s)
+ cloud_account.stub!(:valid_credentials?).and_return(true)
+ cloud_account.save!
+
+ instance = Factory(:instance, :cloud_account_id => cloud_account.id, :pool_id => pool.id)
+ generate_tasks(start_time, interval_length, instance, expected_average)
+ end
+
+ data_points = DataService.qos_task_submission_stats(start_time, end_time, interval_length, pool, InstanceTask::ACTION_CREATE)
+
+ for i in 0...data_points.length
+ dp = data_points[i]
+ dp.average.should == expected_averages[1][i]
+ dp.max.should == (expected_averages[2][i] / 10) * 2 * 9
+ dp.min.should == (expected_averages[0][i] / 10) * 2
+ end
+ end
+
+ def generate_tasks(start_time, interval_length, instance, expected_averages)
+ interval_time = start_time
+ expected_averages.each do |avg|
+ submission_time = interval_time + (interval_length / 2)
+ for i in 1..9 do
+ started_time = submission_time + ((avg / 10) * 2) * i
+
+ task = InstanceTask.new(:instance => instance,
+ :type => "InstanceTask",
+ :state => Task::STATE_QUEUED,
+ :failure_code => nil,
+ :action => InstanceTask::ACTION_CREATE,
+ :task_target_id => instance.id)
+ task.created_at = submission_time
+ task.time_submitted = submission_time
+ task.time_started = started_time
+ task.save!
+ end
+ interval_time += interval_length
+ end
+ end
+
+end
--
1.6.6.1
14 years
[PATCH aggregator] image target polling
by Jan Provazník
From: Jan Provaznik <jprovazn(a)redhat.com>
image builder UI now automatically refreshes targets statuses each 10 s
---
src/app/controllers/image_descriptor_controller.rb | 7 +++++-
.../image_descriptor_target_controller.rb | 8 +++++++
src/app/helpers/image_descriptor_helper.rb | 6 ++--
src/app/views/image_descriptor/_summary.haml | 22 +++++--------------
src/app/views/image_descriptor/_targets.haml | 16 ++++++++++++++
src/app/views/image_descriptor/targets.haml | 1 +
6 files changed, 40 insertions(+), 20 deletions(-)
create mode 100644 src/app/controllers/image_descriptor_target_controller.rb
create mode 100644 src/app/views/image_descriptor/_targets.haml
create mode 100644 src/app/views/image_descriptor/targets.haml
diff --git a/src/app/controllers/image_descriptor_controller.rb b/src/app/controllers/image_descriptor_controller.rb
index 69d3c23..11daa9c 100644
--- a/src/app/controllers/image_descriptor_controller.rb
+++ b/src/app/controllers/image_descriptor_controller.rb
@@ -21,7 +21,7 @@ class ImageDescriptorController < ApplicationController
@tab = 'summary'
if params[:targets]
params[:targets].each do |target|
- ImageDescriptorTarget.new_if_not_exists(:name => target, :image_descriptor_id => params[:image_descriptor][:id], :status => "waiting")
+ ImageDescriptorTarget.new_if_not_exists(:name => target, :image_descriptor_id => params[:image_descriptor][:id], :status => ImageDescriptorTarget::STATE_WAITING)
end
end
end
@@ -55,6 +55,11 @@ class ImageDescriptorController < ApplicationController
redirect_to :action => 'images', :tab => 'show'
end
+ def targets
+ @image_descriptor = ImageDescriptor.find(params[:id])
+ @all_targets = ImageDescriptorTarget.available_targets
+ end
+
def selected_packages
data = ImageDescriptor.find(params[:id]).xml.packages
end
diff --git a/src/app/controllers/image_descriptor_target_controller.rb b/src/app/controllers/image_descriptor_target_controller.rb
new file mode 100644
index 0000000..f2318e3
--- /dev/null
+++ b/src/app/controllers/image_descriptor_target_controller.rb
@@ -0,0 +1,8 @@
+class ImageDescriptorTargetController < ApplicationController
+ before_filter :require_user
+
+ def cancel
+ ImageDescriptorTarget.update(params[:id], :status => ImageDescriptorTarget::STATE_CANCELED)
+ redirect_to :controller => 'image_descriptor', :action => 'new', :params => {'image_descriptor[id]' => params[:descriptor_id], :tab => 'software'}
+ end
+end
diff --git a/src/app/helpers/image_descriptor_helper.rb b/src/app/helpers/image_descriptor_helper.rb
index 7055ce5..73d2502 100644
--- a/src/app/helpers/image_descriptor_helper.rb
+++ b/src/app/helpers/image_descriptor_helper.rb
@@ -21,10 +21,10 @@ module ImageDescriptorHelper
select_tag("repository", ["<option value='all' selected='selected'>All</option>"] + repositories.map{|repid, rep| "<option value=\"#{repid}\">#{rep['name']}</option>"}, {:onchange => "get_repository(event)"})
end
- def image_target_actions(target)
- str = ''
+ def image_target_actions(descriptor, target)
+ str = ' '
if ImageDescriptorTarget::ACTIVE_STATES.include?(target.status)
- str = link_to 'Cancel', {:controller => 'image_descriptor_target', :action => 'cancel', :id => target.id}
+ str = link_to 'Cancel', {:controller => 'image_descriptor_target', :action => 'cancel', :id => target.id, :descriptor_id => descriptor.id}
end
return str
end
diff --git a/src/app/views/image_descriptor/_summary.haml b/src/app/views/image_descriptor/_summary.haml
index 670dbce..632669d 100644
--- a/src/app/views/image_descriptor/_summary.haml
+++ b/src/app/views/image_descriptor/_summary.haml
@@ -1,5 +1,10 @@
- content_for :scripts do
:javascript
+ $(document).ready(function() {
+ var refreshId = setInterval(function() {
+ $('#image_target_list').load('#{url_for :controller => 'image_descriptor', :action => 'targets', :id => @image_descriptor.id, :ajax => true}');
+ }, 10000);
+ });
function toggle_build_form_visibility(ev) {
if ($("#image_build_form").css('display') == 'none') {
$("#image_build_form").css('display', 'block');
@@ -45,22 +50,7 @@
%span
= target['name']
= submit_tag "Build"
- .target_list
- - if @image_descriptor.image_descriptor_targets.empty?
- .empty No images have been created yet.
- -else
- %ul
- - @image_descriptor.image_descriptor_targets.each do |target|
- %li
- %span{:class => 'actions'}
- = image_target_actions(target)
- %span{:class => 'status'}
- = target.status
- = @all_targets[target.name]['name']
- %p
- = target.created_at
- //%span{:style => 'float:right'}
- // Cancel
+ = render :partial => 'targets'
.submit
= submit_tag "Cancel"
diff --git a/src/app/views/image_descriptor/_targets.haml b/src/app/views/image_descriptor/_targets.haml
new file mode 100644
index 0000000..be66513
--- /dev/null
+++ b/src/app/views/image_descriptor/_targets.haml
@@ -0,0 +1,16 @@
+#image_target_list{:class => 'target_list'}
+ - if @image_descriptor.image_descriptor_targets.empty?
+ .empty No images have been created yet.
+ -else
+ %ul
+ - @image_descriptor.image_descriptor_targets.each do |target|
+ %li
+ %span{:class => 'actions'}
+ = image_target_actions(@image_descriptor, target)
+ %span{:class => 'status'}
+ = target.status
+ = @all_targets[target.name]['name']
+ %p
+ = target.created_at
+ //%span{:style => 'float:right'}
+ // Cancel
diff --git a/src/app/views/image_descriptor/targets.haml b/src/app/views/image_descriptor/targets.haml
new file mode 100644
index 0000000..b4351d5
--- /dev/null
+++ b/src/app/views/image_descriptor/targets.haml
@@ -0,0 +1 @@
+= render :partial => 'targets'
--
1.7.0.1
14 years
Rake Task for Creating Demo Data Tasks
by Martyn Taylor
Creates a rake task that allows creation of historical tasks, in order to create QoS statistics
To run
`rake db:demo_data[cloud_account_id, user_id, time_scale, numbder_of_tasks]`
params:
cloud_account_id : The Cloud Account ID to which these tasks are associated
user_id : The user ID for whom these tasks are created
time_scale : the past number of hours these tasks should span across
number_of_tasks : The amount of tasks to create
14 years
Update the general look and layout (revision 2)
by Tomas Sedovic
Please ignore the earlier patches from me.
This set replaces it and provides:
* simplifying the right navigation bar
* adding tabs to the provider and pool pages
* dashboard portlets to occupy the all available space
* jeremy's CSS changes
* loads the dashboard for the root url request
Thomas
14 years
re-sending last two patches after rebase
by Scott Seago
Re-sending last two patches after rebase.
[PATCH aggregator 1/2] Added provider account matchmaking to current task scheduler.
[PATCH aggregator 2/2] don't ask user to select pool if it's already done via the URL/link
14 years
Update the general look and layout
by Tomas Sedovic
Please disregard the previous patchset ("Right nav bar and provider page layout
changes").
This set replaces it and provides:
* simplifying the right navigation bar
* adding tabs to the provider and pool pages
* dashboard portlets to occupy the all available space
* jeremy's CSS changes
Thomas
14 years
[PATCH aggregator] instances index/list view and backend (revision 2)
by Mo Morsi
---
src/app/controllers/instance_controller.rb | 36 ++++++++++++++++++++++++++++
src/app/views/dashboard/summary.haml | 2 +
src/app/views/instance/_list.html.erb | 25 +++++++++++++++++++
src/app/views/instance/index.html.erb | 3 +-
4 files changed, 64 insertions(+), 2 deletions(-)
create mode 100644 src/app/views/instance/_list.html.erb
diff --git a/src/app/controllers/instance_controller.rb b/src/app/controllers/instance_controller.rb
index c898691..d01cb10 100644
--- a/src/app/controllers/instance_controller.rb
+++ b/src/app/controllers/instance_controller.rb
@@ -27,6 +27,42 @@ class InstanceController < ApplicationController
def index
end
+ def paginated
+ # datatables sends pagination in format:
+ # iDisplayStart - start index
+ # iDisplayLength - num of recs
+ # => we need to count page num
+ page = params[:iDisplayStart].to_i / Instance::per_page
+
+ order_col_rec = Instance::COLUMNS[params[:iSortCol_0].to_i]
+ order_col = Instance::COLUMNS[2] unless order_col_rec && order_col_rec[:opts][:searchable]
+ order = order_col[:id] + " " + (params[:sSortDir_0] == 'desc' ? 'desc' : 'asc')
+
+ # FIXME only return those instances in pools which user has instance_view
+ @instances = Instance.search_filter(params[:sSearch], Instance::SEARCHABLE_COLUMNS).paginate(
+ :page => page + 1,
+ :order => order
+ )
+
+ recs = @instances.map do |i|
+ [
+ i.id,
+ i.get_action_list.map {|action| "<a href=\"#{url_for :controller => "instance", :action => "instance_action", :id => i.id, :instance_action => action}\">#{action}</a>"}.join(" | "),
+ i.name,
+ i.state,
+ i.hardware_profile.name,
+ i.image.name
+ ]
+ end
+
+ render :json => {
+ :sEcho => params[:sEcho],
+ :iTotalRecords => @instances.total_entries,
+ :iTotalDisplayRecords => @instances.total_entries,
+ :aaData => recs
+ }
+ end
+
# Right now this is essentially a duplicate of PoolController#show,
# but really it should be a single instance should we decide to have a page
# for that. Redirect on create was all that brought you here anyway, so
diff --git a/src/app/views/dashboard/summary.haml b/src/app/views/dashboard/summary.haml
index f3d6ab6..d8b9e20 100644
--- a/src/app/views/dashboard/summary.haml
+++ b/src/app/views/dashboard/summary.haml
@@ -38,6 +38,8 @@
Create a Template
%a{:href => url_for(:controller => "users", :action => "new")}
Create a User
+ %a{:href => url_for(:controller => "instance", :action => "index")}
+ View Instances
- else
%a{:href => url_for(:controller => "", :action => "")}
Launch Instances
diff --git a/src/app/views/instance/_list.html.erb b/src/app/views/instance/_list.html.erb
new file mode 100644
index 0000000..9d12c76
--- /dev/null
+++ b/src/app/views/instance/_list.html.erb
@@ -0,0 +1,25 @@
+<%= datatable(
+ Instance::COLUMNS.map {|c| c[:opts]},
+ {
+ :table_dom_id => 'instances_table',
+ :per_page => Instance::per_page,
+ :sort_by => "[3, 'asc']",
+ :serverside => true,
+ :ajax_source => url_for(:controller => 'instance', :action => 'paginated'),
+ :append => ".fnSetFilteringDelay()",
+ :persist_state => false,
+ :click_callback => "function(ev) {clickRow(ev);}",
+ }
+) %>
+
+<table class="datatable display" id="instances_table">
+ <thead>
+ <tr>
+ <% Instance::COLUMNS.each do |c| %>
+ <%= "<th>#{c[:header]}</th>" %>
+ <% end %>
+ </tr>
+ </thead>
+ <tbody>
+ </tbody>
+</table>
diff --git a/src/app/views/instance/index.html.erb b/src/app/views/instance/index.html.erb
index a632f6c..c19c3fa 100644
--- a/src/app/views/instance/index.html.erb
+++ b/src/app/views/instance/index.html.erb
@@ -1,2 +1 @@
-<h1>Instance#index</h1>
-<p>Find me in app/views/instance/index.html.erb</p>
+<%= render :partial => 'list' %>
--
1.6.2.5
14 years
Added Utility DataService
by Martyn Taylor
This patch adds a DataService Class that allows the collection of statistics in order to produce QoS and Quota Stats
14 years