New Configuration System

This page describes a new configuration system that will be added to VCL that can be used to dynamically configure deployed systems.

Database schema

config:

id - id of record
name - name of record
ownerid - owner of this record (reference to user.id)
configtypeid - type of record (reference to configtype.id)
data - any data associated with this (ex. puppet manifest)
optional - 0/1 - when this config is mapped to something, can the user specify to apply it or not

configtype:

id - id of record
name - name of record
moduleid - id of module that handles this type of record (reference to module.id)

initial types: puppet, subimage, shellcommand, perlfunction, software

configvariable:

id - id of record
name - name of record
configid - config this is associated with (reference to config.id)
value - default value of this variable
required - 0/1 - when mapped to something, is this variable always applied
ask - 0/1 - will the user be prompted for a value for this variable, or will the default value always be used
key - string in config.data to replace with the value of this variable
datatype - enum(int, multiint, string) - type of this variable so that the frontend knows how to validate it

There are some initial, special names:
- subimage - used for clusters, specifies a subimage to be deployed; when deployed, configinstancevariable.value will be reservation.id of the subimage
- min - used in conjunction with subimage to specify a minimum number of those subimages
- max - used in conjunction with subimage to specify a maximum number of those subimages
- runbefore - used to relate to other configs to specify that this should be run before the ones specified in ‘value’
- runafter - used to relate to other configs to specify that this should be run after the ones specified in ‘value’
- getdata - ??

configinstance:

This is similar to the config table, but is for deployed systems.

id - id of record
reservationid - reservation this is associated with (references reservation.id)
configid - config entry being applied (references config.id)
status - new/processing/completed??

configinstancevariable:

This is similar to the configvariable table, but is for deployed systems.

configinstanceid - config instance this is associated with (references configinstance.id)
configvariableid - config variable being applied (references configvariable.id)
value - similar to configvariable.value, but can be set by user to different value if configvariable.ask is set to 1

configmap:

This is for mapping configs to various resources or other items in VCL.

configid - config being mapped (references config.id)
subid - id from specific resource table (ex. image.id)
configmaptypeid - type of resource or other item in VCL this is mapped to (references configmaptype.id)
affiliationid - allows configs to only be mapped for a specific affiliation, use the Global affilation to map to all (references affiliation.id)
disabled - 0/1 (a little complicated, normally 0) allows exceptions to a config being applied; set to 1 if a config would be mapped due to a general mapping, but want to disable for a specific instance (ex. if a config is applied for everything deployed by a certain provisioning module, but you don’t want it applied for a certain image, you would have an entry in this table mapping it to the provisioning module with disabled set to 0, then you would also have an entry for the image with disabled set to 1)
stage - start_load/post_load/(others?) which stage in the provisioning process where this record should be applied

configmaptype:

id - id of this record
name - name of this record

Initial types:
- image
- OStype
- provisioning

Examples

Assigning a VLAN to an image

This example shows how to assign a VLAN to an image.

For this example, we’ll use the following values from other tables:

module.id for handling VLAN config type: 67
image.id for the image in this example: 524
affiliation.id for the desired affiliation: 6
reservation.id for the image: 2748
user.id that owns the configs: 54

configtype:

id	name	moduleid
5	VLAN	67

config:

id	name	ownerid	configtypeid	data	optional
77	VLAN 30	54	5	30	0

configvariable:

id	name	configid	value	required	ask	key	datatype
486	VLAN	77	30	1	0	(empty)	int

configmaptype:

id	name
1	image

configmap:

configid	subid	configmaptypeid	affiliationid	disabled	stage
77	524	1	6	0	post_load

configinstance:

id	reservationid	configid	status
6854	2748	77	new

configinstancevariable:

configinstanceid	configvariableid	value
6854	486	30

Hadoop cluster with variable amount of slave nodes

This example shows how a hadoop cluster can be requested with 5-10 slave nodes. It can be useful to have the variable amount because 10 nodes may be desired, but you may want to cluster anyway if only 5 nodes are available or if 10 are requested, but 2 of them fail at deploy time.

For this example, we’ll use the following values from other tables:

module.id for handling subimage config type: 58
image.id for the Hadoop master image: 453
image.id for the Hadoop slave image: 454
affiliation.id for the desired affiliation: 5
reservation.id for the Hadoop master image: 2351
reservation.id for the Hadoop slave images: 2352-2361
user.id that owns the configs: 9

configtype:

id	name	moduleid
2	subimage	58

config:

id	name	ownerid	configtypeid	data	optional
59	hadoop cluster	9	2	(empty)	0

configvariable:

id	name	configid	value	required	ask	key	datatype
146	subimage	59	454	1	0	(empty)	int
147	min	59	1	1	1	(empty)	int
148	max	59	50	1	1	(empty)	int

configmaptype:

id	name
1	image

configmap:

configid	subid	configmaptypeid	affiliationid	disabled	stage
59	453	1	5	0	start_load

configinstance:

id	reservationid	configid	status
5023	2351	59	new

configinstancevariable:

configinstanceid	configvariableid	value
5023	146	2352
5023	146	2353
5023	146	2354
5023	146	2355
5023	146	2356
5023	146	2357
5023	146	2358
5023	146	2359
5023	146	2360
5023	146	2361
5023	147	5
5023	148	10

SAS cluster

This example shows how to configure a SAS cluster of 3 nodes: meta, midtier, and apps so that they are all deployed and then started in the correct order.

For this example, we’ll use the following values from other tables:

module.id for handling puppet config type: 57
module.id for handling subimage config type: 58
module.id for handling shellcommand config type: 59
image.id for the SAS meta image: 728
image.id for the SAS midtier image: 729
image.id for the SAS apps image: 730
affiliation.id for the desired affiliation: 7
reservation.id for the SAS meta image: 2466
reservation.id for the SAS midtier image: 2467
reservation.id for the SAS apps image: 2468
user.id that owns the configs: 15

configtype:

id	name	moduleid
1	puppet	57
2	subimage	58
3	shellcommand	59

config:

id	name	ownerid	configtypeid	data
76	SAS apps	15	2	(empty)
77	SAS midtier	15	2	(empty)
78	SAS meta config	15	1	(puppet manifest)
79	SAS apps config	15	1	(puppet manifest)
80	SAS midtier config	15	1	(puppet manifest)
81	SAS meta start	15	3	(startup commands)
82	SAS apps start	15	3	(startup commands)
83	SAS midtier start	15	3	(startup commands)

configvariable:

id	name	configid	value	required	key	datatype
268	subimage	76	729	1	(empty)	int
269	subimage	77	730	1	(empty)	int
270	runafter	78	268,269	1	(empty)	int
271	runafter	79	270	1	(empty)	int
272	runafter	80	271	1	(empty)	int
273	runafter	81	272	1	(empty)	int
274	runafter	82	273	1	(empty)	int
275	runafter	83	274	1	(empty)	int

configmaptype:

id	name
1	image

configmap:

configid	subid	configmaptypeid	affiliationid	stage
76	728	1	7	start_load
77	728	1	7	start_load
78	728	1	7	post_load
79	728	1	7	post_load
80	728	1	7	post_load
81	728	1	7	post_load
82	728	1	7	post_load
83	728	1	7	post_load

configinstance:

id	reservationid	configid	status
6005	2466	76	new
6006	2466	77	new
6007	2466	78	new
6008	2466	79	new
6009	2466	80	new
6010	2466	81	new
6011	2466	82	new
6012	2466	83	new

configinstancevariable:

configinstanceid	configvariableid	value
6005	268	2467
6006	269	2468
6007	270	6005,6006
6008	271	6007
6009	272	6008
6010	273	6009
6011	274	6010
6012	275	6011