@brandonvin While I of course support any innovative thinking/work here, I suspect (but do not know) that it is possible that the server side, scheduling and the elastic agent plugin model itself may in some way assume that an elastic agent can only complete one job even if that wasn't the original design intention. Just based on my wondering why none of the elastic agent plugins seem to work in this way already 😅 So would suggest trying to validate the whole idea in as quick-and-dirty a way as possible before going too deep into perfection 🙏

I do however note in the plugin API that https://plugin-api.gocd.org/current/elastic-agents/#job-completion specifically notes that plugins might want to keep agents running longer and that https://plugin-api.gocd.org/current/elastic-agents/#server-ping would be able to use it as a trigger for "kill idle agents" (one minor bit possibly missing from your list above).

As you've probably figured out, the current termination happens at

kubernetes-elastic-agents/src/main/java/cd/go/contrib/elasticagent/executors/JobCompletionRequestExecutor.java

Lines 43 to 57 in d898615

    
           ClusterProfileProperties clusterProfileProperties = jobCompletionRequest.clusterProfileProperties(); 
        
           String elasticAgentId = jobCompletionRequest.getElasticAgentId(); 
        
           Agent agent = new Agent(); 
        
           agent.setElasticAgentId(elasticAgentId); 
        
           LOG.info(format("[Job Completion] Terminating elastic agent with id {0} on job completion {1}.", agent.elasticAgentId(), jobCompletionRequest.jobIdentifier())); 
        
           List<Agent> agents = Arrays.asList(agent); 
        
           pluginRequest.disableAgents(agents); 
        
           agentInstances.terminate(agent.elasticAgentId(), clusterProfileProperties); 
        
           pluginRequest.deleteAgents(agents); 
        
           return DefaultGoPluginApiResponse.success("");

- I would suggest if the plugin is enabled to allow re-use, you could comment out the termination (forgetting about idle time and eventually killing for now) and see how much of it "just works".

I do suspect dealing with possible race conditions and interactions with https://plugin-api.gocd.org/current/elastic-agents/#should-assign-work might be challenging/interesting, but I have not personally looked in detail at the elastic agent area (other than trying to fix some bugs with the ECS elastic agent). I wonder if the plugin will have enough event hooks to know whether an agent is truly idle rather than "working on a long build", or "taking a long time to boot up and register with the server".

ShouldAssignWork is quite dumb right now, seems to keep the jobId with the pod metadata, implying it assumes a pod is only used for a single job.

kubernetes-elastic-agents/src/main/java/cd/go/contrib/elasticagent/executors/ShouldAssignWorkRequestExecutor.java

Lines 41 to 50 in d898615

    
           KubernetesInstance pod = agentInstances.find(request.agent().elasticAgentId()); 
        
           if (pod == null) { 
        
               return DefaultGoPluginApiResponse.success("false"); 
        
           } 
        
           if (request.jobIdentifier().getJobId().equals(pod.jobId())) { 
        
               LOG.debug(format("[should-assign-work] Job with identifier {0} can be assigned to an agent {1}.", request.jobIdentifier(), pod.name())); 
        
               return DefaultGoPluginApiResponse.success("true"); 
        
           }

	ClusterProfileProperties clusterProfileProperties = jobCompletionRequest.clusterProfileProperties();

	String elasticAgentId = jobCompletionRequest.getElasticAgentId();

	Agent agent = new Agent();
	agent.setElasticAgentId(elasticAgentId);

	LOG.info(format("[Job Completion] Terminating elastic agent with id {0} on job completion {1}.", agent.elasticAgentId(), jobCompletionRequest.jobIdentifier()));

	List<Agent> agents = Arrays.asList(agent);
	pluginRequest.disableAgents(agents);
	agentInstances.terminate(agent.elasticAgentId(), clusterProfileProperties);
	pluginRequest.deleteAgents(agents);

	return DefaultGoPluginApiResponse.success("");

	KubernetesInstance pod = agentInstances.find(request.agent().elasticAgentId());

	if (pod == null) {
	return DefaultGoPluginApiResponse.success("false");
	}

	if (request.jobIdentifier().getJobId().equals(pod.jobId())) {
	LOG.debug(format("[should-assign-work] Job with identifier {0} can be assigned to an agent {1}.", request.jobIdentifier(), pod.name()));
	return DefaultGoPluginApiResponse.success("true");
	}

Elastic Agent Reuse #53

Description

Activity

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions