- Reference >
- Troubleshooting the Kubernetes Operator
Troubleshooting the Kubernetes Operator¶
On this page
- Get Status of a Deployed Resource
- Review the Logs
- View All MongoDB Kubernetes resource Specifications
- Restore StatefulSet that Failed to Deploy
- Remove a MongoDB Kubernetes resource
- Remove the Kubernetes Operator
- Remove the namespace
- Remove the CustomResourceDefinitions
- Disable Ops Manager Feature Controls
Get Status of a Deployed Resource¶
To find the status of a resource deployed with the Kubernetes Operator, invoke one of the following commands:
For Ops Manager resource deployments:
The
status.applicationDatabase.phase
field displays the Application Database resource deployment status. Thestatus.opsManager.phase
field displays the Ops Manager resource deployment status.For MongoDB resource deployments:
The
status.phase
field displays the MongoDB resource deployment status.
The following key-value pairs describe the resource deployment statuses:
Key | Value | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
message |
Message explaining why the resource is in a Pending or
Failed state. |
||||||||||
phase |
|
||||||||||
lastTransition |
Timestamp in ISO 8601 date and time format in UTC when the last reconciliation happened. | ||||||||||
link |
Deployment URL in Ops Manager. | ||||||||||
Resource specific fields | For descriptions of these fields, see MongoDB Database Resource Specification. |
Example
If you want to see what the status of a replica set named
my-replica-set
in the developer
namespace, run:
If my-replica-set
is running, you should see:
If my-replica-set
is not running, you should see:
Review the Logs¶
Review Logs from the Kubernetes Operator¶
To review the Kubernetes Operator logs, invoke this command:
You could check the Ops Manager Logs as well to see if any issues were reported to Ops Manager.
Find a Specific Pod¶
To find which pods are available, invoke this command first:
See also
Kubernetes documentation on kubectl get.
Review Logs from Specific Pod¶
If you want to narrow your review to a specific pod, you can invoke this command:
Example
If your replica set is labeled myrs
, the pod log
command is invoked as:
This returns the Automation Agent Log for this replica set.
View All MongoDB Kubernetes resource Specifications¶
To view all MongoDB Kubernetes resource specifications in the provided namespace:
Example
To read details about the dublin
standalone resource, invoke
this command:
This returns the following response:
Restore StatefulSet that Failed to Deploy¶
A StatefulSet pod may hang with a status of Pending
if it
encounters an error during deployment.
Pending
pods do not automatically terminate, even if you
make and apply configuration changes to resolve the error.
To return the StatefulSet to a healthy state, apply the configuration
changes to the MongoDB resource in the Pending
state, then delete
those pods.
Example
A host system has a number of running pods:
my-replica-set-2
is stuck in the Pending
stage. To gather
more data on the error, run the following:
The output indicates an error in memory allocation.
Updating the memory allocations in the MongoDB resource is insufficient, as the pod does not terminate automatically after applying configuration updates.
To remedy this issue, update the configuration, apply the configuration, then delete the hung pod:
Once this hung pod is deleted, the other pods restart with your new configuration as part of rolling upgrade of the Statefulset.
Note
To learn more about this issue, see Kubernetes Issue 67250.
Remove a MongoDB Kubernetes resource¶
To remove any instance that Kubernetes deployed, you must use Kubernetes.
Important
You can only use the Kubernetes Operator to remove Kubernetes-deployed instances. If you use Ops Manager to remove the instance, Ops Manager throws an error.
Example
To remove a single MongoDB instance you created using Kubernetes:
To remove all MongoDB instances you created using Kubernetes:
Remove the CustomResourceDefinitions¶
To remove the CustomResourceDefinitions:
Remove the CustomResourceDefinitions:
Disable Ops Manager Feature Controls¶
When you manage an Ops Manager project through the Kubernetes Operator, the
Kubernetes Operator places the EXTERNALLY_MANAGED_LOCK
feature
control policy
on the project. This policy disables certain features in the Ops Manager
application that might compromise your Kubernetes Operator configuration. If
you need to use these blocked features, you can remove the policy
through the feature controls API,
make changes in the Ops Manager application, and then restore the original
policy through the API.
Warning
The following procedure enables you to use features in the Ops Manager application that are otherwise blocked by the Kubernetes Operator.
Retrieve the feature control policies for your Ops Manager project.
Important
Save the response that the API returns. After you make changes in the Ops Manager application, you must add these policies back to the project.
Your response should be similar to:
Update the
policies
array with an empty list:The previously blocked features are now available in the Ops Manager application.
Make your changes in the Ops Manager application.
Update the
policies
array with the original feature control policies:The features are now blocked again, preventing you from making further changes through the Ops Manager application. However, the Kubernetes Operator retains any changes you made in the Ops Manager application while features were available.