ArgoCD / GitOps

This guide covers deploying Astromesh across multiple environments using ArgoCD and GitOps principles. Push configuration changes to Git, and ArgoCD automatically syncs them to your Kubernetes clusters.

What and Why

GitOps is an operational model where the desired state of your infrastructure is stored in Git. Instead of running helm install or kubectl apply manually, you:

1. Commit configuration changes to a Git repository
2. ArgoCD detects the change
3. ArgoCD syncs the Kubernetes cluster to match the desired state
4. Rolling updates happen automatically

This provides:

• Audit trail — every change is a Git commit with author, timestamp, and diff
• Reproducibility — any environment can be recreated from the Git state
• Rollback — revert a Git commit to roll back a deployment
• Multi-environment consistency — dev, staging, and prod all deploy from the same chart with different values

Prerequisites

Requirement	Version	Check command
Kubernetes	1.26+	kubectl version
ArgoCD	2.8+	argocd version
Helm	3.12+	helm version
Git repo	accessible from ArgoCD	—

Verify ArgoCD is installed

kubectl get pods -n argocd

Expected output:

NAME                                               READY   STATUS    RESTARTS   AGE
argocd-application-controller-0                    1/1     Running   0          2d
argocd-applicationset-controller-7b74965f7-x2k4m   1/1     Running   0          2d
argocd-dex-server-6dcf645b6b-abc12                 1/1     Running   0          2d
argocd-notifications-controller-5c4d48f9b-def34    1/1     Running   0          2d
argocd-redis-6976fc7dfc-ghi56                      1/1     Running   0          2d
argocd-repo-server-7c4f568f7-jkl78                 1/1     Running   0          2d
argocd-server-7bc7684f8d-mno90                     1/1     Running   0          2d

If ArgoCD is not installed, install it:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Verify ArgoCD can access the repository

ArgoCD needs read access to your Git repository. For private repos, add credentials:

argocd repo add https://github.com/monaccode/astromesh.git \
  --username git \
  --password ghp_your_token

Expected output:

Repository 'https://github.com/monaccode/astromesh.git' added

Step-by-step Setup

1. Review the ApplicationSet

The Astromesh repository includes an ArgoCD ApplicationSet at deploy/gitops/argocd/applicationset.yaml:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: astromesh
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - env: dev
            namespace: astromesh-dev
            valuesFile: values-dev.yaml
          - env: staging
            namespace: astromesh-staging
            valuesFile: values-staging.yaml
          - env: prod
            namespace: astromesh-prod
            valuesFile: values-prod.yaml
  template:
    metadata:
      name: astromesh-{{env}}
    spec:
      project: default
      source:
        repoURL: https://github.com/monaccode/astromesh.git
        targetRevision: HEAD
        path: deploy/helm/astromesh
        helm:
          valueFiles:
            - "{{valuesFile}}"
      destination:
        server: https://kubernetes.default.svc
        namespace: "{{namespace}}"
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

This creates one ArgoCD Application per environment, each deploying the same Helm chart with different values files.

2. Deploy the ApplicationSet

kubectl apply -f deploy/gitops/argocd/applicationset.yaml

Expected output:

applicationset.argoproj.io/astromesh created

3. Verify the Applications

argocd app list

Expected output:

NAME              CLUSTER                         NAMESPACE          PROJECT  STATUS  HEALTH   SYNCPOLICY  CONDITIONS
astromesh-dev     https://kubernetes.default.svc  astromesh-dev      default  Synced  Healthy  Auto-Prune  <none>
astromesh-staging https://kubernetes.default.svc  astromesh-staging  default  Synced  Healthy  Auto-Prune  <none>
astromesh-prod    https://kubernetes.default.svc  astromesh-prod     default  Synced  Healthy  Auto-Prune  <none>

All three environments show Synced and Healthy.

4. Check pods in each namespace

kubectl get pods -n astromesh-dev

Expected output:

NAME                                    READY   STATUS    RESTARTS   AGE
astromesh-5d8f9c7b6-x2k4m              1/1     Running   0          2m
astromesh-postgresql-0                  1/1     Running   0          2m
astromesh-redis-master-0                1/1     Running   0          2m

kubectl get pods -n astromesh-prod

Expected output:

NAME                                    READY   STATUS    RESTARTS   AGE
astromesh-5d8f9c7b6-a1b2c              1/1     Running   0          2m
astromesh-5d8f9c7b6-d3e4f              1/1     Running   0          2m
astromesh-5d8f9c7b6-g5h6i              1/1     Running   0          2m
astromesh-vllm-7f8g9h0i-j1k2l          1/1     Running   0          2m
astromesh-tei-embeddings-m3n4o-p5q6r   1/1     Running   0          2m
astromesh-tei-reranker-s7t8u-v9w0x     1/1     Running   0          2m

Production has 3 Astromesh replicas, vLLM, and two TEI instances as defined in values-prod.yaml.

Configuration

Environments

The ApplicationSet uses a list generator to create one Application per environment:

Environment	Namespace	Values File	Auto-sync	Description
dev	astromesh-dev	values-dev.yaml	Yes	Development with all subcharts, no GPU limits
staging	astromesh-staging	values-staging.yaml	Yes	Pre-production validation
prod	astromesh-prod	values-prod.yaml	Yes	Production with external DBs, GPU, TLS

Sync Policy

syncPolicy:
  automated:
    prune: true       # Delete resources removed from Git
    selfHeal: true    # Revert manual cluster changes to match Git
  syncOptions:
    - CreateNamespace=true  # Create namespace if it doesn't exist

• prune: true — if you remove a resource from your Helm chart, ArgoCD deletes it from the cluster
• selfHeal: true — if someone manually edits a resource in the cluster, ArgoCD reverts it to match Git

Workflow

The GitOps workflow for making changes:

1. Edit configuration in Git

For example, to scale production to 5 replicas, edit deploy/helm/astromesh/values-prod.yaml:

replicaCount: 5

2. Commit and push

git add deploy/helm/astromesh/values-prod.yaml
git commit -m "feat: scale production to 5 replicas"
git push origin main

3. ArgoCD detects the change

ArgoCD polls the repository (default every 3 minutes) or receives a webhook notification. It detects the diff between the desired state (Git) and the live state (cluster).

4. ArgoCD syncs

Because automated.selfHeal: true is set, ArgoCD automatically syncs:

argocd app get astromesh-prod

Expected output:

Name:               astromesh-prod
Server:             https://kubernetes.default.svc
Namespace:          astromesh-prod
Repo:               https://github.com/monaccode/astromesh.git
Path:               deploy/helm/astromesh
Target:             HEAD
Status:             Synced
Health:             Progressing

5. Rolling update completes

kubectl get pods -n astromesh-prod

Expected output:

NAME                                    READY   STATUS    RESTARTS   AGE
astromesh-5d8f9c7b6-a1b2c              1/1     Running   0          10m
astromesh-5d8f9c7b6-d3e4f              1/1     Running   0          10m
astromesh-5d8f9c7b6-g5h6i              1/1     Running   0          10m
astromesh-5d8f9c7b6-j7k8l              1/1     Running   0          30s
astromesh-5d8f9c7b6-m9n0o              1/1     Running   0          30s

Five replicas running.

Customizing the ApplicationSet

Change repository URL

Edit deploy/gitops/argocd/applicationset.yaml:

source:
  repoURL: https://github.com/your-org/your-repo.git

Change target branch

To deploy from a specific branch instead of HEAD:

source:
  targetRevision: main

Or use a tag:

source:
  targetRevision: v0.10.0

Deploy to a different cluster

destination:
  server: https://remote-cluster-api.example.com
  namespace: "{{namespace}}"

Disable auto-sync for production

For manual approval before production deployments:

generators:
  - list:
      elements:
        - env: dev
          namespace: astromesh-dev
          valuesFile: values-dev.yaml
        - env: staging
          namespace: astromesh-staging
          valuesFile: values-staging.yaml
        - env: prod
          namespace: astromesh-prod
          valuesFile: values-prod.yaml

Then override sync policy per-environment by creating separate ApplicationSets or using a matrix generator with sync policy conditions. For simpler control, disable auto-sync on prod after creation:

argocd app set astromesh-prod --sync-policy none

Adding New Environments

To add a new environment (e.g., qa):

1. Create a values file

cp deploy/helm/astromesh/values-staging.yaml deploy/helm/astromesh/values-qa.yaml

Edit values-qa.yaml with QA-specific settings.

2. Add to the ApplicationSet

Edit deploy/gitops/argocd/applicationset.yaml, add to the elements list:

generators:
  - list:
      elements:
        - env: dev
          namespace: astromesh-dev
          valuesFile: values-dev.yaml
        - env: qa
          namespace: astromesh-qa
          valuesFile: values-qa.yaml
        - env: staging
          namespace: astromesh-staging
          valuesFile: values-staging.yaml
        - env: prod
          namespace: astromesh-prod
          valuesFile: values-prod.yaml

3. Commit and apply

git add deploy/
git commit -m "feat: add QA environment"
git push origin main
kubectl apply -f deploy/gitops/argocd/applicationset.yaml

ArgoCD creates the new astromesh-qa Application and syncs it.

Promotion Workflow

Promote changes from dev to staging to production:

1. Make changes in dev values

Edit deploy/helm/astromesh/values-dev.yaml:

config:
  agents:
    new-agent.agent.yaml: |
      apiVersion: astromesh/v1
      kind: Agent
      metadata:
        name: new-agent
      spec:
        identity:
          display_name: "New Agent"
        model:
          primary:
            provider: ollama
            model: llama3.1:8b
        orchestration:
          pattern: react

Commit, push, and verify in dev.

2. Promote to staging

Copy the agent definition to values-staging.yaml:

# Copy the new agent config section
git diff values-dev.yaml  # Review what was added
# Apply the same change to values-staging.yaml

Commit and push. ArgoCD syncs staging.

3. Promote to production

After validating in staging, apply the same change to values-prod.yaml. Commit and push.

Each promotion is a Git commit, providing a clear audit trail of what changed and when.

Rollback

Using ArgoCD

Roll back to a previous sync:

argocd app history astromesh-prod

Expected output:

ID  DATE                           REVISION
2   2026-03-09 11:30:00 +0000 UTC  abc1234
1   2026-03-09 10:00:00 +0000 UTC  def5678

argocd app rollback astromesh-prod 1

Expected output:

TIMESTAMP                  GROUP/KIND      NAMESPACE        NAME          STATUS    HEALTH
2026-03-09T10:00:00+00:00  apps/Deployment astromesh-prod   astromesh     Synced    Healthy

Using Git

Revert the commit that introduced the problem:

git revert abc1234
git push origin main

ArgoCD detects the revert and syncs the cluster back to the previous state.

Common Operations

View sync status

# All environments
argocd app list

# Specific environment
argocd app get astromesh-prod

Manual sync

If auto-sync is disabled or you want to trigger immediately:

argocd app sync astromesh-prod

Expected output:

TIMESTAMP                  GROUP/KIND      NAMESPACE        NAME          STATUS    HEALTH
2026-03-09T11:00:00+00:00  /Service        astromesh-prod   astromesh     Synced    Healthy
2026-03-09T11:00:00+00:00  apps/Deployment astromesh-prod   astromesh     Synced    Healthy

Message:  successfully synced (all tasks run)

View diff before sync

argocd app diff astromesh-prod

Access ArgoCD UI

kubectl port-forward svc/argocd-server 8080:443 -n argocd

Open https://localhost:8080. Get the initial admin password:

argocd admin initial-password -n argocd

Troubleshooting

Sync failed

argocd app get astromesh-prod

If status shows OutOfSync or SyncFailed:

argocd app sync astromesh-prod --retry-limit 3

Check the sync details:

argocd app get astromesh-prod --show-operation

Common causes:

• Helm template error: invalid YAML in values file. Test locally:

helm template astromesh ./deploy/helm/astromesh -f deploy/helm/astromesh/values-prod.yaml

• Resource conflict: another controller manages the same resource. Check annotations.

• Dependency not downloaded: Helm chart dependencies need to be in the charts/ directory or resolvable by ArgoCD.

Secret issues

ArgoCD stores manifests in Git, which means secrets should not be committed in plain text. Use one of these approaches:

1. External Secrets Operator — secrets synced from AWS/GCP/Vault (see Helm guide)
2. Sealed Secrets — encrypted in Git, decrypted in cluster
3. existingSecret — create secrets out-of-band, reference by name in values

Do not put API keys or passwords in values files that are committed to Git.

Configuration drift

If someone manually changes a resource in the cluster, ArgoCD detects drift and shows OutOfSync. With selfHeal: true, it automatically reverts the change.

To check for drift:

argocd app diff astromesh-prod

ArgoCD cannot reach the repository

rpc error: code = Unknown desc = authentication required

Add or update repository credentials:

argocd repo add https://github.com/monaccode/astromesh.git \
  --username git \
  --password ghp_new_token

Application stuck in “Progressing”

Check pod status:

kubectl get pods -n astromesh-prod
kubectl describe pod <pod-name> -n astromesh-prod

Common causes:

• Image pull errors (wrong tag or registry auth)
• Insufficient resources (CPU/memory limits too low)
• Failed health checks (readiness probe failing)

Webhook setup (faster sync)

By default, ArgoCD polls every 3 minutes. For faster sync, configure a GitHub webhook:

1. In the ArgoCD UI, go to Settings > Repositories
2. Note the webhook URL: https://argocd.example.com/api/webhook
3. In GitHub, go to your repo Settings > Webhooks > Add webhook
4. Set the Payload URL to the ArgoCD webhook endpoint
5. Set Content type to application/json
6. Select “Just the push event”

Now ArgoCD syncs within seconds of a push.