GitOps Broke CI/CD! Here's How to Fix It With Argo Events

Interesting approach. What I would do, is add a new stage in the main pipeline, just after the stage where the pipeline updates the git repo with the new image tag. On that new stage, I would (continuously) send a request to the healthcheck endpoint of the application (until it is healthy) and when it is healthy, I would trigger a new pipeline called "functional tests". That way, there is no need to install more stuff to my cluster (like Argo events), and I have one place to check what happens after the deployment of a new version of my application (the main pipeline stages)

Im thinking of using Argo events as first step in a workflow pipeline which is triggered when a new cluster (which is basically a K8s secret) is created and then start deploying apps and managing sequence between them

OR..... Use git branches or tags to model release state.

Merge to release branch, test and build container. If this successful, pipeline merges git branch 'dev_ready_to_test'

Another pipeline triggers on 'dev_ready_to test'. It deploys and runs functional tests and if it passes, it merges to 'prod_ready_to_deploy'.

Add steps as needed for your release process.

ALL this is tracked with in the system of record not some external event bus.

Viktor, it is possible to use this Argo Events to trigger Jenkins Pipelines?I have installed ArgoCD on my Kubernetes and using it, do I need to install any addiitonal plugin to be enable to use Argo Events?

Hmmm, can I trigger the event only once the ArgoCD app was successfully synced and healthy?

Thanks for another great video 🎉 this is definitively a good approach and it perfectly marries event based architectures which I like and use a lot myself. I'll give it a try

Why trigger a pipeline in GitHub when you can trigger a Workflow in Argo Workflows without ever leaving Kubernetes :P

Thanks for introducing this great tool! Personally, I've never heard of it until watching your video. At any rate, I'm about an hour deep into researching this topic, and while it looks promising initially, I have to say I see some potential issues with it; unless, of course, one looks to Kubernetes as their single source of truth.

Thank you for the great content. The gitops approach complicated the delivery pipeline. Even with create replicaset approach, we need to check if new pods are OK or crashing. Not sure if we can use the argocd Application (CRD) alerts also or hooks to trigger the pipeline.

thanks

I used ArgoCD PostSync hooks for something similar.

Maybe I too old-fashioned, but I just use "flux reconcile --with-source" in CI deploy job after pushing changes to git. If this command fails to succeed in specified timeout, script calls "flux events" to show what's wrong, and exits with code 1 (to mark job as failed). This way developers promptly can see if their changes lead to non-working code, and also, what specifically was broken, just by one click in CI pipelines panel.
In event-based architecture, connecting notifications and logs to specific git commits and tags still is a challenge.
(By the way, Argo has "argo app sync" and "argo app wait" commands serving the same purpose.)

Hi, great video, how would you compare this aproach to cd with the Kargo tool. It also solves the promotion problem in gitops, not only decoupling and post deployment steps...

love it, thanks for share it

the replication set is created, but in production it means that pods are just starting to rollout, it can be minutes before the rollout is completed. that is not the time to run the functional test against the service.

Great video viktor

When we started with Flux, we had the same problem. Our pipeline renders a helm chart with helm template and pushes this to a git ops repo. Then the same pipeline sends a webhook to flux to trigger reconcilation. Flux will update the commit status after that and the pipeline checks that for a maximum of 5mins. Then we continue with selenium and zap and then the same for production stage. I like having this within a single stream of events controlled by a pipeline, because otherwiss you never know where you currently are in the process and if it fails somewhere you‘ll have to start searching. Not a good approach when you have loads of pipelines to manage.

Great great video. One way to trigger a workflow is via ArgoCD notification service. When a new deployment sync happens, the notification service sends a message containing actual new version to a webhook. The question is if the notification service is fully reliable. It’s up to your receiving service to do whatever it needs to do . I’m thinking loud here, but have done some other triggers based on the notification service messages.

Hi, one issue with this solution is that if you change, for example, resources or an environment variable, the tests will be rerun on the same image. If the functional tests aren't too costly, this might not be a big problem. Another possible solution could be to use Shell Operator, which would allow writing the validation in shell to check if the change being made to the replicaset is the image. Note: I've never used Shell Operator, so this approach is 100% theoretical.