We have looked at training Stable Diffusion Lora's in a few different ways, however there are great autotrainers out there. CivitAI offers a simple way to train your Lora models. It's easy to train SDXL or 1.5 using the on site interface. We can use our existing datasets as they share the same format.
We will walkthrough the process of training and then downloading our model for use in comfyUI or Automatic1111. For this I am using my Diffusion Training test datasets, found on my Github, so you can repeat the results if you like.